Jul 26 15:03:23.159285 ip-10-0-130-21 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 26 15:03:23.159298 ip-10-0-130-21 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 26 15:03:23.159307 ip-10-0-130-21 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 26 15:03:23.159712 ip-10-0-130-21 systemd[1]: Failed to start Kubernetes Kubelet. Jul 26 15:03:32.450009 ip-10-0-130-21 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 26 15:03:32.450023 ip-10-0-130-21 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 55ea934307fe47f496afcf0bad40af8e -- Jul 26 15:05:18.461589 ip-10-0-130-21 systemd[1]: Starting Kubernetes Kubelet... Jul 26 15:05:18.829787 ip-10-0-130-21 kubenswrapper[2555]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 26 15:05:18.829787 ip-10-0-130-21 kubenswrapper[2555]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 26 15:05:18.829787 ip-10-0-130-21 kubenswrapper[2555]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 26 15:05:18.829787 ip-10-0-130-21 kubenswrapper[2555]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 26 15:05:18.829787 ip-10-0-130-21 kubenswrapper[2555]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 26 15:05:18.831702 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.831604 2555 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 26 15:05:18.836222 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836208 2555 feature_gate.go:328] unrecognized feature gate: Example Jul 26 15:05:18.836257 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836223 2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 26 15:05:18.836257 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836228 2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 26 15:05:18.836257 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836231 2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 26 15:05:18.836257 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836234 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 26 15:05:18.836257 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836237 2555 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 26 15:05:18.836257 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836240 2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 26 15:05:18.836257 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836243 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 26 15:05:18.836257 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836245 2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 26 15:05:18.836257 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836248 2555 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 26 15:05:18.836257 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836251 2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 26 15:05:18.836257 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836253 2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 26 15:05:18.836257 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836257 2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 26 15:05:18.836257 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836261 2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 26 15:05:18.836257 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836264 2555 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 26 15:05:18.836590 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836267 2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 26 15:05:18.836590 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836270 2555 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 26 15:05:18.836590 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836273 2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 26 15:05:18.836590 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836275 2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 26 15:05:18.836590 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836278 2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 26 15:05:18.836590 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836281 2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 26 15:05:18.836590 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836283 2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 26 15:05:18.836590 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836285 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 26 15:05:18.836590 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836288 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 26 15:05:18.836590 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836291 2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 26 15:05:18.836590 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836293 2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 26 15:05:18.836590 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836296 2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 26 15:05:18.836590 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836299 2555 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 26 15:05:18.836590 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836301 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 26 15:05:18.836590 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836303 2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 26 15:05:18.836590 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836306 2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 26 15:05:18.836590 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836308 2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 26 15:05:18.836590 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836311 2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 26 15:05:18.836590 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836314 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 26 15:05:18.836590 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836332 2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 26 15:05:18.837053 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836336 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 26 15:05:18.837053 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836340 2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 26 15:05:18.837053 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836344 2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 26 15:05:18.837053 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836347 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 26 15:05:18.837053 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836350 2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 26 15:05:18.837053 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836353 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 26 15:05:18.837053 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836356 2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 26 15:05:18.837053 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836359 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 26 15:05:18.837053 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836362 2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 26 15:05:18.837053 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836364 2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 26 15:05:18.837053 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836366 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 26 15:05:18.837053 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836369 2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 26 15:05:18.837053 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836371 2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 26 15:05:18.837053 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836375 2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 26 15:05:18.837053 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836378 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 26 15:05:18.837053 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836380 2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 26 15:05:18.837053 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836383 2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 26 15:05:18.837053 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836385 2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 26 15:05:18.837053 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836388 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 26 15:05:18.837053 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836390 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 26 15:05:18.837549 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836393 2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 26 15:05:18.837549 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836395 2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 26 15:05:18.837549 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836398 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 26 15:05:18.837549 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836400 2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 26 15:05:18.837549 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836403 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 26 15:05:18.837549 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836405 2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 26 15:05:18.837549 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836407 2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 26 15:05:18.837549 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836410 2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 26 15:05:18.837549 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836413 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 26 15:05:18.837549 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836415 2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 26 15:05:18.837549 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836419 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 26 15:05:18.837549 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836422 2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 26 15:05:18.837549 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836425 2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 26 15:05:18.837549 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836427 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 26 15:05:18.837549 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836430 2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 26 15:05:18.837549 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836432 2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 26 15:05:18.837549 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836436 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 26 15:05:18.837549 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836439 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 26 15:05:18.837549 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836441 2555 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 26 15:05:18.837549 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836444 2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 26 15:05:18.838120 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836446 2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 26 15:05:18.838120 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836448 2555 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 26 15:05:18.838120 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836451 2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 26 15:05:18.838120 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836454 2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 26 15:05:18.838120 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836456 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 26 15:05:18.838120 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836459 2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 26 15:05:18.838120 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836462 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 26 15:05:18.838120 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836464 2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 26 15:05:18.838120 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836467 2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 26 15:05:18.838120 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836469 2555 feature_gate.go:328] unrecognized feature gate: Example2 Jul 26 15:05:18.838120 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836472 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 26 15:05:18.838120 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836474 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 26 15:05:18.838120 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836477 2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 26 15:05:18.838120 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836479 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 26 15:05:18.838120 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836481 2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 26 15:05:18.838120 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836484 2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 26 15:05:18.838120 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836486 2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 26 15:05:18.838120 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836489 2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 26 15:05:18.838120 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836491 2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 26 15:05:18.838120 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836493 2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 26 15:05:18.838603 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836496 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 26 15:05:18.838603 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836499 2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 26 15:05:18.838603 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836502 2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 26 15:05:18.838603 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.836504 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 26 15:05:18.838603 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837184 2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 26 15:05:18.838603 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837191 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 26 15:05:18.838603 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837194 2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 26 15:05:18.838603 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837198 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 26 15:05:18.838603 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837201 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 26 15:05:18.838603 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837205 2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 26 15:05:18.838603 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837209 2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 26 15:05:18.838603 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837212 2555 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 26 15:05:18.838603 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837214 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 26 15:05:18.838603 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837217 2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 26 15:05:18.838603 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837219 2555 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 26 15:05:18.838603 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837222 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 26 15:05:18.838603 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837224 2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 26 15:05:18.838603 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837227 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 26 15:05:18.838603 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837230 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 26 15:05:18.838603 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837232 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 26 15:05:18.839131 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837234 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 26 15:05:18.839131 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837237 2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 26 15:05:18.839131 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837239 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 26 15:05:18.839131 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837242 2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 26 15:05:18.839131 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837244 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 26 15:05:18.839131 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837246 2555 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 26 15:05:18.839131 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837249 2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 26 15:05:18.839131 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837251 2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 26 15:05:18.839131 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837254 2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 26 15:05:18.839131 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837256 2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 26 15:05:18.839131 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837259 2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 26 15:05:18.839131 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837261 2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 26 15:05:18.839131 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837263 2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 26 15:05:18.839131 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837266 2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 26 15:05:18.839131 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837269 2555 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 26 15:05:18.839131 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837272 2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 26 15:05:18.839131 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837274 2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 26 15:05:18.839131 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837277 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 26 15:05:18.839131 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837279 2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 26 15:05:18.839596 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837282 2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 26 15:05:18.839596 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837285 2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 26 15:05:18.839596 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837287 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 26 15:05:18.839596 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837289 2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 26 15:05:18.839596 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837292 2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 26 15:05:18.839596 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837294 2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 26 15:05:18.839596 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837297 2555 feature_gate.go:328] unrecognized feature gate: Example Jul 26 15:05:18.839596 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837299 2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 26 15:05:18.839596 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837302 2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 26 15:05:18.839596 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837304 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 26 15:05:18.839596 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837307 2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 26 15:05:18.839596 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837309 2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 26 15:05:18.839596 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837311 2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 26 15:05:18.839596 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837313 2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 26 15:05:18.839596 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837332 2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 26 15:05:18.839596 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837335 2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 26 15:05:18.839596 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837337 2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 26 15:05:18.839596 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837340 2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 26 15:05:18.839596 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837345 2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 26 15:05:18.839596 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837348 2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 26 15:05:18.840075 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837351 2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 26 15:05:18.840075 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837354 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 26 15:05:18.840075 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837357 2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 26 15:05:18.840075 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837360 2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 26 15:05:18.840075 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837362 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 26 15:05:18.840075 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837364 2555 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 26 15:05:18.840075 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837371 2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 26 15:05:18.840075 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837374 2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 26 15:05:18.840075 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837377 2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 26 15:05:18.840075 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837379 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 26 15:05:18.840075 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837382 2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 26 15:05:18.840075 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837384 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 26 15:05:18.840075 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837387 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 26 15:05:18.840075 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837390 2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 26 15:05:18.840075 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837393 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 26 15:05:18.840075 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837396 2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 26 15:05:18.840075 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837398 2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 26 15:05:18.840075 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837401 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 26 15:05:18.840075 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837403 2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 26 15:05:18.840546 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837406 2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 26 15:05:18.840546 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837408 2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 26 15:05:18.840546 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837411 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 26 15:05:18.840546 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837413 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 26 15:05:18.840546 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837416 2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 26 15:05:18.840546 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837418 2555 feature_gate.go:328] unrecognized feature gate: Example2 Jul 26 15:05:18.840546 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837421 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 26 15:05:18.840546 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837423 2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 26 15:05:18.840546 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837425 2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 26 15:05:18.840546 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837428 2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 26 15:05:18.840546 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837430 2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 26 15:05:18.840546 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837433 2555 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 26 15:05:18.840546 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837435 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 26 15:05:18.840546 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837439 2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 26 15:05:18.840546 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837442 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 26 15:05:18.840546 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837445 2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 26 15:05:18.840546 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837447 2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 26 15:05:18.840546 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837450 2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 26 15:05:18.840546 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837452 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 26 15:05:18.840546 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837456 2555 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 26 15:05:18.841021 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837458 2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 26 15:05:18.841021 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837461 2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 26 15:05:18.841021 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837464 2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 26 15:05:18.841021 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837466 2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 26 15:05:18.841021 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.837468 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 26 15:05:18.841021 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838536 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 26 15:05:18.841021 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838543 2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 26 15:05:18.841021 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838547 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 26 15:05:18.841021 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838550 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 26 15:05:18.841021 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838553 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 26 15:05:18.841021 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838555 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 26 15:05:18.841021 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838558 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 26 15:05:18.841021 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838561 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 26 15:05:18.841021 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838563 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 26 15:05:18.841021 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838566 2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 26 15:05:18.841021 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838570 2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 26 15:05:18.841021 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838573 2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 26 15:05:18.841021 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838576 2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 26 15:05:18.841021 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838579 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 26 15:05:18.841487 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838582 2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 26 15:05:18.841487 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838584 2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 26 15:05:18.841487 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838587 2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 26 15:05:18.841487 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838590 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 26 15:05:18.841487 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838592 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 26 15:05:18.841487 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838595 2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 26 15:05:18.841487 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838597 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 26 15:05:18.841487 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838600 2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 26 15:05:18.841487 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838603 2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 26 15:05:18.841487 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838605 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 26 15:05:18.841487 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838608 2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 26 15:05:18.841487 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838611 2555 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 26 15:05:18.841487 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838614 2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 26 15:05:18.841487 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838616 2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 26 15:05:18.841487 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838619 2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 26 15:05:18.841487 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838622 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 26 15:05:18.841487 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838624 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 26 15:05:18.841487 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838626 2555 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 26 15:05:18.841487 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838629 2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 26 15:05:18.841487 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838632 2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 26 15:05:18.841941 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838634 2555 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 26 15:05:18.841941 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838637 2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 26 15:05:18.841941 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838639 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 26 15:05:18.841941 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838642 2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 26 15:05:18.841941 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838644 2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 26 15:05:18.841941 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838646 2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 26 15:05:18.841941 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838648 2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 26 15:05:18.841941 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838651 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 26 15:05:18.841941 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838653 2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 26 15:05:18.841941 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838656 2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 26 15:05:18.841941 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838658 2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 26 15:05:18.841941 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838662 2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 26 15:05:18.841941 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838666 2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 26 15:05:18.841941 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838669 2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 26 15:05:18.841941 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838671 2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 26 15:05:18.841941 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838673 2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 26 15:05:18.841941 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838676 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 26 15:05:18.841941 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838678 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 26 15:05:18.841941 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838680 2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 26 15:05:18.841941 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838683 2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 26 15:05:18.842411 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838685 2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 26 15:05:18.842411 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838688 2555 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 26 15:05:18.842411 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838690 2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 26 15:05:18.842411 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838692 2555 feature_gate.go:328] unrecognized feature gate: Example Jul 26 15:05:18.842411 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838695 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 26 15:05:18.842411 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838698 2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 26 15:05:18.842411 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838701 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 26 15:05:18.842411 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838704 2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 26 15:05:18.842411 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838707 2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 26 15:05:18.842411 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838710 2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 26 15:05:18.842411 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838713 2555 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 26 15:05:18.842411 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838717 2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 26 15:05:18.842411 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838720 2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 26 15:05:18.842411 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838722 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 26 15:05:18.842411 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838738 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 26 15:05:18.842411 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838741 2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 26 15:05:18.842411 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838744 2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 26 15:05:18.842411 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838747 2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 26 15:05:18.842411 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838750 2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 26 15:05:18.842411 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838753 2555 feature_gate.go:328] unrecognized feature gate: Example2 Jul 26 15:05:18.842878 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838756 2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 26 15:05:18.842878 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838758 2555 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 26 15:05:18.842878 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838761 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 26 15:05:18.842878 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838763 2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 26 15:05:18.842878 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838766 2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 26 15:05:18.842878 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838768 2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 26 15:05:18.842878 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838771 2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 26 15:05:18.842878 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838773 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 26 15:05:18.842878 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838776 2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 26 15:05:18.842878 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838778 2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 26 15:05:18.842878 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838781 2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 26 15:05:18.842878 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838783 2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 26 15:05:18.842878 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838785 2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 26 15:05:18.842878 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838787 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 26 15:05:18.842878 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838790 2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 26 15:05:18.842878 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838792 2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 26 15:05:18.842878 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838795 2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 26 15:05:18.842878 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838797 2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 26 15:05:18.842878 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838799 2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 26 15:05:18.843324 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838802 2555 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 26 15:05:18.843324 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838805 2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 26 15:05:18.843324 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838807 2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 26 15:05:18.843324 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838809 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 26 15:05:18.843324 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838812 2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 26 15:05:18.843324 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.838814 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 26 15:05:18.843324 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.838878 2555 flags.go:64] FLAG: --address="0.0.0.0" Jul 26 15:05:18.843324 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.838884 2555 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 26 15:05:18.843324 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.838889 2555 flags.go:64] FLAG: --anonymous-auth="true" Jul 26 15:05:18.843324 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.838893 2555 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 26 15:05:18.843324 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.838901 2555 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 26 15:05:18.843324 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.838904 2555 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 26 15:05:18.843324 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.838910 2555 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 26 15:05:18.843324 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.838915 2555 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 26 15:05:18.843324 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.838920 2555 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 26 15:05:18.843324 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.838924 2555 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 26 15:05:18.843324 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.838929 2555 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 26 15:05:18.843324 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.838933 2555 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 26 15:05:18.843324 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.838936 2555 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 26 15:05:18.843324 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.838939 2555 flags.go:64] FLAG: --cgroup-root="" Jul 26 15:05:18.843324 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.838942 2555 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 26 15:05:18.843324 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.838945 2555 flags.go:64] FLAG: --client-ca-file="" Jul 26 15:05:18.843324 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.838947 2555 flags.go:64] FLAG: --cloud-provider="external" Jul 26 15:05:18.843893 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.838950 2555 flags.go:64] FLAG: --cluster-dns="[]" Jul 26 15:05:18.843893 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.838953 2555 flags.go:64] FLAG: --cluster-domain="" Jul 26 15:05:18.843893 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.838956 2555 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 26 15:05:18.843893 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.838959 2555 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 26 15:05:18.843893 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.838962 2555 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 26 15:05:18.843893 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.838965 2555 flags.go:64] FLAG: --container-log-max-files="5" Jul 26 15:05:18.843893 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.838968 2555 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 26 15:05:18.843893 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.838971 2555 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 26 15:05:18.843893 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.838974 2555 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 26 15:05:18.843893 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.838978 2555 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 26 15:05:18.843893 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.838980 2555 flags.go:64] FLAG: --contention-profiling="false" Jul 26 15:05:18.843893 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.838984 2555 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 26 15:05:18.843893 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.838987 2555 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 26 15:05:18.843893 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.838990 2555 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 26 15:05:18.843893 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.838993 2555 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 26 15:05:18.843893 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.838996 2555 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 26 15:05:18.843893 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.838999 2555 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 26 15:05:18.843893 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839002 2555 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 26 15:05:18.843893 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839004 2555 flags.go:64] FLAG: --enable-load-reader="false" Jul 26 15:05:18.843893 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839007 2555 flags.go:64] FLAG: --enable-server="true" Jul 26 15:05:18.843893 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839010 2555 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 26 15:05:18.843893 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839014 2555 flags.go:64] FLAG: --event-burst="100" Jul 26 15:05:18.843893 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839017 2555 flags.go:64] FLAG: --event-qps="50" Jul 26 15:05:18.843893 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839021 2555 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 26 15:05:18.844491 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839024 2555 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 26 15:05:18.844491 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839027 2555 flags.go:64] FLAG: --eviction-hard="" Jul 26 15:05:18.844491 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839030 2555 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 26 15:05:18.844491 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839033 2555 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 26 15:05:18.844491 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839036 2555 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 26 15:05:18.844491 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839039 2555 flags.go:64] FLAG: --eviction-soft="" Jul 26 15:05:18.844491 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839042 2555 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 26 15:05:18.844491 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839044 2555 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 26 15:05:18.844491 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839047 2555 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 26 15:05:18.844491 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839050 2555 flags.go:64] FLAG: --experimental-mounter-path="" Jul 26 15:05:18.844491 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839052 2555 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 26 15:05:18.844491 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839057 2555 flags.go:64] FLAG: --fail-swap-on="true" Jul 26 15:05:18.844491 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839059 2555 flags.go:64] FLAG: --feature-gates="" Jul 26 15:05:18.844491 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839063 2555 flags.go:64] FLAG: --file-check-frequency="20s" Jul 26 15:05:18.844491 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839066 2555 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 26 15:05:18.844491 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839068 2555 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 26 15:05:18.844491 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839071 2555 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 26 15:05:18.844491 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839074 2555 flags.go:64] FLAG: --healthz-port="10248" Jul 26 15:05:18.844491 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839077 2555 flags.go:64] FLAG: --help="false" Jul 26 15:05:18.844491 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839079 2555 flags.go:64] FLAG: --hostname-override="ip-10-0-130-21.ec2.internal" Jul 26 15:05:18.844491 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839083 2555 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 26 15:05:18.844491 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839086 2555 flags.go:64] FLAG: --http-check-frequency="20s" Jul 26 15:05:18.844491 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839089 2555 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 26 15:05:18.845196 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839092 2555 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 26 15:05:18.845196 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839095 2555 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 26 15:05:18.845196 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839098 2555 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 26 15:05:18.845196 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839100 2555 flags.go:64] FLAG: --image-service-endpoint="" Jul 26 15:05:18.845196 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839103 2555 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 26 15:05:18.845196 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839106 2555 flags.go:64] FLAG: --kube-api-burst="100" Jul 26 15:05:18.845196 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839108 2555 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 26 15:05:18.845196 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839111 2555 flags.go:64] FLAG: --kube-api-qps="50" Jul 26 15:05:18.845196 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839115 2555 flags.go:64] FLAG: --kube-reserved="" Jul 26 15:05:18.845196 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839118 2555 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 26 15:05:18.845196 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839120 2555 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 26 15:05:18.845196 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839123 2555 flags.go:64] FLAG: --kubelet-cgroups="" Jul 26 15:05:18.845196 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839126 2555 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 26 15:05:18.845196 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839128 2555 flags.go:64] FLAG: --lock-file="" Jul 26 15:05:18.845196 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839131 2555 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 26 15:05:18.845196 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839134 2555 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 26 15:05:18.845196 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839137 2555 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 26 15:05:18.845196 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839142 2555 flags.go:64] FLAG: --log-json-split-stream="false" Jul 26 15:05:18.845196 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839145 2555 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 26 15:05:18.845196 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839148 2555 flags.go:64] FLAG: --log-text-split-stream="false" Jul 26 15:05:18.845196 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839152 2555 flags.go:64] FLAG: --logging-format="text" Jul 26 15:05:18.845196 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839155 2555 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 26 15:05:18.845196 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839158 2555 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 26 15:05:18.845196 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839160 2555 flags.go:64] FLAG: --manifest-url="" Jul 26 15:05:18.846042 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839163 2555 flags.go:64] FLAG: --manifest-url-header="" Jul 26 15:05:18.846042 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839167 2555 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 26 15:05:18.846042 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839171 2555 flags.go:64] FLAG: --max-open-files="1000000" Jul 26 15:05:18.846042 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839174 2555 flags.go:64] FLAG: --max-pods="110" Jul 26 15:05:18.846042 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839177 2555 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 26 15:05:18.846042 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839180 2555 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 26 15:05:18.846042 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839183 2555 flags.go:64] FLAG: --memory-manager-policy="None" Jul 26 15:05:18.846042 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839185 2555 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 26 15:05:18.846042 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839189 2555 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 26 15:05:18.846042 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839191 2555 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 26 15:05:18.846042 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839194 2555 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 26 15:05:18.846042 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839200 2555 flags.go:64] FLAG: --node-status-max-images="50" Jul 26 15:05:18.846042 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839203 2555 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 26 15:05:18.846042 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839206 2555 flags.go:64] FLAG: --oom-score-adj="-999" Jul 26 15:05:18.846042 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839208 2555 flags.go:64] FLAG: --pod-cidr="" Jul 26 15:05:18.846042 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839211 2555 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 26 15:05:18.846042 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839217 2555 flags.go:64] FLAG: --pod-manifest-path="" Jul 26 15:05:18.846042 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839219 2555 flags.go:64] FLAG: --pod-max-pids="-1" Jul 26 15:05:18.846042 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839222 2555 flags.go:64] FLAG: --pods-per-core="0" Jul 26 15:05:18.846042 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839225 2555 flags.go:64] FLAG: --port="10250" Jul 26 15:05:18.846042 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839228 2555 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 26 15:05:18.846042 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839230 2555 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c8bf87265902aa4f" Jul 26 15:05:18.846042 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839233 2555 flags.go:64] FLAG: --qos-reserved="" Jul 26 15:05:18.846042 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839236 2555 flags.go:64] FLAG: --read-only-port="10255" Jul 26 15:05:18.846623 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839239 2555 flags.go:64] FLAG: --register-node="true" Jul 26 15:05:18.846623 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839241 2555 flags.go:64] FLAG: --register-with-taints="" Jul 26 15:05:18.846623 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839245 2555 flags.go:64] FLAG: --registry-burst="10" Jul 26 15:05:18.846623 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839247 2555 flags.go:64] FLAG: --registry-qps="5" Jul 26 15:05:18.846623 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839250 2555 flags.go:64] FLAG: --reserved-cpus="" Jul 26 15:05:18.846623 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839254 2555 flags.go:64] FLAG: --reserved-memory="" Jul 26 15:05:18.846623 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839257 2555 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 26 15:05:18.846623 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839260 2555 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 26 15:05:18.846623 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839263 2555 flags.go:64] FLAG: --rotate-certificates="false" Jul 26 15:05:18.846623 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839265 2555 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 26 15:05:18.846623 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839268 2555 flags.go:64] FLAG: --runonce="false" Jul 26 15:05:18.846623 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839270 2555 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 26 15:05:18.846623 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839273 2555 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 26 15:05:18.846623 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839276 2555 flags.go:64] FLAG: --seccomp-default="false" Jul 26 15:05:18.846623 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839279 2555 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 26 15:05:18.846623 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839282 2555 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 26 15:05:18.846623 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839285 2555 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 26 15:05:18.846623 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839287 2555 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 26 15:05:18.846623 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839290 2555 flags.go:64] FLAG: --storage-driver-password="root" Jul 26 15:05:18.846623 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839293 2555 flags.go:64] FLAG: --storage-driver-secure="false" Jul 26 15:05:18.846623 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839296 2555 flags.go:64] FLAG: --storage-driver-table="stats" Jul 26 15:05:18.846623 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839298 2555 flags.go:64] FLAG: --storage-driver-user="root" Jul 26 15:05:18.846623 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839301 2555 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 26 15:05:18.846623 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839304 2555 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 26 15:05:18.846623 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839308 2555 flags.go:64] FLAG: --system-cgroups="" Jul 26 15:05:18.847254 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839311 2555 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 26 15:05:18.847254 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839326 2555 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 26 15:05:18.847254 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839330 2555 flags.go:64] FLAG: --tls-cert-file="" Jul 26 15:05:18.847254 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839332 2555 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 26 15:05:18.847254 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839335 2555 flags.go:64] FLAG: --tls-min-version="" Jul 26 15:05:18.847254 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839338 2555 flags.go:64] FLAG: --tls-private-key-file="" Jul 26 15:05:18.847254 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839340 2555 flags.go:64] FLAG: --topology-manager-policy="none" Jul 26 15:05:18.847254 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839343 2555 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 26 15:05:18.847254 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839346 2555 flags.go:64] FLAG: --topology-manager-scope="container" Jul 26 15:05:18.847254 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839348 2555 flags.go:64] FLAG: --v="2" Jul 26 15:05:18.847254 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839352 2555 flags.go:64] FLAG: --version="false" Jul 26 15:05:18.847254 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839355 2555 flags.go:64] FLAG: --vmodule="" Jul 26 15:05:18.847254 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839360 2555 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 26 15:05:18.847254 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839364 2555 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 26 15:05:18.847254 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839461 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 26 15:05:18.847254 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839465 2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 26 15:05:18.847254 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839468 2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 26 15:05:18.847254 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839471 2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 26 15:05:18.847254 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839473 2555 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 26 15:05:18.847254 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839476 2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 26 15:05:18.847254 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839479 2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 26 15:05:18.847254 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839482 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 26 15:05:18.847254 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839484 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 26 15:05:18.847825 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839487 2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 26 15:05:18.847825 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839490 2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 26 15:05:18.847825 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839492 2555 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 26 15:05:18.847825 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839495 2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 26 15:05:18.847825 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839497 2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 26 15:05:18.847825 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839499 2555 feature_gate.go:328] unrecognized feature gate: Example2 Jul 26 15:05:18.847825 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839502 2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 26 15:05:18.847825 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839504 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 26 15:05:18.847825 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839508 2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 26 15:05:18.847825 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839511 2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 26 15:05:18.847825 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839513 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 26 15:05:18.847825 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839515 2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 26 15:05:18.847825 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839518 2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 26 15:05:18.847825 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839520 2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 26 15:05:18.847825 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839522 2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 26 15:05:18.847825 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839525 2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 26 15:05:18.847825 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839528 2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 26 15:05:18.847825 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839530 2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 26 15:05:18.847825 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839533 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 26 15:05:18.847825 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839535 2555 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 26 15:05:18.848310 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839537 2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 26 15:05:18.848310 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839541 2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 26 15:05:18.848310 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839544 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 26 15:05:18.848310 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839546 2555 feature_gate.go:328] unrecognized feature gate: Example Jul 26 15:05:18.848310 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839548 2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 26 15:05:18.848310 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839551 2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 26 15:05:18.848310 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839553 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 26 15:05:18.848310 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839555 2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 26 15:05:18.848310 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839558 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 26 15:05:18.848310 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839562 2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 26 15:05:18.848310 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839565 2555 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 26 15:05:18.848310 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839568 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 26 15:05:18.848310 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839571 2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 26 15:05:18.848310 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839574 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 26 15:05:18.848310 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839577 2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 26 15:05:18.848310 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839580 2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 26 15:05:18.848310 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839582 2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 26 15:05:18.848310 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839584 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 26 15:05:18.848310 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839587 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 26 15:05:18.848310 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839589 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 26 15:05:18.848845 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839593 2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 26 15:05:18.848845 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839595 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 26 15:05:18.848845 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839597 2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 26 15:05:18.848845 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839600 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 26 15:05:18.848845 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839603 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 26 15:05:18.848845 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839605 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 26 15:05:18.848845 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839608 2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 26 15:05:18.848845 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839610 2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 26 15:05:18.848845 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839613 2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 26 15:05:18.848845 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839615 2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 26 15:05:18.848845 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839617 2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 26 15:05:18.848845 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839621 2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 26 15:05:18.848845 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839624 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 26 15:05:18.848845 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839628 2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 26 15:05:18.848845 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839631 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 26 15:05:18.848845 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839633 2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 26 15:05:18.848845 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839636 2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 26 15:05:18.848845 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839638 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 26 15:05:18.848845 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839641 2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 26 15:05:18.849296 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839643 2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 26 15:05:18.849296 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839647 2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 26 15:05:18.849296 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839650 2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 26 15:05:18.849296 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839653 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 26 15:05:18.849296 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839655 2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 26 15:05:18.849296 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839658 2555 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 26 15:05:18.849296 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839661 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 26 15:05:18.849296 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839666 2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 26 15:05:18.849296 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839669 2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 26 15:05:18.849296 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839671 2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 26 15:05:18.849296 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839673 2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 26 15:05:18.849296 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839676 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 26 15:05:18.849296 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839678 2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 26 15:05:18.849296 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839682 2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 26 15:05:18.849296 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839684 2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 26 15:05:18.849296 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839687 2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 26 15:05:18.849296 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839689 2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 26 15:05:18.849296 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839692 2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 26 15:05:18.849296 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839694 2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 26 15:05:18.849296 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839696 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 26 15:05:18.849772 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839699 2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 26 15:05:18.849772 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839701 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 26 15:05:18.849772 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839703 2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 26 15:05:18.849772 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839706 2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 26 15:05:18.849772 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839708 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 26 15:05:18.849772 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839711 2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 26 15:05:18.849772 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839714 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 26 15:05:18.849772 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839717 2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 26 15:05:18.849772 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839719 2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 26 15:05:18.849772 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839722 2555 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 26 15:05:18.849772 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.839724 2555 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 26 15:05:18.849772 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.839729 2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 26 15:05:18.849772 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.846688 2555 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 26 15:05:18.849772 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.846702 2555 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 26 15:05:18.849772 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.846719 2555 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 26 15:05:18.850132 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.846725 2555 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 26 15:05:18.850132 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846778 2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 26 15:05:18.850132 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846784 2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 26 15:05:18.850132 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846788 2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 26 15:05:18.850132 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846791 2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 26 15:05:18.850132 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846795 2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 26 15:05:18.850132 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846797 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 26 15:05:18.850132 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846800 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 26 15:05:18.850132 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846803 2555 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 26 15:05:18.850132 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846806 2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 26 15:05:18.850132 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846808 2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 26 15:05:18.850132 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846811 2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 26 15:05:18.850132 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846813 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 26 15:05:18.850132 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846817 2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 26 15:05:18.850132 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846820 2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 26 15:05:18.850132 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846823 2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 26 15:05:18.850132 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846826 2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 26 15:05:18.850132 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846829 2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 26 15:05:18.850579 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846831 2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 26 15:05:18.850579 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846835 2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 26 15:05:18.850579 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846837 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 26 15:05:18.850579 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846840 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 26 15:05:18.850579 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846842 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 26 15:05:18.850579 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846845 2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 26 15:05:18.850579 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846847 2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 26 15:05:18.850579 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846850 2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 26 15:05:18.850579 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846852 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 26 15:05:18.850579 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846854 2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 26 15:05:18.850579 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846857 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 26 15:05:18.850579 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846860 2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 26 15:05:18.850579 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846863 2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 26 15:05:18.850579 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846866 2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 26 15:05:18.850579 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846868 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 26 15:05:18.850579 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846871 2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 26 15:05:18.850579 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846873 2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 26 15:05:18.850579 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846876 2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 26 15:05:18.850579 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846878 2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 26 15:05:18.851020 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846880 2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 26 15:05:18.851020 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846883 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 26 15:05:18.851020 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846885 2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 26 15:05:18.851020 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846888 2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 26 15:05:18.851020 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846890 2555 feature_gate.go:328] unrecognized feature gate: Example Jul 26 15:05:18.851020 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846893 2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 26 15:05:18.851020 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846895 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 26 15:05:18.851020 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846897 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 26 15:05:18.851020 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846900 2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 26 15:05:18.851020 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846902 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 26 15:05:18.851020 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846905 2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 26 15:05:18.851020 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846907 2555 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 26 15:05:18.851020 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846910 2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 26 15:05:18.851020 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846912 2555 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 26 15:05:18.851020 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846914 2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 26 15:05:18.851020 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846917 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 26 15:05:18.851020 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846919 2555 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 26 15:05:18.851020 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846921 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 26 15:05:18.851020 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846924 2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 26 15:05:18.851020 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846926 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 26 15:05:18.851501 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846929 2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 26 15:05:18.851501 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846932 2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 26 15:05:18.851501 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846934 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 26 15:05:18.851501 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846937 2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 26 15:05:18.851501 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846940 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 26 15:05:18.851501 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846943 2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 26 15:05:18.851501 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846946 2555 feature_gate.go:328] unrecognized feature gate: Example2 Jul 26 15:05:18.851501 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846948 2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 26 15:05:18.851501 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846951 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 26 15:05:18.851501 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846953 2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 26 15:05:18.851501 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846956 2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 26 15:05:18.851501 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846958 2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 26 15:05:18.851501 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846961 2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 26 15:05:18.851501 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846963 2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 26 15:05:18.851501 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846966 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 26 15:05:18.851501 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846968 2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 26 15:05:18.851501 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846971 2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 26 15:05:18.851501 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846974 2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 26 15:05:18.851501 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846978 2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 26 15:05:18.851939 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846980 2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 26 15:05:18.851939 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846983 2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 26 15:05:18.851939 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846986 2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 26 15:05:18.851939 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846988 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 26 15:05:18.851939 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846991 2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 26 15:05:18.851939 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846994 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 26 15:05:18.851939 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846997 2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 26 15:05:18.851939 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.846999 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 26 15:05:18.851939 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847001 2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 26 15:05:18.851939 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847004 2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 26 15:05:18.851939 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847007 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 26 15:05:18.851939 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847009 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 26 15:05:18.851939 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847011 2555 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 26 15:05:18.851939 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847014 2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 26 15:05:18.851939 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847016 2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 26 15:05:18.851939 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847019 2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 26 15:05:18.851939 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847021 2555 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 26 15:05:18.851939 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847024 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 26 15:05:18.851939 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847027 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 26 15:05:18.851939 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847029 2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 26 15:05:18.852523 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847032 2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 26 15:05:18.852523 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847034 2555 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 26 15:05:18.852523 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847047 2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 26 15:05:18.852523 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847050 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 26 15:05:18.852523 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.847055 2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 26 15:05:18.852523 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847169 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 26 15:05:18.852523 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847174 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 26 15:05:18.852523 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847177 2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 26 15:05:18.852523 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847181 2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 26 15:05:18.852523 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847184 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 26 15:05:18.852523 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847187 2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 26 15:05:18.852523 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847190 2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 26 15:05:18.852523 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847192 2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 26 15:05:18.852523 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847195 2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 26 15:05:18.852523 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847198 2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 26 15:05:18.852924 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847201 2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 26 15:05:18.852924 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847204 2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 26 15:05:18.852924 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847207 2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 26 15:05:18.852924 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847210 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 26 15:05:18.852924 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847213 2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 26 15:05:18.852924 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847215 2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 26 15:05:18.852924 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847218 2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 26 15:05:18.852924 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847220 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 26 15:05:18.852924 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847223 2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 26 15:05:18.852924 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847225 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 26 15:05:18.852924 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847228 2555 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 26 15:05:18.852924 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847231 2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 26 15:05:18.852924 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847234 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 26 15:05:18.852924 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847237 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 26 15:05:18.852924 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847240 2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 26 15:05:18.852924 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847243 2555 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 26 15:05:18.852924 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847245 2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 26 15:05:18.852924 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847248 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 26 15:05:18.852924 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847251 2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 26 15:05:18.853394 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847254 2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 26 15:05:18.853394 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847257 2555 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 26 15:05:18.853394 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847260 2555 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 26 15:05:18.853394 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847263 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 26 15:05:18.853394 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847266 2555 feature_gate.go:328] unrecognized feature gate: Example Jul 26 15:05:18.853394 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847268 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 26 15:05:18.853394 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847271 2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 26 15:05:18.853394 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847273 2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 26 15:05:18.853394 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847276 2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 26 15:05:18.853394 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847278 2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 26 15:05:18.853394 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847281 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 26 15:05:18.853394 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847283 2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 26 15:05:18.853394 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847285 2555 feature_gate.go:328] unrecognized feature gate: Example2 Jul 26 15:05:18.853394 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847288 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 26 15:05:18.853394 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847290 2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 26 15:05:18.853394 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847292 2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 26 15:05:18.853394 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847295 2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 26 15:05:18.853394 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847298 2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 26 15:05:18.853394 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847301 2555 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 26 15:05:18.853394 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847303 2555 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 26 15:05:18.853853 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847305 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 26 15:05:18.853853 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847307 2555 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 26 15:05:18.853853 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847310 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 26 15:05:18.853853 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847312 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 26 15:05:18.853853 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847327 2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 26 15:05:18.853853 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847332 2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 26 15:05:18.853853 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847336 2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 26 15:05:18.853853 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847339 2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 26 15:05:18.853853 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847342 2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 26 15:05:18.853853 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847345 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 26 15:05:18.853853 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847347 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 26 15:05:18.853853 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847350 2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 26 15:05:18.853853 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847352 2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 26 15:05:18.853853 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847355 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 26 15:05:18.853853 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847357 2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 26 15:05:18.853853 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847360 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 26 15:05:18.853853 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847362 2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 26 15:05:18.853853 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847365 2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 26 15:05:18.853853 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847367 2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 26 15:05:18.853853 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847369 2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 26 15:05:18.854327 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847372 2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 26 15:05:18.854327 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847374 2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 26 15:05:18.854327 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847376 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 26 15:05:18.854327 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847379 2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 26 15:05:18.854327 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847383 2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 26 15:05:18.854327 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847386 2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 26 15:05:18.854327 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847389 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 26 15:05:18.854327 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847392 2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 26 15:05:18.854327 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847395 2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 26 15:05:18.854327 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847400 2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 26 15:05:18.854327 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847403 2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 26 15:05:18.854327 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847406 2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 26 15:05:18.854327 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847408 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 26 15:05:18.854327 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847411 2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 26 15:05:18.854327 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847413 2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 26 15:05:18.854327 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847416 2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 26 15:05:18.854327 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847418 2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 26 15:05:18.854327 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847421 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 26 15:05:18.854327 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847423 2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 26 15:05:18.854753 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847426 2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 26 15:05:18.854753 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847428 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 26 15:05:18.854753 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847431 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 26 15:05:18.854753 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847433 2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 26 15:05:18.854753 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847435 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 26 15:05:18.854753 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847438 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 26 15:05:18.854753 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847440 2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 26 15:05:18.854753 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847443 2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 26 15:05:18.854753 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847445 2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 26 15:05:18.854753 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847448 2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 26 15:05:18.854753 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:18.847450 2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 26 15:05:18.854753 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.847455 2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 26 15:05:18.854753 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.847571 2555 server.go:962] "Client rotation is on, will bootstrap in background" Jul 26 15:05:18.854753 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.850429 2555 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 26 15:05:18.854753 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.851753 2555 server.go:1019] "Starting client certificate rotation" Jul 26 15:05:18.855096 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.851846 2555 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 26 15:05:18.855096 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.851876 2555 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 26 15:05:18.855096 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.854289 2555 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 26 15:05:18.859265 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.859252 2555 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 26 15:05:18.868426 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.868410 2555 log.go:25] "Validated CRI v1 runtime API" Jul 26 15:05:18.871407 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.871383 2555 log.go:25] "Validated CRI v1 image API" Jul 26 15:05:18.872292 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.872279 2555 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 26 15:05:18.875418 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.875397 2555 fs.go:135] Filesystem UUIDs: map[75987436-f0d2-40c6-b74a-615f969d25e4:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 9e064789-ad95-41df-abb2-000ae837ef37:/dev/nvme0n1p3] Jul 26 15:05:18.875491 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.875415 2555 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 26 15:05:18.879332 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.879299 2555 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 26 15:05:18.881491 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.881389 2555 manager.go:216] Machine: {Timestamp:2026-07-26 15:05:18.880168807 +0000 UTC m=+0.310626998 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3197546 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec283f85419784ba82bcb242ed5a7553 SystemUUID:ec283f85-4197-84ba-82bc-b242ed5a7553 BootID:55ea9343-07fe-47f4-96af-cf0bad40af8e Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:75:9c:18:83:69 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:75:9c:18:83:69 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:12:e6:a9:40:9a:2e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 26 15:05:18.881491 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.881485 2555 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 26 15:05:18.881616 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.881583 2555 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 26 15:05:18.884735 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.884716 2555 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 26 15:05:18.884852 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.884737 2555 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-21.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 26 15:05:18.885432 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.885422 2555 topology_manager.go:138] "Creating topology manager with none policy" Jul 26 15:05:18.885472 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.885434 2555 container_manager_linux.go:309] "Creating device plugin manager" Jul 26 15:05:18.885472 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.885441 2555 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 26 15:05:18.886029 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.886020 2555 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 26 15:05:18.887216 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.887206 2555 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 26 15:05:18.889896 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.889884 2555 state_mem.go:36] "Initialized new in-memory state store" Jul 26 15:05:18.890022 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.890013 2555 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 26 15:05:18.890058 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.890042 2555 kubelet.go:486] "Attempting to sync node with API server" Jul 26 15:05:18.890058 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.890054 2555 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 26 15:05:18.890107 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.890071 2555 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 26 15:05:18.890107 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.890083 2555 kubelet.go:398] "Adding apiserver pod source" Jul 26 15:05:18.890107 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.890094 2555 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 26 15:05:18.890510 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.890499 2555 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 26 15:05:18.890574 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.890564 2555 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 26 15:05:18.892472 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.892456 2555 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 26 15:05:18.893763 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.893751 2555 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 26 15:05:18.893805 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.893779 2555 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 26 15:05:18.894415 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.894399 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 26 15:05:18.894415 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.894415 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 26 15:05:18.894529 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.894427 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 26 15:05:18.894529 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.894432 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 26 15:05:18.894529 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.894440 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 26 15:05:18.894529 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.894445 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 26 15:05:18.894529 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.894452 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 26 15:05:18.894529 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.894462 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 26 15:05:18.894529 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.894468 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 26 15:05:18.894529 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.894473 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 26 15:05:18.894529 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.894485 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 26 15:05:18.894529 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.894490 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 26 15:05:18.894998 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.894979 2555 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tfhrq" Jul 26 15:05:18.896192 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.896181 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 26 15:05:18.896229 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.896193 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 26 15:05:18.899345 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.899309 2555 server.go:1268] "Started kubelet" Jul 26 15:05:18.899441 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.899414 2555 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 26 15:05:18.899518 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.899453 2555 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 26 15:05:18.899577 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.899538 2555 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 26 15:05:18.900292 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.900276 2555 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-21.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 26 15:05:18.900667 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:18.900646 2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-21.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 26 15:05:18.900745 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:18.900711 2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 26 15:05:18.901227 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.901211 2555 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 26 15:05:18.901222 ip-10-0-130-21 systemd[1]: Started Kubernetes Kubelet. Jul 26 15:05:18.902573 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.902554 2555 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tfhrq" Jul 26 15:05:18.902656 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.902602 2555 server.go:310] "Adding debug handlers to kubelet server" Jul 26 15:05:18.904234 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:18.904217 2555 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 26 15:05:18.904480 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:18.903695 2555 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-21.ec2.internal.18c5df45f63e3eee default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-21.ec2.internal,UID:ip-10-0-130-21.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-21.ec2.internal,},FirstTimestamp:2026-07-26 15:05:18.899281646 +0000 UTC m=+0.329739836,LastTimestamp:2026-07-26 15:05:18.899281646 +0000 UTC m=+0.329739836,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-21.ec2.internal,}" Jul 26 15:05:18.907977 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.907960 2555 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 26 15:05:18.907977 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.907966 2555 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 26 15:05:18.908110 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:18.908048 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-21.ec2.internal\" not found" Jul 26 15:05:18.908434 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.908420 2555 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 26 15:05:18.908434 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.908431 2555 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 26 15:05:18.908554 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.908444 2555 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 26 15:05:18.909152 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.909133 2555 factory.go:153] Registering CRI-O factory Jul 26 15:05:18.909152 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.909154 2555 factory.go:223] Registration of the crio container factory successfully Jul 26 15:05:18.909272 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.909222 2555 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 26 15:05:18.909272 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.909231 2555 factory.go:55] Registering systemd factory Jul 26 15:05:18.909272 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.909239 2555 factory.go:223] Registration of the systemd container factory successfully Jul 26 15:05:18.909272 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.909272 2555 factory.go:103] Registering Raw factory Jul 26 15:05:18.909455 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.909286 2555 manager.go:1196] Started watching for new ooms in manager Jul 26 15:05:18.909908 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.909893 2555 manager.go:349] Starting recovery of all containers Jul 26 15:05:18.910105 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:18.910080 2555 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-21.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 26 15:05:18.910208 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:18.910189 2555 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 26 15:05:18.911509 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.911492 2555 reconstruct.go:97] "Volume reconstruction finished" Jul 26 15:05:18.911509 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.911506 2555 reconciler.go:29] "Reconciler: start to sync state" Jul 26 15:05:18.920824 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.920807 2555 manager.go:354] Recovery completed Jul 26 15:05:18.924793 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.924777 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 26 15:05:18.926522 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.926507 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-21.ec2.internal" event="NodeHasSufficientMemory" Jul 26 15:05:18.926579 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.926533 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-21.ec2.internal" event="NodeHasNoDiskPressure" Jul 26 15:05:18.926579 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.926548 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-21.ec2.internal" event="NodeHasSufficientPID" Jul 26 15:05:18.927024 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.927007 2555 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 26 15:05:18.927024 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.927021 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 26 15:05:18.927124 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.927025 2555 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 26 15:05:18.927124 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.927052 2555 state_mem.go:36] "Initialized new in-memory state store" Jul 26 15:05:18.927623 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.927609 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-21.ec2.internal" event="NodeHasSufficientMemory" Jul 26 15:05:18.927692 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.927637 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-21.ec2.internal" event="NodeHasNoDiskPressure" Jul 26 15:05:18.927692 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.927650 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-21.ec2.internal" event="NodeHasSufficientPID" Jul 26 15:05:18.928072 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.928061 2555 policy_none.go:49] "None policy: Start" Jul 26 15:05:18.928121 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.928075 2555 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 26 15:05:18.928121 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.928085 2555 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 26 15:05:18.928968 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.928957 2555 policy_none.go:47] "Start" Jul 26 15:05:18.964600 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.964495 2555 manager.go:337] "Starting Device Plugin manager" Jul 26 15:05:18.964670 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:18.964618 2555 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 26 15:05:18.964670 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.964632 2555 server.go:87] "Starting device plugin registration server" Jul 26 15:05:18.964828 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.964818 2555 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 26 15:05:18.964828 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.964829 2555 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 26 15:05:18.964950 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.964914 2555 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 26 15:05:18.965016 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.964978 2555 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 26 15:05:18.965016 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:18.964986 2555 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 26 15:05:18.965485 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:18.965468 2555 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 26 15:05:18.965542 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:18.965513 2555 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-21.ec2.internal\" not found" Jul 26 15:05:19.049885 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:19.049859 2555 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 26 15:05:19.051018 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:19.050991 2555 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 26 15:05:19.051082 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:19.051035 2555 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 26 15:05:19.051082 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:19.051060 2555 kubelet.go:2448] "Starting kubelet main sync loop" Jul 26 15:05:19.051167 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:19.051097 2555 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 26 15:05:19.053605 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:19.053588 2555 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 26 15:05:19.065789 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:19.065772 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 26 15:05:19.066477 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:19.066463 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-21.ec2.internal" event="NodeHasSufficientMemory" Jul 26 15:05:19.066544 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:19.066489 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-21.ec2.internal" event="NodeHasNoDiskPressure" Jul 26 15:05:19.066544 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:19.066504 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-21.ec2.internal" event="NodeHasSufficientPID" Jul 26 15:05:19.066544 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:19.066529 2555 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-21.ec2.internal" Jul 26 15:05:19.074350 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:19.074336 2555 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-21.ec2.internal" Jul 26 15:05:19.074398 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:19.074355 2555 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-21.ec2.internal\": node \"ip-10-0-130-21.ec2.internal\" not found" Jul 26 15:05:19.090193 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:19.090151 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-21.ec2.internal\" not found" Jul 26 15:05:20.158794 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:20.158774 2555 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-21.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-21.ec2.internal"] Jul 26 15:05:20.158864 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:20.158822 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 26 15:05:20.160054 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:20.160038 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-21.ec2.internal" event="NodeHasSufficientMemory" Jul 26 15:05:20.160121 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:20.160064 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-21.ec2.internal" event="NodeHasNoDiskPressure" Jul 26 15:05:20.160121 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:20.160074 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-21.ec2.internal" event="NodeHasSufficientPID" Jul 26 15:05:20.161207 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:20.161195 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 26 15:05:20.161319 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:20.161305 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-21.ec2.internal" Jul 26 15:05:20.161392 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:20.161337 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 26 15:05:20.161859 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:20.161838 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-21.ec2.internal" event="NodeHasSufficientMemory" Jul 26 15:05:20.161941 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:20.161855 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-21.ec2.internal" event="NodeHasSufficientMemory" Jul 26 15:05:20.161941 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:20.161880 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-21.ec2.internal" event="NodeHasNoDiskPressure" Jul 26 15:05:20.161941 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:20.161889 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-21.ec2.internal" event="NodeHasSufficientPID" Jul 26 15:05:20.161941 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:20.161865 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-21.ec2.internal" event="NodeHasNoDiskPressure" Jul 26 15:05:20.162070 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:20.161957 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-21.ec2.internal" event="NodeHasSufficientPID" Jul 26 15:05:20.163129 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:20.163117 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-21.ec2.internal" Jul 26 15:05:20.163215 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:20.163138 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 26 15:05:20.163775 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:20.163759 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-21.ec2.internal" event="NodeHasSufficientMemory" Jul 26 15:05:20.163853 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:20.163784 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-21.ec2.internal" event="NodeHasNoDiskPressure" Jul 26 15:05:20.163853 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:20.163794 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-21.ec2.internal" event="NodeHasSufficientPID" Jul 26 15:05:20.190456 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:20.190439 2555 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-21.ec2.internal\" not found" node="ip-10-0-130-21.ec2.internal" Jul 26 15:05:20.194565 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:20.194550 2555 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-21.ec2.internal\" not found" node="ip-10-0-130-21.ec2.internal" Jul 26 15:05:20.197013 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:20.196999 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-21.ec2.internal\" not found" Jul 26 15:05:20.219781 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:20.219762 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b006c6a0d5bf9a38bdcc97149639ead8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-21.ec2.internal\" (UID: \"b006c6a0d5bf9a38bdcc97149639ead8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-21.ec2.internal" Jul 26 15:05:20.219839 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:20.219785 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b006c6a0d5bf9a38bdcc97149639ead8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-21.ec2.internal\" (UID: \"b006c6a0d5bf9a38bdcc97149639ead8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-21.ec2.internal" Jul 26 15:05:20.219839 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:20.219815 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b544e0067e3546297215dd780b690606-config\") pod \"kube-apiserver-proxy-ip-10-0-130-21.ec2.internal\" (UID: \"b544e0067e3546297215dd780b690606\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-21.ec2.internal" Jul 26 15:05:20.297567 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:20.297550 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-21.ec2.internal\" not found" Jul 26 15:05:20.320881 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:20.320853 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b006c6a0d5bf9a38bdcc97149639ead8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-21.ec2.internal\" (UID: \"b006c6a0d5bf9a38bdcc97149639ead8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-21.ec2.internal" Jul 26 15:05:20.320960 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:20.320908 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b006c6a0d5bf9a38bdcc97149639ead8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-21.ec2.internal\" (UID: \"b006c6a0d5bf9a38bdcc97149639ead8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-21.ec2.internal" Jul 26 15:05:20.320960 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:20.320945 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b006c6a0d5bf9a38bdcc97149639ead8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-21.ec2.internal\" (UID: \"b006c6a0d5bf9a38bdcc97149639ead8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-21.ec2.internal" Jul 26 15:05:20.320960 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:20.320953 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b006c6a0d5bf9a38bdcc97149639ead8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-21.ec2.internal\" (UID: \"b006c6a0d5bf9a38bdcc97149639ead8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-21.ec2.internal" Jul 26 15:05:20.321065 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:20.320977 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b544e0067e3546297215dd780b690606-config\") pod \"kube-apiserver-proxy-ip-10-0-130-21.ec2.internal\" (UID: \"b544e0067e3546297215dd780b690606\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-21.ec2.internal" Jul 26 15:05:20.321065 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:20.321005 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b544e0067e3546297215dd780b690606-config\") pod \"kube-apiserver-proxy-ip-10-0-130-21.ec2.internal\" (UID: \"b544e0067e3546297215dd780b690606\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-21.ec2.internal" Jul 26 15:05:20.398126 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:20.398105 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-21.ec2.internal\" not found" Jul 26 15:05:20.492523 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:20.492497 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-21.ec2.internal" Jul 26 15:05:20.498039 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:20.498024 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-21.ec2.internal" Jul 26 15:05:20.499058 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:20.499041 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-21.ec2.internal\" not found" Jul 26 15:05:20.599440 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:20.599417 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-21.ec2.internal\" not found" Jul 26 15:05:20.699921 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:20.699871 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-21.ec2.internal\" not found" Jul 26 15:05:20.800394 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:20.800376 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-21.ec2.internal\" not found" Jul 26 15:05:20.809844 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:20.809825 2555 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 26 15:05:20.858160 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:20.858144 2555 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 26 15:05:20.858500 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:20.858248 2555 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 26 15:05:20.858500 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:20.858309 2555 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 26 15:05:20.900572 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:20.900551 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-21.ec2.internal\" not found" Jul 26 15:05:20.913781 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:20.913750 2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-25 15:00:19 +0000 UTC" deadline="2028-01-19 03:01:47.030432302 +0000 UTC" Jul 26 15:05:20.913826 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:20.913783 2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12995h56m26.116652744s" Jul 26 15:05:21.001515 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:21.001462 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-21.ec2.internal\" not found" Jul 26 15:05:21.101813 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:21.101794 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-21.ec2.internal\" not found" Jul 26 15:05:21.169322 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:21.169290 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb006c6a0d5bf9a38bdcc97149639ead8.slice/crio-b38c67ff084ab4fece760467e4ec405f4400a668d69913fe55e17ef5f126386e WatchSource:0}: Error finding container b38c67ff084ab4fece760467e4ec405f4400a668d69913fe55e17ef5f126386e: Status 404 returned error can't find the container with id b38c67ff084ab4fece760467e4ec405f4400a668d69913fe55e17ef5f126386e Jul 26 15:05:21.170862 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:21.170837 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb544e0067e3546297215dd780b690606.slice/crio-49b8ec93df634d80e84be903010f9a022d96baeeb991f779e3e4ae7e82c5bf55 WatchSource:0}: Error finding container 49b8ec93df634d80e84be903010f9a022d96baeeb991f779e3e4ae7e82c5bf55: Status 404 returned error can't find the container with id 49b8ec93df634d80e84be903010f9a022d96baeeb991f779e3e4ae7e82c5bf55 Jul 26 15:05:21.174482 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.174463 2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 26 15:05:21.202206 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:21.202190 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-21.ec2.internal\" not found" Jul 26 15:05:21.258885 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.258850 2555 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 26 15:05:21.302509 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:21.302486 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-21.ec2.internal\" not found" Jul 26 15:05:21.333207 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.333187 2555 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 26 15:05:21.414880 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.414857 2555 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-21.ec2.internal" Jul 26 15:05:21.425266 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.425249 2555 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 26 15:05:21.426397 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.426384 2555 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-21.ec2.internal" Jul 26 15:05:21.432344 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.432328 2555 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 26 15:05:21.696015 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.695992 2555 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 26 15:05:21.897617 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.897593 2555 apiserver.go:52] "Watching apiserver" Jul 26 15:05:21.902414 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.902388 2555 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 26 15:05:21.902842 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.902819 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-dsgxs","openshift-dns/node-resolver-rfxhs","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-21.ec2.internal","openshift-multus/multus-nzw98","openshift-network-diagnostics/network-check-target-k8xdr","kube-system/global-pull-secret-syncer-b8spm","kube-system/konnectivity-agent-bndqd","kube-system/kube-apiserver-proxy-ip-10-0-130-21.ec2.internal","openshift-image-registry/node-ca-j6hp5","openshift-multus/multus-additional-cni-plugins-4kk2l","openshift-multus/network-metrics-daemon-qlw9m","openshift-network-operator/iptables-alerter-frrgx","openshift-ovn-kubernetes/ovnkube-node-9f857","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjswx"] Jul 26 15:05:21.904245 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.904225 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bndqd" Jul 26 15:05:21.905899 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.905839 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:05:21.906790 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.906770 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 26 15:05:21.907886 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.907042 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-vml2n\"" Jul 26 15:05:21.907886 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.907205 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 26 15:05:21.907886 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.907544 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjswx" Jul 26 15:05:21.907886 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.907607 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 26 15:05:21.907886 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.907678 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 26 15:05:21.907886 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.907733 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 26 15:05:21.908349 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.908132 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-p5tnt\"" Jul 26 15:05:21.908349 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.908176 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 26 15:05:21.908349 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.908215 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 26 15:05:21.908349 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.908179 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 26 15:05:21.909553 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.909244 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nzw98" Jul 26 15:05:21.911419 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.910597 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 26 15:05:21.911419 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.910723 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-kcrhs\"" Jul 26 15:05:21.911419 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.910799 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 26 15:05:21.911419 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.910878 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k8xdr" Jul 26 15:05:21.911419 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.910960 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 26 15:05:21.911419 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:21.910958 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k8xdr" podUID="0357f424-c295-4aa7-8330-f551894f81d8" Jul 26 15:05:21.911419 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.911175 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 26 15:05:21.911419 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.911175 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-bbt8m\"" Jul 26 15:05:21.911419 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.911198 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 26 15:05:21.911419 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.911311 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 26 15:05:21.911908 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.911481 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 26 15:05:21.911983 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.911922 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8spm" Jul 26 15:05:21.912032 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:21.911988 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8spm" podUID="706538eb-583d-4fca-8f24-285cd3ab940c" Jul 26 15:05:21.913338 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.913321 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-j6hp5" Jul 26 15:05:21.915721 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.914364 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dsgxs" Jul 26 15:05:21.915721 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.915273 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 26 15:05:21.915721 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.915459 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 26 15:05:21.915721 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.915568 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 26 15:05:21.915721 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.915578 2555 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 26 15:05:21.915721 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.915633 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-knwdb\"" Jul 26 15:05:21.917144 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.917081 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rfxhs" Jul 26 15:05:21.918000 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.917978 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 26 15:05:21.918091 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.918042 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 26 15:05:21.918981 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.918961 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-xb6gx\"" Jul 26 15:05:21.919066 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.919009 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 26 15:05:21.919527 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.919508 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-l8lwl\"" Jul 26 15:05:21.920544 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.920522 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4kk2l" Jul 26 15:05:21.920901 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.920882 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 26 15:05:21.922718 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.922702 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlw9m" Jul 26 15:05:21.922794 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.922723 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 26 15:05:21.922794 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:21.922759 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlw9m" podUID="1b6101bd-09a1-444f-ae02-50384ba66923" Jul 26 15:05:21.922970 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.922953 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 26 15:05:21.923037 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.923001 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6z48m\"" Jul 26 15:05:21.924345 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.924326 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-frrgx" Jul 26 15:05:21.926208 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.926191 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 26 15:05:21.926438 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.926421 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-tcfgc\"" Jul 26 15:05:21.926524 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.926499 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 26 15:05:21.926683 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.926426 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 26 15:05:21.927176 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.927158 2555 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 26 15:05:21.930765 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.930734 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e1c832bc-b107-450a-b19d-ae4848abbad9-etc-selinux\") pod \"aws-ebs-csi-driver-node-fjswx\" (UID: \"e1c832bc-b107-450a-b19d-ae4848abbad9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjswx" Jul 26 15:05:21.930856 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.930767 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1fbafda6-6242-400e-b2d4-24584608e5f5-run-ovn\") pod \"ovnkube-node-9f857\" (UID: \"1fbafda6-6242-400e-b2d4-24584608e5f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:05:21.930856 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.930792 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvtdz\" (UniqueName: \"kubernetes.io/projected/1fbafda6-6242-400e-b2d4-24584608e5f5-kube-api-access-bvtdz\") pod \"ovnkube-node-9f857\" (UID: \"1fbafda6-6242-400e-b2d4-24584608e5f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:05:21.930856 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.930815 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/db621adf-49db-46b5-9224-4c526e9ae434-host-run-k8s-cni-cncf-io\") pod \"multus-nzw98\" (UID: \"db621adf-49db-46b5-9224-4c526e9ae434\") " pod="openshift-multus/multus-nzw98" Jul 26 15:05:21.930856 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.930837 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d648d15e-a430-43e7-bcf9-0ff82b55e678-lib-modules\") pod \"tuned-dsgxs\" (UID: \"d648d15e-a430-43e7-bcf9-0ff82b55e678\") " pod="openshift-cluster-node-tuning-operator/tuned-dsgxs" Jul 26 15:05:21.931073 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.930859 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1fbafda6-6242-400e-b2d4-24584608e5f5-host-run-netns\") pod \"ovnkube-node-9f857\" (UID: \"1fbafda6-6242-400e-b2d4-24584608e5f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:05:21.931073 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.930895 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/db621adf-49db-46b5-9224-4c526e9ae434-multus-cni-dir\") pod \"multus-nzw98\" (UID: \"db621adf-49db-46b5-9224-4c526e9ae434\") " pod="openshift-multus/multus-nzw98" Jul 26 15:05:21.931073 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.930924 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/db621adf-49db-46b5-9224-4c526e9ae434-hostroot\") pod \"multus-nzw98\" (UID: \"db621adf-49db-46b5-9224-4c526e9ae434\") " pod="openshift-multus/multus-nzw98" Jul 26 15:05:21.931073 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.930961 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/db621adf-49db-46b5-9224-4c526e9ae434-multus-daemon-config\") pod \"multus-nzw98\" (UID: \"db621adf-49db-46b5-9224-4c526e9ae434\") " pod="openshift-multus/multus-nzw98" Jul 26 15:05:21.931073 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.930982 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d648d15e-a430-43e7-bcf9-0ff82b55e678-etc-systemd\") pod \"tuned-dsgxs\" (UID: \"d648d15e-a430-43e7-bcf9-0ff82b55e678\") " pod="openshift-cluster-node-tuning-operator/tuned-dsgxs" Jul 26 15:05:21.931073 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.931003 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk4sv\" (UniqueName: \"kubernetes.io/projected/57b3ab2c-f95e-45d9-9caa-6c1eac37919c-kube-api-access-mk4sv\") pod \"node-resolver-rfxhs\" (UID: \"57b3ab2c-f95e-45d9-9caa-6c1eac37919c\") " pod="openshift-dns/node-resolver-rfxhs" Jul 26 15:05:21.931073 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.931024 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1fbafda6-6242-400e-b2d4-24584608e5f5-run-systemd\") pod \"ovnkube-node-9f857\" (UID: \"1fbafda6-6242-400e-b2d4-24584608e5f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:05:21.931073 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.931054 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1fbafda6-6242-400e-b2d4-24584608e5f5-host-cni-netd\") pod \"ovnkube-node-9f857\" (UID: \"1fbafda6-6242-400e-b2d4-24584608e5f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:05:21.931393 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.931106 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/53c30547-0ebf-4139-80c9-e452ed2b9578-serviceca\") pod \"node-ca-j6hp5\" (UID: \"53c30547-0ebf-4139-80c9-e452ed2b9578\") " pod="openshift-image-registry/node-ca-j6hp5" Jul 26 15:05:21.931393 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.931126 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d648d15e-a430-43e7-bcf9-0ff82b55e678-tmp\") pod \"tuned-dsgxs\" (UID: \"d648d15e-a430-43e7-bcf9-0ff82b55e678\") " pod="openshift-cluster-node-tuning-operator/tuned-dsgxs" Jul 26 15:05:21.931393 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.931140 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57b3ab2c-f95e-45d9-9caa-6c1eac37919c-tmp-dir\") pod \"node-resolver-rfxhs\" (UID: \"57b3ab2c-f95e-45d9-9caa-6c1eac37919c\") " pod="openshift-dns/node-resolver-rfxhs" Jul 26 15:05:21.931393 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.931169 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1fbafda6-6242-400e-b2d4-24584608e5f5-ovnkube-config\") pod \"ovnkube-node-9f857\" (UID: \"1fbafda6-6242-400e-b2d4-24584608e5f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:05:21.931393 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.931202 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/db621adf-49db-46b5-9224-4c526e9ae434-os-release\") pod \"multus-nzw98\" (UID: \"db621adf-49db-46b5-9224-4c526e9ae434\") " pod="openshift-multus/multus-nzw98" Jul 26 15:05:21.931393 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.931242 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db621adf-49db-46b5-9224-4c526e9ae434-etc-kubernetes\") pod \"multus-nzw98\" (UID: \"db621adf-49db-46b5-9224-4c526e9ae434\") " pod="openshift-multus/multus-nzw98" Jul 26 15:05:21.931393 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.931265 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/64483b31-45cb-4fa9-93a4-9726dc1bbd99-agent-certs\") pod \"konnectivity-agent-bndqd\" (UID: \"64483b31-45cb-4fa9-93a4-9726dc1bbd99\") " pod="kube-system/konnectivity-agent-bndqd" Jul 26 15:05:21.931393 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.931298 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d648d15e-a430-43e7-bcf9-0ff82b55e678-etc-kubernetes\") pod \"tuned-dsgxs\" (UID: \"d648d15e-a430-43e7-bcf9-0ff82b55e678\") " pod="openshift-cluster-node-tuning-operator/tuned-dsgxs" Jul 26 15:05:21.931393 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.931335 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d648d15e-a430-43e7-bcf9-0ff82b55e678-etc-sysctl-conf\") pod \"tuned-dsgxs\" (UID: \"d648d15e-a430-43e7-bcf9-0ff82b55e678\") " pod="openshift-cluster-node-tuning-operator/tuned-dsgxs" Jul 26 15:05:21.931393 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.931356 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1fbafda6-6242-400e-b2d4-24584608e5f5-node-log\") pod \"ovnkube-node-9f857\" (UID: \"1fbafda6-6242-400e-b2d4-24584608e5f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:05:21.931393 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.931384 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1fbafda6-6242-400e-b2d4-24584608e5f5-log-socket\") pod \"ovnkube-node-9f857\" (UID: \"1fbafda6-6242-400e-b2d4-24584608e5f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:05:21.931692 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.931399 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1fbafda6-6242-400e-b2d4-24584608e5f5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9f857\" (UID: \"1fbafda6-6242-400e-b2d4-24584608e5f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:05:21.931692 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.931421 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/db621adf-49db-46b5-9224-4c526e9ae434-system-cni-dir\") pod \"multus-nzw98\" (UID: \"db621adf-49db-46b5-9224-4c526e9ae434\") " pod="openshift-multus/multus-nzw98" Jul 26 15:05:21.931692 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.931441 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d648d15e-a430-43e7-bcf9-0ff82b55e678-var-lib-kubelet\") pod \"tuned-dsgxs\" (UID: \"d648d15e-a430-43e7-bcf9-0ff82b55e678\") " pod="openshift-cluster-node-tuning-operator/tuned-dsgxs" Jul 26 15:05:21.931692 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.931457 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1fbafda6-6242-400e-b2d4-24584608e5f5-ovn-node-metrics-cert\") pod \"ovnkube-node-9f857\" (UID: \"1fbafda6-6242-400e-b2d4-24584608e5f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:05:21.931692 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.931470 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4mqj\" (UniqueName: \"kubernetes.io/projected/0357f424-c295-4aa7-8330-f551894f81d8-kube-api-access-r4mqj\") pod \"network-check-target-k8xdr\" (UID: \"0357f424-c295-4aa7-8330-f551894f81d8\") " pod="openshift-network-diagnostics/network-check-target-k8xdr" Jul 26 15:05:21.931692 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.931485 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d648d15e-a430-43e7-bcf9-0ff82b55e678-host\") pod \"tuned-dsgxs\" (UID: \"d648d15e-a430-43e7-bcf9-0ff82b55e678\") " pod="openshift-cluster-node-tuning-operator/tuned-dsgxs" Jul 26 15:05:21.931692 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.931502 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1fbafda6-6242-400e-b2d4-24584608e5f5-host-kubelet\") pod \"ovnkube-node-9f857\" (UID: \"1fbafda6-6242-400e-b2d4-24584608e5f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:05:21.931692 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.931519 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1fbafda6-6242-400e-b2d4-24584608e5f5-var-lib-openvswitch\") pod \"ovnkube-node-9f857\" (UID: \"1fbafda6-6242-400e-b2d4-24584608e5f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:05:21.931692 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.931552 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/db621adf-49db-46b5-9224-4c526e9ae434-host-var-lib-cni-bin\") pod \"multus-nzw98\" (UID: \"db621adf-49db-46b5-9224-4c526e9ae434\") " pod="openshift-multus/multus-nzw98" Jul 26 15:05:21.931692 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.931586 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/db621adf-49db-46b5-9224-4c526e9ae434-multus-conf-dir\") pod \"multus-nzw98\" (UID: \"db621adf-49db-46b5-9224-4c526e9ae434\") " pod="openshift-multus/multus-nzw98" Jul 26 15:05:21.931692 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.931617 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e1c832bc-b107-450a-b19d-ae4848abbad9-socket-dir\") pod \"aws-ebs-csi-driver-node-fjswx\" (UID: \"e1c832bc-b107-450a-b19d-ae4848abbad9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjswx" Jul 26 15:05:21.931692 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.931647 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/53c30547-0ebf-4139-80c9-e452ed2b9578-host\") pod \"node-ca-j6hp5\" (UID: \"53c30547-0ebf-4139-80c9-e452ed2b9578\") " pod="openshift-image-registry/node-ca-j6hp5" Jul 26 15:05:21.931692 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.931671 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1fbafda6-6242-400e-b2d4-24584608e5f5-host-slash\") pod \"ovnkube-node-9f857\" (UID: \"1fbafda6-6242-400e-b2d4-24584608e5f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:05:21.932097 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.931692 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d648d15e-a430-43e7-bcf9-0ff82b55e678-sys\") pod \"tuned-dsgxs\" (UID: \"d648d15e-a430-43e7-bcf9-0ff82b55e678\") " pod="openshift-cluster-node-tuning-operator/tuned-dsgxs" Jul 26 15:05:21.932097 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.931734 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1fbafda6-6242-400e-b2d4-24584608e5f5-systemd-units\") pod \"ovnkube-node-9f857\" (UID: \"1fbafda6-6242-400e-b2d4-24584608e5f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:05:21.932097 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.931765 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/db621adf-49db-46b5-9224-4c526e9ae434-host-var-lib-kubelet\") pod \"multus-nzw98\" (UID: \"db621adf-49db-46b5-9224-4c526e9ae434\") " pod="openshift-multus/multus-nzw98" Jul 26 15:05:21.932097 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.931804 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/706538eb-583d-4fca-8f24-285cd3ab940c-original-pull-secret\") pod \"global-pull-secret-syncer-b8spm\" (UID: \"706538eb-583d-4fca-8f24-285cd3ab940c\") " pod="kube-system/global-pull-secret-syncer-b8spm" Jul 26 15:05:21.932097 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.931854 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d648d15e-a430-43e7-bcf9-0ff82b55e678-etc-sysctl-d\") pod \"tuned-dsgxs\" (UID: \"d648d15e-a430-43e7-bcf9-0ff82b55e678\") " pod="openshift-cluster-node-tuning-operator/tuned-dsgxs" Jul 26 15:05:21.932097 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.931883 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1fbafda6-6242-400e-b2d4-24584608e5f5-etc-openvswitch\") pod \"ovnkube-node-9f857\" (UID: \"1fbafda6-6242-400e-b2d4-24584608e5f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:05:21.932097 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.931910 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1fbafda6-6242-400e-b2d4-24584608e5f5-host-cni-bin\") pod \"ovnkube-node-9f857\" (UID: \"1fbafda6-6242-400e-b2d4-24584608e5f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:05:21.932097 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.931950 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1fbafda6-6242-400e-b2d4-24584608e5f5-ovnkube-script-lib\") pod \"ovnkube-node-9f857\" (UID: \"1fbafda6-6242-400e-b2d4-24584608e5f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:05:21.932097 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.931973 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e1c832bc-b107-450a-b19d-ae4848abbad9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fjswx\" (UID: \"e1c832bc-b107-450a-b19d-ae4848abbad9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjswx" Jul 26 15:05:21.932097 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.931998 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e1c832bc-b107-450a-b19d-ae4848abbad9-registration-dir\") pod \"aws-ebs-csi-driver-node-fjswx\" (UID: \"e1c832bc-b107-450a-b19d-ae4848abbad9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjswx" Jul 26 15:05:21.932097 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.932051 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1fbafda6-6242-400e-b2d4-24584608e5f5-env-overrides\") pod \"ovnkube-node-9f857\" (UID: \"1fbafda6-6242-400e-b2d4-24584608e5f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:05:21.932097 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.932077 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/db621adf-49db-46b5-9224-4c526e9ae434-host-var-lib-cni-multus\") pod \"multus-nzw98\" (UID: \"db621adf-49db-46b5-9224-4c526e9ae434\") " pod="openshift-multus/multus-nzw98" Jul 26 15:05:21.932622 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.932107 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5skb\" (UniqueName: \"kubernetes.io/projected/d648d15e-a430-43e7-bcf9-0ff82b55e678-kube-api-access-h5skb\") pod \"tuned-dsgxs\" (UID: \"d648d15e-a430-43e7-bcf9-0ff82b55e678\") " pod="openshift-cluster-node-tuning-operator/tuned-dsgxs" Jul 26 15:05:21.932622 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.932149 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/db621adf-49db-46b5-9224-4c526e9ae434-host-run-netns\") pod \"multus-nzw98\" (UID: \"db621adf-49db-46b5-9224-4c526e9ae434\") " pod="openshift-multus/multus-nzw98" Jul 26 15:05:21.932622 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.932199 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/64483b31-45cb-4fa9-93a4-9726dc1bbd99-konnectivity-ca\") pod \"konnectivity-agent-bndqd\" (UID: \"64483b31-45cb-4fa9-93a4-9726dc1bbd99\") " pod="kube-system/konnectivity-agent-bndqd" Jul 26 15:05:21.932622 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.932232 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9r5b\" (UniqueName: \"kubernetes.io/projected/53c30547-0ebf-4139-80c9-e452ed2b9578-kube-api-access-z9r5b\") pod \"node-ca-j6hp5\" (UID: \"53c30547-0ebf-4139-80c9-e452ed2b9578\") " pod="openshift-image-registry/node-ca-j6hp5" Jul 26 15:05:21.932622 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.932254 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d648d15e-a430-43e7-bcf9-0ff82b55e678-etc-sysconfig\") pod \"tuned-dsgxs\" (UID: \"d648d15e-a430-43e7-bcf9-0ff82b55e678\") " pod="openshift-cluster-node-tuning-operator/tuned-dsgxs" Jul 26 15:05:21.932622 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.932281 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d648d15e-a430-43e7-bcf9-0ff82b55e678-etc-tuned\") pod \"tuned-dsgxs\" (UID: \"d648d15e-a430-43e7-bcf9-0ff82b55e678\") " pod="openshift-cluster-node-tuning-operator/tuned-dsgxs" Jul 26 15:05:21.932622 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.932304 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1fbafda6-6242-400e-b2d4-24584608e5f5-run-openvswitch\") pod \"ovnkube-node-9f857\" (UID: \"1fbafda6-6242-400e-b2d4-24584608e5f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:05:21.932622 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.932326 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/db621adf-49db-46b5-9224-4c526e9ae434-cni-binary-copy\") pod \"multus-nzw98\" (UID: \"db621adf-49db-46b5-9224-4c526e9ae434\") " pod="openshift-multus/multus-nzw98" Jul 26 15:05:21.932622 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.932346 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d648d15e-a430-43e7-bcf9-0ff82b55e678-etc-modprobe-d\") pod \"tuned-dsgxs\" (UID: \"d648d15e-a430-43e7-bcf9-0ff82b55e678\") " pod="openshift-cluster-node-tuning-operator/tuned-dsgxs" Jul 26 15:05:21.932622 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.932375 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d648d15e-a430-43e7-bcf9-0ff82b55e678-run\") pod \"tuned-dsgxs\" (UID: \"d648d15e-a430-43e7-bcf9-0ff82b55e678\") " pod="openshift-cluster-node-tuning-operator/tuned-dsgxs" Jul 26 15:05:21.932622 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.932396 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1fbafda6-6242-400e-b2d4-24584608e5f5-host-run-ovn-kubernetes\") pod \"ovnkube-node-9f857\" (UID: \"1fbafda6-6242-400e-b2d4-24584608e5f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:05:21.932622 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.932586 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/db621adf-49db-46b5-9224-4c526e9ae434-multus-socket-dir-parent\") pod \"multus-nzw98\" (UID: \"db621adf-49db-46b5-9224-4c526e9ae434\") " pod="openshift-multus/multus-nzw98" Jul 26 15:05:21.933076 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.932634 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e1c832bc-b107-450a-b19d-ae4848abbad9-device-dir\") pod \"aws-ebs-csi-driver-node-fjswx\" (UID: \"e1c832bc-b107-450a-b19d-ae4848abbad9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjswx" Jul 26 15:05:21.933076 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.932658 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e1c832bc-b107-450a-b19d-ae4848abbad9-sys-fs\") pod \"aws-ebs-csi-driver-node-fjswx\" (UID: \"e1c832bc-b107-450a-b19d-ae4848abbad9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjswx" Jul 26 15:05:21.933076 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.932681 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbwnr\" (UniqueName: \"kubernetes.io/projected/e1c832bc-b107-450a-b19d-ae4848abbad9-kube-api-access-fbwnr\") pod \"aws-ebs-csi-driver-node-fjswx\" (UID: \"e1c832bc-b107-450a-b19d-ae4848abbad9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjswx" Jul 26 15:05:21.933076 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.932703 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/db621adf-49db-46b5-9224-4c526e9ae434-cnibin\") pod \"multus-nzw98\" (UID: \"db621adf-49db-46b5-9224-4c526e9ae434\") " pod="openshift-multus/multus-nzw98" Jul 26 15:05:21.933076 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.932724 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wtgd\" (UniqueName: \"kubernetes.io/projected/db621adf-49db-46b5-9224-4c526e9ae434-kube-api-access-7wtgd\") pod \"multus-nzw98\" (UID: \"db621adf-49db-46b5-9224-4c526e9ae434\") " pod="openshift-multus/multus-nzw98" Jul 26 15:05:21.933076 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.932746 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/706538eb-583d-4fca-8f24-285cd3ab940c-dbus\") pod \"global-pull-secret-syncer-b8spm\" (UID: \"706538eb-583d-4fca-8f24-285cd3ab940c\") " pod="kube-system/global-pull-secret-syncer-b8spm" Jul 26 15:05:21.933076 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.932768 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/57b3ab2c-f95e-45d9-9caa-6c1eac37919c-hosts-file\") pod \"node-resolver-rfxhs\" (UID: \"57b3ab2c-f95e-45d9-9caa-6c1eac37919c\") " pod="openshift-dns/node-resolver-rfxhs" Jul 26 15:05:21.933076 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.932789 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/db621adf-49db-46b5-9224-4c526e9ae434-host-run-multus-certs\") pod \"multus-nzw98\" (UID: \"db621adf-49db-46b5-9224-4c526e9ae434\") " pod="openshift-multus/multus-nzw98" Jul 26 15:05:21.933076 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.932813 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/706538eb-583d-4fca-8f24-285cd3ab940c-kubelet-config\") pod \"global-pull-secret-syncer-b8spm\" (UID: \"706538eb-583d-4fca-8f24-285cd3ab940c\") " pod="kube-system/global-pull-secret-syncer-b8spm" Jul 26 15:05:21.951791 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.951770 2555 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-7dvgs" Jul 26 15:05:21.958977 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:21.958959 2555 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-7dvgs" Jul 26 15:05:22.016215 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.016191 2555 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 26 15:05:22.033381 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.033354 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/53c30547-0ebf-4139-80c9-e452ed2b9578-serviceca\") pod \"node-ca-j6hp5\" (UID: \"53c30547-0ebf-4139-80c9-e452ed2b9578\") " pod="openshift-image-registry/node-ca-j6hp5" Jul 26 15:05:22.033513 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.033398 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d648d15e-a430-43e7-bcf9-0ff82b55e678-tmp\") pod \"tuned-dsgxs\" (UID: \"d648d15e-a430-43e7-bcf9-0ff82b55e678\") " pod="openshift-cluster-node-tuning-operator/tuned-dsgxs" Jul 26 15:05:22.033513 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.033422 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57b3ab2c-f95e-45d9-9caa-6c1eac37919c-tmp-dir\") pod \"node-resolver-rfxhs\" (UID: \"57b3ab2c-f95e-45d9-9caa-6c1eac37919c\") " pod="openshift-dns/node-resolver-rfxhs" Jul 26 15:05:22.034032 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.033695 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57b3ab2c-f95e-45d9-9caa-6c1eac37919c-tmp-dir\") pod \"node-resolver-rfxhs\" (UID: \"57b3ab2c-f95e-45d9-9caa-6c1eac37919c\") " pod="openshift-dns/node-resolver-rfxhs" Jul 26 15:05:22.034032 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.033765 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1fbafda6-6242-400e-b2d4-24584608e5f5-ovnkube-config\") pod \"ovnkube-node-9f857\" (UID: \"1fbafda6-6242-400e-b2d4-24584608e5f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:05:22.034032 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.033765 2555 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 26 15:05:22.034032 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.033798 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/53c30547-0ebf-4139-80c9-e452ed2b9578-serviceca\") pod \"node-ca-j6hp5\" (UID: \"53c30547-0ebf-4139-80c9-e452ed2b9578\") " pod="openshift-image-registry/node-ca-j6hp5" Jul 26 15:05:22.034032 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.033804 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/db621adf-49db-46b5-9224-4c526e9ae434-os-release\") pod \"multus-nzw98\" (UID: \"db621adf-49db-46b5-9224-4c526e9ae434\") " pod="openshift-multus/multus-nzw98" Jul 26 15:05:22.034032 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.033838 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db621adf-49db-46b5-9224-4c526e9ae434-etc-kubernetes\") pod \"multus-nzw98\" (UID: \"db621adf-49db-46b5-9224-4c526e9ae434\") " pod="openshift-multus/multus-nzw98" Jul 26 15:05:22.034032 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.033881 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/db621adf-49db-46b5-9224-4c526e9ae434-os-release\") pod \"multus-nzw98\" (UID: \"db621adf-49db-46b5-9224-4c526e9ae434\") " pod="openshift-multus/multus-nzw98" Jul 26 15:05:22.034032 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.033883 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/64483b31-45cb-4fa9-93a4-9726dc1bbd99-agent-certs\") pod \"konnectivity-agent-bndqd\" (UID: \"64483b31-45cb-4fa9-93a4-9726dc1bbd99\") " pod="kube-system/konnectivity-agent-bndqd" Jul 26 15:05:22.034032 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.033926 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db621adf-49db-46b5-9224-4c526e9ae434-etc-kubernetes\") pod \"multus-nzw98\" (UID: \"db621adf-49db-46b5-9224-4c526e9ae434\") " pod="openshift-multus/multus-nzw98" Jul 26 15:05:22.034032 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.033957 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d648d15e-a430-43e7-bcf9-0ff82b55e678-etc-kubernetes\") pod \"tuned-dsgxs\" (UID: \"d648d15e-a430-43e7-bcf9-0ff82b55e678\") " pod="openshift-cluster-node-tuning-operator/tuned-dsgxs" Jul 26 15:05:22.034032 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.033982 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d648d15e-a430-43e7-bcf9-0ff82b55e678-etc-sysctl-conf\") pod \"tuned-dsgxs\" (UID: \"d648d15e-a430-43e7-bcf9-0ff82b55e678\") " pod="openshift-cluster-node-tuning-operator/tuned-dsgxs" Jul 26 15:05:22.034032 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.034003 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1fbafda6-6242-400e-b2d4-24584608e5f5-node-log\") pod \"ovnkube-node-9f857\" (UID: \"1fbafda6-6242-400e-b2d4-24584608e5f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:05:22.034032 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.034036 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d648d15e-a430-43e7-bcf9-0ff82b55e678-etc-kubernetes\") pod \"tuned-dsgxs\" (UID: \"d648d15e-a430-43e7-bcf9-0ff82b55e678\") " pod="openshift-cluster-node-tuning-operator/tuned-dsgxs" Jul 26 15:05:22.034662 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.034072 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1fbafda6-6242-400e-b2d4-24584608e5f5-log-socket\") pod \"ovnkube-node-9f857\" (UID: \"1fbafda6-6242-400e-b2d4-24584608e5f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:05:22.034662 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.034110 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1fbafda6-6242-400e-b2d4-24584608e5f5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9f857\" (UID: \"1fbafda6-6242-400e-b2d4-24584608e5f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:05:22.034662 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.034136 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/db621adf-49db-46b5-9224-4c526e9ae434-system-cni-dir\") pod \"multus-nzw98\" (UID: \"db621adf-49db-46b5-9224-4c526e9ae434\") " pod="openshift-multus/multus-nzw98" Jul 26 15:05:22.034662 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.034151 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d648d15e-a430-43e7-bcf9-0ff82b55e678-etc-sysctl-conf\") pod \"tuned-dsgxs\" (UID: \"d648d15e-a430-43e7-bcf9-0ff82b55e678\") " pod="openshift-cluster-node-tuning-operator/tuned-dsgxs" Jul 26 15:05:22.034662 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.034167 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/96d47908-2b56-4fbe-a6e4-6472cd3a5cd4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4kk2l\" (UID: \"96d47908-2b56-4fbe-a6e4-6472cd3a5cd4\") " pod="openshift-multus/multus-additional-cni-plugins-4kk2l" Jul 26 15:05:22.034662 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.034195 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d648d15e-a430-43e7-bcf9-0ff82b55e678-var-lib-kubelet\") pod \"tuned-dsgxs\" (UID: \"d648d15e-a430-43e7-bcf9-0ff82b55e678\") " pod="openshift-cluster-node-tuning-operator/tuned-dsgxs" Jul 26 15:05:22.034662 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.034217 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1fbafda6-6242-400e-b2d4-24584608e5f5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9f857\" (UID: \"1fbafda6-6242-400e-b2d4-24584608e5f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:05:22.034662 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.034224 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1fbafda6-6242-400e-b2d4-24584608e5f5-ovn-node-metrics-cert\") pod \"ovnkube-node-9f857\" (UID: \"1fbafda6-6242-400e-b2d4-24584608e5f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:05:22.034662 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.034237 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1fbafda6-6242-400e-b2d4-24584608e5f5-log-socket\") pod \"ovnkube-node-9f857\" (UID: \"1fbafda6-6242-400e-b2d4-24584608e5f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:05:22.034662 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.034196 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1fbafda6-6242-400e-b2d4-24584608e5f5-ovnkube-config\") pod \"ovnkube-node-9f857\" (UID: \"1fbafda6-6242-400e-b2d4-24584608e5f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:05:22.034662 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.034237 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1fbafda6-6242-400e-b2d4-24584608e5f5-node-log\") pod \"ovnkube-node-9f857\" (UID: \"1fbafda6-6242-400e-b2d4-24584608e5f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:05:22.034662 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.034272 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5xq4\" (UniqueName: \"kubernetes.io/projected/96d47908-2b56-4fbe-a6e4-6472cd3a5cd4-kube-api-access-q5xq4\") pod \"multus-additional-cni-plugins-4kk2l\" (UID: \"96d47908-2b56-4fbe-a6e4-6472cd3a5cd4\") " pod="openshift-multus/multus-additional-cni-plugins-4kk2l" Jul 26 15:05:22.034662 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.034298 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/db621adf-49db-46b5-9224-4c526e9ae434-system-cni-dir\") pod \"multus-nzw98\" (UID: \"db621adf-49db-46b5-9224-4c526e9ae434\") " pod="openshift-multus/multus-nzw98" Jul 26 15:05:22.034662 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.034302 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4mqj\" (UniqueName: \"kubernetes.io/projected/0357f424-c295-4aa7-8330-f551894f81d8-kube-api-access-r4mqj\") pod \"network-check-target-k8xdr\" (UID: \"0357f424-c295-4aa7-8330-f551894f81d8\") " pod="openshift-network-diagnostics/network-check-target-k8xdr" Jul 26 15:05:22.034662 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.034324 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d648d15e-a430-43e7-bcf9-0ff82b55e678-host\") pod \"tuned-dsgxs\" (UID: \"d648d15e-a430-43e7-bcf9-0ff82b55e678\") " pod="openshift-cluster-node-tuning-operator/tuned-dsgxs" Jul 26 15:05:22.034662 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.034380 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d648d15e-a430-43e7-bcf9-0ff82b55e678-var-lib-kubelet\") pod \"tuned-dsgxs\" (UID: \"d648d15e-a430-43e7-bcf9-0ff82b55e678\") " pod="openshift-cluster-node-tuning-operator/tuned-dsgxs" Jul 26 15:05:22.034662 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.034361 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1fbafda6-6242-400e-b2d4-24584608e5f5-host-kubelet\") pod \"ovnkube-node-9f857\" (UID: \"1fbafda6-6242-400e-b2d4-24584608e5f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:05:22.035527 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.034416 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1fbafda6-6242-400e-b2d4-24584608e5f5-var-lib-openvswitch\") pod \"ovnkube-node-9f857\" (UID: \"1fbafda6-6242-400e-b2d4-24584608e5f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:05:22.035527 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.034439 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/db621adf-49db-46b5-9224-4c526e9ae434-host-var-lib-cni-bin\") pod \"multus-nzw98\" (UID: \"db621adf-49db-46b5-9224-4c526e9ae434\") " pod="openshift-multus/multus-nzw98" Jul 26 15:05:22.035527 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.034455 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d648d15e-a430-43e7-bcf9-0ff82b55e678-host\") pod \"tuned-dsgxs\" (UID: \"d648d15e-a430-43e7-bcf9-0ff82b55e678\") " pod="openshift-cluster-node-tuning-operator/tuned-dsgxs" Jul 26 15:05:22.035527 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.034463 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/db621adf-49db-46b5-9224-4c526e9ae434-multus-conf-dir\") pod \"multus-nzw98\" (UID: \"db621adf-49db-46b5-9224-4c526e9ae434\") " pod="openshift-multus/multus-nzw98" Jul 26 15:05:22.035527 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.034490 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/96d47908-2b56-4fbe-a6e4-6472cd3a5cd4-system-cni-dir\") pod \"multus-additional-cni-plugins-4kk2l\" (UID: \"96d47908-2b56-4fbe-a6e4-6472cd3a5cd4\") " pod="openshift-multus/multus-additional-cni-plugins-4kk2l" Jul 26 15:05:22.035527 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.034521 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/96d47908-2b56-4fbe-a6e4-6472cd3a5cd4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4kk2l\" (UID: \"96d47908-2b56-4fbe-a6e4-6472cd3a5cd4\") " pod="openshift-multus/multus-additional-cni-plugins-4kk2l" Jul 26 15:05:22.035527 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.034525 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/db621adf-49db-46b5-9224-4c526e9ae434-multus-conf-dir\") pod \"multus-nzw98\" (UID: \"db621adf-49db-46b5-9224-4c526e9ae434\") " pod="openshift-multus/multus-nzw98" Jul 26 15:05:22.035527 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.034550 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e1c832bc-b107-450a-b19d-ae4848abbad9-socket-dir\") pod \"aws-ebs-csi-driver-node-fjswx\" (UID: \"e1c832bc-b107-450a-b19d-ae4848abbad9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjswx" Jul 26 15:05:22.035527 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.034417 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1fbafda6-6242-400e-b2d4-24584608e5f5-host-kubelet\") pod \"ovnkube-node-9f857\" (UID: \"1fbafda6-6242-400e-b2d4-24584608e5f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:05:22.035527 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.034493 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1fbafda6-6242-400e-b2d4-24584608e5f5-var-lib-openvswitch\") pod \"ovnkube-node-9f857\" (UID: \"1fbafda6-6242-400e-b2d4-24584608e5f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:05:22.035527 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.034575 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/53c30547-0ebf-4139-80c9-e452ed2b9578-host\") pod \"node-ca-j6hp5\" (UID: \"53c30547-0ebf-4139-80c9-e452ed2b9578\") " pod="openshift-image-registry/node-ca-j6hp5" Jul 26 15:05:22.035527 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.034585 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/db621adf-49db-46b5-9224-4c526e9ae434-host-var-lib-cni-bin\") pod \"multus-nzw98\" (UID: \"db621adf-49db-46b5-9224-4c526e9ae434\") " pod="openshift-multus/multus-nzw98" Jul 26 15:05:22.035527 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.034613 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1fbafda6-6242-400e-b2d4-24584608e5f5-host-slash\") pod \"ovnkube-node-9f857\" (UID: \"1fbafda6-6242-400e-b2d4-24584608e5f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:05:22.035527 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.034627 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/53c30547-0ebf-4139-80c9-e452ed2b9578-host\") pod \"node-ca-j6hp5\" (UID: \"53c30547-0ebf-4139-80c9-e452ed2b9578\") " pod="openshift-image-registry/node-ca-j6hp5" Jul 26 15:05:22.035527 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.034654 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e1c832bc-b107-450a-b19d-ae4848abbad9-socket-dir\") pod \"aws-ebs-csi-driver-node-fjswx\" (UID: \"e1c832bc-b107-450a-b19d-ae4848abbad9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjswx" Jul 26 15:05:22.035527 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.034667 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b6101bd-09a1-444f-ae02-50384ba66923-metrics-certs\") pod \"network-metrics-daemon-qlw9m\" (UID: \"1b6101bd-09a1-444f-ae02-50384ba66923\") " pod="openshift-multus/network-metrics-daemon-qlw9m" Jul 26 15:05:22.035527 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.034682 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1fbafda6-6242-400e-b2d4-24584608e5f5-host-slash\") pod \"ovnkube-node-9f857\" (UID: \"1fbafda6-6242-400e-b2d4-24584608e5f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:05:22.036335 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.034713 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d648d15e-a430-43e7-bcf9-0ff82b55e678-sys\") pod \"tuned-dsgxs\" (UID: \"d648d15e-a430-43e7-bcf9-0ff82b55e678\") " pod="openshift-cluster-node-tuning-operator/tuned-dsgxs" Jul 26 15:05:22.036335 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.034737 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1fbafda6-6242-400e-b2d4-24584608e5f5-systemd-units\") pod \"ovnkube-node-9f857\" (UID: \"1fbafda6-6242-400e-b2d4-24584608e5f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:05:22.036335 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.034765 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d648d15e-a430-43e7-bcf9-0ff82b55e678-sys\") pod \"tuned-dsgxs\" (UID: \"d648d15e-a430-43e7-bcf9-0ff82b55e678\") " pod="openshift-cluster-node-tuning-operator/tuned-dsgxs" Jul 26 15:05:22.036335 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.034783 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/db621adf-49db-46b5-9224-4c526e9ae434-host-var-lib-kubelet\") pod \"multus-nzw98\" (UID: \"db621adf-49db-46b5-9224-4c526e9ae434\") " pod="openshift-multus/multus-nzw98" Jul 26 15:05:22.036335 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.034802 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1fbafda6-6242-400e-b2d4-24584608e5f5-systemd-units\") pod \"ovnkube-node-9f857\" (UID: \"1fbafda6-6242-400e-b2d4-24584608e5f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:05:22.036335 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.034808 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/706538eb-583d-4fca-8f24-285cd3ab940c-original-pull-secret\") pod \"global-pull-secret-syncer-b8spm\" (UID: \"706538eb-583d-4fca-8f24-285cd3ab940c\") " pod="kube-system/global-pull-secret-syncer-b8spm" Jul 26 15:05:22.036335 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.034835 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/96d47908-2b56-4fbe-a6e4-6472cd3a5cd4-os-release\") pod \"multus-additional-cni-plugins-4kk2l\" (UID: \"96d47908-2b56-4fbe-a6e4-6472cd3a5cd4\") " pod="openshift-multus/multus-additional-cni-plugins-4kk2l" Jul 26 15:05:22.036335 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.034838 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/db621adf-49db-46b5-9224-4c526e9ae434-host-var-lib-kubelet\") pod \"multus-nzw98\" (UID: \"db621adf-49db-46b5-9224-4c526e9ae434\") " pod="openshift-multus/multus-nzw98" Jul 26 15:05:22.036335 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.034859 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gdv6\" (UniqueName: \"kubernetes.io/projected/1b6101bd-09a1-444f-ae02-50384ba66923-kube-api-access-9gdv6\") pod \"network-metrics-daemon-qlw9m\" (UID: \"1b6101bd-09a1-444f-ae02-50384ba66923\") " pod="openshift-multus/network-metrics-daemon-qlw9m" Jul 26 15:05:22.036335 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:22.034917 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 26 15:05:22.036335 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.034961 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d648d15e-a430-43e7-bcf9-0ff82b55e678-etc-sysctl-d\") pod \"tuned-dsgxs\" (UID: \"d648d15e-a430-43e7-bcf9-0ff82b55e678\") " pod="openshift-cluster-node-tuning-operator/tuned-dsgxs" Jul 26 15:05:22.036335 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:22.034993 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/706538eb-583d-4fca-8f24-285cd3ab940c-original-pull-secret podName:706538eb-583d-4fca-8f24-285cd3ab940c nodeName:}" failed. No retries permitted until 2026-07-26 15:05:22.534974471 +0000 UTC m=+2.958821341 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/706538eb-583d-4fca-8f24-285cd3ab940c-original-pull-secret") pod "global-pull-secret-syncer-b8spm" (UID: "706538eb-583d-4fca-8f24-285cd3ab940c") : object "kube-system"/"original-pull-secret" not registered Jul 26 15:05:22.036335 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.035008 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1fbafda6-6242-400e-b2d4-24584608e5f5-etc-openvswitch\") pod \"ovnkube-node-9f857\" (UID: \"1fbafda6-6242-400e-b2d4-24584608e5f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:05:22.036335 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.035039 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1fbafda6-6242-400e-b2d4-24584608e5f5-host-cni-bin\") pod \"ovnkube-node-9f857\" (UID: \"1fbafda6-6242-400e-b2d4-24584608e5f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:05:22.036335 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.035064 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1fbafda6-6242-400e-b2d4-24584608e5f5-ovnkube-script-lib\") pod \"ovnkube-node-9f857\" (UID: \"1fbafda6-6242-400e-b2d4-24584608e5f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:05:22.036335 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.035066 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d648d15e-a430-43e7-bcf9-0ff82b55e678-etc-sysctl-d\") pod \"tuned-dsgxs\" (UID: \"d648d15e-a430-43e7-bcf9-0ff82b55e678\") " pod="openshift-cluster-node-tuning-operator/tuned-dsgxs" Jul 26 15:05:22.036335 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.035088 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/96d47908-2b56-4fbe-a6e4-6472cd3a5cd4-cni-binary-copy\") pod \"multus-additional-cni-plugins-4kk2l\" (UID: \"96d47908-2b56-4fbe-a6e4-6472cd3a5cd4\") " pod="openshift-multus/multus-additional-cni-plugins-4kk2l" Jul 26 15:05:22.037133 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.035114 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/96d47908-2b56-4fbe-a6e4-6472cd3a5cd4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4kk2l\" (UID: \"96d47908-2b56-4fbe-a6e4-6472cd3a5cd4\") " pod="openshift-multus/multus-additional-cni-plugins-4kk2l" Jul 26 15:05:22.037133 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.035141 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e1c832bc-b107-450a-b19d-ae4848abbad9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fjswx\" (UID: \"e1c832bc-b107-450a-b19d-ae4848abbad9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjswx" Jul 26 15:05:22.037133 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.035145 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1fbafda6-6242-400e-b2d4-24584608e5f5-etc-openvswitch\") pod \"ovnkube-node-9f857\" (UID: \"1fbafda6-6242-400e-b2d4-24584608e5f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:05:22.037133 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.035166 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e1c832bc-b107-450a-b19d-ae4848abbad9-registration-dir\") pod \"aws-ebs-csi-driver-node-fjswx\" (UID: \"e1c832bc-b107-450a-b19d-ae4848abbad9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjswx" Jul 26 15:05:22.037133 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.035105 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1fbafda6-6242-400e-b2d4-24584608e5f5-host-cni-bin\") pod \"ovnkube-node-9f857\" (UID: \"1fbafda6-6242-400e-b2d4-24584608e5f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:05:22.037133 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.035212 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e1c832bc-b107-450a-b19d-ae4848abbad9-registration-dir\") pod \"aws-ebs-csi-driver-node-fjswx\" (UID: \"e1c832bc-b107-450a-b19d-ae4848abbad9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjswx" Jul 26 15:05:22.037133 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.035947 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1fbafda6-6242-400e-b2d4-24584608e5f5-ovnkube-script-lib\") pod \"ovnkube-node-9f857\" (UID: \"1fbafda6-6242-400e-b2d4-24584608e5f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:05:22.037133 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.036005 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1fbafda6-6242-400e-b2d4-24584608e5f5-env-overrides\") pod \"ovnkube-node-9f857\" (UID: \"1fbafda6-6242-400e-b2d4-24584608e5f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:05:22.037133 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.036277 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e1c832bc-b107-450a-b19d-ae4848abbad9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fjswx\" (UID: \"e1c832bc-b107-450a-b19d-ae4848abbad9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjswx" Jul 26 15:05:22.037133 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.036661 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/db621adf-49db-46b5-9224-4c526e9ae434-host-var-lib-cni-multus\") pod \"multus-nzw98\" (UID: \"db621adf-49db-46b5-9224-4c526e9ae434\") " pod="openshift-multus/multus-nzw98" Jul 26 15:05:22.037133 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.036692 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1fbafda6-6242-400e-b2d4-24584608e5f5-env-overrides\") pod \"ovnkube-node-9f857\" (UID: \"1fbafda6-6242-400e-b2d4-24584608e5f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:05:22.037638 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.037308 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1fbafda6-6242-400e-b2d4-24584608e5f5-ovn-node-metrics-cert\") pod \"ovnkube-node-9f857\" (UID: \"1fbafda6-6242-400e-b2d4-24584608e5f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:05:22.037638 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.037435 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/db621adf-49db-46b5-9224-4c526e9ae434-host-var-lib-cni-multus\") pod \"multus-nzw98\" (UID: \"db621adf-49db-46b5-9224-4c526e9ae434\") " pod="openshift-multus/multus-nzw98" Jul 26 15:05:22.037638 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.037474 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bf799819-067d-4c2c-891c-ddcb8a800c2e-iptables-alerter-script\") pod \"iptables-alerter-frrgx\" (UID: \"bf799819-067d-4c2c-891c-ddcb8a800c2e\") " pod="openshift-network-operator/iptables-alerter-frrgx" Jul 26 15:05:22.037638 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.037514 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5skb\" (UniqueName: \"kubernetes.io/projected/d648d15e-a430-43e7-bcf9-0ff82b55e678-kube-api-access-h5skb\") pod \"tuned-dsgxs\" (UID: \"d648d15e-a430-43e7-bcf9-0ff82b55e678\") " pod="openshift-cluster-node-tuning-operator/tuned-dsgxs" Jul 26 15:05:22.037638 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.037540 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/db621adf-49db-46b5-9224-4c526e9ae434-host-run-netns\") pod \"multus-nzw98\" (UID: \"db621adf-49db-46b5-9224-4c526e9ae434\") " pod="openshift-multus/multus-nzw98" Jul 26 15:05:22.037638 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.037570 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/64483b31-45cb-4fa9-93a4-9726dc1bbd99-konnectivity-ca\") pod \"konnectivity-agent-bndqd\" (UID: \"64483b31-45cb-4fa9-93a4-9726dc1bbd99\") " pod="kube-system/konnectivity-agent-bndqd" Jul 26 15:05:22.037638 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.037598 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9r5b\" (UniqueName: \"kubernetes.io/projected/53c30547-0ebf-4139-80c9-e452ed2b9578-kube-api-access-z9r5b\") pod \"node-ca-j6hp5\" (UID: \"53c30547-0ebf-4139-80c9-e452ed2b9578\") " pod="openshift-image-registry/node-ca-j6hp5" Jul 26 15:05:22.037638 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.037625 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d648d15e-a430-43e7-bcf9-0ff82b55e678-etc-sysconfig\") pod \"tuned-dsgxs\" (UID: \"d648d15e-a430-43e7-bcf9-0ff82b55e678\") " pod="openshift-cluster-node-tuning-operator/tuned-dsgxs" Jul 26 15:05:22.038046 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.037647 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d648d15e-a430-43e7-bcf9-0ff82b55e678-etc-tuned\") pod \"tuned-dsgxs\" (UID: \"d648d15e-a430-43e7-bcf9-0ff82b55e678\") " pod="openshift-cluster-node-tuning-operator/tuned-dsgxs" Jul 26 15:05:22.038046 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.037643 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/64483b31-45cb-4fa9-93a4-9726dc1bbd99-agent-certs\") pod \"konnectivity-agent-bndqd\" (UID: \"64483b31-45cb-4fa9-93a4-9726dc1bbd99\") " pod="kube-system/konnectivity-agent-bndqd" Jul 26 15:05:22.038046 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.037675 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1fbafda6-6242-400e-b2d4-24584608e5f5-run-openvswitch\") pod \"ovnkube-node-9f857\" (UID: \"1fbafda6-6242-400e-b2d4-24584608e5f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:05:22.038046 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.037723 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/db621adf-49db-46b5-9224-4c526e9ae434-cni-binary-copy\") pod \"multus-nzw98\" (UID: \"db621adf-49db-46b5-9224-4c526e9ae434\") " pod="openshift-multus/multus-nzw98" Jul 26 15:05:22.038046 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.037742 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/db621adf-49db-46b5-9224-4c526e9ae434-host-run-netns\") pod \"multus-nzw98\" (UID: \"db621adf-49db-46b5-9224-4c526e9ae434\") " pod="openshift-multus/multus-nzw98" Jul 26 15:05:22.038046 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.037754 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d648d15e-a430-43e7-bcf9-0ff82b55e678-etc-modprobe-d\") pod \"tuned-dsgxs\" (UID: \"d648d15e-a430-43e7-bcf9-0ff82b55e678\") " pod="openshift-cluster-node-tuning-operator/tuned-dsgxs" Jul 26 15:05:22.038046 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.037804 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1fbafda6-6242-400e-b2d4-24584608e5f5-run-openvswitch\") pod \"ovnkube-node-9f857\" (UID: \"1fbafda6-6242-400e-b2d4-24584608e5f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:05:22.038362 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.038306 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/db621adf-49db-46b5-9224-4c526e9ae434-cni-binary-copy\") pod \"multus-nzw98\" (UID: \"db621adf-49db-46b5-9224-4c526e9ae434\") " pod="openshift-multus/multus-nzw98" Jul 26 15:05:22.038411 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.038402 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d648d15e-a430-43e7-bcf9-0ff82b55e678-etc-modprobe-d\") pod \"tuned-dsgxs\" (UID: \"d648d15e-a430-43e7-bcf9-0ff82b55e678\") " pod="openshift-cluster-node-tuning-operator/tuned-dsgxs" Jul 26 15:05:22.038465 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.038407 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/64483b31-45cb-4fa9-93a4-9726dc1bbd99-konnectivity-ca\") pod \"konnectivity-agent-bndqd\" (UID: \"64483b31-45cb-4fa9-93a4-9726dc1bbd99\") " pod="kube-system/konnectivity-agent-bndqd" Jul 26 15:05:22.038465 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.038458 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d648d15e-a430-43e7-bcf9-0ff82b55e678-etc-sysconfig\") pod \"tuned-dsgxs\" (UID: \"d648d15e-a430-43e7-bcf9-0ff82b55e678\") " pod="openshift-cluster-node-tuning-operator/tuned-dsgxs" Jul 26 15:05:22.038983 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.038706 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d648d15e-a430-43e7-bcf9-0ff82b55e678-run\") pod \"tuned-dsgxs\" (UID: \"d648d15e-a430-43e7-bcf9-0ff82b55e678\") " pod="openshift-cluster-node-tuning-operator/tuned-dsgxs" Jul 26 15:05:22.038983 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.038746 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1fbafda6-6242-400e-b2d4-24584608e5f5-host-run-ovn-kubernetes\") pod \"ovnkube-node-9f857\" (UID: \"1fbafda6-6242-400e-b2d4-24584608e5f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:05:22.038983 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.038776 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/db621adf-49db-46b5-9224-4c526e9ae434-multus-socket-dir-parent\") pod \"multus-nzw98\" (UID: \"db621adf-49db-46b5-9224-4c526e9ae434\") " pod="openshift-multus/multus-nzw98" Jul 26 15:05:22.038983 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.038806 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bf799819-067d-4c2c-891c-ddcb8a800c2e-host-slash\") pod \"iptables-alerter-frrgx\" (UID: \"bf799819-067d-4c2c-891c-ddcb8a800c2e\") " pod="openshift-network-operator/iptables-alerter-frrgx" Jul 26 15:05:22.038983 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.038832 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sncpb\" (UniqueName: \"kubernetes.io/projected/bf799819-067d-4c2c-891c-ddcb8a800c2e-kube-api-access-sncpb\") pod \"iptables-alerter-frrgx\" (UID: \"bf799819-067d-4c2c-891c-ddcb8a800c2e\") " pod="openshift-network-operator/iptables-alerter-frrgx" Jul 26 15:05:22.038983 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.038861 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e1c832bc-b107-450a-b19d-ae4848abbad9-device-dir\") pod \"aws-ebs-csi-driver-node-fjswx\" (UID: \"e1c832bc-b107-450a-b19d-ae4848abbad9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjswx" Jul 26 15:05:22.038983 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.038891 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e1c832bc-b107-450a-b19d-ae4848abbad9-sys-fs\") pod \"aws-ebs-csi-driver-node-fjswx\" (UID: \"e1c832bc-b107-450a-b19d-ae4848abbad9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjswx" Jul 26 15:05:22.038983 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.038919 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbwnr\" (UniqueName: \"kubernetes.io/projected/e1c832bc-b107-450a-b19d-ae4848abbad9-kube-api-access-fbwnr\") pod \"aws-ebs-csi-driver-node-fjswx\" (UID: \"e1c832bc-b107-450a-b19d-ae4848abbad9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjswx" Jul 26 15:05:22.038983 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.038969 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/db621adf-49db-46b5-9224-4c526e9ae434-cnibin\") pod \"multus-nzw98\" (UID: \"db621adf-49db-46b5-9224-4c526e9ae434\") " pod="openshift-multus/multus-nzw98" Jul 26 15:05:22.038983 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:22.038975 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 26 15:05:22.039573 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.038992 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wtgd\" (UniqueName: \"kubernetes.io/projected/db621adf-49db-46b5-9224-4c526e9ae434-kube-api-access-7wtgd\") pod \"multus-nzw98\" (UID: \"db621adf-49db-46b5-9224-4c526e9ae434\") " pod="openshift-multus/multus-nzw98" Jul 26 15:05:22.039573 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:22.038997 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 26 15:05:22.039573 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:22.039027 2555 projected.go:196] Error preparing data for projected volume kube-api-access-r4mqj for pod openshift-network-diagnostics/network-check-target-k8xdr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 15:05:22.039573 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.039035 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/706538eb-583d-4fca-8f24-285cd3ab940c-dbus\") pod \"global-pull-secret-syncer-b8spm\" (UID: \"706538eb-583d-4fca-8f24-285cd3ab940c\") " pod="kube-system/global-pull-secret-syncer-b8spm" Jul 26 15:05:22.039573 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.039072 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/57b3ab2c-f95e-45d9-9caa-6c1eac37919c-hosts-file\") pod \"node-resolver-rfxhs\" (UID: \"57b3ab2c-f95e-45d9-9caa-6c1eac37919c\") " pod="openshift-dns/node-resolver-rfxhs" Jul 26 15:05:22.039573 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:22.039090 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0357f424-c295-4aa7-8330-f551894f81d8-kube-api-access-r4mqj podName:0357f424-c295-4aa7-8330-f551894f81d8 nodeName:}" failed. No retries permitted until 2026-07-26 15:05:22.539072085 +0000 UTC m=+2.962918938 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-r4mqj" (UniqueName: "kubernetes.io/projected/0357f424-c295-4aa7-8330-f551894f81d8-kube-api-access-r4mqj") pod "network-check-target-k8xdr" (UID: "0357f424-c295-4aa7-8330-f551894f81d8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 15:05:22.039573 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.039125 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/57b3ab2c-f95e-45d9-9caa-6c1eac37919c-hosts-file\") pod \"node-resolver-rfxhs\" (UID: \"57b3ab2c-f95e-45d9-9caa-6c1eac37919c\") " pod="openshift-dns/node-resolver-rfxhs" Jul 26 15:05:22.039573 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.039126 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/db621adf-49db-46b5-9224-4c526e9ae434-host-run-multus-certs\") pod \"multus-nzw98\" (UID: \"db621adf-49db-46b5-9224-4c526e9ae434\") " pod="openshift-multus/multus-nzw98" Jul 26 15:05:22.039573 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.039163 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/706538eb-583d-4fca-8f24-285cd3ab940c-kubelet-config\") pod \"global-pull-secret-syncer-b8spm\" (UID: \"706538eb-583d-4fca-8f24-285cd3ab940c\") " pod="kube-system/global-pull-secret-syncer-b8spm" Jul 26 15:05:22.039573 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.039166 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/db621adf-49db-46b5-9224-4c526e9ae434-host-run-multus-certs\") pod \"multus-nzw98\" (UID: \"db621adf-49db-46b5-9224-4c526e9ae434\") " pod="openshift-multus/multus-nzw98" Jul 26 15:05:22.039573 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.039193 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e1c832bc-b107-450a-b19d-ae4848abbad9-etc-selinux\") pod \"aws-ebs-csi-driver-node-fjswx\" (UID: \"e1c832bc-b107-450a-b19d-ae4848abbad9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjswx" Jul 26 15:05:22.039573 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.039217 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1fbafda6-6242-400e-b2d4-24584608e5f5-run-ovn\") pod \"ovnkube-node-9f857\" (UID: \"1fbafda6-6242-400e-b2d4-24584608e5f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:05:22.039573 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.039234 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e1c832bc-b107-450a-b19d-ae4848abbad9-device-dir\") pod \"aws-ebs-csi-driver-node-fjswx\" (UID: \"e1c832bc-b107-450a-b19d-ae4848abbad9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjswx" Jul 26 15:05:22.039573 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.039245 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvtdz\" (UniqueName: \"kubernetes.io/projected/1fbafda6-6242-400e-b2d4-24584608e5f5-kube-api-access-bvtdz\") pod \"ovnkube-node-9f857\" (UID: \"1fbafda6-6242-400e-b2d4-24584608e5f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:05:22.039573 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.039275 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/db621adf-49db-46b5-9224-4c526e9ae434-host-run-k8s-cni-cncf-io\") pod \"multus-nzw98\" (UID: \"db621adf-49db-46b5-9224-4c526e9ae434\") " pod="openshift-multus/multus-nzw98" Jul 26 15:05:22.039573 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.039288 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e1c832bc-b107-450a-b19d-ae4848abbad9-sys-fs\") pod \"aws-ebs-csi-driver-node-fjswx\" (UID: \"e1c832bc-b107-450a-b19d-ae4848abbad9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjswx" Jul 26 15:05:22.039573 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.039309 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d648d15e-a430-43e7-bcf9-0ff82b55e678-lib-modules\") pod \"tuned-dsgxs\" (UID: \"d648d15e-a430-43e7-bcf9-0ff82b55e678\") " pod="openshift-cluster-node-tuning-operator/tuned-dsgxs" Jul 26 15:05:22.040671 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.039333 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1fbafda6-6242-400e-b2d4-24584608e5f5-host-run-netns\") pod \"ovnkube-node-9f857\" (UID: \"1fbafda6-6242-400e-b2d4-24584608e5f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:05:22.040671 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.039360 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/db621adf-49db-46b5-9224-4c526e9ae434-multus-cni-dir\") pod \"multus-nzw98\" (UID: \"db621adf-49db-46b5-9224-4c526e9ae434\") " pod="openshift-multus/multus-nzw98" Jul 26 15:05:22.040671 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.039387 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/db621adf-49db-46b5-9224-4c526e9ae434-hostroot\") pod \"multus-nzw98\" (UID: \"db621adf-49db-46b5-9224-4c526e9ae434\") " pod="openshift-multus/multus-nzw98" Jul 26 15:05:22.040671 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.039414 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/db621adf-49db-46b5-9224-4c526e9ae434-multus-daemon-config\") pod \"multus-nzw98\" (UID: \"db621adf-49db-46b5-9224-4c526e9ae434\") " pod="openshift-multus/multus-nzw98" Jul 26 15:05:22.040671 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.039444 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d648d15e-a430-43e7-bcf9-0ff82b55e678-etc-systemd\") pod \"tuned-dsgxs\" (UID: \"d648d15e-a430-43e7-bcf9-0ff82b55e678\") " pod="openshift-cluster-node-tuning-operator/tuned-dsgxs" Jul 26 15:05:22.040671 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.039467 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk4sv\" (UniqueName: \"kubernetes.io/projected/57b3ab2c-f95e-45d9-9caa-6c1eac37919c-kube-api-access-mk4sv\") pod \"node-resolver-rfxhs\" (UID: \"57b3ab2c-f95e-45d9-9caa-6c1eac37919c\") " pod="openshift-dns/node-resolver-rfxhs" Jul 26 15:05:22.040671 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.039493 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1fbafda6-6242-400e-b2d4-24584608e5f5-run-systemd\") pod \"ovnkube-node-9f857\" (UID: \"1fbafda6-6242-400e-b2d4-24584608e5f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:05:22.040671 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.039543 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/db621adf-49db-46b5-9224-4c526e9ae434-cnibin\") pod \"multus-nzw98\" (UID: \"db621adf-49db-46b5-9224-4c526e9ae434\") " pod="openshift-multus/multus-nzw98" Jul 26 15:05:22.040671 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.039600 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1fbafda6-6242-400e-b2d4-24584608e5f5-host-run-ovn-kubernetes\") pod \"ovnkube-node-9f857\" (UID: \"1fbafda6-6242-400e-b2d4-24584608e5f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:05:22.040671 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.039691 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d648d15e-a430-43e7-bcf9-0ff82b55e678-run\") pod \"tuned-dsgxs\" (UID: \"d648d15e-a430-43e7-bcf9-0ff82b55e678\") " pod="openshift-cluster-node-tuning-operator/tuned-dsgxs" Jul 26 15:05:22.040671 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.039748 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/db621adf-49db-46b5-9224-4c526e9ae434-multus-socket-dir-parent\") pod \"multus-nzw98\" (UID: \"db621adf-49db-46b5-9224-4c526e9ae434\") " pod="openshift-multus/multus-nzw98" Jul 26 15:05:22.040671 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.039757 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1fbafda6-6242-400e-b2d4-24584608e5f5-host-cni-netd\") pod \"ovnkube-node-9f857\" (UID: \"1fbafda6-6242-400e-b2d4-24584608e5f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:05:22.040671 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.039788 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/96d47908-2b56-4fbe-a6e4-6472cd3a5cd4-cnibin\") pod \"multus-additional-cni-plugins-4kk2l\" (UID: \"96d47908-2b56-4fbe-a6e4-6472cd3a5cd4\") " pod="openshift-multus/multus-additional-cni-plugins-4kk2l" Jul 26 15:05:22.040671 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.039847 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/706538eb-583d-4fca-8f24-285cd3ab940c-dbus\") pod \"global-pull-secret-syncer-b8spm\" (UID: \"706538eb-583d-4fca-8f24-285cd3ab940c\") " pod="kube-system/global-pull-secret-syncer-b8spm" Jul 26 15:05:22.040671 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.039543 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1fbafda6-6242-400e-b2d4-24584608e5f5-run-systemd\") pod \"ovnkube-node-9f857\" (UID: \"1fbafda6-6242-400e-b2d4-24584608e5f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:05:22.040671 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.039894 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1fbafda6-6242-400e-b2d4-24584608e5f5-host-cni-netd\") pod \"ovnkube-node-9f857\" (UID: \"1fbafda6-6242-400e-b2d4-24584608e5f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:05:22.040671 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.040011 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1fbafda6-6242-400e-b2d4-24584608e5f5-run-ovn\") pod \"ovnkube-node-9f857\" (UID: \"1fbafda6-6242-400e-b2d4-24584608e5f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:05:22.040671 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.040029 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/706538eb-583d-4fca-8f24-285cd3ab940c-kubelet-config\") pod \"global-pull-secret-syncer-b8spm\" (UID: \"706538eb-583d-4fca-8f24-285cd3ab940c\") " pod="kube-system/global-pull-secret-syncer-b8spm" Jul 26 15:05:22.043567 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.040089 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/db621adf-49db-46b5-9224-4c526e9ae434-multus-cni-dir\") pod \"multus-nzw98\" (UID: \"db621adf-49db-46b5-9224-4c526e9ae434\") " pod="openshift-multus/multus-nzw98" Jul 26 15:05:22.043567 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.040188 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d648d15e-a430-43e7-bcf9-0ff82b55e678-lib-modules\") pod \"tuned-dsgxs\" (UID: \"d648d15e-a430-43e7-bcf9-0ff82b55e678\") " pod="openshift-cluster-node-tuning-operator/tuned-dsgxs" Jul 26 15:05:22.043567 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.040234 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1fbafda6-6242-400e-b2d4-24584608e5f5-host-run-netns\") pod \"ovnkube-node-9f857\" (UID: \"1fbafda6-6242-400e-b2d4-24584608e5f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:05:22.043567 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.040348 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/db621adf-49db-46b5-9224-4c526e9ae434-host-run-k8s-cni-cncf-io\") pod \"multus-nzw98\" (UID: \"db621adf-49db-46b5-9224-4c526e9ae434\") " pod="openshift-multus/multus-nzw98" Jul 26 15:05:22.043567 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.040353 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d648d15e-a430-43e7-bcf9-0ff82b55e678-etc-systemd\") pod \"tuned-dsgxs\" (UID: \"d648d15e-a430-43e7-bcf9-0ff82b55e678\") " pod="openshift-cluster-node-tuning-operator/tuned-dsgxs" Jul 26 15:05:22.043567 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.040416 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/db621adf-49db-46b5-9224-4c526e9ae434-hostroot\") pod \"multus-nzw98\" (UID: \"db621adf-49db-46b5-9224-4c526e9ae434\") " pod="openshift-multus/multus-nzw98" Jul 26 15:05:22.043567 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.040486 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e1c832bc-b107-450a-b19d-ae4848abbad9-etc-selinux\") pod \"aws-ebs-csi-driver-node-fjswx\" (UID: \"e1c832bc-b107-450a-b19d-ae4848abbad9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjswx" Jul 26 15:05:22.043567 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.040598 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d648d15e-a430-43e7-bcf9-0ff82b55e678-tmp\") pod \"tuned-dsgxs\" (UID: \"d648d15e-a430-43e7-bcf9-0ff82b55e678\") " pod="openshift-cluster-node-tuning-operator/tuned-dsgxs" Jul 26 15:05:22.043567 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.040613 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/db621adf-49db-46b5-9224-4c526e9ae434-multus-daemon-config\") pod \"multus-nzw98\" (UID: \"db621adf-49db-46b5-9224-4c526e9ae434\") " pod="openshift-multus/multus-nzw98" Jul 26 15:05:22.043567 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.041256 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d648d15e-a430-43e7-bcf9-0ff82b55e678-etc-tuned\") pod \"tuned-dsgxs\" (UID: \"d648d15e-a430-43e7-bcf9-0ff82b55e678\") " pod="openshift-cluster-node-tuning-operator/tuned-dsgxs" Jul 26 15:05:22.047504 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.047478 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5skb\" (UniqueName: \"kubernetes.io/projected/d648d15e-a430-43e7-bcf9-0ff82b55e678-kube-api-access-h5skb\") pod \"tuned-dsgxs\" (UID: \"d648d15e-a430-43e7-bcf9-0ff82b55e678\") " pod="openshift-cluster-node-tuning-operator/tuned-dsgxs" Jul 26 15:05:22.052846 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.052782 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk4sv\" (UniqueName: \"kubernetes.io/projected/57b3ab2c-f95e-45d9-9caa-6c1eac37919c-kube-api-access-mk4sv\") pod \"node-resolver-rfxhs\" (UID: \"57b3ab2c-f95e-45d9-9caa-6c1eac37919c\") " pod="openshift-dns/node-resolver-rfxhs" Jul 26 15:05:22.053322 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.053296 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvtdz\" (UniqueName: \"kubernetes.io/projected/1fbafda6-6242-400e-b2d4-24584608e5f5-kube-api-access-bvtdz\") pod \"ovnkube-node-9f857\" (UID: \"1fbafda6-6242-400e-b2d4-24584608e5f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:05:22.053734 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.053691 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wtgd\" (UniqueName: \"kubernetes.io/projected/db621adf-49db-46b5-9224-4c526e9ae434-kube-api-access-7wtgd\") pod \"multus-nzw98\" (UID: \"db621adf-49db-46b5-9224-4c526e9ae434\") " pod="openshift-multus/multus-nzw98" Jul 26 15:05:22.053830 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.053743 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9r5b\" (UniqueName: \"kubernetes.io/projected/53c30547-0ebf-4139-80c9-e452ed2b9578-kube-api-access-z9r5b\") pod \"node-ca-j6hp5\" (UID: \"53c30547-0ebf-4139-80c9-e452ed2b9578\") " pod="openshift-image-registry/node-ca-j6hp5" Jul 26 15:05:22.054551 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.054509 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbwnr\" (UniqueName: \"kubernetes.io/projected/e1c832bc-b107-450a-b19d-ae4848abbad9-kube-api-access-fbwnr\") pod \"aws-ebs-csi-driver-node-fjswx\" (UID: \"e1c832bc-b107-450a-b19d-ae4848abbad9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjswx" Jul 26 15:05:22.063158 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.063115 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-21.ec2.internal" event={"ID":"b544e0067e3546297215dd780b690606","Type":"ContainerStarted","Data":"49b8ec93df634d80e84be903010f9a022d96baeeb991f779e3e4ae7e82c5bf55"} Jul 26 15:05:22.064524 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.064498 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-21.ec2.internal" event={"ID":"b006c6a0d5bf9a38bdcc97149639ead8","Type":"ContainerStarted","Data":"b38c67ff084ab4fece760467e4ec405f4400a668d69913fe55e17ef5f126386e"} Jul 26 15:05:22.140573 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.140550 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/96d47908-2b56-4fbe-a6e4-6472cd3a5cd4-cni-binary-copy\") pod \"multus-additional-cni-plugins-4kk2l\" (UID: \"96d47908-2b56-4fbe-a6e4-6472cd3a5cd4\") " pod="openshift-multus/multus-additional-cni-plugins-4kk2l" Jul 26 15:05:22.140706 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.140578 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/96d47908-2b56-4fbe-a6e4-6472cd3a5cd4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4kk2l\" (UID: \"96d47908-2b56-4fbe-a6e4-6472cd3a5cd4\") " pod="openshift-multus/multus-additional-cni-plugins-4kk2l" Jul 26 15:05:22.140706 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.140596 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bf799819-067d-4c2c-891c-ddcb8a800c2e-iptables-alerter-script\") pod \"iptables-alerter-frrgx\" (UID: \"bf799819-067d-4c2c-891c-ddcb8a800c2e\") " pod="openshift-network-operator/iptables-alerter-frrgx" Jul 26 15:05:22.140706 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.140626 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bf799819-067d-4c2c-891c-ddcb8a800c2e-host-slash\") pod \"iptables-alerter-frrgx\" (UID: \"bf799819-067d-4c2c-891c-ddcb8a800c2e\") " pod="openshift-network-operator/iptables-alerter-frrgx" Jul 26 15:05:22.140706 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.140646 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sncpb\" (UniqueName: \"kubernetes.io/projected/bf799819-067d-4c2c-891c-ddcb8a800c2e-kube-api-access-sncpb\") pod \"iptables-alerter-frrgx\" (UID: \"bf799819-067d-4c2c-891c-ddcb8a800c2e\") " pod="openshift-network-operator/iptables-alerter-frrgx" Jul 26 15:05:22.140706 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.140672 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/96d47908-2b56-4fbe-a6e4-6472cd3a5cd4-cnibin\") pod \"multus-additional-cni-plugins-4kk2l\" (UID: \"96d47908-2b56-4fbe-a6e4-6472cd3a5cd4\") " pod="openshift-multus/multus-additional-cni-plugins-4kk2l" Jul 26 15:05:22.140706 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.140696 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/96d47908-2b56-4fbe-a6e4-6472cd3a5cd4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4kk2l\" (UID: \"96d47908-2b56-4fbe-a6e4-6472cd3a5cd4\") " pod="openshift-multus/multus-additional-cni-plugins-4kk2l" Jul 26 15:05:22.141030 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.140717 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5xq4\" (UniqueName: \"kubernetes.io/projected/96d47908-2b56-4fbe-a6e4-6472cd3a5cd4-kube-api-access-q5xq4\") pod \"multus-additional-cni-plugins-4kk2l\" (UID: \"96d47908-2b56-4fbe-a6e4-6472cd3a5cd4\") " pod="openshift-multus/multus-additional-cni-plugins-4kk2l" Jul 26 15:05:22.141030 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.140751 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/96d47908-2b56-4fbe-a6e4-6472cd3a5cd4-system-cni-dir\") pod \"multus-additional-cni-plugins-4kk2l\" (UID: \"96d47908-2b56-4fbe-a6e4-6472cd3a5cd4\") " pod="openshift-multus/multus-additional-cni-plugins-4kk2l" Jul 26 15:05:22.141030 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.140771 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/96d47908-2b56-4fbe-a6e4-6472cd3a5cd4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4kk2l\" (UID: \"96d47908-2b56-4fbe-a6e4-6472cd3a5cd4\") " pod="openshift-multus/multus-additional-cni-plugins-4kk2l" Jul 26 15:05:22.141030 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.140803 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/96d47908-2b56-4fbe-a6e4-6472cd3a5cd4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4kk2l\" (UID: \"96d47908-2b56-4fbe-a6e4-6472cd3a5cd4\") " pod="openshift-multus/multus-additional-cni-plugins-4kk2l" Jul 26 15:05:22.141030 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.140795 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b6101bd-09a1-444f-ae02-50384ba66923-metrics-certs\") pod \"network-metrics-daemon-qlw9m\" (UID: \"1b6101bd-09a1-444f-ae02-50384ba66923\") " pod="openshift-multus/network-metrics-daemon-qlw9m" Jul 26 15:05:22.141030 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.140876 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/96d47908-2b56-4fbe-a6e4-6472cd3a5cd4-os-release\") pod \"multus-additional-cni-plugins-4kk2l\" (UID: \"96d47908-2b56-4fbe-a6e4-6472cd3a5cd4\") " pod="openshift-multus/multus-additional-cni-plugins-4kk2l" Jul 26 15:05:22.141030 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:22.140888 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 15:05:22.141030 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.140893 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/96d47908-2b56-4fbe-a6e4-6472cd3a5cd4-cnibin\") pod \"multus-additional-cni-plugins-4kk2l\" (UID: \"96d47908-2b56-4fbe-a6e4-6472cd3a5cd4\") " pod="openshift-multus/multus-additional-cni-plugins-4kk2l" Jul 26 15:05:22.141030 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.140904 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gdv6\" (UniqueName: \"kubernetes.io/projected/1b6101bd-09a1-444f-ae02-50384ba66923-kube-api-access-9gdv6\") pod \"network-metrics-daemon-qlw9m\" (UID: \"1b6101bd-09a1-444f-ae02-50384ba66923\") " pod="openshift-multus/network-metrics-daemon-qlw9m" Jul 26 15:05:22.141030 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:22.140946 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b6101bd-09a1-444f-ae02-50384ba66923-metrics-certs podName:1b6101bd-09a1-444f-ae02-50384ba66923 nodeName:}" failed. No retries permitted until 2026-07-26 15:05:22.640916887 +0000 UTC m=+3.064763753 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1b6101bd-09a1-444f-ae02-50384ba66923-metrics-certs") pod "network-metrics-daemon-qlw9m" (UID: "1b6101bd-09a1-444f-ae02-50384ba66923") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 15:05:22.141030 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.140966 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bf799819-067d-4c2c-891c-ddcb8a800c2e-host-slash\") pod \"iptables-alerter-frrgx\" (UID: \"bf799819-067d-4c2c-891c-ddcb8a800c2e\") " pod="openshift-network-operator/iptables-alerter-frrgx" Jul 26 15:05:22.141030 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.140975 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/96d47908-2b56-4fbe-a6e4-6472cd3a5cd4-system-cni-dir\") pod \"multus-additional-cni-plugins-4kk2l\" (UID: \"96d47908-2b56-4fbe-a6e4-6472cd3a5cd4\") " pod="openshift-multus/multus-additional-cni-plugins-4kk2l" Jul 26 15:05:22.141592 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.141107 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/96d47908-2b56-4fbe-a6e4-6472cd3a5cd4-os-release\") pod \"multus-additional-cni-plugins-4kk2l\" (UID: \"96d47908-2b56-4fbe-a6e4-6472cd3a5cd4\") " pod="openshift-multus/multus-additional-cni-plugins-4kk2l" Jul 26 15:05:22.141592 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.141171 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/96d47908-2b56-4fbe-a6e4-6472cd3a5cd4-cni-binary-copy\") pod \"multus-additional-cni-plugins-4kk2l\" (UID: \"96d47908-2b56-4fbe-a6e4-6472cd3a5cd4\") " pod="openshift-multus/multus-additional-cni-plugins-4kk2l" Jul 26 15:05:22.141592 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.141518 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/96d47908-2b56-4fbe-a6e4-6472cd3a5cd4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4kk2l\" (UID: \"96d47908-2b56-4fbe-a6e4-6472cd3a5cd4\") " pod="openshift-multus/multus-additional-cni-plugins-4kk2l" Jul 26 15:05:22.141592 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.141529 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/96d47908-2b56-4fbe-a6e4-6472cd3a5cd4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4kk2l\" (UID: \"96d47908-2b56-4fbe-a6e4-6472cd3a5cd4\") " pod="openshift-multus/multus-additional-cni-plugins-4kk2l" Jul 26 15:05:22.141753 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.141620 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bf799819-067d-4c2c-891c-ddcb8a800c2e-iptables-alerter-script\") pod \"iptables-alerter-frrgx\" (UID: \"bf799819-067d-4c2c-891c-ddcb8a800c2e\") " pod="openshift-network-operator/iptables-alerter-frrgx" Jul 26 15:05:22.150719 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.150691 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gdv6\" (UniqueName: \"kubernetes.io/projected/1b6101bd-09a1-444f-ae02-50384ba66923-kube-api-access-9gdv6\") pod \"network-metrics-daemon-qlw9m\" (UID: \"1b6101bd-09a1-444f-ae02-50384ba66923\") " pod="openshift-multus/network-metrics-daemon-qlw9m" Jul 26 15:05:22.151095 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.151074 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sncpb\" (UniqueName: \"kubernetes.io/projected/bf799819-067d-4c2c-891c-ddcb8a800c2e-kube-api-access-sncpb\") pod \"iptables-alerter-frrgx\" (UID: \"bf799819-067d-4c2c-891c-ddcb8a800c2e\") " pod="openshift-network-operator/iptables-alerter-frrgx" Jul 26 15:05:22.151185 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.151074 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5xq4\" (UniqueName: \"kubernetes.io/projected/96d47908-2b56-4fbe-a6e4-6472cd3a5cd4-kube-api-access-q5xq4\") pod \"multus-additional-cni-plugins-4kk2l\" (UID: \"96d47908-2b56-4fbe-a6e4-6472cd3a5cd4\") " pod="openshift-multus/multus-additional-cni-plugins-4kk2l" Jul 26 15:05:22.222152 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.222089 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bndqd" Jul 26 15:05:22.230869 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.230844 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:05:22.241427 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.241408 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjswx" Jul 26 15:05:22.246016 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.245995 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nzw98" Jul 26 15:05:22.253636 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.253617 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-j6hp5" Jul 26 15:05:22.261435 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.261413 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dsgxs" Jul 26 15:05:22.268230 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.268212 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rfxhs" Jul 26 15:05:22.274070 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.274052 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4kk2l" Jul 26 15:05:22.280825 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.280807 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-frrgx" Jul 26 15:05:22.543536 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.543456 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4mqj\" (UniqueName: \"kubernetes.io/projected/0357f424-c295-4aa7-8330-f551894f81d8-kube-api-access-r4mqj\") pod \"network-check-target-k8xdr\" (UID: \"0357f424-c295-4aa7-8330-f551894f81d8\") " pod="openshift-network-diagnostics/network-check-target-k8xdr" Jul 26 15:05:22.543536 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.543498 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/706538eb-583d-4fca-8f24-285cd3ab940c-original-pull-secret\") pod \"global-pull-secret-syncer-b8spm\" (UID: \"706538eb-583d-4fca-8f24-285cd3ab940c\") " pod="kube-system/global-pull-secret-syncer-b8spm" Jul 26 15:05:22.543738 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:22.543614 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 26 15:05:22.543738 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:22.543631 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 26 15:05:22.543738 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:22.543654 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 26 15:05:22.543738 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:22.543667 2555 projected.go:196] Error preparing data for projected volume kube-api-access-r4mqj for pod openshift-network-diagnostics/network-check-target-k8xdr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 15:05:22.543738 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:22.543693 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/706538eb-583d-4fca-8f24-285cd3ab940c-original-pull-secret podName:706538eb-583d-4fca-8f24-285cd3ab940c nodeName:}" failed. No retries permitted until 2026-07-26 15:05:23.543678616 +0000 UTC m=+3.967525472 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/706538eb-583d-4fca-8f24-285cd3ab940c-original-pull-secret") pod "global-pull-secret-syncer-b8spm" (UID: "706538eb-583d-4fca-8f24-285cd3ab940c") : object "kube-system"/"original-pull-secret" not registered Jul 26 15:05:22.543738 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:22.543713 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0357f424-c295-4aa7-8330-f551894f81d8-kube-api-access-r4mqj podName:0357f424-c295-4aa7-8330-f551894f81d8 nodeName:}" failed. No retries permitted until 2026-07-26 15:05:23.54370034 +0000 UTC m=+3.967547199 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-r4mqj" (UniqueName: "kubernetes.io/projected/0357f424-c295-4aa7-8330-f551894f81d8-kube-api-access-r4mqj") pod "network-check-target-k8xdr" (UID: "0357f424-c295-4aa7-8330-f551894f81d8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 15:05:22.644816 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.644792 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b6101bd-09a1-444f-ae02-50384ba66923-metrics-certs\") pod \"network-metrics-daemon-qlw9m\" (UID: \"1b6101bd-09a1-444f-ae02-50384ba66923\") " pod="openshift-multus/network-metrics-daemon-qlw9m" Jul 26 15:05:22.644950 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:22.644915 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 15:05:22.645000 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:22.644984 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b6101bd-09a1-444f-ae02-50384ba66923-metrics-certs podName:1b6101bd-09a1-444f-ae02-50384ba66923 nodeName:}" failed. No retries permitted until 2026-07-26 15:05:23.64496979 +0000 UTC m=+4.068816639 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1b6101bd-09a1-444f-ae02-50384ba66923-metrics-certs") pod "network-metrics-daemon-qlw9m" (UID: "1b6101bd-09a1-444f-ae02-50384ba66923") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 15:05:22.960669 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.960632 2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-25 15:00:21 +0000 UTC" deadline="2028-01-04 15:40:39.598690783 +0000 UTC" Jul 26 15:05:22.960669 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:22.960668 2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12648h35m16.638025607s" Jul 26 15:05:22.995817 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:22.995792 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53c30547_0ebf_4139_80c9_e452ed2b9578.slice/crio-a3c7fa5259639bc5fa90ead4a6cfbc4719c6bb1cc664a8aaca19c89a0d8c00b8 WatchSource:0}: Error finding container a3c7fa5259639bc5fa90ead4a6cfbc4719c6bb1cc664a8aaca19c89a0d8c00b8: Status 404 returned error can't find the container with id a3c7fa5259639bc5fa90ead4a6cfbc4719c6bb1cc664a8aaca19c89a0d8c00b8 Jul 26 15:05:22.997168 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:22.997141 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb621adf_49db_46b5_9224_4c526e9ae434.slice/crio-d4b932b7cbe9872ce76c2a89d852dfe09ce34e69656bac3df5d51a673705a2b3 WatchSource:0}: Error finding container d4b932b7cbe9872ce76c2a89d852dfe09ce34e69656bac3df5d51a673705a2b3: Status 404 returned error can't find the container with id d4b932b7cbe9872ce76c2a89d852dfe09ce34e69656bac3df5d51a673705a2b3 Jul 26 15:05:23.002087 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:23.002055 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fbafda6_6242_400e_b2d4_24584608e5f5.slice/crio-ff240ec4f938f627f3fde25cbb1829159934a141d5a55fbaac165a7ee004c25d WatchSource:0}: Error finding container ff240ec4f938f627f3fde25cbb1829159934a141d5a55fbaac165a7ee004c25d: Status 404 returned error can't find the container with id ff240ec4f938f627f3fde25cbb1829159934a141d5a55fbaac165a7ee004c25d Jul 26 15:05:23.002920 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:23.002896 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64483b31_45cb_4fa9_93a4_9726dc1bbd99.slice/crio-e2467a396f9a82d2146c9a40994a486e9f11a1fe4cd984b7770e6c0a8c1b0569 WatchSource:0}: Error finding container e2467a396f9a82d2146c9a40994a486e9f11a1fe4cd984b7770e6c0a8c1b0569: Status 404 returned error can't find the container with id e2467a396f9a82d2146c9a40994a486e9f11a1fe4cd984b7770e6c0a8c1b0569 Jul 26 15:05:23.003338 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:23.003311 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf799819_067d_4c2c_891c_ddcb8a800c2e.slice/crio-a345be7b52629c3f088a3161272e8feea7774cb83c81bdde8cf0e7ea2ec46dbe WatchSource:0}: Error finding container a345be7b52629c3f088a3161272e8feea7774cb83c81bdde8cf0e7ea2ec46dbe: Status 404 returned error can't find the container with id a345be7b52629c3f088a3161272e8feea7774cb83c81bdde8cf0e7ea2ec46dbe Jul 26 15:05:23.004259 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:23.004240 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57b3ab2c_f95e_45d9_9caa_6c1eac37919c.slice/crio-3a6efd123bac5cd283c6cd14a2b5a959d4ad6b3c8c0ea5e2ffce85e4c1442725 WatchSource:0}: Error finding container 3a6efd123bac5cd283c6cd14a2b5a959d4ad6b3c8c0ea5e2ffce85e4c1442725: Status 404 returned error can't find the container with id 3a6efd123bac5cd283c6cd14a2b5a959d4ad6b3c8c0ea5e2ffce85e4c1442725 Jul 26 15:05:23.005836 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:23.005627 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd648d15e_a430_43e7_bcf9_0ff82b55e678.slice/crio-dfc0acf15ce4646aefbf10a36ca7019bf8b53fcba960c5b696f0e67e6a167757 WatchSource:0}: Error finding container dfc0acf15ce4646aefbf10a36ca7019bf8b53fcba960c5b696f0e67e6a167757: Status 404 returned error can't find the container with id dfc0acf15ce4646aefbf10a36ca7019bf8b53fcba960c5b696f0e67e6a167757 Jul 26 15:05:23.006595 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:23.006578 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96d47908_2b56_4fbe_a6e4_6472cd3a5cd4.slice/crio-531aa9c59497b138aca0a736771230781aef5cee8b34ecda078168f2b149bfc4 WatchSource:0}: Error finding container 531aa9c59497b138aca0a736771230781aef5cee8b34ecda078168f2b149bfc4: Status 404 returned error can't find the container with id 531aa9c59497b138aca0a736771230781aef5cee8b34ecda078168f2b149bfc4 Jul 26 15:05:23.008078 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:23.008042 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1c832bc_b107_450a_b19d_ae4848abbad9.slice/crio-79178d02c28815ca092e0ece68552ad2586877eb478957c1835879499e139223 WatchSource:0}: Error finding container 79178d02c28815ca092e0ece68552ad2586877eb478957c1835879499e139223: Status 404 returned error can't find the container with id 79178d02c28815ca092e0ece68552ad2586877eb478957c1835879499e139223 Jul 26 15:05:23.057838 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:23.057819 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k8xdr" Jul 26 15:05:23.057942 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:23.057914 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k8xdr" podUID="0357f424-c295-4aa7-8330-f551894f81d8" Jul 26 15:05:23.058009 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:23.057958 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlw9m" Jul 26 15:05:23.058091 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:23.058075 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlw9m" podUID="1b6101bd-09a1-444f-ae02-50384ba66923" Jul 26 15:05:23.071220 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:23.071195 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjswx" event={"ID":"e1c832bc-b107-450a-b19d-ae4848abbad9","Type":"ContainerStarted","Data":"79178d02c28815ca092e0ece68552ad2586877eb478957c1835879499e139223"} Jul 26 15:05:23.072393 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:23.072370 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4kk2l" event={"ID":"96d47908-2b56-4fbe-a6e4-6472cd3a5cd4","Type":"ContainerStarted","Data":"531aa9c59497b138aca0a736771230781aef5cee8b34ecda078168f2b149bfc4"} Jul 26 15:05:23.073435 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:23.073412 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dsgxs" event={"ID":"d648d15e-a430-43e7-bcf9-0ff82b55e678","Type":"ContainerStarted","Data":"dfc0acf15ce4646aefbf10a36ca7019bf8b53fcba960c5b696f0e67e6a167757"} Jul 26 15:05:23.075042 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:23.075019 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9f857" event={"ID":"1fbafda6-6242-400e-b2d4-24584608e5f5","Type":"ContainerStarted","Data":"ff240ec4f938f627f3fde25cbb1829159934a141d5a55fbaac165a7ee004c25d"} Jul 26 15:05:23.076166 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:23.076146 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nzw98" event={"ID":"db621adf-49db-46b5-9224-4c526e9ae434","Type":"ContainerStarted","Data":"d4b932b7cbe9872ce76c2a89d852dfe09ce34e69656bac3df5d51a673705a2b3"} Jul 26 15:05:23.077588 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:23.077571 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bndqd" event={"ID":"64483b31-45cb-4fa9-93a4-9726dc1bbd99","Type":"ContainerStarted","Data":"e2467a396f9a82d2146c9a40994a486e9f11a1fe4cd984b7770e6c0a8c1b0569"} Jul 26 15:05:23.078674 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:23.078656 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rfxhs" event={"ID":"57b3ab2c-f95e-45d9-9caa-6c1eac37919c","Type":"ContainerStarted","Data":"3a6efd123bac5cd283c6cd14a2b5a959d4ad6b3c8c0ea5e2ffce85e4c1442725"} Jul 26 15:05:23.081221 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:23.081200 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-frrgx" event={"ID":"bf799819-067d-4c2c-891c-ddcb8a800c2e","Type":"ContainerStarted","Data":"a345be7b52629c3f088a3161272e8feea7774cb83c81bdde8cf0e7ea2ec46dbe"} Jul 26 15:05:23.084399 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:23.084381 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-j6hp5" event={"ID":"53c30547-0ebf-4139-80c9-e452ed2b9578","Type":"ContainerStarted","Data":"a3c7fa5259639bc5fa90ead4a6cfbc4719c6bb1cc664a8aaca19c89a0d8c00b8"} Jul 26 15:05:23.311602 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:23.311573 2555 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 26 15:05:23.554392 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:23.553567 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4mqj\" (UniqueName: \"kubernetes.io/projected/0357f424-c295-4aa7-8330-f551894f81d8-kube-api-access-r4mqj\") pod \"network-check-target-k8xdr\" (UID: \"0357f424-c295-4aa7-8330-f551894f81d8\") " pod="openshift-network-diagnostics/network-check-target-k8xdr" Jul 26 15:05:23.554392 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:23.553625 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/706538eb-583d-4fca-8f24-285cd3ab940c-original-pull-secret\") pod \"global-pull-secret-syncer-b8spm\" (UID: \"706538eb-583d-4fca-8f24-285cd3ab940c\") " pod="kube-system/global-pull-secret-syncer-b8spm" Jul 26 15:05:23.554392 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:23.553761 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 26 15:05:23.554392 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:23.553814 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/706538eb-583d-4fca-8f24-285cd3ab940c-original-pull-secret podName:706538eb-583d-4fca-8f24-285cd3ab940c nodeName:}" failed. No retries permitted until 2026-07-26 15:05:25.553796893 +0000 UTC m=+5.977643743 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/706538eb-583d-4fca-8f24-285cd3ab940c-original-pull-secret") pod "global-pull-secret-syncer-b8spm" (UID: "706538eb-583d-4fca-8f24-285cd3ab940c") : object "kube-system"/"original-pull-secret" not registered Jul 26 15:05:23.554392 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:23.554205 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 26 15:05:23.554392 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:23.554225 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 26 15:05:23.554392 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:23.554237 2555 projected.go:196] Error preparing data for projected volume kube-api-access-r4mqj for pod openshift-network-diagnostics/network-check-target-k8xdr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 15:05:23.554392 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:23.554279 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0357f424-c295-4aa7-8330-f551894f81d8-kube-api-access-r4mqj podName:0357f424-c295-4aa7-8330-f551894f81d8 nodeName:}" failed. No retries permitted until 2026-07-26 15:05:25.554265399 +0000 UTC m=+5.978112253 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-r4mqj" (UniqueName: "kubernetes.io/projected/0357f424-c295-4aa7-8330-f551894f81d8-kube-api-access-r4mqj") pod "network-check-target-k8xdr" (UID: "0357f424-c295-4aa7-8330-f551894f81d8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 15:05:23.654501 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:23.654459 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b6101bd-09a1-444f-ae02-50384ba66923-metrics-certs\") pod \"network-metrics-daemon-qlw9m\" (UID: \"1b6101bd-09a1-444f-ae02-50384ba66923\") " pod="openshift-multus/network-metrics-daemon-qlw9m" Jul 26 15:05:23.654661 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:23.654600 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 15:05:23.654661 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:23.654653 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b6101bd-09a1-444f-ae02-50384ba66923-metrics-certs podName:1b6101bd-09a1-444f-ae02-50384ba66923 nodeName:}" failed. No retries permitted until 2026-07-26 15:05:25.654636222 +0000 UTC m=+6.078483076 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1b6101bd-09a1-444f-ae02-50384ba66923-metrics-certs") pod "network-metrics-daemon-qlw9m" (UID: "1b6101bd-09a1-444f-ae02-50384ba66923") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 15:05:23.961209 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:23.961145 2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-25 15:00:21 +0000 UTC" deadline="2028-04-24 07:11:23.1534133 +0000 UTC" Jul 26 15:05:23.961209 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:23.961206 2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15304h5m59.192211313s" Jul 26 15:05:24.060507 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:24.060462 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8spm" Jul 26 15:05:24.060671 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:24.060587 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8spm" podUID="706538eb-583d-4fca-8f24-285cd3ab940c" Jul 26 15:05:24.100296 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:24.099574 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-21.ec2.internal" event={"ID":"b544e0067e3546297215dd780b690606","Type":"ContainerStarted","Data":"f931056d78b6179e8bb61eb4a10a1d2b95c5576ee6d2d58fb273151a47cb0c0e"} Jul 26 15:05:24.108243 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:24.107729 2555 generic.go:356] "Generic (PLEG): container finished" podID="b006c6a0d5bf9a38bdcc97149639ead8" containerID="5a88588524eff92f16e4506fa8501e599ea1d2b7312d0a134f29526e675e8fa2" exitCode=0 Jul 26 15:05:24.108243 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:24.107773 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-21.ec2.internal" event={"ID":"b006c6a0d5bf9a38bdcc97149639ead8","Type":"ContainerDied","Data":"5a88588524eff92f16e4506fa8501e599ea1d2b7312d0a134f29526e675e8fa2"} Jul 26 15:05:24.112701 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:24.112592 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-21.ec2.internal" podStartSLOduration=3.112577688 podStartE2EDuration="3.112577688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 15:05:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:05:24.112422679 +0000 UTC m=+4.536269575" watchObservedRunningTime="2026-07-26 15:05:24.112577688 +0000 UTC m=+4.536424561" Jul 26 15:05:25.058808 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:25.058169 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlw9m" Jul 26 15:05:25.058808 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:25.058302 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlw9m" podUID="1b6101bd-09a1-444f-ae02-50384ba66923" Jul 26 15:05:25.058808 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:25.058666 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k8xdr" Jul 26 15:05:25.058808 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:25.058747 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k8xdr" podUID="0357f424-c295-4aa7-8330-f551894f81d8" Jul 26 15:05:25.125129 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:25.125040 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-21.ec2.internal" event={"ID":"b006c6a0d5bf9a38bdcc97149639ead8","Type":"ContainerStarted","Data":"d1f454dce52f77e89411c6387f56a8cc6fef008817d71fce449a715e6fdef046"} Jul 26 15:05:25.138924 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:25.138874 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-21.ec2.internal" podStartSLOduration=4.138854762 podStartE2EDuration="4.138854762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 15:05:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:05:25.138470307 +0000 UTC m=+5.562317179" watchObservedRunningTime="2026-07-26 15:05:25.138854762 +0000 UTC m=+5.562701636" Jul 26 15:05:25.571244 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:25.571206 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4mqj\" (UniqueName: \"kubernetes.io/projected/0357f424-c295-4aa7-8330-f551894f81d8-kube-api-access-r4mqj\") pod \"network-check-target-k8xdr\" (UID: \"0357f424-c295-4aa7-8330-f551894f81d8\") " pod="openshift-network-diagnostics/network-check-target-k8xdr" Jul 26 15:05:25.571417 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:25.571272 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/706538eb-583d-4fca-8f24-285cd3ab940c-original-pull-secret\") pod \"global-pull-secret-syncer-b8spm\" (UID: \"706538eb-583d-4fca-8f24-285cd3ab940c\") " pod="kube-system/global-pull-secret-syncer-b8spm" Jul 26 15:05:25.571417 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:25.571412 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 26 15:05:25.571529 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:25.571477 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/706538eb-583d-4fca-8f24-285cd3ab940c-original-pull-secret podName:706538eb-583d-4fca-8f24-285cd3ab940c nodeName:}" failed. No retries permitted until 2026-07-26 15:05:29.571457373 +0000 UTC m=+9.995304236 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/706538eb-583d-4fca-8f24-285cd3ab940c-original-pull-secret") pod "global-pull-secret-syncer-b8spm" (UID: "706538eb-583d-4fca-8f24-285cd3ab940c") : object "kube-system"/"original-pull-secret" not registered Jul 26 15:05:25.571924 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:25.571899 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 26 15:05:25.571924 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:25.571942 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 26 15:05:25.572100 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:25.571955 2555 projected.go:196] Error preparing data for projected volume kube-api-access-r4mqj for pod openshift-network-diagnostics/network-check-target-k8xdr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 15:05:25.572100 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:25.572001 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0357f424-c295-4aa7-8330-f551894f81d8-kube-api-access-r4mqj podName:0357f424-c295-4aa7-8330-f551894f81d8 nodeName:}" failed. No retries permitted until 2026-07-26 15:05:29.571986796 +0000 UTC m=+9.995833661 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-r4mqj" (UniqueName: "kubernetes.io/projected/0357f424-c295-4aa7-8330-f551894f81d8-kube-api-access-r4mqj") pod "network-check-target-k8xdr" (UID: "0357f424-c295-4aa7-8330-f551894f81d8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 15:05:25.672640 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:25.672603 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b6101bd-09a1-444f-ae02-50384ba66923-metrics-certs\") pod \"network-metrics-daemon-qlw9m\" (UID: \"1b6101bd-09a1-444f-ae02-50384ba66923\") " pod="openshift-multus/network-metrics-daemon-qlw9m" Jul 26 15:05:25.672810 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:25.672792 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 15:05:25.672878 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:25.672865 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b6101bd-09a1-444f-ae02-50384ba66923-metrics-certs podName:1b6101bd-09a1-444f-ae02-50384ba66923 nodeName:}" failed. No retries permitted until 2026-07-26 15:05:29.67284488 +0000 UTC m=+10.096691752 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1b6101bd-09a1-444f-ae02-50384ba66923-metrics-certs") pod "network-metrics-daemon-qlw9m" (UID: "1b6101bd-09a1-444f-ae02-50384ba66923") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 15:05:26.059806 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:26.059771 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8spm" Jul 26 15:05:26.060261 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:26.059921 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8spm" podUID="706538eb-583d-4fca-8f24-285cd3ab940c" Jul 26 15:05:27.058547 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:27.058516 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k8xdr" Jul 26 15:05:27.058730 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:27.058652 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k8xdr" podUID="0357f424-c295-4aa7-8330-f551894f81d8" Jul 26 15:05:27.059178 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:27.058991 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlw9m" Jul 26 15:05:27.059178 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:27.059095 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlw9m" podUID="1b6101bd-09a1-444f-ae02-50384ba66923" Jul 26 15:05:28.058781 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:28.058750 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8spm" Jul 26 15:05:28.059236 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:28.058881 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8spm" podUID="706538eb-583d-4fca-8f24-285cd3ab940c" Jul 26 15:05:29.058039 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:29.058003 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k8xdr" Jul 26 15:05:29.058039 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:29.058029 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlw9m" Jul 26 15:05:29.058255 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:29.058136 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k8xdr" podUID="0357f424-c295-4aa7-8330-f551894f81d8" Jul 26 15:05:29.058355 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:29.058290 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlw9m" podUID="1b6101bd-09a1-444f-ae02-50384ba66923" Jul 26 15:05:29.607520 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:29.607482 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4mqj\" (UniqueName: \"kubernetes.io/projected/0357f424-c295-4aa7-8330-f551894f81d8-kube-api-access-r4mqj\") pod \"network-check-target-k8xdr\" (UID: \"0357f424-c295-4aa7-8330-f551894f81d8\") " pod="openshift-network-diagnostics/network-check-target-k8xdr" Jul 26 15:05:29.607945 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:29.607547 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/706538eb-583d-4fca-8f24-285cd3ab940c-original-pull-secret\") pod \"global-pull-secret-syncer-b8spm\" (UID: \"706538eb-583d-4fca-8f24-285cd3ab940c\") " pod="kube-system/global-pull-secret-syncer-b8spm" Jul 26 15:05:29.607945 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:29.607677 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 26 15:05:29.607945 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:29.607737 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/706538eb-583d-4fca-8f24-285cd3ab940c-original-pull-secret podName:706538eb-583d-4fca-8f24-285cd3ab940c nodeName:}" failed. No retries permitted until 2026-07-26 15:05:37.607719446 +0000 UTC m=+18.031566376 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/706538eb-583d-4fca-8f24-285cd3ab940c-original-pull-secret") pod "global-pull-secret-syncer-b8spm" (UID: "706538eb-583d-4fca-8f24-285cd3ab940c") : object "kube-system"/"original-pull-secret" not registered Jul 26 15:05:29.608143 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:29.608121 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 26 15:05:29.608195 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:29.608144 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 26 15:05:29.608195 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:29.608156 2555 projected.go:196] Error preparing data for projected volume kube-api-access-r4mqj for pod openshift-network-diagnostics/network-check-target-k8xdr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 15:05:29.608290 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:29.608207 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0357f424-c295-4aa7-8330-f551894f81d8-kube-api-access-r4mqj podName:0357f424-c295-4aa7-8330-f551894f81d8 nodeName:}" failed. No retries permitted until 2026-07-26 15:05:37.608192376 +0000 UTC m=+18.032039245 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-r4mqj" (UniqueName: "kubernetes.io/projected/0357f424-c295-4aa7-8330-f551894f81d8-kube-api-access-r4mqj") pod "network-check-target-k8xdr" (UID: "0357f424-c295-4aa7-8330-f551894f81d8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 15:05:29.708771 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:29.708728 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b6101bd-09a1-444f-ae02-50384ba66923-metrics-certs\") pod \"network-metrics-daemon-qlw9m\" (UID: \"1b6101bd-09a1-444f-ae02-50384ba66923\") " pod="openshift-multus/network-metrics-daemon-qlw9m" Jul 26 15:05:29.708975 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:29.708902 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 15:05:29.708975 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:29.708974 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b6101bd-09a1-444f-ae02-50384ba66923-metrics-certs podName:1b6101bd-09a1-444f-ae02-50384ba66923 nodeName:}" failed. No retries permitted until 2026-07-26 15:05:37.708957166 +0000 UTC m=+18.132804028 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1b6101bd-09a1-444f-ae02-50384ba66923-metrics-certs") pod "network-metrics-daemon-qlw9m" (UID: "1b6101bd-09a1-444f-ae02-50384ba66923") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 15:05:30.059761 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:30.059729 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8spm" Jul 26 15:05:30.059951 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:30.059832 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8spm" podUID="706538eb-583d-4fca-8f24-285cd3ab940c" Jul 26 15:05:31.058952 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:31.058896 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k8xdr" Jul 26 15:05:31.059578 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:31.058896 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlw9m" Jul 26 15:05:31.059578 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:31.059134 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlw9m" podUID="1b6101bd-09a1-444f-ae02-50384ba66923" Jul 26 15:05:31.059794 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:31.059764 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k8xdr" podUID="0357f424-c295-4aa7-8330-f551894f81d8" Jul 26 15:05:32.058398 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:32.058364 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8spm" Jul 26 15:05:32.058577 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:32.058488 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8spm" podUID="706538eb-583d-4fca-8f24-285cd3ab940c" Jul 26 15:05:33.058098 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:33.058072 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlw9m" Jul 26 15:05:33.058460 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:33.058106 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k8xdr" Jul 26 15:05:33.058460 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:33.058194 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlw9m" podUID="1b6101bd-09a1-444f-ae02-50384ba66923" Jul 26 15:05:33.058460 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:33.058309 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k8xdr" podUID="0357f424-c295-4aa7-8330-f551894f81d8" Jul 26 15:05:34.058440 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:34.058405 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8spm" Jul 26 15:05:34.058838 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:34.058534 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8spm" podUID="706538eb-583d-4fca-8f24-285cd3ab940c" Jul 26 15:05:35.058370 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:35.058336 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k8xdr" Jul 26 15:05:35.058370 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:35.058373 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlw9m" Jul 26 15:05:35.058814 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:35.058448 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k8xdr" podUID="0357f424-c295-4aa7-8330-f551894f81d8" Jul 26 15:05:35.058814 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:35.058571 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlw9m" podUID="1b6101bd-09a1-444f-ae02-50384ba66923" Jul 26 15:05:36.058642 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:36.058608 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8spm" Jul 26 15:05:36.059089 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:36.058729 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8spm" podUID="706538eb-583d-4fca-8f24-285cd3ab940c" Jul 26 15:05:37.057873 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:37.057795 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k8xdr" Jul 26 15:05:37.057873 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:37.057816 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlw9m" Jul 26 15:05:37.058097 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:37.057908 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k8xdr" podUID="0357f424-c295-4aa7-8330-f551894f81d8" Jul 26 15:05:37.058097 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:37.058041 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlw9m" podUID="1b6101bd-09a1-444f-ae02-50384ba66923" Jul 26 15:05:37.671138 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:37.671105 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4mqj\" (UniqueName: \"kubernetes.io/projected/0357f424-c295-4aa7-8330-f551894f81d8-kube-api-access-r4mqj\") pod \"network-check-target-k8xdr\" (UID: \"0357f424-c295-4aa7-8330-f551894f81d8\") " pod="openshift-network-diagnostics/network-check-target-k8xdr" Jul 26 15:05:37.671579 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:37.671159 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/706538eb-583d-4fca-8f24-285cd3ab940c-original-pull-secret\") pod \"global-pull-secret-syncer-b8spm\" (UID: \"706538eb-583d-4fca-8f24-285cd3ab940c\") " pod="kube-system/global-pull-secret-syncer-b8spm" Jul 26 15:05:37.671579 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:37.671261 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 26 15:05:37.671579 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:37.671297 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 26 15:05:37.671579 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:37.671325 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 26 15:05:37.671579 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:37.671340 2555 projected.go:196] Error preparing data for projected volume kube-api-access-r4mqj for pod openshift-network-diagnostics/network-check-target-k8xdr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 15:05:37.671579 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:37.671326 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/706538eb-583d-4fca-8f24-285cd3ab940c-original-pull-secret podName:706538eb-583d-4fca-8f24-285cd3ab940c nodeName:}" failed. No retries permitted until 2026-07-26 15:05:53.671312275 +0000 UTC m=+34.095159125 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/706538eb-583d-4fca-8f24-285cd3ab940c-original-pull-secret") pod "global-pull-secret-syncer-b8spm" (UID: "706538eb-583d-4fca-8f24-285cd3ab940c") : object "kube-system"/"original-pull-secret" not registered Jul 26 15:05:37.671579 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:37.671403 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0357f424-c295-4aa7-8330-f551894f81d8-kube-api-access-r4mqj podName:0357f424-c295-4aa7-8330-f551894f81d8 nodeName:}" failed. No retries permitted until 2026-07-26 15:05:53.671387107 +0000 UTC m=+34.095233963 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-r4mqj" (UniqueName: "kubernetes.io/projected/0357f424-c295-4aa7-8330-f551894f81d8-kube-api-access-r4mqj") pod "network-check-target-k8xdr" (UID: "0357f424-c295-4aa7-8330-f551894f81d8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 15:05:37.771580 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:37.771546 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b6101bd-09a1-444f-ae02-50384ba66923-metrics-certs\") pod \"network-metrics-daemon-qlw9m\" (UID: \"1b6101bd-09a1-444f-ae02-50384ba66923\") " pod="openshift-multus/network-metrics-daemon-qlw9m" Jul 26 15:05:37.771759 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:37.771736 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 15:05:37.771833 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:37.771821 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b6101bd-09a1-444f-ae02-50384ba66923-metrics-certs podName:1b6101bd-09a1-444f-ae02-50384ba66923 nodeName:}" failed. No retries permitted until 2026-07-26 15:05:53.771800791 +0000 UTC m=+34.195647642 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1b6101bd-09a1-444f-ae02-50384ba66923-metrics-certs") pod "network-metrics-daemon-qlw9m" (UID: "1b6101bd-09a1-444f-ae02-50384ba66923") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 15:05:38.058151 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:38.058082 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8spm" Jul 26 15:05:38.058307 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:38.058215 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8spm" podUID="706538eb-583d-4fca-8f24-285cd3ab940c" Jul 26 15:05:39.058334 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:39.058298 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k8xdr" Jul 26 15:05:39.058334 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:39.058334 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlw9m" Jul 26 15:05:39.058810 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:39.058422 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k8xdr" podUID="0357f424-c295-4aa7-8330-f551894f81d8" Jul 26 15:05:39.058810 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:39.058527 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlw9m" podUID="1b6101bd-09a1-444f-ae02-50384ba66923" Jul 26 15:05:40.058798 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:40.058766 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8spm" Jul 26 15:05:40.059254 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:40.058876 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8spm" podUID="706538eb-583d-4fca-8f24-285cd3ab940c" Jul 26 15:05:41.058581 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:41.058554 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlw9m" Jul 26 15:05:41.058721 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:41.058696 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlw9m" podUID="1b6101bd-09a1-444f-ae02-50384ba66923" Jul 26 15:05:41.058814 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:41.058763 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k8xdr" Jul 26 15:05:41.058877 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:41.058857 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k8xdr" podUID="0357f424-c295-4aa7-8330-f551894f81d8" Jul 26 15:05:41.154064 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:41.154036 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjswx" event={"ID":"e1c832bc-b107-450a-b19d-ae4848abbad9","Type":"ContainerStarted","Data":"f926a2755656adc99306fcd9598280fe991cb1d6bbc4609ac15e5f64392e7a3b"} Jul 26 15:05:41.155196 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:41.155174 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4kk2l" event={"ID":"96d47908-2b56-4fbe-a6e4-6472cd3a5cd4","Type":"ContainerStarted","Data":"bc57b91aedc856f59269d4efb365edca792a167dcbb3815128f7c88d1ce586b9"} Jul 26 15:05:41.156563 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:41.156539 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dsgxs" event={"ID":"d648d15e-a430-43e7-bcf9-0ff82b55e678","Type":"ContainerStarted","Data":"532d065c999a44a562c1f78a0f1f70663397894ec87da24c83a5e5318fb20c6d"} Jul 26 15:05:41.157807 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:41.157785 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9f857" event={"ID":"1fbafda6-6242-400e-b2d4-24584608e5f5","Type":"ContainerStarted","Data":"996359d3f21f57168d6f880c94343d38e282a2ef909d3f6bd280784b910a8a1f"} Jul 26 15:05:41.159044 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:41.159024 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bndqd" event={"ID":"64483b31-45cb-4fa9-93a4-9726dc1bbd99","Type":"ContainerStarted","Data":"4b6acb44eda5ac7cd10a943a70bd07d23f90d2feb545155cb178a2bc2e39be1f"} Jul 26 15:05:41.160333 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:41.160312 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rfxhs" event={"ID":"57b3ab2c-f95e-45d9-9caa-6c1eac37919c","Type":"ContainerStarted","Data":"194ed29a0dbb64f152850a50ca3ee933fb2eee7054cd627e43da8083fb615ab8"} Jul 26 15:05:41.161568 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:41.161549 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-j6hp5" event={"ID":"53c30547-0ebf-4139-80c9-e452ed2b9578","Type":"ContainerStarted","Data":"c7aa03d358d6614db153769393ab7b95fa60fb748a6839c95bcceb556dc3de20"} Jul 26 15:05:41.184424 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:41.184390 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-rfxhs" podStartSLOduration=3.394153084 podStartE2EDuration="21.184376869s" totalImagesPullingTime="17.790223785s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 15:05:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:05:41.183914687 +0000 UTC m=+21.607761558" watchObservedRunningTime="2026-07-26 15:05:41.184376869 +0000 UTC m=+21.608223739" Jul 26 15:05:41.206666 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:41.206632 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-bndqd" podStartSLOduration=3.433392213 podStartE2EDuration="21.206618455s" totalImagesPullingTime="17.773226242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:05:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:05:41.206490859 +0000 UTC m=+21.630337733" watchObservedRunningTime="2026-07-26 15:05:41.206618455 +0000 UTC m=+21.630465325" Jul 26 15:05:41.206743 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:41.206716 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-j6hp5" podStartSLOduration=3.413450141 podStartE2EDuration="21.206709445s" totalImagesPullingTime="17.793259304s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 15:05:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:05:41.195200464 +0000 UTC m=+21.619047334" watchObservedRunningTime="2026-07-26 15:05:41.206709445 +0000 UTC m=+21.630556316" Jul 26 15:05:41.225190 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:41.225153 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-dsgxs" podStartSLOduration=3.443840635 podStartE2EDuration="21.225138088s" totalImagesPullingTime="17.781297453s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 15:05:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:05:41.2247624 +0000 UTC m=+21.648609282" watchObservedRunningTime="2026-07-26 15:05:41.225138088 +0000 UTC m=+21.648984960" Jul 26 15:05:41.594214 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:41.593908 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-bndqd" Jul 26 15:05:41.594680 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:41.594658 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-bndqd" Jul 26 15:05:42.058179 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:42.058160 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8spm" Jul 26 15:05:42.058270 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:42.058253 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8spm" podUID="706538eb-583d-4fca-8f24-285cd3ab940c" Jul 26 15:05:42.136126 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:42.136104 2555 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 26 15:05:42.165562 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:42.165542 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9f857_1fbafda6-6242-400e-b2d4-24584608e5f5/ovn-acl-logging/0.log" Jul 26 15:05:42.165844 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:42.165821 2555 generic.go:356] "Generic (PLEG): container finished" podID="1fbafda6-6242-400e-b2d4-24584608e5f5" containerID="3cb0b4e327d09e217693e7a635b3aea26f206866615d1b20f6352bf169fdde3b" exitCode=1 Jul 26 15:05:42.165956 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:42.165888 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9f857" event={"ID":"1fbafda6-6242-400e-b2d4-24584608e5f5","Type":"ContainerStarted","Data":"dd019f3ce1eeacc5c868b9cc5298d690007a0dfa0ee6498d8f50ed846cdac47f"} Jul 26 15:05:42.166012 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:42.165986 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9f857" event={"ID":"1fbafda6-6242-400e-b2d4-24584608e5f5","Type":"ContainerStarted","Data":"91b058af1c6472f923ee0193843b1f1b1651967268eccc6ec5d29256a2a4cb16"} Jul 26 15:05:42.166048 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:42.166039 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9f857" event={"ID":"1fbafda6-6242-400e-b2d4-24584608e5f5","Type":"ContainerStarted","Data":"daa0819ed440eabfc858366e6d6770e63970155e3aca8a31d8da5a89259d7657"} Jul 26 15:05:42.166090 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:42.166082 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9f857" event={"ID":"1fbafda6-6242-400e-b2d4-24584608e5f5","Type":"ContainerStarted","Data":"ff592afc156c7e1be0368f075f37d41b77c9daf2b479b764cc872178330b95b5"} Jul 26 15:05:42.166132 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:42.166119 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9f857" event={"ID":"1fbafda6-6242-400e-b2d4-24584608e5f5","Type":"ContainerDied","Data":"3cb0b4e327d09e217693e7a635b3aea26f206866615d1b20f6352bf169fdde3b"} Jul 26 15:05:42.167207 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:42.167188 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nzw98" event={"ID":"db621adf-49db-46b5-9224-4c526e9ae434","Type":"ContainerStarted","Data":"ebeafc7c8443e04f2ec826dda981de19fa47cfbfc7017692151256b273916638"} Jul 26 15:05:42.168849 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:42.168831 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjswx" event={"ID":"e1c832bc-b107-450a-b19d-ae4848abbad9","Type":"ContainerStarted","Data":"54573b3372b9a49a34752c06dc38bca9495ba04a17a5168dd3912ef90febd909"} Jul 26 15:05:42.170121 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:42.170102 2555 generic.go:356] "Generic (PLEG): container finished" podID="96d47908-2b56-4fbe-a6e4-6472cd3a5cd4" containerID="bc57b91aedc856f59269d4efb365edca792a167dcbb3815128f7c88d1ce586b9" exitCode=0 Jul 26 15:05:42.170199 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:42.170172 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4kk2l" event={"ID":"96d47908-2b56-4fbe-a6e4-6472cd3a5cd4","Type":"ContainerDied","Data":"bc57b91aedc856f59269d4efb365edca792a167dcbb3815128f7c88d1ce586b9"} Jul 26 15:05:42.180676 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:42.180639 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-nzw98" podStartSLOduration=3.8945026929999997 podStartE2EDuration="22.180628143s" totalImagesPullingTime="18.28612545s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 15:05:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:05:42.180287936 +0000 UTC m=+22.604134807" watchObservedRunningTime="2026-07-26 15:05:42.180628143 +0000 UTC m=+22.604475031" Jul 26 15:05:42.984069 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:42.983969 2555 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-26T15:05:42.136121665Z","UUID":"4935454b-0b00-4e42-bc3a-4add5b895db3","Handler":null,"Name":"","Endpoint":""} Jul 26 15:05:42.985763 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:42.985740 2555 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 26 15:05:42.985899 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:42.985783 2555 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 26 15:05:43.058038 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:43.058015 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k8xdr" Jul 26 15:05:43.058238 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:43.058046 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlw9m" Jul 26 15:05:43.058238 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:43.058115 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k8xdr" podUID="0357f424-c295-4aa7-8330-f551894f81d8" Jul 26 15:05:43.058323 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:43.058253 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlw9m" podUID="1b6101bd-09a1-444f-ae02-50384ba66923" Jul 26 15:05:43.173469 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:43.173434 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-frrgx" event={"ID":"bf799819-067d-4c2c-891c-ddcb8a800c2e","Type":"ContainerStarted","Data":"27ae3dcc147f70607b63602df386a0f17e746a4295f53fb16df52f87bac5e2fd"} Jul 26 15:05:43.173469 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:43.173464 2555 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 26 15:05:43.187532 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:43.187480 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-frrgx" podStartSLOduration=5.41519829 podStartE2EDuration="23.187459256s" totalImagesPullingTime="17.772260966s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 15:05:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:05:43.186910044 +0000 UTC m=+23.610756916" watchObservedRunningTime="2026-07-26 15:05:43.187459256 +0000 UTC m=+23.611306132" Jul 26 15:05:44.058888 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:44.058642 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8spm" Jul 26 15:05:44.059129 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:44.059011 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8spm" podUID="706538eb-583d-4fca-8f24-285cd3ab940c" Jul 26 15:05:44.177533 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:44.177495 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjswx" event={"ID":"e1c832bc-b107-450a-b19d-ae4848abbad9","Type":"ContainerStarted","Data":"14d2e14acea788b4612c25e653a59a282eeefb6214631e333b9bb85ca014c775"} Jul 26 15:05:44.180231 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:44.180211 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9f857_1fbafda6-6242-400e-b2d4-24584608e5f5/ovn-acl-logging/0.log" Jul 26 15:05:44.180600 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:44.180575 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9f857" event={"ID":"1fbafda6-6242-400e-b2d4-24584608e5f5","Type":"ContainerStarted","Data":"4232e697c9d6a52625779810b35e34e4f238bd88a847104f2919b8fa1111bd4e"} Jul 26 15:05:44.193300 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:44.193248 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjswx" podStartSLOduration=4.271599362 podStartE2EDuration="24.193236706s" totalImagesPullingTime="19.921637344s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 15:05:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:05:44.192969765 +0000 UTC m=+24.616816646" watchObservedRunningTime="2026-07-26 15:05:44.193236706 +0000 UTC m=+24.617083576" Jul 26 15:05:45.058261 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:45.058229 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlw9m" Jul 26 15:05:45.058469 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:45.058357 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlw9m" podUID="1b6101bd-09a1-444f-ae02-50384ba66923" Jul 26 15:05:45.058469 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:45.058419 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k8xdr" Jul 26 15:05:45.058579 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:45.058516 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k8xdr" podUID="0357f424-c295-4aa7-8330-f551894f81d8" Jul 26 15:05:46.058075 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:46.057969 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8spm" Jul 26 15:05:46.058685 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:46.058154 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8spm" podUID="706538eb-583d-4fca-8f24-285cd3ab940c" Jul 26 15:05:46.186860 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:46.186791 2555 generic.go:356] "Generic (PLEG): container finished" podID="96d47908-2b56-4fbe-a6e4-6472cd3a5cd4" containerID="463d33e6fdad125f65582bfb21dfd74ca4b743f4db8f661c66543985276d10cd" exitCode=0 Jul 26 15:05:46.186968 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:46.186875 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4kk2l" event={"ID":"96d47908-2b56-4fbe-a6e4-6472cd3a5cd4","Type":"ContainerDied","Data":"463d33e6fdad125f65582bfb21dfd74ca4b743f4db8f661c66543985276d10cd"} Jul 26 15:05:46.190334 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:46.190318 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9f857_1fbafda6-6242-400e-b2d4-24584608e5f5/ovn-acl-logging/0.log" Jul 26 15:05:46.190673 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:46.190651 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9f857" event={"ID":"1fbafda6-6242-400e-b2d4-24584608e5f5","Type":"ContainerStarted","Data":"f099edd37025eedfe6364db407e4845ae8e4d8ef2e76d60876fc26b53440800f"} Jul 26 15:05:46.190951 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:46.190913 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:05:46.191225 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:46.191112 2555 scope.go:117] "RemoveContainer" containerID="3cb0b4e327d09e217693e7a635b3aea26f206866615d1b20f6352bf169fdde3b" Jul 26 15:05:46.206105 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:46.206079 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:05:46.405794 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:46.405772 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-bndqd" Jul 26 15:05:46.405901 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:46.405889 2555 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 26 15:05:46.406247 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:46.406228 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-bndqd" Jul 26 15:05:46.494782 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:46.494733 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:05:46.586688 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:46.586667 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:05:46.642959 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:46.642923 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:05:47.058625 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:47.058599 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlw9m" Jul 26 15:05:47.059028 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:47.058640 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k8xdr" Jul 26 15:05:47.059028 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:47.058704 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlw9m" podUID="1b6101bd-09a1-444f-ae02-50384ba66923" Jul 26 15:05:47.059028 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:47.058756 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k8xdr" podUID="0357f424-c295-4aa7-8330-f551894f81d8" Jul 26 15:05:47.196762 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:47.196737 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9f857_1fbafda6-6242-400e-b2d4-24584608e5f5/ovn-acl-logging/0.log" Jul 26 15:05:47.197205 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:47.197184 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9f857" event={"ID":"1fbafda6-6242-400e-b2d4-24584608e5f5","Type":"ContainerStarted","Data":"9959238371d12e9d98d72525e377f5e61a3d3f1e9ac483ecbfc61db589b01493"} Jul 26 15:05:47.197403 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:47.197389 2555 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 26 15:05:47.197669 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:47.197646 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:05:47.221419 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:47.221359 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-9f857" podStartSLOduration=9.209518716 podStartE2EDuration="27.2213404s" totalImagesPullingTime="18.011821684s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 15:05:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:05:47.220980369 +0000 UTC m=+27.644827243" watchObservedRunningTime="2026-07-26 15:05:47.2213404 +0000 UTC m=+27.645187272" Jul 26 15:05:47.228426 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:47.228398 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:05:47.932046 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:47.931888 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:05:47.935718 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:47.935695 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-b8spm"] Jul 26 15:05:47.935862 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:47.935849 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8spm" Jul 26 15:05:47.935994 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:47.935970 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8spm" podUID="706538eb-583d-4fca-8f24-285cd3ab940c" Jul 26 15:05:47.939141 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:47.939122 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-k8xdr"] Jul 26 15:05:47.939243 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:47.939203 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k8xdr" Jul 26 15:05:47.939304 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:47.939283 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k8xdr" podUID="0357f424-c295-4aa7-8330-f551894f81d8" Jul 26 15:05:47.939966 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:47.939944 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qlw9m"] Jul 26 15:05:47.940064 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:47.940051 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlw9m" Jul 26 15:05:47.940171 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:47.940149 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlw9m" podUID="1b6101bd-09a1-444f-ae02-50384ba66923" Jul 26 15:05:48.032917 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:48.032855 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:05:48.201127 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:48.201099 2555 generic.go:356] "Generic (PLEG): container finished" podID="96d47908-2b56-4fbe-a6e4-6472cd3a5cd4" containerID="b9808e20e170c8fb753fff9120c66b98ff52e44b43615be6f208d29b221c3289" exitCode=0 Jul 26 15:05:48.201471 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:48.201190 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4kk2l" event={"ID":"96d47908-2b56-4fbe-a6e4-6472cd3a5cd4","Type":"ContainerDied","Data":"b9808e20e170c8fb753fff9120c66b98ff52e44b43615be6f208d29b221c3289"} Jul 26 15:05:48.201471 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:48.201437 2555 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 26 15:05:48.263552 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:48.263524 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:05:48.836823 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:48.836795 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:05:48.893501 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:48.893425 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:05:48.993577 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:48.993547 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:05:49.058679 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:49.058655 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlw9m" Jul 26 15:05:49.058794 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:49.058748 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlw9m" podUID="1b6101bd-09a1-444f-ae02-50384ba66923" Jul 26 15:05:49.602920 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:49.602859 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:05:49.653068 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:49.653047 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:05:49.921967 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:49.921943 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:05:49.973380 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:49.973336 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:05:50.058780 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:50.058755 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k8xdr" Jul 26 15:05:50.058894 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:50.058851 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k8xdr" podUID="0357f424-c295-4aa7-8330-f551894f81d8" Jul 26 15:05:50.058962 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:50.058901 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8spm" Jul 26 15:05:50.059019 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:50.059001 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8spm" podUID="706538eb-583d-4fca-8f24-285cd3ab940c" Jul 26 15:05:50.208426 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:50.208362 2555 generic.go:356] "Generic (PLEG): container finished" podID="96d47908-2b56-4fbe-a6e4-6472cd3a5cd4" containerID="6cbebf9d10850912d8a80837261eae04bc527dd49aec2a781db65efe79b13e70" exitCode=0 Jul 26 15:05:50.208562 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:50.208454 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4kk2l" event={"ID":"96d47908-2b56-4fbe-a6e4-6472cd3a5cd4","Type":"ContainerDied","Data":"6cbebf9d10850912d8a80837261eae04bc527dd49aec2a781db65efe79b13e70"} Jul 26 15:05:50.218827 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:50.218787 2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-9f857" podUID="1fbafda6-6242-400e-b2d4-24584608e5f5" containerName="ovnkube-controller" probeResult="failure" output="" Jul 26 15:05:51.058905 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:51.058874 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlw9m" Jul 26 15:05:51.059593 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:51.059017 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlw9m" podUID="1b6101bd-09a1-444f-ae02-50384ba66923" Jul 26 15:05:52.058041 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:52.057954 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k8xdr" Jul 26 15:05:52.058187 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:52.058066 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k8xdr" podUID="0357f424-c295-4aa7-8330-f551894f81d8" Jul 26 15:05:52.058187 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:52.058106 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8spm" Jul 26 15:05:52.058283 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:52.058220 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8spm" podUID="706538eb-583d-4fca-8f24-285cd3ab940c" Jul 26 15:05:52.873604 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:52.873571 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:05:52.887077 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:52.887055 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-21.ec2.internal" event="NodeReady" Jul 26 15:05:52.887212 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:52.887168 2555 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 26 15:05:52.918630 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:52.918579 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7fd4475f4c-z475j"] Jul 26 15:05:52.950995 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:52.950962 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-x57rg"] Jul 26 15:05:52.951467 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:52.951449 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fd4475f4c-z475j" Jul 26 15:05:52.953653 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:52.953424 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 26 15:05:52.953653 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:52.953460 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6qhrv\"" Jul 26 15:05:52.953653 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:52.953508 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 26 15:05:52.953653 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:52.953508 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 26 15:05:52.957723 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:52.957675 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 26 15:05:52.964838 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:52.964777 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vmblz"] Jul 26 15:05:52.964951 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:52.964850 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-x57rg" Jul 26 15:05:52.966806 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:52.966790 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 26 15:05:52.966910 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:52.966892 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8qc59\"" Jul 26 15:05:52.966993 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:52.966897 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 26 15:05:52.986894 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:52.986875 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7fd4475f4c-z475j"] Jul 26 15:05:52.987033 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:52.986910 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-x57rg"] Jul 26 15:05:52.987033 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:52.986953 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vmblz"] Jul 26 15:05:52.987141 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:52.987058 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vmblz" Jul 26 15:05:52.988996 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:52.988977 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 26 15:05:52.989094 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:52.989030 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-fz4kh\"" Jul 26 15:05:52.989173 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:52.989156 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 26 15:05:52.989257 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:52.989240 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 26 15:05:53.038971 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:53.038942 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:05:53.057824 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:53.057801 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlw9m" Jul 26 15:05:53.060211 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:53.060194 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 26 15:05:53.060324 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:53.060304 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tgnb2\"" Jul 26 15:05:53.088939 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:53.088906 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm97b\" (UniqueName: \"kubernetes.io/projected/184d66ee-e80d-4766-9642-c9de748e1d7b-kube-api-access-cm97b\") pod \"dns-default-x57rg\" (UID: \"184d66ee-e80d-4766-9642-c9de748e1d7b\") " pod="openshift-dns/dns-default-x57rg" Jul 26 15:05:53.089022 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:53.088950 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnbzm\" (UniqueName: \"kubernetes.io/projected/da36cb43-fdd5-41dc-a3d9-e799d58a9ae4-kube-api-access-mnbzm\") pod \"ingress-canary-vmblz\" (UID: \"da36cb43-fdd5-41dc-a3d9-e799d58a9ae4\") " pod="openshift-ingress-canary/ingress-canary-vmblz" Jul 26 15:05:53.089022 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:53.088996 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3183398e-9b01-4309-a64d-0c37a9c20303-image-registry-private-configuration\") pod \"image-registry-7fd4475f4c-z475j\" (UID: \"3183398e-9b01-4309-a64d-0c37a9c20303\") " pod="openshift-image-registry/image-registry-7fd4475f4c-z475j" Jul 26 15:05:53.089139 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:53.089051 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da36cb43-fdd5-41dc-a3d9-e799d58a9ae4-cert\") pod \"ingress-canary-vmblz\" (UID: \"da36cb43-fdd5-41dc-a3d9-e799d58a9ae4\") " pod="openshift-ingress-canary/ingress-canary-vmblz" Jul 26 15:05:53.089139 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:53.089077 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3183398e-9b01-4309-a64d-0c37a9c20303-trusted-ca\") pod \"image-registry-7fd4475f4c-z475j\" (UID: \"3183398e-9b01-4309-a64d-0c37a9c20303\") " pod="openshift-image-registry/image-registry-7fd4475f4c-z475j" Jul 26 15:05:53.089139 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:53.089129 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3183398e-9b01-4309-a64d-0c37a9c20303-registry-certificates\") pod \"image-registry-7fd4475f4c-z475j\" (UID: \"3183398e-9b01-4309-a64d-0c37a9c20303\") " pod="openshift-image-registry/image-registry-7fd4475f4c-z475j" Jul 26 15:05:53.089283 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:53.089199 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3183398e-9b01-4309-a64d-0c37a9c20303-registry-tls\") pod \"image-registry-7fd4475f4c-z475j\" (UID: \"3183398e-9b01-4309-a64d-0c37a9c20303\") " pod="openshift-image-registry/image-registry-7fd4475f4c-z475j" Jul 26 15:05:53.089283 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:53.089226 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3183398e-9b01-4309-a64d-0c37a9c20303-bound-sa-token\") pod \"image-registry-7fd4475f4c-z475j\" (UID: \"3183398e-9b01-4309-a64d-0c37a9c20303\") " pod="openshift-image-registry/image-registry-7fd4475f4c-z475j" Jul 26 15:05:53.089283 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:53.089266 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/184d66ee-e80d-4766-9642-c9de748e1d7b-config-volume\") pod \"dns-default-x57rg\" (UID: \"184d66ee-e80d-4766-9642-c9de748e1d7b\") " pod="openshift-dns/dns-default-x57rg" Jul 26 15:05:53.089417 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:53.089290 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/184d66ee-e80d-4766-9642-c9de748e1d7b-metrics-tls\") pod \"dns-default-x57rg\" (UID: \"184d66ee-e80d-4766-9642-c9de748e1d7b\") " pod="openshift-dns/dns-default-x57rg" Jul 26 15:05:53.089417 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:53.089317 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3183398e-9b01-4309-a64d-0c37a9c20303-installation-pull-secrets\") pod \"image-registry-7fd4475f4c-z475j\" (UID: \"3183398e-9b01-4309-a64d-0c37a9c20303\") " pod="openshift-image-registry/image-registry-7fd4475f4c-z475j" Jul 26 15:05:53.089417 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:53.089356 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3183398e-9b01-4309-a64d-0c37a9c20303-ca-trust-extracted\") pod \"image-registry-7fd4475f4c-z475j\" (UID: \"3183398e-9b01-4309-a64d-0c37a9c20303\") " pod="openshift-image-registry/image-registry-7fd4475f4c-z475j" Jul 26 15:05:53.089417 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:53.089386 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/184d66ee-e80d-4766-9642-c9de748e1d7b-tmp-dir\") pod \"dns-default-x57rg\" (UID: \"184d66ee-e80d-4766-9642-c9de748e1d7b\") " pod="openshift-dns/dns-default-x57rg" Jul 26 15:05:53.089417 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:53.089407 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbkhj\" (UniqueName: \"kubernetes.io/projected/3183398e-9b01-4309-a64d-0c37a9c20303-kube-api-access-dbkhj\") pod \"image-registry-7fd4475f4c-z475j\" (UID: \"3183398e-9b01-4309-a64d-0c37a9c20303\") " pod="openshift-image-registry/image-registry-7fd4475f4c-z475j" Jul 26 15:05:53.139961 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:53.139921 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:05:53.189893 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:53.189875 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3183398e-9b01-4309-a64d-0c37a9c20303-ca-trust-extracted\") pod \"image-registry-7fd4475f4c-z475j\" (UID: \"3183398e-9b01-4309-a64d-0c37a9c20303\") " pod="openshift-image-registry/image-registry-7fd4475f4c-z475j" Jul 26 15:05:53.190006 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:53.189909 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/184d66ee-e80d-4766-9642-c9de748e1d7b-tmp-dir\") pod \"dns-default-x57rg\" (UID: \"184d66ee-e80d-4766-9642-c9de748e1d7b\") " pod="openshift-dns/dns-default-x57rg" Jul 26 15:05:53.190006 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:53.189952 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbkhj\" (UniqueName: \"kubernetes.io/projected/3183398e-9b01-4309-a64d-0c37a9c20303-kube-api-access-dbkhj\") pod \"image-registry-7fd4475f4c-z475j\" (UID: \"3183398e-9b01-4309-a64d-0c37a9c20303\") " pod="openshift-image-registry/image-registry-7fd4475f4c-z475j" Jul 26 15:05:53.190006 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:53.189979 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm97b\" (UniqueName: \"kubernetes.io/projected/184d66ee-e80d-4766-9642-c9de748e1d7b-kube-api-access-cm97b\") pod \"dns-default-x57rg\" (UID: \"184d66ee-e80d-4766-9642-c9de748e1d7b\") " pod="openshift-dns/dns-default-x57rg" Jul 26 15:05:53.190006 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:53.189995 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnbzm\" (UniqueName: \"kubernetes.io/projected/da36cb43-fdd5-41dc-a3d9-e799d58a9ae4-kube-api-access-mnbzm\") pod \"ingress-canary-vmblz\" (UID: \"da36cb43-fdd5-41dc-a3d9-e799d58a9ae4\") " pod="openshift-ingress-canary/ingress-canary-vmblz" Jul 26 15:05:53.190203 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:53.190021 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3183398e-9b01-4309-a64d-0c37a9c20303-image-registry-private-configuration\") pod \"image-registry-7fd4475f4c-z475j\" (UID: \"3183398e-9b01-4309-a64d-0c37a9c20303\") " pod="openshift-image-registry/image-registry-7fd4475f4c-z475j" Jul 26 15:05:53.190203 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:53.190048 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da36cb43-fdd5-41dc-a3d9-e799d58a9ae4-cert\") pod \"ingress-canary-vmblz\" (UID: \"da36cb43-fdd5-41dc-a3d9-e799d58a9ae4\") " pod="openshift-ingress-canary/ingress-canary-vmblz" Jul 26 15:05:53.190203 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:53.190071 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3183398e-9b01-4309-a64d-0c37a9c20303-trusted-ca\") pod \"image-registry-7fd4475f4c-z475j\" (UID: \"3183398e-9b01-4309-a64d-0c37a9c20303\") " pod="openshift-image-registry/image-registry-7fd4475f4c-z475j" Jul 26 15:05:53.190344 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:53.190225 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3183398e-9b01-4309-a64d-0c37a9c20303-registry-certificates\") pod \"image-registry-7fd4475f4c-z475j\" (UID: \"3183398e-9b01-4309-a64d-0c37a9c20303\") " pod="openshift-image-registry/image-registry-7fd4475f4c-z475j" Jul 26 15:05:53.190344 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:53.190243 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/184d66ee-e80d-4766-9642-c9de748e1d7b-tmp-dir\") pod \"dns-default-x57rg\" (UID: \"184d66ee-e80d-4766-9642-c9de748e1d7b\") " pod="openshift-dns/dns-default-x57rg" Jul 26 15:05:53.190432 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:53.190378 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3183398e-9b01-4309-a64d-0c37a9c20303-ca-trust-extracted\") pod \"image-registry-7fd4475f4c-z475j\" (UID: \"3183398e-9b01-4309-a64d-0c37a9c20303\") " pod="openshift-image-registry/image-registry-7fd4475f4c-z475j" Jul 26 15:05:53.190491 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:53.190429 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3183398e-9b01-4309-a64d-0c37a9c20303-registry-tls\") pod \"image-registry-7fd4475f4c-z475j\" (UID: \"3183398e-9b01-4309-a64d-0c37a9c20303\") " pod="openshift-image-registry/image-registry-7fd4475f4c-z475j" Jul 26 15:05:53.190491 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:53.190459 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3183398e-9b01-4309-a64d-0c37a9c20303-bound-sa-token\") pod \"image-registry-7fd4475f4c-z475j\" (UID: \"3183398e-9b01-4309-a64d-0c37a9c20303\") " pod="openshift-image-registry/image-registry-7fd4475f4c-z475j" Jul 26 15:05:53.190586 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:53.190564 2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 26 15:05:53.190586 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:53.190577 2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fd4475f4c-z475j: secret "image-registry-tls" not found Jul 26 15:05:53.190682 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:53.190638 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3183398e-9b01-4309-a64d-0c37a9c20303-registry-tls podName:3183398e-9b01-4309-a64d-0c37a9c20303 nodeName:}" failed. No retries permitted until 2026-07-26 15:05:53.690620906 +0000 UTC m=+34.114467767 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3183398e-9b01-4309-a64d-0c37a9c20303-registry-tls") pod "image-registry-7fd4475f4c-z475j" (UID: "3183398e-9b01-4309-a64d-0c37a9c20303") : secret "image-registry-tls" not found Jul 26 15:05:53.190791 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:53.190770 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3183398e-9b01-4309-a64d-0c37a9c20303-registry-certificates\") pod \"image-registry-7fd4475f4c-z475j\" (UID: \"3183398e-9b01-4309-a64d-0c37a9c20303\") " pod="openshift-image-registry/image-registry-7fd4475f4c-z475j" Jul 26 15:05:53.190856 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:53.190821 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/184d66ee-e80d-4766-9642-c9de748e1d7b-config-volume\") pod \"dns-default-x57rg\" (UID: \"184d66ee-e80d-4766-9642-c9de748e1d7b\") " pod="openshift-dns/dns-default-x57rg" Jul 26 15:05:53.190992 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:53.190947 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3183398e-9b01-4309-a64d-0c37a9c20303-trusted-ca\") pod \"image-registry-7fd4475f4c-z475j\" (UID: \"3183398e-9b01-4309-a64d-0c37a9c20303\") " pod="openshift-image-registry/image-registry-7fd4475f4c-z475j" Jul 26 15:05:53.190992 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:53.190964 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/184d66ee-e80d-4766-9642-c9de748e1d7b-metrics-tls\") pod \"dns-default-x57rg\" (UID: \"184d66ee-e80d-4766-9642-c9de748e1d7b\") " pod="openshift-dns/dns-default-x57rg" Jul 26 15:05:53.191160 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:53.191002 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3183398e-9b01-4309-a64d-0c37a9c20303-installation-pull-secrets\") pod \"image-registry-7fd4475f4c-z475j\" (UID: \"3183398e-9b01-4309-a64d-0c37a9c20303\") " pod="openshift-image-registry/image-registry-7fd4475f4c-z475j" Jul 26 15:05:53.191160 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:53.191060 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 26 15:05:53.191160 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:53.191114 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/184d66ee-e80d-4766-9642-c9de748e1d7b-metrics-tls podName:184d66ee-e80d-4766-9642-c9de748e1d7b nodeName:}" failed. No retries permitted until 2026-07-26 15:05:53.691099608 +0000 UTC m=+34.114946483 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/184d66ee-e80d-4766-9642-c9de748e1d7b-metrics-tls") pod "dns-default-x57rg" (UID: "184d66ee-e80d-4766-9642-c9de748e1d7b") : secret "dns-default-metrics-tls" not found Jul 26 15:05:53.191366 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:53.191342 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/184d66ee-e80d-4766-9642-c9de748e1d7b-config-volume\") pod \"dns-default-x57rg\" (UID: \"184d66ee-e80d-4766-9642-c9de748e1d7b\") " pod="openshift-dns/dns-default-x57rg" Jul 26 15:05:53.194721 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:53.194701 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3183398e-9b01-4309-a64d-0c37a9c20303-image-registry-private-configuration\") pod \"image-registry-7fd4475f4c-z475j\" (UID: \"3183398e-9b01-4309-a64d-0c37a9c20303\") " pod="openshift-image-registry/image-registry-7fd4475f4c-z475j" Jul 26 15:05:53.194809 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:53.194703 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3183398e-9b01-4309-a64d-0c37a9c20303-installation-pull-secrets\") pod \"image-registry-7fd4475f4c-z475j\" (UID: \"3183398e-9b01-4309-a64d-0c37a9c20303\") " pod="openshift-image-registry/image-registry-7fd4475f4c-z475j" Jul 26 15:05:53.194888 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:53.194836 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da36cb43-fdd5-41dc-a3d9-e799d58a9ae4-cert\") pod \"ingress-canary-vmblz\" (UID: \"da36cb43-fdd5-41dc-a3d9-e799d58a9ae4\") " pod="openshift-ingress-canary/ingress-canary-vmblz" Jul 26 15:05:53.199054 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:53.199034 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm97b\" (UniqueName: \"kubernetes.io/projected/184d66ee-e80d-4766-9642-c9de748e1d7b-kube-api-access-cm97b\") pod \"dns-default-x57rg\" (UID: \"184d66ee-e80d-4766-9642-c9de748e1d7b\") " pod="openshift-dns/dns-default-x57rg" Jul 26 15:05:53.199230 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:53.199210 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3183398e-9b01-4309-a64d-0c37a9c20303-bound-sa-token\") pod \"image-registry-7fd4475f4c-z475j\" (UID: \"3183398e-9b01-4309-a64d-0c37a9c20303\") " pod="openshift-image-registry/image-registry-7fd4475f4c-z475j" Jul 26 15:05:53.199292 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:53.199269 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbkhj\" (UniqueName: \"kubernetes.io/projected/3183398e-9b01-4309-a64d-0c37a9c20303-kube-api-access-dbkhj\") pod \"image-registry-7fd4475f4c-z475j\" (UID: \"3183398e-9b01-4309-a64d-0c37a9c20303\") " pod="openshift-image-registry/image-registry-7fd4475f4c-z475j" Jul 26 15:05:53.199352 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:53.199269 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnbzm\" (UniqueName: \"kubernetes.io/projected/da36cb43-fdd5-41dc-a3d9-e799d58a9ae4-kube-api-access-mnbzm\") pod \"ingress-canary-vmblz\" (UID: \"da36cb43-fdd5-41dc-a3d9-e799d58a9ae4\") " pod="openshift-ingress-canary/ingress-canary-vmblz" Jul 26 15:05:53.296106 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:53.296020 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vmblz" Jul 26 15:05:53.693886 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:53.693847 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4mqj\" (UniqueName: \"kubernetes.io/projected/0357f424-c295-4aa7-8330-f551894f81d8-kube-api-access-r4mqj\") pod \"network-check-target-k8xdr\" (UID: \"0357f424-c295-4aa7-8330-f551894f81d8\") " pod="openshift-network-diagnostics/network-check-target-k8xdr" Jul 26 15:05:53.694087 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:53.693911 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3183398e-9b01-4309-a64d-0c37a9c20303-registry-tls\") pod \"image-registry-7fd4475f4c-z475j\" (UID: \"3183398e-9b01-4309-a64d-0c37a9c20303\") " pod="openshift-image-registry/image-registry-7fd4475f4c-z475j" Jul 26 15:05:53.694087 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:53.693960 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/706538eb-583d-4fca-8f24-285cd3ab940c-original-pull-secret\") pod \"global-pull-secret-syncer-b8spm\" (UID: \"706538eb-583d-4fca-8f24-285cd3ab940c\") " pod="kube-system/global-pull-secret-syncer-b8spm" Jul 26 15:05:53.694087 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:53.693987 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/184d66ee-e80d-4766-9642-c9de748e1d7b-metrics-tls\") pod \"dns-default-x57rg\" (UID: \"184d66ee-e80d-4766-9642-c9de748e1d7b\") " pod="openshift-dns/dns-default-x57rg" Jul 26 15:05:53.694087 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:53.694052 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 26 15:05:53.694087 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:53.694074 2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 26 15:05:53.694087 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:53.694080 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 26 15:05:53.694087 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:53.694088 2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fd4475f4c-z475j: secret "image-registry-tls" not found Jul 26 15:05:53.694087 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:53.694090 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 26 15:05:53.694469 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:53.694093 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 26 15:05:53.694469 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:53.694137 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/184d66ee-e80d-4766-9642-c9de748e1d7b-metrics-tls podName:184d66ee-e80d-4766-9642-c9de748e1d7b nodeName:}" failed. No retries permitted until 2026-07-26 15:05:54.694124083 +0000 UTC m=+35.117970932 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/184d66ee-e80d-4766-9642-c9de748e1d7b-metrics-tls") pod "dns-default-x57rg" (UID: "184d66ee-e80d-4766-9642-c9de748e1d7b") : secret "dns-default-metrics-tls" not found Jul 26 15:05:53.694469 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:53.694093 2555 projected.go:196] Error preparing data for projected volume kube-api-access-r4mqj for pod openshift-network-diagnostics/network-check-target-k8xdr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 15:05:53.694469 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:53.694154 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3183398e-9b01-4309-a64d-0c37a9c20303-registry-tls podName:3183398e-9b01-4309-a64d-0c37a9c20303 nodeName:}" failed. No retries permitted until 2026-07-26 15:05:54.694144813 +0000 UTC m=+35.117991664 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3183398e-9b01-4309-a64d-0c37a9c20303-registry-tls") pod "image-registry-7fd4475f4c-z475j" (UID: "3183398e-9b01-4309-a64d-0c37a9c20303") : secret "image-registry-tls" not found Jul 26 15:05:53.694469 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:53.694169 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/706538eb-583d-4fca-8f24-285cd3ab940c-original-pull-secret podName:706538eb-583d-4fca-8f24-285cd3ab940c nodeName:}" failed. No retries permitted until 2026-07-26 15:06:25.694160922 +0000 UTC m=+66.118007774 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/706538eb-583d-4fca-8f24-285cd3ab940c-original-pull-secret") pod "global-pull-secret-syncer-b8spm" (UID: "706538eb-583d-4fca-8f24-285cd3ab940c") : object "kube-system"/"original-pull-secret" not registered Jul 26 15:05:53.694469 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:53.694183 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0357f424-c295-4aa7-8330-f551894f81d8-kube-api-access-r4mqj podName:0357f424-c295-4aa7-8330-f551894f81d8 nodeName:}" failed. No retries permitted until 2026-07-26 15:06:25.694175861 +0000 UTC m=+66.118022715 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-r4mqj" (UniqueName: "kubernetes.io/projected/0357f424-c295-4aa7-8330-f551894f81d8-kube-api-access-r4mqj") pod "network-check-target-k8xdr" (UID: "0357f424-c295-4aa7-8330-f551894f81d8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 15:05:53.795193 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:53.795160 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b6101bd-09a1-444f-ae02-50384ba66923-metrics-certs\") pod \"network-metrics-daemon-qlw9m\" (UID: \"1b6101bd-09a1-444f-ae02-50384ba66923\") " pod="openshift-multus/network-metrics-daemon-qlw9m" Jul 26 15:05:53.795361 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:53.795335 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 26 15:05:53.795414 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:53.795390 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b6101bd-09a1-444f-ae02-50384ba66923-metrics-certs podName:1b6101bd-09a1-444f-ae02-50384ba66923 nodeName:}" failed. No retries permitted until 2026-07-26 15:06:25.79537507 +0000 UTC m=+66.219221923 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1b6101bd-09a1-444f-ae02-50384ba66923-metrics-certs") pod "network-metrics-daemon-qlw9m" (UID: "1b6101bd-09a1-444f-ae02-50384ba66923") : secret "metrics-daemon-secret" not found Jul 26 15:05:54.058091 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:54.058018 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k8xdr" Jul 26 15:05:54.058496 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:54.058018 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8spm" Jul 26 15:05:54.061790 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:54.061691 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 26 15:05:54.061790 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:54.061692 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 26 15:05:54.061790 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:54.061743 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4dzfg\"" Jul 26 15:05:54.062048 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:54.061837 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 26 15:05:54.257101 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:54.257074 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vmblz"] Jul 26 15:05:54.298215 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:05:54.298188 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda36cb43_fdd5_41dc_a3d9_e799d58a9ae4.slice/crio-d6808b8bd4c6844bae971a50bdcba52fac4141a389cf28389982c6d226790c83 WatchSource:0}: Error finding container d6808b8bd4c6844bae971a50bdcba52fac4141a389cf28389982c6d226790c83: Status 404 returned error can't find the container with id d6808b8bd4c6844bae971a50bdcba52fac4141a389cf28389982c6d226790c83 Jul 26 15:05:54.701330 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:54.701302 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3183398e-9b01-4309-a64d-0c37a9c20303-registry-tls\") pod \"image-registry-7fd4475f4c-z475j\" (UID: \"3183398e-9b01-4309-a64d-0c37a9c20303\") " pod="openshift-image-registry/image-registry-7fd4475f4c-z475j" Jul 26 15:05:54.701442 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:54.701342 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/184d66ee-e80d-4766-9642-c9de748e1d7b-metrics-tls\") pod \"dns-default-x57rg\" (UID: \"184d66ee-e80d-4766-9642-c9de748e1d7b\") " pod="openshift-dns/dns-default-x57rg" Jul 26 15:05:54.701442 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:54.701431 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 26 15:05:54.701519 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:54.701449 2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 26 15:05:54.701519 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:54.701467 2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fd4475f4c-z475j: secret "image-registry-tls" not found Jul 26 15:05:54.701519 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:54.701482 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/184d66ee-e80d-4766-9642-c9de748e1d7b-metrics-tls podName:184d66ee-e80d-4766-9642-c9de748e1d7b nodeName:}" failed. No retries permitted until 2026-07-26 15:05:56.701467889 +0000 UTC m=+37.125314738 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/184d66ee-e80d-4766-9642-c9de748e1d7b-metrics-tls") pod "dns-default-x57rg" (UID: "184d66ee-e80d-4766-9642-c9de748e1d7b") : secret "dns-default-metrics-tls" not found Jul 26 15:05:54.701519 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:54.701509 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3183398e-9b01-4309-a64d-0c37a9c20303-registry-tls podName:3183398e-9b01-4309-a64d-0c37a9c20303 nodeName:}" failed. No retries permitted until 2026-07-26 15:05:56.701496951 +0000 UTC m=+37.125343827 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3183398e-9b01-4309-a64d-0c37a9c20303-registry-tls") pod "image-registry-7fd4475f4c-z475j" (UID: "3183398e-9b01-4309-a64d-0c37a9c20303") : secret "image-registry-tls" not found Jul 26 15:05:55.223268 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:55.223230 2555 generic.go:356] "Generic (PLEG): container finished" podID="96d47908-2b56-4fbe-a6e4-6472cd3a5cd4" containerID="b5508a929e30adcb04f5d8ee0bd2c6a9b5815cd9e80bb0b18a1914b28d1fcd56" exitCode=0 Jul 26 15:05:55.223700 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:55.223317 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4kk2l" event={"ID":"96d47908-2b56-4fbe-a6e4-6472cd3a5cd4","Type":"ContainerDied","Data":"b5508a929e30adcb04f5d8ee0bd2c6a9b5815cd9e80bb0b18a1914b28d1fcd56"} Jul 26 15:05:55.224751 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:55.224726 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vmblz" event={"ID":"da36cb43-fdd5-41dc-a3d9-e799d58a9ae4","Type":"ContainerStarted","Data":"d6808b8bd4c6844bae971a50bdcba52fac4141a389cf28389982c6d226790c83"} Jul 26 15:05:56.230703 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:56.230494 2555 generic.go:356] "Generic (PLEG): container finished" podID="96d47908-2b56-4fbe-a6e4-6472cd3a5cd4" containerID="44701c0b4bd588cd16de4bdd8f8580b5eac0da3ae69391f48380e13366040731" exitCode=0 Jul 26 15:05:56.230703 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:56.230584 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4kk2l" event={"ID":"96d47908-2b56-4fbe-a6e4-6472cd3a5cd4","Type":"ContainerDied","Data":"44701c0b4bd588cd16de4bdd8f8580b5eac0da3ae69391f48380e13366040731"} Jul 26 15:05:56.718250 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:56.718215 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3183398e-9b01-4309-a64d-0c37a9c20303-registry-tls\") pod \"image-registry-7fd4475f4c-z475j\" (UID: \"3183398e-9b01-4309-a64d-0c37a9c20303\") " pod="openshift-image-registry/image-registry-7fd4475f4c-z475j" Jul 26 15:05:56.718426 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:56.718265 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/184d66ee-e80d-4766-9642-c9de748e1d7b-metrics-tls\") pod \"dns-default-x57rg\" (UID: \"184d66ee-e80d-4766-9642-c9de748e1d7b\") " pod="openshift-dns/dns-default-x57rg" Jul 26 15:05:56.718426 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:56.718374 2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 26 15:05:56.718426 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:56.718385 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 26 15:05:56.718426 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:56.718393 2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fd4475f4c-z475j: secret "image-registry-tls" not found Jul 26 15:05:56.718569 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:56.718453 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3183398e-9b01-4309-a64d-0c37a9c20303-registry-tls podName:3183398e-9b01-4309-a64d-0c37a9c20303 nodeName:}" failed. No retries permitted until 2026-07-26 15:06:00.718434666 +0000 UTC m=+41.142281516 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3183398e-9b01-4309-a64d-0c37a9c20303-registry-tls") pod "image-registry-7fd4475f4c-z475j" (UID: "3183398e-9b01-4309-a64d-0c37a9c20303") : secret "image-registry-tls" not found Jul 26 15:05:56.718569 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:05:56.718473 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/184d66ee-e80d-4766-9642-c9de748e1d7b-metrics-tls podName:184d66ee-e80d-4766-9642-c9de748e1d7b nodeName:}" failed. No retries permitted until 2026-07-26 15:06:00.718462953 +0000 UTC m=+41.142309808 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/184d66ee-e80d-4766-9642-c9de748e1d7b-metrics-tls") pod "dns-default-x57rg" (UID: "184d66ee-e80d-4766-9642-c9de748e1d7b") : secret "dns-default-metrics-tls" not found Jul 26 15:05:57.235654 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:57.235621 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4kk2l" event={"ID":"96d47908-2b56-4fbe-a6e4-6472cd3a5cd4","Type":"ContainerStarted","Data":"435f80cd2b281c5d1a582d71515c36589ed760931bf1ef4dfc2a4e0c66c74436"} Jul 26 15:05:57.237094 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:57.237074 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vmblz" event={"ID":"da36cb43-fdd5-41dc-a3d9-e799d58a9ae4","Type":"ContainerStarted","Data":"6d61b6ddfbac1f251bdff2009f6af6ab66f1455985f8131399297a43aced2d7a"} Jul 26 15:05:57.255342 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:57.255298 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4kk2l" podStartSLOduration=9.47550658 podStartE2EDuration="37.2552848s" totalImagesPullingTime="27.77977822s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 15:05:20 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:05:57.254172192 +0000 UTC m=+37.678019066" watchObservedRunningTime="2026-07-26 15:05:57.2552848 +0000 UTC m=+37.679131671" Jul 26 15:05:57.267828 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:57.267792 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vmblz" podStartSLOduration=3.129473473 podStartE2EDuration="5.267780361s" totalImagesPullingTime="2.138306888s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:05:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:05:57.266920102 +0000 UTC m=+37.690766987" watchObservedRunningTime="2026-07-26 15:05:57.267780361 +0000 UTC m=+37.691627210" Jul 26 15:05:57.381909 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:57.381875 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:05:57.428527 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:57.428511 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:05:57.722169 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:57.722127 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:05:57.770616 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:57.770583 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:05:58.332715 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:58.332680 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:05:58.380477 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:05:58.380436 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:06:00.549846 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:06:00.549814 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:06:00.746408 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:06:00.746378 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3183398e-9b01-4309-a64d-0c37a9c20303-registry-tls\") pod \"image-registry-7fd4475f4c-z475j\" (UID: \"3183398e-9b01-4309-a64d-0c37a9c20303\") " pod="openshift-image-registry/image-registry-7fd4475f4c-z475j" Jul 26 15:06:00.746549 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:06:00.746416 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/184d66ee-e80d-4766-9642-c9de748e1d7b-metrics-tls\") pod \"dns-default-x57rg\" (UID: \"184d66ee-e80d-4766-9642-c9de748e1d7b\") " pod="openshift-dns/dns-default-x57rg" Jul 26 15:06:00.746549 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:06:00.746501 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 26 15:06:00.746549 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:06:00.746506 2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 26 15:06:00.746549 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:06:00.746521 2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fd4475f4c-z475j: secret "image-registry-tls" not found Jul 26 15:06:00.746549 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:06:00.746549 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/184d66ee-e80d-4766-9642-c9de748e1d7b-metrics-tls podName:184d66ee-e80d-4766-9642-c9de748e1d7b nodeName:}" failed. No retries permitted until 2026-07-26 15:06:08.746533605 +0000 UTC m=+49.170380454 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/184d66ee-e80d-4766-9642-c9de748e1d7b-metrics-tls") pod "dns-default-x57rg" (UID: "184d66ee-e80d-4766-9642-c9de748e1d7b") : secret "dns-default-metrics-tls" not found Jul 26 15:06:00.746701 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:06:00.746562 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3183398e-9b01-4309-a64d-0c37a9c20303-registry-tls podName:3183398e-9b01-4309-a64d-0c37a9c20303 nodeName:}" failed. No retries permitted until 2026-07-26 15:06:08.746556494 +0000 UTC m=+49.170403343 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3183398e-9b01-4309-a64d-0c37a9c20303-registry-tls") pod "image-registry-7fd4475f4c-z475j" (UID: "3183398e-9b01-4309-a64d-0c37a9c20303") : secret "image-registry-tls" not found Jul 26 15:06:03.964410 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:06:03.964381 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:06:04.021727 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:06:04.021700 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:06:05.393100 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:06:05.393067 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:06:05.437937 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:06:05.437911 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:06:08.798543 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:06:08.798512 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3183398e-9b01-4309-a64d-0c37a9c20303-registry-tls\") pod \"image-registry-7fd4475f4c-z475j\" (UID: \"3183398e-9b01-4309-a64d-0c37a9c20303\") " pod="openshift-image-registry/image-registry-7fd4475f4c-z475j" Jul 26 15:06:08.798951 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:06:08.798553 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/184d66ee-e80d-4766-9642-c9de748e1d7b-metrics-tls\") pod \"dns-default-x57rg\" (UID: \"184d66ee-e80d-4766-9642-c9de748e1d7b\") " pod="openshift-dns/dns-default-x57rg" Jul 26 15:06:08.798951 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:06:08.798648 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 26 15:06:08.798951 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:06:08.798653 2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 26 15:06:08.798951 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:06:08.798670 2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fd4475f4c-z475j: secret "image-registry-tls" not found Jul 26 15:06:08.798951 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:06:08.798711 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/184d66ee-e80d-4766-9642-c9de748e1d7b-metrics-tls podName:184d66ee-e80d-4766-9642-c9de748e1d7b nodeName:}" failed. No retries permitted until 2026-07-26 15:06:24.798696194 +0000 UTC m=+65.222543043 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/184d66ee-e80d-4766-9642-c9de748e1d7b-metrics-tls") pod "dns-default-x57rg" (UID: "184d66ee-e80d-4766-9642-c9de748e1d7b") : secret "dns-default-metrics-tls" not found Jul 26 15:06:08.798951 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:06:08.798724 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3183398e-9b01-4309-a64d-0c37a9c20303-registry-tls podName:3183398e-9b01-4309-a64d-0c37a9c20303 nodeName:}" failed. No retries permitted until 2026-07-26 15:06:24.798718383 +0000 UTC m=+65.222565232 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3183398e-9b01-4309-a64d-0c37a9c20303-registry-tls") pod "image-registry-7fd4475f4c-z475j" (UID: "3183398e-9b01-4309-a64d-0c37a9c20303") : secret "image-registry-tls" not found Jul 26 15:06:11.588756 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:06:11.588726 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:06:11.643022 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:06:11.642998 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:06:20.217418 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:06:20.217393 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9f857" Jul 26 15:06:20.366405 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:06:20.366363 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:06:20.421678 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:06:20.421652 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:06:24.807120 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:06:24.807071 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3183398e-9b01-4309-a64d-0c37a9c20303-registry-tls\") pod \"image-registry-7fd4475f4c-z475j\" (UID: \"3183398e-9b01-4309-a64d-0c37a9c20303\") " pod="openshift-image-registry/image-registry-7fd4475f4c-z475j" Jul 26 15:06:24.807120 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:06:24.807130 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/184d66ee-e80d-4766-9642-c9de748e1d7b-metrics-tls\") pod \"dns-default-x57rg\" (UID: \"184d66ee-e80d-4766-9642-c9de748e1d7b\") " pod="openshift-dns/dns-default-x57rg" Jul 26 15:06:24.807587 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:06:24.807225 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 26 15:06:24.807587 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:06:24.807235 2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 26 15:06:24.807587 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:06:24.807258 2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fd4475f4c-z475j: secret "image-registry-tls" not found Jul 26 15:06:24.807587 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:06:24.807285 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/184d66ee-e80d-4766-9642-c9de748e1d7b-metrics-tls podName:184d66ee-e80d-4766-9642-c9de748e1d7b nodeName:}" failed. No retries permitted until 2026-07-26 15:06:56.807272169 +0000 UTC m=+97.231119018 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/184d66ee-e80d-4766-9642-c9de748e1d7b-metrics-tls") pod "dns-default-x57rg" (UID: "184d66ee-e80d-4766-9642-c9de748e1d7b") : secret "dns-default-metrics-tls" not found Jul 26 15:06:24.807587 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:06:24.807330 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3183398e-9b01-4309-a64d-0c37a9c20303-registry-tls podName:3183398e-9b01-4309-a64d-0c37a9c20303 nodeName:}" failed. No retries permitted until 2026-07-26 15:06:56.807313609 +0000 UTC m=+97.231160458 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3183398e-9b01-4309-a64d-0c37a9c20303-registry-tls") pod "image-registry-7fd4475f4c-z475j" (UID: "3183398e-9b01-4309-a64d-0c37a9c20303") : secret "image-registry-tls" not found Jul 26 15:06:25.714054 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:06:25.714023 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4mqj\" (UniqueName: \"kubernetes.io/projected/0357f424-c295-4aa7-8330-f551894f81d8-kube-api-access-r4mqj\") pod \"network-check-target-k8xdr\" (UID: \"0357f424-c295-4aa7-8330-f551894f81d8\") " pod="openshift-network-diagnostics/network-check-target-k8xdr" Jul 26 15:06:25.714316 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:06:25.714076 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/706538eb-583d-4fca-8f24-285cd3ab940c-original-pull-secret\") pod \"global-pull-secret-syncer-b8spm\" (UID: \"706538eb-583d-4fca-8f24-285cd3ab940c\") " pod="kube-system/global-pull-secret-syncer-b8spm" Jul 26 15:06:25.716383 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:06:25.716358 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 26 15:06:25.716509 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:06:25.716391 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 26 15:06:25.726358 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:06:25.726341 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 26 15:06:25.728477 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:06:25.728453 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/706538eb-583d-4fca-8f24-285cd3ab940c-original-pull-secret\") pod \"global-pull-secret-syncer-b8spm\" (UID: \"706538eb-583d-4fca-8f24-285cd3ab940c\") " pod="kube-system/global-pull-secret-syncer-b8spm" Jul 26 15:06:25.737544 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:06:25.737516 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4mqj\" (UniqueName: \"kubernetes.io/projected/0357f424-c295-4aa7-8330-f551894f81d8-kube-api-access-r4mqj\") pod \"network-check-target-k8xdr\" (UID: \"0357f424-c295-4aa7-8330-f551894f81d8\") " pod="openshift-network-diagnostics/network-check-target-k8xdr" Jul 26 15:06:25.814642 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:06:25.814619 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b6101bd-09a1-444f-ae02-50384ba66923-metrics-certs\") pod \"network-metrics-daemon-qlw9m\" (UID: \"1b6101bd-09a1-444f-ae02-50384ba66923\") " pod="openshift-multus/network-metrics-daemon-qlw9m" Jul 26 15:06:25.814910 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:06:25.814736 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 26 15:06:25.814910 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:06:25.814790 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b6101bd-09a1-444f-ae02-50384ba66923-metrics-certs podName:1b6101bd-09a1-444f-ae02-50384ba66923 nodeName:}" failed. No retries permitted until 2026-07-26 15:07:29.814773458 +0000 UTC m=+130.238620309 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1b6101bd-09a1-444f-ae02-50384ba66923-metrics-certs") pod "network-metrics-daemon-qlw9m" (UID: "1b6101bd-09a1-444f-ae02-50384ba66923") : secret "metrics-daemon-secret" not found Jul 26 15:06:25.870760 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:06:25.870743 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4dzfg\"" Jul 26 15:06:25.873092 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:06:25.873074 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8spm" Jul 26 15:06:25.879737 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:06:25.879704 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k8xdr" Jul 26 15:06:25.998511 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:06:25.998487 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-b8spm"] Jul 26 15:06:26.001538 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:06:26.001515 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod706538eb_583d_4fca_8f24_285cd3ab940c.slice/crio-8a73d098d2b296f18deb15464a61b3021ec6c264450bb4679201962924390362 WatchSource:0}: Error finding container 8a73d098d2b296f18deb15464a61b3021ec6c264450bb4679201962924390362: Status 404 returned error can't find the container with id 8a73d098d2b296f18deb15464a61b3021ec6c264450bb4679201962924390362 Jul 26 15:06:26.018362 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:06:26.018304 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-k8xdr"] Jul 26 15:06:26.020523 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:06:26.020504 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0357f424_c295_4aa7_8330_f551894f81d8.slice/crio-1d862d49fd3773840420f32248f14953c928799eab1545f2d38e75fe317474bd WatchSource:0}: Error finding container 1d862d49fd3773840420f32248f14953c928799eab1545f2d38e75fe317474bd: Status 404 returned error can't find the container with id 1d862d49fd3773840420f32248f14953c928799eab1545f2d38e75fe317474bd Jul 26 15:06:26.300354 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:06:26.300280 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-k8xdr" event={"ID":"0357f424-c295-4aa7-8330-f551894f81d8","Type":"ContainerStarted","Data":"1d862d49fd3773840420f32248f14953c928799eab1545f2d38e75fe317474bd"} Jul 26 15:06:26.301329 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:06:26.301303 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-b8spm" event={"ID":"706538eb-583d-4fca-8f24-285cd3ab940c","Type":"ContainerStarted","Data":"8a73d098d2b296f18deb15464a61b3021ec6c264450bb4679201962924390362"} Jul 26 15:06:26.948760 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:06:26.948727 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:06:27.003286 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:06:27.003249 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:06:31.319373 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:06:31.319324 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-k8xdr" event={"ID":"0357f424-c295-4aa7-8330-f551894f81d8","Type":"ContainerStarted","Data":"909327399c1ae9483b5d26870fd4c6208fe98668500f147a497439655a59aad2"} Jul 26 15:06:31.319971 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:06:31.319429 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-k8xdr" Jul 26 15:06:31.320917 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:06:31.320884 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-b8spm" event={"ID":"706538eb-583d-4fca-8f24-285cd3ab940c","Type":"ContainerStarted","Data":"c898fd5420e2a1992871df709207be0ba97931b8c54da6bdc19cf0ea10fee21c"} Jul 26 15:06:31.333348 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:06:31.333301 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-k8xdr" podStartSLOduration=66.989826458 podStartE2EDuration="1m11.333288699s" totalImagesPullingTime="4.343462241s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:05:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:06:31.332240182 +0000 UTC m=+71.756087052" watchObservedRunningTime="2026-07-26 15:06:31.333288699 +0000 UTC m=+71.757135569" Jul 26 15:06:31.344445 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:06:31.344391 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-b8spm" podStartSLOduration=66.978718875 podStartE2EDuration="1m11.344377964s" totalImagesPullingTime="4.365659089s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 15:05:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:06:31.343945847 +0000 UTC m=+71.767792710" watchObservedRunningTime="2026-07-26 15:06:31.344377964 +0000 UTC m=+71.768224834" Jul 26 15:06:45.436560 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:06:45.436443 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65987bb4d4-bs78b"] Jul 26 15:06:45.440318 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:06:45.440299 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65987bb4d4-bs78b" Jul 26 15:06:45.442553 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:06:45.442531 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 26 15:06:45.443533 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:06:45.443510 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 26 15:06:45.443533 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:06:45.443525 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-vbzdj\"" Jul 26 15:06:45.443659 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:06:45.443565 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 26 15:06:45.443659 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:06:45.443520 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 26 15:06:45.446748 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:06:45.446725 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65987bb4d4-bs78b"] Jul 26 15:06:45.547646 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:06:45.547619 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9cqp\" (UniqueName: \"kubernetes.io/projected/8f0717ea-6fb7-4af6-ac2a-9344789f2491-kube-api-access-f9cqp\") pod \"managed-serviceaccount-addon-agent-65987bb4d4-bs78b\" (UID: \"8f0717ea-6fb7-4af6-ac2a-9344789f2491\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65987bb4d4-bs78b" Jul 26 15:06:45.547758 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:06:45.547649 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8f0717ea-6fb7-4af6-ac2a-9344789f2491-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-65987bb4d4-bs78b\" (UID: \"8f0717ea-6fb7-4af6-ac2a-9344789f2491\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65987bb4d4-bs78b" Jul 26 15:06:45.648898 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:06:45.648875 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9cqp\" (UniqueName: \"kubernetes.io/projected/8f0717ea-6fb7-4af6-ac2a-9344789f2491-kube-api-access-f9cqp\") pod \"managed-serviceaccount-addon-agent-65987bb4d4-bs78b\" (UID: \"8f0717ea-6fb7-4af6-ac2a-9344789f2491\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65987bb4d4-bs78b" Jul 26 15:06:45.648975 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:06:45.648905 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8f0717ea-6fb7-4af6-ac2a-9344789f2491-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-65987bb4d4-bs78b\" (UID: \"8f0717ea-6fb7-4af6-ac2a-9344789f2491\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65987bb4d4-bs78b" Jul 26 15:06:45.651177 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:06:45.651156 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8f0717ea-6fb7-4af6-ac2a-9344789f2491-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-65987bb4d4-bs78b\" (UID: \"8f0717ea-6fb7-4af6-ac2a-9344789f2491\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65987bb4d4-bs78b" Jul 26 15:06:45.655827 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:06:45.655805 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9cqp\" (UniqueName: \"kubernetes.io/projected/8f0717ea-6fb7-4af6-ac2a-9344789f2491-kube-api-access-f9cqp\") pod \"managed-serviceaccount-addon-agent-65987bb4d4-bs78b\" (UID: \"8f0717ea-6fb7-4af6-ac2a-9344789f2491\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65987bb4d4-bs78b" Jul 26 15:06:45.757014 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:06:45.756910 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65987bb4d4-bs78b" Jul 26 15:06:45.871109 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:06:45.871082 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65987bb4d4-bs78b"] Jul 26 15:06:45.874172 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:06:45.874138 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f0717ea_6fb7_4af6_ac2a_9344789f2491.slice/crio-5b304afc67520094905a458b914485f6a5c5d41f823a57a8a746603e4e7a945d WatchSource:0}: Error finding container 5b304afc67520094905a458b914485f6a5c5d41f823a57a8a746603e4e7a945d: Status 404 returned error can't find the container with id 5b304afc67520094905a458b914485f6a5c5d41f823a57a8a746603e4e7a945d Jul 26 15:06:46.355746 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:06:46.355712 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65987bb4d4-bs78b" event={"ID":"8f0717ea-6fb7-4af6-ac2a-9344789f2491","Type":"ContainerStarted","Data":"5b304afc67520094905a458b914485f6a5c5d41f823a57a8a746603e4e7a945d"} Jul 26 15:06:49.367554 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:06:49.367510 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65987bb4d4-bs78b" event={"ID":"8f0717ea-6fb7-4af6-ac2a-9344789f2491","Type":"ContainerStarted","Data":"2cc76180103e8b9600648eaa66353de5bd9001176547d9791200c1213f6f99a6"} Jul 26 15:06:56.755900 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:06:56.755864 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:06:56.822057 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:06:56.822028 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3183398e-9b01-4309-a64d-0c37a9c20303-registry-tls\") pod \"image-registry-7fd4475f4c-z475j\" (UID: \"3183398e-9b01-4309-a64d-0c37a9c20303\") " pod="openshift-image-registry/image-registry-7fd4475f4c-z475j" Jul 26 15:06:56.822203 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:06:56.822067 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/184d66ee-e80d-4766-9642-c9de748e1d7b-metrics-tls\") pod \"dns-default-x57rg\" (UID: \"184d66ee-e80d-4766-9642-c9de748e1d7b\") " pod="openshift-dns/dns-default-x57rg" Jul 26 15:06:56.822203 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:06:56.822176 2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 26 15:06:56.822203 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:06:56.822200 2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fd4475f4c-z475j: secret "image-registry-tls" not found Jul 26 15:06:56.822340 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:06:56.822256 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3183398e-9b01-4309-a64d-0c37a9c20303-registry-tls podName:3183398e-9b01-4309-a64d-0c37a9c20303 nodeName:}" failed. No retries permitted until 2026-07-26 15:08:00.822239831 +0000 UTC m=+161.246086680 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3183398e-9b01-4309-a64d-0c37a9c20303-registry-tls") pod "image-registry-7fd4475f4c-z475j" (UID: "3183398e-9b01-4309-a64d-0c37a9c20303") : secret "image-registry-tls" not found Jul 26 15:06:56.822340 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:06:56.822183 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 26 15:06:56.822340 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:06:56.822339 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/184d66ee-e80d-4766-9642-c9de748e1d7b-metrics-tls podName:184d66ee-e80d-4766-9642-c9de748e1d7b nodeName:}" failed. No retries permitted until 2026-07-26 15:08:00.822320891 +0000 UTC m=+161.246167755 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/184d66ee-e80d-4766-9642-c9de748e1d7b-metrics-tls") pod "dns-default-x57rg" (UID: "184d66ee-e80d-4766-9642-c9de748e1d7b") : secret "dns-default-metrics-tls" not found Jul 26 15:07:02.326034 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:07:02.325994 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-k8xdr" Jul 26 15:07:02.338510 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:07:02.338466 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65987bb4d4-bs78b" podStartSLOduration=14.558192258 podStartE2EDuration="17.338452325s" totalImagesPullingTime="2.780260067s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:06:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:06:49.379750967 +0000 UTC m=+89.803597836" watchObservedRunningTime="2026-07-26 15:07:02.338452325 +0000 UTC m=+102.762299195" Jul 26 15:07:29.830071 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:07:29.830034 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b6101bd-09a1-444f-ae02-50384ba66923-metrics-certs\") pod \"network-metrics-daemon-qlw9m\" (UID: \"1b6101bd-09a1-444f-ae02-50384ba66923\") " pod="openshift-multus/network-metrics-daemon-qlw9m" Jul 26 15:07:29.830511 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:07:29.830184 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 26 15:07:29.830511 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:07:29.830246 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b6101bd-09a1-444f-ae02-50384ba66923-metrics-certs podName:1b6101bd-09a1-444f-ae02-50384ba66923 nodeName:}" failed. No retries permitted until 2026-07-26 15:09:31.830228471 +0000 UTC m=+252.254075325 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1b6101bd-09a1-444f-ae02-50384ba66923-metrics-certs") pod "network-metrics-daemon-qlw9m" (UID: "1b6101bd-09a1-444f-ae02-50384ba66923") : secret "metrics-daemon-secret" not found Jul 26 15:07:55.963316 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:07:55.963270 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-7fd4475f4c-z475j" podUID="3183398e-9b01-4309-a64d-0c37a9c20303" Jul 26 15:07:55.973378 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:07:55.973351 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-x57rg" podUID="184d66ee-e80d-4766-9642-c9de748e1d7b" Jul 26 15:07:56.066603 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:07:56.066567 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-qlw9m" podUID="1b6101bd-09a1-444f-ae02-50384ba66923" Jul 26 15:07:56.532544 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:07:56.532515 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fd4475f4c-z475j" Jul 26 15:07:56.532693 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:07:56.532515 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-x57rg" Jul 26 15:08:00.829336 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:08:00.829296 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3183398e-9b01-4309-a64d-0c37a9c20303-registry-tls\") pod \"image-registry-7fd4475f4c-z475j\" (UID: \"3183398e-9b01-4309-a64d-0c37a9c20303\") " pod="openshift-image-registry/image-registry-7fd4475f4c-z475j" Jul 26 15:08:00.829336 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:08:00.829340 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/184d66ee-e80d-4766-9642-c9de748e1d7b-metrics-tls\") pod \"dns-default-x57rg\" (UID: \"184d66ee-e80d-4766-9642-c9de748e1d7b\") " pod="openshift-dns/dns-default-x57rg" Jul 26 15:08:00.829806 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:08:00.829431 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 26 15:08:00.829806 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:08:00.829436 2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 26 15:08:00.829806 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:08:00.829454 2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fd4475f4c-z475j: secret "image-registry-tls" not found Jul 26 15:08:00.829806 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:08:00.829489 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/184d66ee-e80d-4766-9642-c9de748e1d7b-metrics-tls podName:184d66ee-e80d-4766-9642-c9de748e1d7b nodeName:}" failed. No retries permitted until 2026-07-26 15:10:02.82947502 +0000 UTC m=+283.253321869 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/184d66ee-e80d-4766-9642-c9de748e1d7b-metrics-tls") pod "dns-default-x57rg" (UID: "184d66ee-e80d-4766-9642-c9de748e1d7b") : secret "dns-default-metrics-tls" not found Jul 26 15:08:00.829806 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:08:00.829501 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3183398e-9b01-4309-a64d-0c37a9c20303-registry-tls podName:3183398e-9b01-4309-a64d-0c37a9c20303 nodeName:}" failed. No retries permitted until 2026-07-26 15:10:02.829495989 +0000 UTC m=+283.253342838 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3183398e-9b01-4309-a64d-0c37a9c20303-registry-tls") pod "image-registry-7fd4475f4c-z475j" (UID: "3183398e-9b01-4309-a64d-0c37a9c20303") : secret "image-registry-tls" not found Jul 26 15:08:11.058670 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:08:11.058638 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlw9m" Jul 26 15:08:49.656768 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:08:49.656735 2555 generic.go:356] "Generic (PLEG): container finished" podID="8f0717ea-6fb7-4af6-ac2a-9344789f2491" containerID="2cc76180103e8b9600648eaa66353de5bd9001176547d9791200c1213f6f99a6" exitCode=255 Jul 26 15:08:49.657146 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:08:49.656807 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65987bb4d4-bs78b" event={"ID":"8f0717ea-6fb7-4af6-ac2a-9344789f2491","Type":"ContainerDied","Data":"2cc76180103e8b9600648eaa66353de5bd9001176547d9791200c1213f6f99a6"} Jul 26 15:08:49.662378 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:08:49.662360 2555 scope.go:117] "RemoveContainer" containerID="2cc76180103e8b9600648eaa66353de5bd9001176547d9791200c1213f6f99a6" Jul 26 15:08:50.663509 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:08:50.663471 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65987bb4d4-bs78b" event={"ID":"8f0717ea-6fb7-4af6-ac2a-9344789f2491","Type":"ContainerStarted","Data":"27640789fc160fb1f743bbd8edd58820b144b540f4763167f423e3b4fbcb97e2"} Jul 26 15:09:31.834135 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:31.834102 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b6101bd-09a1-444f-ae02-50384ba66923-metrics-certs\") pod \"network-metrics-daemon-qlw9m\" (UID: \"1b6101bd-09a1-444f-ae02-50384ba66923\") " pod="openshift-multus/network-metrics-daemon-qlw9m" Jul 26 15:09:31.834568 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:09:31.834235 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 26 15:09:31.834568 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:09:31.834289 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b6101bd-09a1-444f-ae02-50384ba66923-metrics-certs podName:1b6101bd-09a1-444f-ae02-50384ba66923 nodeName:}" failed. No retries permitted until 2026-07-26 15:11:33.834275568 +0000 UTC m=+374.258122420 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1b6101bd-09a1-444f-ae02-50384ba66923-metrics-certs") pod "network-metrics-daemon-qlw9m" (UID: "1b6101bd-09a1-444f-ae02-50384ba66923") : secret "metrics-daemon-secret" not found Jul 26 15:09:36.439730 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.439691 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wmjfd"] Jul 26 15:09:36.441639 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.441616 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-qklsw"] Jul 26 15:09:36.441774 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.441751 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wmjfd" Jul 26 15:09:36.443726 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.443700 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-r7x5d"] Jul 26 15:09:36.443881 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.443857 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 26 15:09:36.443978 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.443918 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-qklsw" Jul 26 15:09:36.443978 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.443922 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 26 15:09:36.443978 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.443955 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 26 15:09:36.443978 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.443867 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 26 15:09:36.444189 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.443962 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-znjrd\"" Jul 26 15:09:36.445887 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.445844 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-r7x5d" Jul 26 15:09:36.446028 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.445920 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 26 15:09:36.446028 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.445997 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 26 15:09:36.446145 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.446044 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 26 15:09:36.446423 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.446402 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-kv4pm\"" Jul 26 15:09:36.446518 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.446411 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 26 15:09:36.448318 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.448295 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 26 15:09:36.448443 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.448316 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 26 15:09:36.448443 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.448348 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-vzfks\"" Jul 26 15:09:36.448443 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.448424 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 26 15:09:36.451113 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.451086 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wmjfd"] Jul 26 15:09:36.455947 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.455903 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-qklsw"] Jul 26 15:09:36.456726 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.456697 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-r7x5d"] Jul 26 15:09:36.534849 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.534800 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-rxf4b"] Jul 26 15:09:36.536665 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.536631 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-qkmnq"] Jul 26 15:09:36.536777 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.536760 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-rxf4b" Jul 26 15:09:36.538226 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.538211 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-qkmnq" Jul 26 15:09:36.539337 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.539321 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 26 15:09:36.539428 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.539392 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 26 15:09:36.539494 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.539449 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 26 15:09:36.539911 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.539894 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 26 15:09:36.539997 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.539914 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 26 15:09:36.539997 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.539925 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-b2hrw\"" Jul 26 15:09:36.540146 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.540132 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 26 15:09:36.541698 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.541682 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-5jg2j\"" Jul 26 15:09:36.545914 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.545897 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 26 15:09:36.547666 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.547618 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-rxf4b"] Jul 26 15:09:36.559408 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.559368 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-qkmnq"] Jul 26 15:09:36.564851 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.564833 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/750ed7a9-694b-4f9b-afb5-6a3b99fc3834-serving-cert\") pod \"service-ca-operator-5d6c75f84f-wmjfd\" (UID: \"750ed7a9-694b-4f9b-afb5-6a3b99fc3834\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wmjfd" Jul 26 15:09:36.564919 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.564859 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/750ed7a9-694b-4f9b-afb5-6a3b99fc3834-config\") pod \"service-ca-operator-5d6c75f84f-wmjfd\" (UID: \"750ed7a9-694b-4f9b-afb5-6a3b99fc3834\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wmjfd" Jul 26 15:09:36.564919 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.564879 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng9rr\" (UniqueName: \"kubernetes.io/projected/1ea98438-fbc7-40a7-89d1-3b0a167119f4-kube-api-access-ng9rr\") pod \"cluster-samples-operator-6748489656-r7x5d\" (UID: \"1ea98438-fbc7-40a7-89d1-3b0a167119f4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-r7x5d" Jul 26 15:09:36.564919 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.564903 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ea98438-fbc7-40a7-89d1-3b0a167119f4-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-r7x5d\" (UID: \"1ea98438-fbc7-40a7-89d1-3b0a167119f4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-r7x5d" Jul 26 15:09:36.565035 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.564978 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e5f83acc-8163-48d2-94de-bf52dd9c0e42-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-qklsw\" (UID: \"e5f83acc-8163-48d2-94de-bf52dd9c0e42\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-qklsw" Jul 26 15:09:36.565035 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.564998 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e5f83acc-8163-48d2-94de-bf52dd9c0e42-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-qklsw\" (UID: \"e5f83acc-8163-48d2-94de-bf52dd9c0e42\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-qklsw" Jul 26 15:09:36.565035 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.565017 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59kjk\" (UniqueName: \"kubernetes.io/projected/750ed7a9-694b-4f9b-afb5-6a3b99fc3834-kube-api-access-59kjk\") pod \"service-ca-operator-5d6c75f84f-wmjfd\" (UID: \"750ed7a9-694b-4f9b-afb5-6a3b99fc3834\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wmjfd" Jul 26 15:09:36.565122 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.565043 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnjt9\" (UniqueName: \"kubernetes.io/projected/e5f83acc-8163-48d2-94de-bf52dd9c0e42-kube-api-access-cnjt9\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-qklsw\" (UID: \"e5f83acc-8163-48d2-94de-bf52dd9c0e42\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-qklsw" Jul 26 15:09:36.656323 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.656301 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:09:36.665998 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.665969 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ea98438-fbc7-40a7-89d1-3b0a167119f4-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-r7x5d\" (UID: \"1ea98438-fbc7-40a7-89d1-3b0a167119f4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-r7x5d" Jul 26 15:09:36.666095 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.666007 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84fece67-b584-4229-800c-e74f02667c8d-service-ca-bundle\") pod \"insights-operator-5885fdb759-rxf4b\" (UID: \"84fece67-b584-4229-800c-e74f02667c8d\") " pod="openshift-insights/insights-operator-5885fdb759-rxf4b" Jul 26 15:09:36.666095 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.666026 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e5f83acc-8163-48d2-94de-bf52dd9c0e42-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-qklsw\" (UID: \"e5f83acc-8163-48d2-94de-bf52dd9c0e42\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-qklsw" Jul 26 15:09:36.666095 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.666042 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e5f83acc-8163-48d2-94de-bf52dd9c0e42-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-qklsw\" (UID: \"e5f83acc-8163-48d2-94de-bf52dd9c0e42\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-qklsw" Jul 26 15:09:36.666095 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.666059 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fbgt\" (UniqueName: \"kubernetes.io/projected/84fece67-b584-4229-800c-e74f02667c8d-kube-api-access-7fbgt\") pod \"insights-operator-5885fdb759-rxf4b\" (UID: \"84fece67-b584-4229-800c-e74f02667c8d\") " pod="openshift-insights/insights-operator-5885fdb759-rxf4b" Jul 26 15:09:36.666322 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.666111 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59kjk\" (UniqueName: \"kubernetes.io/projected/750ed7a9-694b-4f9b-afb5-6a3b99fc3834-kube-api-access-59kjk\") pod \"service-ca-operator-5d6c75f84f-wmjfd\" (UID: \"750ed7a9-694b-4f9b-afb5-6a3b99fc3834\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wmjfd" Jul 26 15:09:36.666322 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:09:36.666120 2555 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 26 15:09:36.666322 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.666144 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnjt9\" (UniqueName: \"kubernetes.io/projected/e5f83acc-8163-48d2-94de-bf52dd9c0e42-kube-api-access-cnjt9\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-qklsw\" (UID: \"e5f83acc-8163-48d2-94de-bf52dd9c0e42\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-qklsw" Jul 26 15:09:36.666322 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:09:36.666197 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ea98438-fbc7-40a7-89d1-3b0a167119f4-samples-operator-tls podName:1ea98438-fbc7-40a7-89d1-3b0a167119f4 nodeName:}" failed. No retries permitted until 2026-07-26 15:09:37.166162855 +0000 UTC m=+257.590009715 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1ea98438-fbc7-40a7-89d1-3b0a167119f4-samples-operator-tls") pod "cluster-samples-operator-6748489656-r7x5d" (UID: "1ea98438-fbc7-40a7-89d1-3b0a167119f4") : secret "samples-operator-tls" not found Jul 26 15:09:36.666322 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.666230 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/750ed7a9-694b-4f9b-afb5-6a3b99fc3834-config\") pod \"service-ca-operator-5d6c75f84f-wmjfd\" (UID: \"750ed7a9-694b-4f9b-afb5-6a3b99fc3834\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wmjfd" Jul 26 15:09:36.666322 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.666274 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/84fece67-b584-4229-800c-e74f02667c8d-snapshots\") pod \"insights-operator-5885fdb759-rxf4b\" (UID: \"84fece67-b584-4229-800c-e74f02667c8d\") " pod="openshift-insights/insights-operator-5885fdb759-rxf4b" Jul 26 15:09:36.666618 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.666327 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84fece67-b584-4229-800c-e74f02667c8d-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-rxf4b\" (UID: \"84fece67-b584-4229-800c-e74f02667c8d\") " pod="openshift-insights/insights-operator-5885fdb759-rxf4b" Jul 26 15:09:36.666618 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.666383 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/84fece67-b584-4229-800c-e74f02667c8d-serving-cert\") pod \"insights-operator-5885fdb759-rxf4b\" (UID: \"84fece67-b584-4229-800c-e74f02667c8d\") " pod="openshift-insights/insights-operator-5885fdb759-rxf4b" Jul 26 15:09:36.666618 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.666415 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/750ed7a9-694b-4f9b-afb5-6a3b99fc3834-serving-cert\") pod \"service-ca-operator-5d6c75f84f-wmjfd\" (UID: \"750ed7a9-694b-4f9b-afb5-6a3b99fc3834\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wmjfd" Jul 26 15:09:36.666618 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.666441 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng9rr\" (UniqueName: \"kubernetes.io/projected/1ea98438-fbc7-40a7-89d1-3b0a167119f4-kube-api-access-ng9rr\") pod \"cluster-samples-operator-6748489656-r7x5d\" (UID: \"1ea98438-fbc7-40a7-89d1-3b0a167119f4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-r7x5d" Jul 26 15:09:36.666618 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.666466 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/84fece67-b584-4229-800c-e74f02667c8d-tmp\") pod \"insights-operator-5885fdb759-rxf4b\" (UID: \"84fece67-b584-4229-800c-e74f02667c8d\") " pod="openshift-insights/insights-operator-5885fdb759-rxf4b" Jul 26 15:09:36.666618 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.666538 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhg6j\" (UniqueName: \"kubernetes.io/projected/aafc3a3d-6f48-4f65-86b8-760837db363d-kube-api-access-vhg6j\") pod \"volume-data-source-validator-549c6d54cf-qkmnq\" (UID: \"aafc3a3d-6f48-4f65-86b8-760837db363d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-qkmnq" Jul 26 15:09:36.666618 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.666591 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e5f83acc-8163-48d2-94de-bf52dd9c0e42-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-qklsw\" (UID: \"e5f83acc-8163-48d2-94de-bf52dd9c0e42\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-qklsw" Jul 26 15:09:36.666852 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.666835 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/750ed7a9-694b-4f9b-afb5-6a3b99fc3834-config\") pod \"service-ca-operator-5d6c75f84f-wmjfd\" (UID: \"750ed7a9-694b-4f9b-afb5-6a3b99fc3834\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wmjfd" Jul 26 15:09:36.668327 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.668308 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e5f83acc-8163-48d2-94de-bf52dd9c0e42-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-qklsw\" (UID: \"e5f83acc-8163-48d2-94de-bf52dd9c0e42\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-qklsw" Jul 26 15:09:36.668446 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.668430 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/750ed7a9-694b-4f9b-afb5-6a3b99fc3834-serving-cert\") pod \"service-ca-operator-5d6c75f84f-wmjfd\" (UID: \"750ed7a9-694b-4f9b-afb5-6a3b99fc3834\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wmjfd" Jul 26 15:09:36.674179 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.674150 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59kjk\" (UniqueName: \"kubernetes.io/projected/750ed7a9-694b-4f9b-afb5-6a3b99fc3834-kube-api-access-59kjk\") pod \"service-ca-operator-5d6c75f84f-wmjfd\" (UID: \"750ed7a9-694b-4f9b-afb5-6a3b99fc3834\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wmjfd" Jul 26 15:09:36.674276 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.674242 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnjt9\" (UniqueName: \"kubernetes.io/projected/e5f83acc-8163-48d2-94de-bf52dd9c0e42-kube-api-access-cnjt9\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-qklsw\" (UID: \"e5f83acc-8163-48d2-94de-bf52dd9c0e42\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-qklsw" Jul 26 15:09:36.674668 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.674649 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng9rr\" (UniqueName: \"kubernetes.io/projected/1ea98438-fbc7-40a7-89d1-3b0a167119f4-kube-api-access-ng9rr\") pod \"cluster-samples-operator-6748489656-r7x5d\" (UID: \"1ea98438-fbc7-40a7-89d1-3b0a167119f4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-r7x5d" Jul 26 15:09:36.742470 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.742420 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:09:36.753274 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.753259 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wmjfd" Jul 26 15:09:36.761598 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.761450 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-qklsw" Jul 26 15:09:36.767511 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.767482 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84fece67-b584-4229-800c-e74f02667c8d-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-rxf4b\" (UID: \"84fece67-b584-4229-800c-e74f02667c8d\") " pod="openshift-insights/insights-operator-5885fdb759-rxf4b" Jul 26 15:09:36.767615 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.767526 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/84fece67-b584-4229-800c-e74f02667c8d-serving-cert\") pod \"insights-operator-5885fdb759-rxf4b\" (UID: \"84fece67-b584-4229-800c-e74f02667c8d\") " pod="openshift-insights/insights-operator-5885fdb759-rxf4b" Jul 26 15:09:36.767615 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.767557 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/84fece67-b584-4229-800c-e74f02667c8d-tmp\") pod \"insights-operator-5885fdb759-rxf4b\" (UID: \"84fece67-b584-4229-800c-e74f02667c8d\") " pod="openshift-insights/insights-operator-5885fdb759-rxf4b" Jul 26 15:09:36.767615 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.767592 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhg6j\" (UniqueName: \"kubernetes.io/projected/aafc3a3d-6f48-4f65-86b8-760837db363d-kube-api-access-vhg6j\") pod \"volume-data-source-validator-549c6d54cf-qkmnq\" (UID: \"aafc3a3d-6f48-4f65-86b8-760837db363d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-qkmnq" Jul 26 15:09:36.767775 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.767633 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84fece67-b584-4229-800c-e74f02667c8d-service-ca-bundle\") pod \"insights-operator-5885fdb759-rxf4b\" (UID: \"84fece67-b584-4229-800c-e74f02667c8d\") " pod="openshift-insights/insights-operator-5885fdb759-rxf4b" Jul 26 15:09:36.767775 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.767660 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fbgt\" (UniqueName: \"kubernetes.io/projected/84fece67-b584-4229-800c-e74f02667c8d-kube-api-access-7fbgt\") pod \"insights-operator-5885fdb759-rxf4b\" (UID: \"84fece67-b584-4229-800c-e74f02667c8d\") " pod="openshift-insights/insights-operator-5885fdb759-rxf4b" Jul 26 15:09:36.767775 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.767717 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/84fece67-b584-4229-800c-e74f02667c8d-snapshots\") pod \"insights-operator-5885fdb759-rxf4b\" (UID: \"84fece67-b584-4229-800c-e74f02667c8d\") " pod="openshift-insights/insights-operator-5885fdb759-rxf4b" Jul 26 15:09:36.768318 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.768296 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/84fece67-b584-4229-800c-e74f02667c8d-tmp\") pod \"insights-operator-5885fdb759-rxf4b\" (UID: \"84fece67-b584-4229-800c-e74f02667c8d\") " pod="openshift-insights/insights-operator-5885fdb759-rxf4b" Jul 26 15:09:36.768424 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.768342 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/84fece67-b584-4229-800c-e74f02667c8d-snapshots\") pod \"insights-operator-5885fdb759-rxf4b\" (UID: \"84fece67-b584-4229-800c-e74f02667c8d\") " pod="openshift-insights/insights-operator-5885fdb759-rxf4b" Jul 26 15:09:36.768659 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.768614 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84fece67-b584-4229-800c-e74f02667c8d-service-ca-bundle\") pod \"insights-operator-5885fdb759-rxf4b\" (UID: \"84fece67-b584-4229-800c-e74f02667c8d\") " pod="openshift-insights/insights-operator-5885fdb759-rxf4b" Jul 26 15:09:36.769028 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.768985 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84fece67-b584-4229-800c-e74f02667c8d-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-rxf4b\" (UID: \"84fece67-b584-4229-800c-e74f02667c8d\") " pod="openshift-insights/insights-operator-5885fdb759-rxf4b" Jul 26 15:09:36.770135 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.770118 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/84fece67-b584-4229-800c-e74f02667c8d-serving-cert\") pod \"insights-operator-5885fdb759-rxf4b\" (UID: \"84fece67-b584-4229-800c-e74f02667c8d\") " pod="openshift-insights/insights-operator-5885fdb759-rxf4b" Jul 26 15:09:36.775676 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.775656 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fbgt\" (UniqueName: \"kubernetes.io/projected/84fece67-b584-4229-800c-e74f02667c8d-kube-api-access-7fbgt\") pod \"insights-operator-5885fdb759-rxf4b\" (UID: \"84fece67-b584-4229-800c-e74f02667c8d\") " pod="openshift-insights/insights-operator-5885fdb759-rxf4b" Jul 26 15:09:36.775771 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.775747 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhg6j\" (UniqueName: \"kubernetes.io/projected/aafc3a3d-6f48-4f65-86b8-760837db363d-kube-api-access-vhg6j\") pod \"volume-data-source-validator-549c6d54cf-qkmnq\" (UID: \"aafc3a3d-6f48-4f65-86b8-760837db363d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-qkmnq" Jul 26 15:09:36.847355 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.847327 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-rxf4b" Jul 26 15:09:36.853251 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.853226 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-qkmnq" Jul 26 15:09:36.879716 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.879608 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wmjfd"] Jul 26 15:09:36.882025 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:09:36.881999 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod750ed7a9_694b_4f9b_afb5_6a3b99fc3834.slice/crio-dd15af13c91e17efd007a72bf09ff2da6dfddb8becc76ff511742d9afb2f28aa WatchSource:0}: Error finding container dd15af13c91e17efd007a72bf09ff2da6dfddb8becc76ff511742d9afb2f28aa: Status 404 returned error can't find the container with id dd15af13c91e17efd007a72bf09ff2da6dfddb8becc76ff511742d9afb2f28aa Jul 26 15:09:36.896327 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.896275 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-qklsw"] Jul 26 15:09:36.905458 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:09:36.905406 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5f83acc_8163_48d2_94de_bf52dd9c0e42.slice/crio-5b73c1baf73a8fdc1dfc182ab7229da26655a5e5dacff3837b70e709b21d0928 WatchSource:0}: Error finding container 5b73c1baf73a8fdc1dfc182ab7229da26655a5e5dacff3837b70e709b21d0928: Status 404 returned error can't find the container with id 5b73c1baf73a8fdc1dfc182ab7229da26655a5e5dacff3837b70e709b21d0928 Jul 26 15:09:36.975605 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.975578 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-rxf4b"] Jul 26 15:09:36.978405 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:09:36.978380 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84fece67_b584_4229_800c_e74f02667c8d.slice/crio-513b71677399e751f0163361f4fadb468d2e6190ecef034579dcfed0ffac012b WatchSource:0}: Error finding container 513b71677399e751f0163361f4fadb468d2e6190ecef034579dcfed0ffac012b: Status 404 returned error can't find the container with id 513b71677399e751f0163361f4fadb468d2e6190ecef034579dcfed0ffac012b Jul 26 15:09:36.992946 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:36.992880 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-qkmnq"] Jul 26 15:09:36.995534 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:09:36.995516 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaafc3a3d_6f48_4f65_86b8_760837db363d.slice/crio-c8483338d2961d71aa0db0601b85caf32ddb452b30cc40c2a53b2eee6ca26c32 WatchSource:0}: Error finding container c8483338d2961d71aa0db0601b85caf32ddb452b30cc40c2a53b2eee6ca26c32: Status 404 returned error can't find the container with id c8483338d2961d71aa0db0601b85caf32ddb452b30cc40c2a53b2eee6ca26c32 Jul 26 15:09:37.172211 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:37.172189 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ea98438-fbc7-40a7-89d1-3b0a167119f4-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-r7x5d\" (UID: \"1ea98438-fbc7-40a7-89d1-3b0a167119f4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-r7x5d" Jul 26 15:09:37.172359 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:09:37.172340 2555 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 26 15:09:37.172409 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:09:37.172400 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ea98438-fbc7-40a7-89d1-3b0a167119f4-samples-operator-tls podName:1ea98438-fbc7-40a7-89d1-3b0a167119f4 nodeName:}" failed. No retries permitted until 2026-07-26 15:09:38.172383639 +0000 UTC m=+258.596230493 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1ea98438-fbc7-40a7-89d1-3b0a167119f4-samples-operator-tls") pod "cluster-samples-operator-6748489656-r7x5d" (UID: "1ea98438-fbc7-40a7-89d1-3b0a167119f4") : secret "samples-operator-tls" not found Jul 26 15:09:37.799950 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:37.799903 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-qklsw" event={"ID":"e5f83acc-8163-48d2-94de-bf52dd9c0e42","Type":"ContainerStarted","Data":"5b73c1baf73a8fdc1dfc182ab7229da26655a5e5dacff3837b70e709b21d0928"} Jul 26 15:09:37.801303 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:37.801272 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-rxf4b" event={"ID":"84fece67-b584-4229-800c-e74f02667c8d","Type":"ContainerStarted","Data":"513b71677399e751f0163361f4fadb468d2e6190ecef034579dcfed0ffac012b"} Jul 26 15:09:37.802896 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:37.802871 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wmjfd" event={"ID":"750ed7a9-694b-4f9b-afb5-6a3b99fc3834","Type":"ContainerStarted","Data":"dd15af13c91e17efd007a72bf09ff2da6dfddb8becc76ff511742d9afb2f28aa"} Jul 26 15:09:37.804276 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:37.804246 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-qkmnq" event={"ID":"aafc3a3d-6f48-4f65-86b8-760837db363d","Type":"ContainerStarted","Data":"c8483338d2961d71aa0db0601b85caf32ddb452b30cc40c2a53b2eee6ca26c32"} Jul 26 15:09:38.180394 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:38.180354 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ea98438-fbc7-40a7-89d1-3b0a167119f4-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-r7x5d\" (UID: \"1ea98438-fbc7-40a7-89d1-3b0a167119f4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-r7x5d" Jul 26 15:09:38.180581 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:09:38.180512 2555 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 26 15:09:38.180644 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:09:38.180582 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ea98438-fbc7-40a7-89d1-3b0a167119f4-samples-operator-tls podName:1ea98438-fbc7-40a7-89d1-3b0a167119f4 nodeName:}" failed. No retries permitted until 2026-07-26 15:09:40.180563182 +0000 UTC m=+260.604410032 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1ea98438-fbc7-40a7-89d1-3b0a167119f4-samples-operator-tls") pod "cluster-samples-operator-6748489656-r7x5d" (UID: "1ea98438-fbc7-40a7-89d1-3b0a167119f4") : secret "samples-operator-tls" not found Jul 26 15:09:38.809632 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:38.809594 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-qkmnq" event={"ID":"aafc3a3d-6f48-4f65-86b8-760837db363d","Type":"ContainerStarted","Data":"ef60025114cddfea8fea0cc9b1e59d64f1060874c5dc883b1f2974ec250dff3d"} Jul 26 15:09:38.823815 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:38.823765 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-qkmnq" podStartSLOduration=1.6262246930000002 podStartE2EDuration="2.823746864s" totalImagesPullingTime="1.197522171s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:09:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:09:38.823527259 +0000 UTC m=+259.247374130" watchObservedRunningTime="2026-07-26 15:09:38.823746864 +0000 UTC m=+259.247593739" Jul 26 15:09:38.896335 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:38.896296 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:09:38.997600 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:38.997564 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:09:40.195540 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:40.195507 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ea98438-fbc7-40a7-89d1-3b0a167119f4-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-r7x5d\" (UID: \"1ea98438-fbc7-40a7-89d1-3b0a167119f4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-r7x5d" Jul 26 15:09:40.195982 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:09:40.195640 2555 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 26 15:09:40.195982 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:09:40.195700 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ea98438-fbc7-40a7-89d1-3b0a167119f4-samples-operator-tls podName:1ea98438-fbc7-40a7-89d1-3b0a167119f4 nodeName:}" failed. No retries permitted until 2026-07-26 15:09:44.19568676 +0000 UTC m=+264.619533610 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1ea98438-fbc7-40a7-89d1-3b0a167119f4-samples-operator-tls") pod "cluster-samples-operator-6748489656-r7x5d" (UID: "1ea98438-fbc7-40a7-89d1-3b0a167119f4") : secret "samples-operator-tls" not found Jul 26 15:09:40.817970 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:40.817922 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-qklsw" event={"ID":"e5f83acc-8163-48d2-94de-bf52dd9c0e42","Type":"ContainerStarted","Data":"79ae2e51530aa1fa35b435214949599560254f12dcc765c64c8df97f32bce972"} Jul 26 15:09:40.819420 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:40.819379 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-rxf4b" event={"ID":"84fece67-b584-4229-800c-e74f02667c8d","Type":"ContainerStarted","Data":"db38f44262c37140954d48efcc5e815a80425878d71e43a40aaf74f1f8f2602e"} Jul 26 15:09:40.820901 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:40.820877 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wmjfd" event={"ID":"750ed7a9-694b-4f9b-afb5-6a3b99fc3834","Type":"ContainerStarted","Data":"cf1f7145b672323c9562091c2464522606232e5c0e6eb4f0337279505ad679eb"} Jul 26 15:09:40.845803 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:40.845757 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-qklsw" podStartSLOduration=1.8908858039999998 podStartE2EDuration="4.84573937s" totalImagesPullingTime="2.954853566s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:09:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:09:40.831769197 +0000 UTC m=+261.255616069" watchObservedRunningTime="2026-07-26 15:09:40.84573937 +0000 UTC m=+261.269586247" Jul 26 15:09:40.845939 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:40.845852 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-rxf4b" podStartSLOduration=1.959708961 podStartE2EDuration="4.845843303s" totalImagesPullingTime="2.886134342s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:09:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:09:40.844754521 +0000 UTC m=+261.268601391" watchObservedRunningTime="2026-07-26 15:09:40.845843303 +0000 UTC m=+261.269690177" Jul 26 15:09:40.857944 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:40.857901 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wmjfd" podStartSLOduration=1.875344819 podStartE2EDuration="4.857888697s" totalImagesPullingTime="2.982543878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:09:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:09:40.857317909 +0000 UTC m=+261.281164779" watchObservedRunningTime="2026-07-26 15:09:40.857888697 +0000 UTC m=+261.281735568" Jul 26 15:09:41.019349 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:41.019316 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hbjsw"] Jul 26 15:09:41.021963 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:41.021944 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hbjsw" Jul 26 15:09:41.024395 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:41.024366 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 26 15:09:41.024544 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:41.024527 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 26 15:09:41.024879 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:41.024864 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-m4q6m\"" Jul 26 15:09:41.032501 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:41.032477 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hbjsw"] Jul 26 15:09:41.055890 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:41.055862 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:09:41.103472 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:41.103417 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-przg8\" (UniqueName: \"kubernetes.io/projected/a65a53c9-99a1-4097-9686-73738e044085-kube-api-access-przg8\") pod \"insights-runtime-extractor-hbjsw\" (UID: \"a65a53c9-99a1-4097-9686-73738e044085\") " pod="openshift-insights/insights-runtime-extractor-hbjsw" Jul 26 15:09:41.103472 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:41.103445 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a65a53c9-99a1-4097-9686-73738e044085-crio-socket\") pod \"insights-runtime-extractor-hbjsw\" (UID: \"a65a53c9-99a1-4097-9686-73738e044085\") " pod="openshift-insights/insights-runtime-extractor-hbjsw" Jul 26 15:09:41.103611 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:41.103482 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a65a53c9-99a1-4097-9686-73738e044085-data-volume\") pod \"insights-runtime-extractor-hbjsw\" (UID: \"a65a53c9-99a1-4097-9686-73738e044085\") " pod="openshift-insights/insights-runtime-extractor-hbjsw" Jul 26 15:09:41.103611 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:41.103541 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a65a53c9-99a1-4097-9686-73738e044085-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hbjsw\" (UID: \"a65a53c9-99a1-4097-9686-73738e044085\") " pod="openshift-insights/insights-runtime-extractor-hbjsw" Jul 26 15:09:41.103611 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:41.103566 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a65a53c9-99a1-4097-9686-73738e044085-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hbjsw\" (UID: \"a65a53c9-99a1-4097-9686-73738e044085\") " pod="openshift-insights/insights-runtime-extractor-hbjsw" Jul 26 15:09:41.132569 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:41.132530 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:09:41.204809 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:41.204776 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-przg8\" (UniqueName: \"kubernetes.io/projected/a65a53c9-99a1-4097-9686-73738e044085-kube-api-access-przg8\") pod \"insights-runtime-extractor-hbjsw\" (UID: \"a65a53c9-99a1-4097-9686-73738e044085\") " pod="openshift-insights/insights-runtime-extractor-hbjsw" Jul 26 15:09:41.205206 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:41.204814 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a65a53c9-99a1-4097-9686-73738e044085-crio-socket\") pod \"insights-runtime-extractor-hbjsw\" (UID: \"a65a53c9-99a1-4097-9686-73738e044085\") " pod="openshift-insights/insights-runtime-extractor-hbjsw" Jul 26 15:09:41.205206 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:41.204865 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a65a53c9-99a1-4097-9686-73738e044085-data-volume\") pod \"insights-runtime-extractor-hbjsw\" (UID: \"a65a53c9-99a1-4097-9686-73738e044085\") " pod="openshift-insights/insights-runtime-extractor-hbjsw" Jul 26 15:09:41.205206 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:41.204922 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a65a53c9-99a1-4097-9686-73738e044085-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hbjsw\" (UID: \"a65a53c9-99a1-4097-9686-73738e044085\") " pod="openshift-insights/insights-runtime-extractor-hbjsw" Jul 26 15:09:41.205206 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:41.204963 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a65a53c9-99a1-4097-9686-73738e044085-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hbjsw\" (UID: \"a65a53c9-99a1-4097-9686-73738e044085\") " pod="openshift-insights/insights-runtime-extractor-hbjsw" Jul 26 15:09:41.205206 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:41.204984 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a65a53c9-99a1-4097-9686-73738e044085-crio-socket\") pod \"insights-runtime-extractor-hbjsw\" (UID: \"a65a53c9-99a1-4097-9686-73738e044085\") " pod="openshift-insights/insights-runtime-extractor-hbjsw" Jul 26 15:09:41.205206 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:09:41.205076 2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 26 15:09:41.205206 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:09:41.205088 2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 26 15:09:41.205206 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:09:41.205127 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a65a53c9-99a1-4097-9686-73738e044085-insights-runtime-extractor-tls podName:a65a53c9-99a1-4097-9686-73738e044085 nodeName:}" failed. No retries permitted until 2026-07-26 15:09:41.705111341 +0000 UTC m=+262.128958194 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a65a53c9-99a1-4097-9686-73738e044085-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hbjsw" (UID: "a65a53c9-99a1-4097-9686-73738e044085") : secret "insights-runtime-extractor-tls" not found Jul 26 15:09:41.205206 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:09:41.205147 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a65a53c9-99a1-4097-9686-73738e044085-kube-rbac-proxy-cm podName:a65a53c9-99a1-4097-9686-73738e044085 nodeName:}" failed. No retries permitted until 2026-07-26 15:09:41.705137069 +0000 UTC m=+262.128983919 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a65a53c9-99a1-4097-9686-73738e044085-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hbjsw" (UID: "a65a53c9-99a1-4097-9686-73738e044085") : configmap "kube-rbac-proxy" not found Jul 26 15:09:41.205544 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:41.205305 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a65a53c9-99a1-4097-9686-73738e044085-data-volume\") pod \"insights-runtime-extractor-hbjsw\" (UID: \"a65a53c9-99a1-4097-9686-73738e044085\") " pod="openshift-insights/insights-runtime-extractor-hbjsw" Jul 26 15:09:41.212085 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:41.212061 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-przg8\" (UniqueName: \"kubernetes.io/projected/a65a53c9-99a1-4097-9686-73738e044085-kube-api-access-przg8\") pod \"insights-runtime-extractor-hbjsw\" (UID: \"a65a53c9-99a1-4097-9686-73738e044085\") " pod="openshift-insights/insights-runtime-extractor-hbjsw" Jul 26 15:09:41.709485 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:41.709454 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a65a53c9-99a1-4097-9686-73738e044085-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hbjsw\" (UID: \"a65a53c9-99a1-4097-9686-73738e044085\") " pod="openshift-insights/insights-runtime-extractor-hbjsw" Jul 26 15:09:41.709628 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:41.709490 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a65a53c9-99a1-4097-9686-73738e044085-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hbjsw\" (UID: \"a65a53c9-99a1-4097-9686-73738e044085\") " pod="openshift-insights/insights-runtime-extractor-hbjsw" Jul 26 15:09:41.709628 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:09:41.709580 2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 26 15:09:41.709628 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:09:41.709600 2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 26 15:09:41.709761 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:09:41.709642 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a65a53c9-99a1-4097-9686-73738e044085-insights-runtime-extractor-tls podName:a65a53c9-99a1-4097-9686-73738e044085 nodeName:}" failed. No retries permitted until 2026-07-26 15:09:42.709627861 +0000 UTC m=+263.133474710 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a65a53c9-99a1-4097-9686-73738e044085-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hbjsw" (UID: "a65a53c9-99a1-4097-9686-73738e044085") : secret "insights-runtime-extractor-tls" not found Jul 26 15:09:41.709761 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:09:41.709654 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a65a53c9-99a1-4097-9686-73738e044085-kube-rbac-proxy-cm podName:a65a53c9-99a1-4097-9686-73738e044085 nodeName:}" failed. No retries permitted until 2026-07-26 15:09:42.709648702 +0000 UTC m=+263.133495551 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a65a53c9-99a1-4097-9686-73738e044085-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hbjsw" (UID: "a65a53c9-99a1-4097-9686-73738e044085") : configmap "kube-rbac-proxy" not found Jul 26 15:09:41.741818 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:41.741787 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:09:41.843703 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:41.843630 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:09:41.893714 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:41.893686 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-6v4n5"] Jul 26 15:09:41.896607 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:41.896588 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-6v4n5" Jul 26 15:09:41.899498 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:41.899479 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 26 15:09:41.899844 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:41.899807 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 26 15:09:41.900103 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:41.900089 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-sk5qf\"" Jul 26 15:09:41.905586 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:41.905568 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-6v4n5"] Jul 26 15:09:41.922417 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:41.922396 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:09:41.980466 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:41.980404 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:09:42.012048 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:42.012022 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtd9n\" (UniqueName: \"kubernetes.io/projected/cc080b47-4ca2-4dfb-aaeb-239c0918e09d-kube-api-access-rtd9n\") pod \"migrator-6865c9967-6v4n5\" (UID: \"cc080b47-4ca2-4dfb-aaeb-239c0918e09d\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-6v4n5" Jul 26 15:09:42.112672 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:42.112642 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtd9n\" (UniqueName: \"kubernetes.io/projected/cc080b47-4ca2-4dfb-aaeb-239c0918e09d-kube-api-access-rtd9n\") pod \"migrator-6865c9967-6v4n5\" (UID: \"cc080b47-4ca2-4dfb-aaeb-239c0918e09d\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-6v4n5" Jul 26 15:09:42.120213 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:42.120185 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtd9n\" (UniqueName: \"kubernetes.io/projected/cc080b47-4ca2-4dfb-aaeb-239c0918e09d-kube-api-access-rtd9n\") pod \"migrator-6865c9967-6v4n5\" (UID: \"cc080b47-4ca2-4dfb-aaeb-239c0918e09d\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-6v4n5" Jul 26 15:09:42.207020 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:42.206995 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-6v4n5" Jul 26 15:09:42.320838 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:42.320811 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-6v4n5"] Jul 26 15:09:42.323443 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:09:42.323416 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc080b47_4ca2_4dfb_aaeb_239c0918e09d.slice/crio-90ffb212eee480be75f9e6a16dae33a5206bdd7a0a6754db782a52d3044f0852 WatchSource:0}: Error finding container 90ffb212eee480be75f9e6a16dae33a5206bdd7a0a6754db782a52d3044f0852: Status 404 returned error can't find the container with id 90ffb212eee480be75f9e6a16dae33a5206bdd7a0a6754db782a52d3044f0852 Jul 26 15:09:42.451442 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:42.451417 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:09:42.552918 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:42.552865 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:09:42.717038 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:42.717008 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a65a53c9-99a1-4097-9686-73738e044085-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hbjsw\" (UID: \"a65a53c9-99a1-4097-9686-73738e044085\") " pod="openshift-insights/insights-runtime-extractor-hbjsw" Jul 26 15:09:42.717038 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:42.717038 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a65a53c9-99a1-4097-9686-73738e044085-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hbjsw\" (UID: \"a65a53c9-99a1-4097-9686-73738e044085\") " pod="openshift-insights/insights-runtime-extractor-hbjsw" Jul 26 15:09:42.717225 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:09:42.717125 2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 26 15:09:42.717225 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:09:42.717186 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a65a53c9-99a1-4097-9686-73738e044085-kube-rbac-proxy-cm podName:a65a53c9-99a1-4097-9686-73738e044085 nodeName:}" failed. No retries permitted until 2026-07-26 15:09:44.717169608 +0000 UTC m=+265.141016458 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a65a53c9-99a1-4097-9686-73738e044085-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hbjsw" (UID: "a65a53c9-99a1-4097-9686-73738e044085") : configmap "kube-rbac-proxy" not found Jul 26 15:09:42.717301 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:09:42.717231 2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 26 15:09:42.717301 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:09:42.717273 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a65a53c9-99a1-4097-9686-73738e044085-insights-runtime-extractor-tls podName:a65a53c9-99a1-4097-9686-73738e044085 nodeName:}" failed. No retries permitted until 2026-07-26 15:09:44.717262794 +0000 UTC m=+265.141109644 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a65a53c9-99a1-4097-9686-73738e044085-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hbjsw" (UID: "a65a53c9-99a1-4097-9686-73738e044085") : secret "insights-runtime-extractor-tls" not found Jul 26 15:09:42.828784 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:42.828696 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-6v4n5" event={"ID":"cc080b47-4ca2-4dfb-aaeb-239c0918e09d","Type":"ContainerStarted","Data":"90ffb212eee480be75f9e6a16dae33a5206bdd7a0a6754db782a52d3044f0852"} Jul 26 15:09:43.206047 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:43.206017 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rfxhs_57b3ab2c-f95e-45d9-9caa-6c1eac37919c/dns-node-resolver/0.log" Jul 26 15:09:43.835179 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:43.835144 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-6v4n5" event={"ID":"cc080b47-4ca2-4dfb-aaeb-239c0918e09d","Type":"ContainerStarted","Data":"2e75551e50a5ea2639a7920f68879af44edb6d969cb02f6b77f24235f19ffed8"} Jul 26 15:09:43.835548 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:43.835215 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-6v4n5" event={"ID":"cc080b47-4ca2-4dfb-aaeb-239c0918e09d","Type":"ContainerStarted","Data":"031d62a754289876d5e6a33e0d9b00b4fd0fdf60fe01d9cf00a99ecfec5e7fbc"} Jul 26 15:09:43.869399 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:43.869355 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-6v4n5" podStartSLOduration=1.887754793 podStartE2EDuration="2.869339515s" totalImagesPullingTime="981.584722ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:09:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:09:43.860436883 +0000 UTC m=+264.284283753" watchObservedRunningTime="2026-07-26 15:09:43.869339515 +0000 UTC m=+264.293186386" Jul 26 15:09:43.953847 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:43.953821 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:09:44.003819 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:44.003795 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:09:44.006337 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:44.006308 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-j6hp5_53c30547-0ebf-4139-80c9-e452ed2b9578/node-ca/0.log" Jul 26 15:09:44.083550 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:44.083526 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:09:44.137567 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:44.137540 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:09:44.229544 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:44.229519 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ea98438-fbc7-40a7-89d1-3b0a167119f4-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-r7x5d\" (UID: \"1ea98438-fbc7-40a7-89d1-3b0a167119f4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-r7x5d" Jul 26 15:09:44.229643 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:09:44.229616 2555 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 26 15:09:44.229688 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:09:44.229662 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ea98438-fbc7-40a7-89d1-3b0a167119f4-samples-operator-tls podName:1ea98438-fbc7-40a7-89d1-3b0a167119f4 nodeName:}" failed. No retries permitted until 2026-07-26 15:09:52.229649087 +0000 UTC m=+272.653495936 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1ea98438-fbc7-40a7-89d1-3b0a167119f4-samples-operator-tls") pod "cluster-samples-operator-6748489656-r7x5d" (UID: "1ea98438-fbc7-40a7-89d1-3b0a167119f4") : secret "samples-operator-tls" not found Jul 26 15:09:44.733442 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:44.733412 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a65a53c9-99a1-4097-9686-73738e044085-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hbjsw\" (UID: \"a65a53c9-99a1-4097-9686-73738e044085\") " pod="openshift-insights/insights-runtime-extractor-hbjsw" Jul 26 15:09:44.733588 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:44.733446 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a65a53c9-99a1-4097-9686-73738e044085-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hbjsw\" (UID: \"a65a53c9-99a1-4097-9686-73738e044085\") " pod="openshift-insights/insights-runtime-extractor-hbjsw" Jul 26 15:09:44.733588 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:09:44.733534 2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 26 15:09:44.733588 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:09:44.733564 2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 26 15:09:44.733705 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:09:44.733601 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a65a53c9-99a1-4097-9686-73738e044085-kube-rbac-proxy-cm podName:a65a53c9-99a1-4097-9686-73738e044085 nodeName:}" failed. No retries permitted until 2026-07-26 15:09:48.733584435 +0000 UTC m=+269.157431304 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a65a53c9-99a1-4097-9686-73738e044085-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hbjsw" (UID: "a65a53c9-99a1-4097-9686-73738e044085") : configmap "kube-rbac-proxy" not found Jul 26 15:09:44.733705 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:09:44.733614 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a65a53c9-99a1-4097-9686-73738e044085-insights-runtime-extractor-tls podName:a65a53c9-99a1-4097-9686-73738e044085 nodeName:}" failed. No retries permitted until 2026-07-26 15:09:48.733608786 +0000 UTC m=+269.157455635 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a65a53c9-99a1-4097-9686-73738e044085-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hbjsw" (UID: "a65a53c9-99a1-4097-9686-73738e044085") : secret "insights-runtime-extractor-tls" not found Jul 26 15:09:44.870823 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:44.870793 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:09:45.006287 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:45.006224 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vmblz_da36cb43-fdd5-41dc-a3d9-e799d58a9ae4/serve-healthcheck-canary/0.log" Jul 26 15:09:47.443710 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:47.443678 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:09:47.552011 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:47.551984 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:09:47.666570 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:47.666544 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:09:47.772675 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:47.772603 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:09:47.873904 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:47.873867 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:09:47.983511 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:47.983484 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:09:48.633067 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:48.633035 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:09:48.763938 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:48.763909 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a65a53c9-99a1-4097-9686-73738e044085-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hbjsw\" (UID: \"a65a53c9-99a1-4097-9686-73738e044085\") " pod="openshift-insights/insights-runtime-extractor-hbjsw" Jul 26 15:09:48.764084 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:48.763957 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a65a53c9-99a1-4097-9686-73738e044085-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hbjsw\" (UID: \"a65a53c9-99a1-4097-9686-73738e044085\") " pod="openshift-insights/insights-runtime-extractor-hbjsw" Jul 26 15:09:48.764084 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:09:48.764035 2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 26 15:09:48.764152 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:09:48.764101 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a65a53c9-99a1-4097-9686-73738e044085-kube-rbac-proxy-cm podName:a65a53c9-99a1-4097-9686-73738e044085 nodeName:}" failed. No retries permitted until 2026-07-26 15:09:56.764082672 +0000 UTC m=+277.187929527 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a65a53c9-99a1-4097-9686-73738e044085-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hbjsw" (UID: "a65a53c9-99a1-4097-9686-73738e044085") : configmap "kube-rbac-proxy" not found Jul 26 15:09:48.764152 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:09:48.764124 2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 26 15:09:48.764217 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:09:48.764175 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a65a53c9-99a1-4097-9686-73738e044085-insights-runtime-extractor-tls podName:a65a53c9-99a1-4097-9686-73738e044085 nodeName:}" failed. No retries permitted until 2026-07-26 15:09:56.764163377 +0000 UTC m=+277.188010227 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a65a53c9-99a1-4097-9686-73738e044085-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hbjsw" (UID: "a65a53c9-99a1-4097-9686-73738e044085") : secret "insights-runtime-extractor-tls" not found Jul 26 15:09:49.242368 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:49.242336 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:09:49.443947 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:49.443898 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:09:49.638270 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:49.638238 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:09:50.037733 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:50.037654 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:09:50.449461 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:50.449432 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:09:50.616880 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:50.616847 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:09:50.670261 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:50.670231 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:09:50.901575 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:50.901548 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:09:51.044832 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:51.044796 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:09:51.341356 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:51.341270 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:09:51.441784 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:51.441752 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:09:51.542231 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:51.542202 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:09:51.642096 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:51.642062 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:09:52.040253 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:52.040187 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:09:52.290244 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:52.290217 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ea98438-fbc7-40a7-89d1-3b0a167119f4-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-r7x5d\" (UID: \"1ea98438-fbc7-40a7-89d1-3b0a167119f4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-r7x5d" Jul 26 15:09:52.292490 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:52.292449 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ea98438-fbc7-40a7-89d1-3b0a167119f4-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-r7x5d\" (UID: \"1ea98438-fbc7-40a7-89d1-3b0a167119f4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-r7x5d" Jul 26 15:09:52.366810 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:52.366777 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-r7x5d" Jul 26 15:09:52.484853 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:52.484823 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-r7x5d"] Jul 26 15:09:52.634235 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:52.634209 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:09:52.863274 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:52.863241 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-r7x5d" event={"ID":"1ea98438-fbc7-40a7-89d1-3b0a167119f4","Type":"ContainerStarted","Data":"a7c6a949b349ff41e35f23b0924c7f27e5f3350d8d72f5addf85f912709e8fe3"} Jul 26 15:09:53.041072 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:53.041000 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:09:53.438640 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:53.438596 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:09:53.636951 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:53.636901 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:09:53.936834 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:53.936783 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:09:54.239868 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:54.239805 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:09:54.340785 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:54.340758 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:09:54.636474 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:54.636443 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:09:54.870694 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:54.870658 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-r7x5d" event={"ID":"1ea98438-fbc7-40a7-89d1-3b0a167119f4","Type":"ContainerStarted","Data":"3756dab4e7faeaa272c9be452b14e95cd6d69821eec81ab760d17f761205361a"} Jul 26 15:09:54.870834 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:54.870716 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-r7x5d" event={"ID":"1ea98438-fbc7-40a7-89d1-3b0a167119f4","Type":"ContainerStarted","Data":"3f433c766de511be645ddfeea89888bb6fd94627a486337635dc4e56dbad9d40"} Jul 26 15:09:54.884266 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:54.884224 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-r7x5d" podStartSLOduration=17.159811928 podStartE2EDuration="18.884211481s" totalImagesPullingTime="1.724399553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:09:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:09:54.883706722 +0000 UTC m=+275.307553592" watchObservedRunningTime="2026-07-26 15:09:54.884211481 +0000 UTC m=+275.308058352" Jul 26 15:09:55.040865 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:55.040794 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:09:55.436808 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:55.436772 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:09:55.639312 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:55.639281 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:09:56.036587 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:56.036560 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:09:56.823596 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:56.823567 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a65a53c9-99a1-4097-9686-73738e044085-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hbjsw\" (UID: \"a65a53c9-99a1-4097-9686-73738e044085\") " pod="openshift-insights/insights-runtime-extractor-hbjsw" Jul 26 15:09:56.823974 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:56.823596 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a65a53c9-99a1-4097-9686-73738e044085-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hbjsw\" (UID: \"a65a53c9-99a1-4097-9686-73738e044085\") " pod="openshift-insights/insights-runtime-extractor-hbjsw" Jul 26 15:09:56.823974 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:09:56.823712 2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 26 15:09:56.823974 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:09:56.823771 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a65a53c9-99a1-4097-9686-73738e044085-insights-runtime-extractor-tls podName:a65a53c9-99a1-4097-9686-73738e044085 nodeName:}" failed. No retries permitted until 2026-07-26 15:10:12.823748768 +0000 UTC m=+293.247595618 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a65a53c9-99a1-4097-9686-73738e044085-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hbjsw" (UID: "a65a53c9-99a1-4097-9686-73738e044085") : secret "insights-runtime-extractor-tls" not found Jul 26 15:09:56.824108 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:56.824092 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a65a53c9-99a1-4097-9686-73738e044085-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hbjsw\" (UID: \"a65a53c9-99a1-4097-9686-73738e044085\") " pod="openshift-insights/insights-runtime-extractor-hbjsw" Jul 26 15:09:56.839975 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:56.839949 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:09:57.033018 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:57.032991 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:09:57.444466 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:57.444426 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:09:57.638617 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:57.638591 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:09:57.739227 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:57.739156 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:09:57.789226 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:57.789202 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:09:58.037749 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:58.037654 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:09:58.241644 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:58.241611 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:09:59.532945 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:09:59.532892 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-7fd4475f4c-z475j" podUID="3183398e-9b01-4309-a64d-0c37a9c20303" Jul 26 15:09:59.533280 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:09:59.533074 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-x57rg" podUID="184d66ee-e80d-4766-9642-c9de748e1d7b" Jul 26 15:09:59.884875 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:59.884843 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-x57rg" Jul 26 15:09:59.885020 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:09:59.884843 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fd4475f4c-z475j" Jul 26 15:10:02.537342 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:02.537300 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:02.644828 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:02.644793 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:02.859020 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:02.858920 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3183398e-9b01-4309-a64d-0c37a9c20303-registry-tls\") pod \"image-registry-7fd4475f4c-z475j\" (UID: \"3183398e-9b01-4309-a64d-0c37a9c20303\") " pod="openshift-image-registry/image-registry-7fd4475f4c-z475j" Jul 26 15:10:02.859020 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:02.858982 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/184d66ee-e80d-4766-9642-c9de748e1d7b-metrics-tls\") pod \"dns-default-x57rg\" (UID: \"184d66ee-e80d-4766-9642-c9de748e1d7b\") " pod="openshift-dns/dns-default-x57rg" Jul 26 15:10:02.861156 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:02.861133 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/184d66ee-e80d-4766-9642-c9de748e1d7b-metrics-tls\") pod \"dns-default-x57rg\" (UID: \"184d66ee-e80d-4766-9642-c9de748e1d7b\") " pod="openshift-dns/dns-default-x57rg" Jul 26 15:10:02.861375 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:02.861355 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3183398e-9b01-4309-a64d-0c37a9c20303-registry-tls\") pod \"image-registry-7fd4475f4c-z475j\" (UID: \"3183398e-9b01-4309-a64d-0c37a9c20303\") " pod="openshift-image-registry/image-registry-7fd4475f4c-z475j" Jul 26 15:10:02.887669 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:02.887653 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8qc59\"" Jul 26 15:10:02.887782 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:02.887654 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6qhrv\"" Jul 26 15:10:02.895341 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:02.895327 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-x57rg" Jul 26 15:10:02.895445 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:02.895429 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fd4475f4c-z475j" Jul 26 15:10:03.027846 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:03.027818 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7fd4475f4c-z475j"] Jul 26 15:10:03.030416 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:10:03.030392 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3183398e_9b01_4309_a64d_0c37a9c20303.slice/crio-189fee21f5c03ef7514faa3369422b597234a89fd6c8e7791cce465a02d3db73 WatchSource:0}: Error finding container 189fee21f5c03ef7514faa3369422b597234a89fd6c8e7791cce465a02d3db73: Status 404 returned error can't find the container with id 189fee21f5c03ef7514faa3369422b597234a89fd6c8e7791cce465a02d3db73 Jul 26 15:10:03.039217 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:03.039178 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-x57rg"] Jul 26 15:10:03.043455 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:10:03.043434 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod184d66ee_e80d_4766_9642_c9de748e1d7b.slice/crio-1a0b6f6d087bc5fb26526eeb93676709ef9ceb766a3222cffadf076cddbda12f WatchSource:0}: Error finding container 1a0b6f6d087bc5fb26526eeb93676709ef9ceb766a3222cffadf076cddbda12f: Status 404 returned error can't find the container with id 1a0b6f6d087bc5fb26526eeb93676709ef9ceb766a3222cffadf076cddbda12f Jul 26 15:10:03.048641 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:03.048617 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:03.227895 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:03.227864 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:03.328320 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:03.328288 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:03.447960 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:03.447893 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:03.528445 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:03.528362 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:03.621967 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:03.621922 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7fd4475f4c-z475j"] Jul 26 15:10:03.757129 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:03.757082 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:03.845718 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:03.845637 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:03.898288 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:03.898255 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7fd4475f4c-z475j" event={"ID":"3183398e-9b01-4309-a64d-0c37a9c20303","Type":"ContainerStarted","Data":"9b843bd9063c0bd15059b3ef11c5705d53b8970ecd2946290a59a60bbab44ccd"} Jul 26 15:10:03.898451 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:03.898312 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7fd4475f4c-z475j" event={"ID":"3183398e-9b01-4309-a64d-0c37a9c20303","Type":"ContainerStarted","Data":"189fee21f5c03ef7514faa3369422b597234a89fd6c8e7791cce465a02d3db73"} Jul 26 15:10:03.898451 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:03.898336 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7fd4475f4c-z475j" Jul 26 15:10:03.899304 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:03.899282 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-x57rg" event={"ID":"184d66ee-e80d-4766-9642-c9de748e1d7b","Type":"ContainerStarted","Data":"1a0b6f6d087bc5fb26526eeb93676709ef9ceb766a3222cffadf076cddbda12f"} Jul 26 15:10:03.917622 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:03.917569 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7fd4475f4c-z475j" podStartSLOduration=283.917550323 podStartE2EDuration="4m43.917550323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:05:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:10:03.916875152 +0000 UTC m=+284.340722022" watchObservedRunningTime="2026-07-26 15:10:03.917550323 +0000 UTC m=+284.341397194" Jul 26 15:10:03.962132 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:03.962101 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:04.062499 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:04.062449 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:04.241429 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:04.241392 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:04.639156 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:04.639131 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:04.739852 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:04.739823 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:04.904600 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:04.904572 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-x57rg" event={"ID":"184d66ee-e80d-4766-9642-c9de748e1d7b","Type":"ContainerStarted","Data":"2c2659332ff9dd67cf631c91274c5773b8cb3670e3c9e2d0faee11ff7a280a51"} Jul 26 15:10:04.904705 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:04.904622 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-x57rg" event={"ID":"184d66ee-e80d-4766-9642-c9de748e1d7b","Type":"ContainerStarted","Data":"a342dd074be631138a097129ff7fffe8cefc4f1bfd9043a60244007b878d47a1"} Jul 26 15:10:04.904772 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:04.904750 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-x57rg" Jul 26 15:10:04.936023 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:04.935986 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-x57rg" podStartSLOduration=251.518304689 podStartE2EDuration="4m12.935973996s" totalImagesPullingTime="1.417669307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:05:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:10:04.935260792 +0000 UTC m=+285.359107673" watchObservedRunningTime="2026-07-26 15:10:04.935973996 +0000 UTC m=+285.359820909" Jul 26 15:10:05.452440 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:05.452413 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:05.552940 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:05.552899 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:11.092987 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:11.092955 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:11.157087 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:11.157054 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:11.239025 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:11.238997 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:11.243667 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:11.243645 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-6brxk"] Jul 26 15:10:11.248203 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:11.248184 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-6brxk" Jul 26 15:10:11.250175 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:11.250156 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 26 15:10:11.250267 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:11.250250 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-p6nmt\"" Jul 26 15:10:11.254591 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:11.254563 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-6brxk"] Jul 26 15:10:11.422633 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:11.422606 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4461086d-e84a-46f8-bbf7-a8c9cba0aabb-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-6brxk\" (UID: \"4461086d-e84a-46f8-bbf7-a8c9cba0aabb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-6brxk" Jul 26 15:10:11.469466 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:11.469445 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:11.523447 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:11.523420 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4461086d-e84a-46f8-bbf7-a8c9cba0aabb-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-6brxk\" (UID: \"4461086d-e84a-46f8-bbf7-a8c9cba0aabb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-6brxk" Jul 26 15:10:11.525882 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:11.525859 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4461086d-e84a-46f8-bbf7-a8c9cba0aabb-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-6brxk\" (UID: \"4461086d-e84a-46f8-bbf7-a8c9cba0aabb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-6brxk" Jul 26 15:10:11.556474 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:11.556452 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-6brxk" Jul 26 15:10:11.674119 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:11.674047 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-6brxk"] Jul 26 15:10:11.677251 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:10:11.677224 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4461086d_e84a_46f8_bbf7_a8c9cba0aabb.slice/crio-ddcdb999e98f86c1eb5e14d55dafe49f4f3196f3bcc8ebfda11b6b97ddecda53 WatchSource:0}: Error finding container ddcdb999e98f86c1eb5e14d55dafe49f4f3196f3bcc8ebfda11b6b97ddecda53: Status 404 returned error can't find the container with id ddcdb999e98f86c1eb5e14d55dafe49f4f3196f3bcc8ebfda11b6b97ddecda53 Jul 26 15:10:11.928618 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:11.928562 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-6brxk" event={"ID":"4461086d-e84a-46f8-bbf7-a8c9cba0aabb","Type":"ContainerStarted","Data":"ddcdb999e98f86c1eb5e14d55dafe49f4f3196f3bcc8ebfda11b6b97ddecda53"} Jul 26 15:10:12.662216 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:12.662195 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5947968457-clxm2"] Jul 26 15:10:12.663830 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:12.663806 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:12.666218 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:12.666202 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5947968457-clxm2" Jul 26 15:10:12.668606 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:12.668581 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 26 15:10:12.668606 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:12.668599 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 26 15:10:12.669422 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:12.669221 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 26 15:10:12.669422 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:12.669247 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 26 15:10:12.669422 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:12.669247 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-r8ctd\"" Jul 26 15:10:12.669422 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:12.669300 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 26 15:10:12.669422 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:12.669330 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 26 15:10:12.669782 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:12.669536 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 26 15:10:12.674561 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:12.674538 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5947968457-clxm2"] Jul 26 15:10:12.720067 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:12.720040 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:12.834190 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:12.834169 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a65a53c9-99a1-4097-9686-73738e044085-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hbjsw\" (UID: \"a65a53c9-99a1-4097-9686-73738e044085\") " pod="openshift-insights/insights-runtime-extractor-hbjsw" Jul 26 15:10:12.834308 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:12.834199 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/94b38fca-3d2b-496c-acde-6e3ac99d76fa-oauth-serving-cert\") pod \"console-5947968457-clxm2\" (UID: \"94b38fca-3d2b-496c-acde-6e3ac99d76fa\") " pod="openshift-console/console-5947968457-clxm2" Jul 26 15:10:12.834308 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:12.834231 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nmtn\" (UniqueName: \"kubernetes.io/projected/94b38fca-3d2b-496c-acde-6e3ac99d76fa-kube-api-access-5nmtn\") pod \"console-5947968457-clxm2\" (UID: \"94b38fca-3d2b-496c-acde-6e3ac99d76fa\") " pod="openshift-console/console-5947968457-clxm2" Jul 26 15:10:12.834308 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:12.834291 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/94b38fca-3d2b-496c-acde-6e3ac99d76fa-service-ca\") pod \"console-5947968457-clxm2\" (UID: \"94b38fca-3d2b-496c-acde-6e3ac99d76fa\") " pod="openshift-console/console-5947968457-clxm2" Jul 26 15:10:12.834417 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:12.834326 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/94b38fca-3d2b-496c-acde-6e3ac99d76fa-console-oauth-config\") pod \"console-5947968457-clxm2\" (UID: \"94b38fca-3d2b-496c-acde-6e3ac99d76fa\") " pod="openshift-console/console-5947968457-clxm2" Jul 26 15:10:12.834417 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:12.834350 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/94b38fca-3d2b-496c-acde-6e3ac99d76fa-console-serving-cert\") pod \"console-5947968457-clxm2\" (UID: \"94b38fca-3d2b-496c-acde-6e3ac99d76fa\") " pod="openshift-console/console-5947968457-clxm2" Jul 26 15:10:12.834486 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:12.834408 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/94b38fca-3d2b-496c-acde-6e3ac99d76fa-console-config\") pod \"console-5947968457-clxm2\" (UID: \"94b38fca-3d2b-496c-acde-6e3ac99d76fa\") " pod="openshift-console/console-5947968457-clxm2" Jul 26 15:10:12.836444 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:12.836420 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a65a53c9-99a1-4097-9686-73738e044085-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hbjsw\" (UID: \"a65a53c9-99a1-4097-9686-73738e044085\") " pod="openshift-insights/insights-runtime-extractor-hbjsw" Jul 26 15:10:12.934363 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:12.934309 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-6brxk" event={"ID":"4461086d-e84a-46f8-bbf7-a8c9cba0aabb","Type":"ContainerStarted","Data":"34534c386079abdacca98db3335f575c4cdef5f6157b8cc41f389730de7f38e3"} Jul 26 15:10:12.934514 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:12.934498 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-6brxk" Jul 26 15:10:12.934642 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:12.934627 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/94b38fca-3d2b-496c-acde-6e3ac99d76fa-service-ca\") pod \"console-5947968457-clxm2\" (UID: \"94b38fca-3d2b-496c-acde-6e3ac99d76fa\") " pod="openshift-console/console-5947968457-clxm2" Jul 26 15:10:12.934683 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:12.934653 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/94b38fca-3d2b-496c-acde-6e3ac99d76fa-console-oauth-config\") pod \"console-5947968457-clxm2\" (UID: \"94b38fca-3d2b-496c-acde-6e3ac99d76fa\") " pod="openshift-console/console-5947968457-clxm2" Jul 26 15:10:12.934683 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:12.934672 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/94b38fca-3d2b-496c-acde-6e3ac99d76fa-console-serving-cert\") pod \"console-5947968457-clxm2\" (UID: \"94b38fca-3d2b-496c-acde-6e3ac99d76fa\") " pod="openshift-console/console-5947968457-clxm2" Jul 26 15:10:12.934747 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:12.934696 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/94b38fca-3d2b-496c-acde-6e3ac99d76fa-console-config\") pod \"console-5947968457-clxm2\" (UID: \"94b38fca-3d2b-496c-acde-6e3ac99d76fa\") " pod="openshift-console/console-5947968457-clxm2" Jul 26 15:10:12.934747 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:12.934716 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/94b38fca-3d2b-496c-acde-6e3ac99d76fa-oauth-serving-cert\") pod \"console-5947968457-clxm2\" (UID: \"94b38fca-3d2b-496c-acde-6e3ac99d76fa\") " pod="openshift-console/console-5947968457-clxm2" Jul 26 15:10:12.934867 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:12.934850 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nmtn\" (UniqueName: \"kubernetes.io/projected/94b38fca-3d2b-496c-acde-6e3ac99d76fa-kube-api-access-5nmtn\") pod \"console-5947968457-clxm2\" (UID: \"94b38fca-3d2b-496c-acde-6e3ac99d76fa\") " pod="openshift-console/console-5947968457-clxm2" Jul 26 15:10:12.935469 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:12.935447 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/94b38fca-3d2b-496c-acde-6e3ac99d76fa-console-config\") pod \"console-5947968457-clxm2\" (UID: \"94b38fca-3d2b-496c-acde-6e3ac99d76fa\") " pod="openshift-console/console-5947968457-clxm2" Jul 26 15:10:12.935567 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:12.935447 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/94b38fca-3d2b-496c-acde-6e3ac99d76fa-oauth-serving-cert\") pod \"console-5947968457-clxm2\" (UID: \"94b38fca-3d2b-496c-acde-6e3ac99d76fa\") " pod="openshift-console/console-5947968457-clxm2" Jul 26 15:10:12.935567 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:12.935513 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/94b38fca-3d2b-496c-acde-6e3ac99d76fa-service-ca\") pod \"console-5947968457-clxm2\" (UID: \"94b38fca-3d2b-496c-acde-6e3ac99d76fa\") " pod="openshift-console/console-5947968457-clxm2" Jul 26 15:10:12.937002 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:12.936975 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/94b38fca-3d2b-496c-acde-6e3ac99d76fa-console-serving-cert\") pod \"console-5947968457-clxm2\" (UID: \"94b38fca-3d2b-496c-acde-6e3ac99d76fa\") " pod="openshift-console/console-5947968457-clxm2" Jul 26 15:10:12.937280 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:12.937262 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/94b38fca-3d2b-496c-acde-6e3ac99d76fa-console-oauth-config\") pod \"console-5947968457-clxm2\" (UID: \"94b38fca-3d2b-496c-acde-6e3ac99d76fa\") " pod="openshift-console/console-5947968457-clxm2" Jul 26 15:10:12.939589 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:12.939571 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-6brxk" Jul 26 15:10:12.943001 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:12.942982 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nmtn\" (UniqueName: \"kubernetes.io/projected/94b38fca-3d2b-496c-acde-6e3ac99d76fa-kube-api-access-5nmtn\") pod \"console-5947968457-clxm2\" (UID: \"94b38fca-3d2b-496c-acde-6e3ac99d76fa\") " pod="openshift-console/console-5947968457-clxm2" Jul 26 15:10:12.949797 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:12.949753 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-6brxk" podStartSLOduration=0.997843742 podStartE2EDuration="1.949740717s" totalImagesPullingTime="951.896975ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:10:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:10:12.948781187 +0000 UTC m=+293.372628057" watchObservedRunningTime="2026-07-26 15:10:12.949740717 +0000 UTC m=+293.373587587" Jul 26 15:10:12.978613 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:12.978589 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5947968457-clxm2" Jul 26 15:10:13.101147 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:13.101117 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5947968457-clxm2"] Jul 26 15:10:13.103679 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:10:13.103649 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94b38fca_3d2b_496c_acde_6e3ac99d76fa.slice/crio-8e2219e05d056a1ef5b00462f567dfd6bbc09aad1a3ec12abe1f1624f4424d48 WatchSource:0}: Error finding container 8e2219e05d056a1ef5b00462f567dfd6bbc09aad1a3ec12abe1f1624f4424d48: Status 404 returned error can't find the container with id 8e2219e05d056a1ef5b00462f567dfd6bbc09aad1a3ec12abe1f1624f4424d48 Jul 26 15:10:13.132000 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:13.131967 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hbjsw" Jul 26 15:10:13.251850 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:13.251820 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hbjsw"] Jul 26 15:10:13.254533 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:10:13.254509 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda65a53c9_99a1_4097_9686_73738e044085.slice/crio-9c3d96c8af17defe0b8ff7b9fc0c9ff8b973213fb9a1803fc19c3a3a01141253 WatchSource:0}: Error finding container 9c3d96c8af17defe0b8ff7b9fc0c9ff8b973213fb9a1803fc19c3a3a01141253: Status 404 returned error can't find the container with id 9c3d96c8af17defe0b8ff7b9fc0c9ff8b973213fb9a1803fc19c3a3a01141253 Jul 26 15:10:13.752259 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:13.752224 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:13.940282 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:13.940247 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hbjsw" event={"ID":"a65a53c9-99a1-4097-9686-73738e044085","Type":"ContainerStarted","Data":"9c3d96c8af17defe0b8ff7b9fc0c9ff8b973213fb9a1803fc19c3a3a01141253"} Jul 26 15:10:13.941942 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:13.941901 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5947968457-clxm2" event={"ID":"94b38fca-3d2b-496c-acde-6e3ac99d76fa","Type":"ContainerStarted","Data":"8e2219e05d056a1ef5b00462f567dfd6bbc09aad1a3ec12abe1f1624f4424d48"} Jul 26 15:10:14.060041 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:10:14.059957 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-qlw9m" podUID="1b6101bd-09a1-444f-ae02-50384ba66923" Jul 26 15:10:14.910980 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:14.910851 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-x57rg" Jul 26 15:10:15.067912 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:15.067875 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:17.692262 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:17.692225 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:17.794710 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:17.794674 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:18.250683 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:18.250651 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:18.352020 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:18.351986 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:18.953123 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:18.953072 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:19.218733 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:19.218665 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:19.269700 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:19.269671 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:19.957435 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:19.957407 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9f857_1fbafda6-6242-400e-b2d4-24584608e5f5/ovn-acl-logging/0.log" Jul 26 15:10:19.960880 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:19.960855 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9f857_1fbafda6-6242-400e-b2d4-24584608e5f5/ovn-acl-logging/0.log" Jul 26 15:10:19.962611 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:19.962593 2555 kubelet.go:1648] "Image garbage collection succeeded" Jul 26 15:10:19.969759 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:19.969728 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hbjsw" event={"ID":"a65a53c9-99a1-4097-9686-73738e044085","Type":"ContainerStarted","Data":"abcac999a8e87ee65f5b8ace5ce83b11ebac4df534672e1d9289f8d36aa191c7"} Jul 26 15:10:19.971457 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:19.971433 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5947968457-clxm2" event={"ID":"94b38fca-3d2b-496c-acde-6e3ac99d76fa","Type":"ContainerStarted","Data":"807f3bdc59eb71b64fafb56ea89b9e1ff873c319c7ae0eedd31015c37103c542"} Jul 26 15:10:19.989329 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:19.989289 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5947968457-clxm2" podStartSLOduration=2.038860759 podStartE2EDuration="7.989276148s" totalImagesPullingTime="5.950415389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:10:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:10:19.987569295 +0000 UTC m=+300.411416166" watchObservedRunningTime="2026-07-26 15:10:19.989276148 +0000 UTC m=+300.413123019" Jul 26 15:10:20.042792 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:20.042657 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:20.147351 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:20.147312 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:20.243211 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:20.243132 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:20.428479 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:20.428436 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:20.641148 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:20.641111 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:20.665165 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:20.665134 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-54pc5"] Jul 26 15:10:20.671592 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:20.671571 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-54pc5" Jul 26 15:10:20.673815 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:20.673785 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 26 15:10:20.673815 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:20.673785 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 26 15:10:20.674018 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:20.673790 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 26 15:10:20.674358 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:20.674337 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 26 15:10:20.674447 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:20.674403 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-n6j8p\"" Jul 26 15:10:20.674503 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:20.674471 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 26 15:10:20.678982 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:20.678959 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-54pc5"] Jul 26 15:10:20.682593 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:20.682573 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-ljbtm"] Jul 26 15:10:20.685122 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:20.685094 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/43ab9a7f-db10-4445-b2b6-b6793c88f75d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-54pc5\" (UID: \"43ab9a7f-db10-4445-b2b6-b6793c88f75d\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-54pc5" Jul 26 15:10:20.685227 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:20.685131 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs6ql\" (UniqueName: \"kubernetes.io/projected/43ab9a7f-db10-4445-b2b6-b6793c88f75d-kube-api-access-bs6ql\") pod \"openshift-state-metrics-b8f49f5c6-54pc5\" (UID: \"43ab9a7f-db10-4445-b2b6-b6793c88f75d\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-54pc5" Jul 26 15:10:20.685227 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:20.685203 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43ab9a7f-db10-4445-b2b6-b6793c88f75d-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-54pc5\" (UID: \"43ab9a7f-db10-4445-b2b6-b6793c88f75d\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-54pc5" Jul 26 15:10:20.685319 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:20.685251 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/43ab9a7f-db10-4445-b2b6-b6793c88f75d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-54pc5\" (UID: \"43ab9a7f-db10-4445-b2b6-b6793c88f75d\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-54pc5" Jul 26 15:10:20.686156 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:20.686139 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ljbtm" Jul 26 15:10:20.688096 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:20.688077 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 26 15:10:20.688199 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:20.688105 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gn4ps\"" Jul 26 15:10:20.688199 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:20.688155 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 26 15:10:20.688300 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:20.688232 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 26 15:10:20.734702 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:20.734679 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:20.786091 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:20.786064 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs6ql\" (UniqueName: \"kubernetes.io/projected/43ab9a7f-db10-4445-b2b6-b6793c88f75d-kube-api-access-bs6ql\") pod \"openshift-state-metrics-b8f49f5c6-54pc5\" (UID: \"43ab9a7f-db10-4445-b2b6-b6793c88f75d\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-54pc5" Jul 26 15:10:20.786226 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:20.786105 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c6ef366b-7f71-4f17-8383-7c1c87a3a1f1-root\") pod \"node-exporter-ljbtm\" (UID: \"c6ef366b-7f71-4f17-8383-7c1c87a3a1f1\") " pod="openshift-monitoring/node-exporter-ljbtm" Jul 26 15:10:20.786226 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:20.786131 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c6ef366b-7f71-4f17-8383-7c1c87a3a1f1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ljbtm\" (UID: \"c6ef366b-7f71-4f17-8383-7c1c87a3a1f1\") " pod="openshift-monitoring/node-exporter-ljbtm" Jul 26 15:10:20.786226 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:20.786194 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c6ef366b-7f71-4f17-8383-7c1c87a3a1f1-node-exporter-wtmp\") pod \"node-exporter-ljbtm\" (UID: \"c6ef366b-7f71-4f17-8383-7c1c87a3a1f1\") " pod="openshift-monitoring/node-exporter-ljbtm" Jul 26 15:10:20.786362 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:20.786272 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43ab9a7f-db10-4445-b2b6-b6793c88f75d-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-54pc5\" (UID: \"43ab9a7f-db10-4445-b2b6-b6793c88f75d\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-54pc5" Jul 26 15:10:20.786362 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:20.786297 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz7kk\" (UniqueName: \"kubernetes.io/projected/c6ef366b-7f71-4f17-8383-7c1c87a3a1f1-kube-api-access-dz7kk\") pod \"node-exporter-ljbtm\" (UID: \"c6ef366b-7f71-4f17-8383-7c1c87a3a1f1\") " pod="openshift-monitoring/node-exporter-ljbtm" Jul 26 15:10:20.786362 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:20.786334 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/43ab9a7f-db10-4445-b2b6-b6793c88f75d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-54pc5\" (UID: \"43ab9a7f-db10-4445-b2b6-b6793c88f75d\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-54pc5" Jul 26 15:10:20.786506 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:20.786383 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6ef366b-7f71-4f17-8383-7c1c87a3a1f1-metrics-client-ca\") pod \"node-exporter-ljbtm\" (UID: \"c6ef366b-7f71-4f17-8383-7c1c87a3a1f1\") " pod="openshift-monitoring/node-exporter-ljbtm" Jul 26 15:10:20.786506 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:20.786403 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c6ef366b-7f71-4f17-8383-7c1c87a3a1f1-node-exporter-tls\") pod \"node-exporter-ljbtm\" (UID: \"c6ef366b-7f71-4f17-8383-7c1c87a3a1f1\") " pod="openshift-monitoring/node-exporter-ljbtm" Jul 26 15:10:20.786506 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:20.786421 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c6ef366b-7f71-4f17-8383-7c1c87a3a1f1-node-exporter-textfile\") pod \"node-exporter-ljbtm\" (UID: \"c6ef366b-7f71-4f17-8383-7c1c87a3a1f1\") " pod="openshift-monitoring/node-exporter-ljbtm" Jul 26 15:10:20.786506 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:20.786453 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c6ef366b-7f71-4f17-8383-7c1c87a3a1f1-node-exporter-accelerators-collector-config\") pod \"node-exporter-ljbtm\" (UID: \"c6ef366b-7f71-4f17-8383-7c1c87a3a1f1\") " pod="openshift-monitoring/node-exporter-ljbtm" Jul 26 15:10:20.786506 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:10:20.786476 2555 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 26 15:10:20.786736 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:20.786510 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c6ef366b-7f71-4f17-8383-7c1c87a3a1f1-sys\") pod \"node-exporter-ljbtm\" (UID: \"c6ef366b-7f71-4f17-8383-7c1c87a3a1f1\") " pod="openshift-monitoring/node-exporter-ljbtm" Jul 26 15:10:20.786736 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:10:20.786539 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43ab9a7f-db10-4445-b2b6-b6793c88f75d-openshift-state-metrics-tls podName:43ab9a7f-db10-4445-b2b6-b6793c88f75d nodeName:}" failed. No retries permitted until 2026-07-26 15:10:21.28651946 +0000 UTC m=+301.710366310 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/43ab9a7f-db10-4445-b2b6-b6793c88f75d-openshift-state-metrics-tls") pod "openshift-state-metrics-b8f49f5c6-54pc5" (UID: "43ab9a7f-db10-4445-b2b6-b6793c88f75d") : secret "openshift-state-metrics-tls" not found Jul 26 15:10:20.786736 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:20.786638 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/43ab9a7f-db10-4445-b2b6-b6793c88f75d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-54pc5\" (UID: \"43ab9a7f-db10-4445-b2b6-b6793c88f75d\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-54pc5" Jul 26 15:10:20.786943 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:20.786912 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43ab9a7f-db10-4445-b2b6-b6793c88f75d-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-54pc5\" (UID: \"43ab9a7f-db10-4445-b2b6-b6793c88f75d\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-54pc5" Jul 26 15:10:20.789136 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:20.789112 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/43ab9a7f-db10-4445-b2b6-b6793c88f75d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-54pc5\" (UID: \"43ab9a7f-db10-4445-b2b6-b6793c88f75d\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-54pc5" Jul 26 15:10:20.797184 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:20.797163 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs6ql\" (UniqueName: \"kubernetes.io/projected/43ab9a7f-db10-4445-b2b6-b6793c88f75d-kube-api-access-bs6ql\") pod \"openshift-state-metrics-b8f49f5c6-54pc5\" (UID: \"43ab9a7f-db10-4445-b2b6-b6793c88f75d\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-54pc5" Jul 26 15:10:20.806259 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:20.806232 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:20.887402 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:20.887378 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6ef366b-7f71-4f17-8383-7c1c87a3a1f1-metrics-client-ca\") pod \"node-exporter-ljbtm\" (UID: \"c6ef366b-7f71-4f17-8383-7c1c87a3a1f1\") " pod="openshift-monitoring/node-exporter-ljbtm" Jul 26 15:10:20.887529 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:20.887413 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c6ef366b-7f71-4f17-8383-7c1c87a3a1f1-node-exporter-tls\") pod \"node-exporter-ljbtm\" (UID: \"c6ef366b-7f71-4f17-8383-7c1c87a3a1f1\") " pod="openshift-monitoring/node-exporter-ljbtm" Jul 26 15:10:20.887529 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:20.887432 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c6ef366b-7f71-4f17-8383-7c1c87a3a1f1-node-exporter-textfile\") pod \"node-exporter-ljbtm\" (UID: \"c6ef366b-7f71-4f17-8383-7c1c87a3a1f1\") " pod="openshift-monitoring/node-exporter-ljbtm" Jul 26 15:10:20.887529 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:20.887479 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c6ef366b-7f71-4f17-8383-7c1c87a3a1f1-node-exporter-accelerators-collector-config\") pod \"node-exporter-ljbtm\" (UID: \"c6ef366b-7f71-4f17-8383-7c1c87a3a1f1\") " pod="openshift-monitoring/node-exporter-ljbtm" Jul 26 15:10:20.887732 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:20.887531 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c6ef366b-7f71-4f17-8383-7c1c87a3a1f1-sys\") pod \"node-exporter-ljbtm\" (UID: \"c6ef366b-7f71-4f17-8383-7c1c87a3a1f1\") " pod="openshift-monitoring/node-exporter-ljbtm" Jul 26 15:10:20.887732 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:10:20.887544 2555 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 26 15:10:20.887732 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:10:20.887604 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6ef366b-7f71-4f17-8383-7c1c87a3a1f1-node-exporter-tls podName:c6ef366b-7f71-4f17-8383-7c1c87a3a1f1 nodeName:}" failed. No retries permitted until 2026-07-26 15:10:21.38758656 +0000 UTC m=+301.811433431 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/c6ef366b-7f71-4f17-8383-7c1c87a3a1f1-node-exporter-tls") pod "node-exporter-ljbtm" (UID: "c6ef366b-7f71-4f17-8383-7c1c87a3a1f1") : secret "node-exporter-tls" not found Jul 26 15:10:20.887732 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:20.887623 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c6ef366b-7f71-4f17-8383-7c1c87a3a1f1-sys\") pod \"node-exporter-ljbtm\" (UID: \"c6ef366b-7f71-4f17-8383-7c1c87a3a1f1\") " pod="openshift-monitoring/node-exporter-ljbtm" Jul 26 15:10:20.887732 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:20.887625 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c6ef366b-7f71-4f17-8383-7c1c87a3a1f1-root\") pod \"node-exporter-ljbtm\" (UID: \"c6ef366b-7f71-4f17-8383-7c1c87a3a1f1\") " pod="openshift-monitoring/node-exporter-ljbtm" Jul 26 15:10:20.887732 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:20.887657 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c6ef366b-7f71-4f17-8383-7c1c87a3a1f1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ljbtm\" (UID: \"c6ef366b-7f71-4f17-8383-7c1c87a3a1f1\") " pod="openshift-monitoring/node-exporter-ljbtm" Jul 26 15:10:20.887732 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:20.887691 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c6ef366b-7f71-4f17-8383-7c1c87a3a1f1-node-exporter-textfile\") pod \"node-exporter-ljbtm\" (UID: \"c6ef366b-7f71-4f17-8383-7c1c87a3a1f1\") " pod="openshift-monitoring/node-exporter-ljbtm" Jul 26 15:10:20.888091 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:20.887739 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c6ef366b-7f71-4f17-8383-7c1c87a3a1f1-root\") pod \"node-exporter-ljbtm\" (UID: \"c6ef366b-7f71-4f17-8383-7c1c87a3a1f1\") " pod="openshift-monitoring/node-exporter-ljbtm" Jul 26 15:10:20.888091 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:20.887745 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c6ef366b-7f71-4f17-8383-7c1c87a3a1f1-node-exporter-wtmp\") pod \"node-exporter-ljbtm\" (UID: \"c6ef366b-7f71-4f17-8383-7c1c87a3a1f1\") " pod="openshift-monitoring/node-exporter-ljbtm" Jul 26 15:10:20.888091 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:20.887797 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz7kk\" (UniqueName: \"kubernetes.io/projected/c6ef366b-7f71-4f17-8383-7c1c87a3a1f1-kube-api-access-dz7kk\") pod \"node-exporter-ljbtm\" (UID: \"c6ef366b-7f71-4f17-8383-7c1c87a3a1f1\") " pod="openshift-monitoring/node-exporter-ljbtm" Jul 26 15:10:20.888091 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:20.887841 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c6ef366b-7f71-4f17-8383-7c1c87a3a1f1-node-exporter-wtmp\") pod \"node-exporter-ljbtm\" (UID: \"c6ef366b-7f71-4f17-8383-7c1c87a3a1f1\") " pod="openshift-monitoring/node-exporter-ljbtm" Jul 26 15:10:20.888284 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:20.888138 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c6ef366b-7f71-4f17-8383-7c1c87a3a1f1-node-exporter-accelerators-collector-config\") pod \"node-exporter-ljbtm\" (UID: \"c6ef366b-7f71-4f17-8383-7c1c87a3a1f1\") " pod="openshift-monitoring/node-exporter-ljbtm" Jul 26 15:10:20.888437 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:20.888418 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6ef366b-7f71-4f17-8383-7c1c87a3a1f1-metrics-client-ca\") pod \"node-exporter-ljbtm\" (UID: \"c6ef366b-7f71-4f17-8383-7c1c87a3a1f1\") " pod="openshift-monitoring/node-exporter-ljbtm" Jul 26 15:10:20.890007 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:20.889985 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c6ef366b-7f71-4f17-8383-7c1c87a3a1f1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ljbtm\" (UID: \"c6ef366b-7f71-4f17-8383-7c1c87a3a1f1\") " pod="openshift-monitoring/node-exporter-ljbtm" Jul 26 15:10:20.895739 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:20.895694 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz7kk\" (UniqueName: \"kubernetes.io/projected/c6ef366b-7f71-4f17-8383-7c1c87a3a1f1-kube-api-access-dz7kk\") pod \"node-exporter-ljbtm\" (UID: \"c6ef366b-7f71-4f17-8383-7c1c87a3a1f1\") " pod="openshift-monitoring/node-exporter-ljbtm" Jul 26 15:10:20.906138 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:20.906098 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:21.008074 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:21.007793 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:21.106668 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:21.106625 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:21.226056 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:21.225981 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:21.291419 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:21.291382 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/43ab9a7f-db10-4445-b2b6-b6793c88f75d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-54pc5\" (UID: \"43ab9a7f-db10-4445-b2b6-b6793c88f75d\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-54pc5" Jul 26 15:10:21.293876 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:21.293844 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/43ab9a7f-db10-4445-b2b6-b6793c88f75d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-54pc5\" (UID: \"43ab9a7f-db10-4445-b2b6-b6793c88f75d\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-54pc5" Jul 26 15:10:21.392779 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:21.392743 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c6ef366b-7f71-4f17-8383-7c1c87a3a1f1-node-exporter-tls\") pod \"node-exporter-ljbtm\" (UID: \"c6ef366b-7f71-4f17-8383-7c1c87a3a1f1\") " pod="openshift-monitoring/node-exporter-ljbtm" Jul 26 15:10:21.395306 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:21.395273 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c6ef366b-7f71-4f17-8383-7c1c87a3a1f1-node-exporter-tls\") pod \"node-exporter-ljbtm\" (UID: \"c6ef366b-7f71-4f17-8383-7c1c87a3a1f1\") " pod="openshift-monitoring/node-exporter-ljbtm" Jul 26 15:10:21.398837 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:21.398808 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:21.583200 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:21.583111 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-54pc5" Jul 26 15:10:21.598724 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:21.598699 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ljbtm" Jul 26 15:10:21.661847 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:21.661814 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:21.737712 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:21.737676 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:22.214915 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:22.214884 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:22.452495 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:10:22.452398 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6ef366b_7f71_4f17_8383_7c1c87a3a1f1.slice/crio-dec4828d91f30ef7ee4338455737e11c2c3b729e5b52dfd49004e9082e55167a WatchSource:0}: Error finding container dec4828d91f30ef7ee4338455737e11c2c3b729e5b52dfd49004e9082e55167a: Status 404 returned error can't find the container with id dec4828d91f30ef7ee4338455737e11c2c3b729e5b52dfd49004e9082e55167a Jul 26 15:10:22.454278 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:22.454256 2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 26 15:10:22.571032 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:22.570986 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-54pc5"] Jul 26 15:10:22.573227 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:10:22.573198 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43ab9a7f_db10_4445_b2b6_b6793c88f75d.slice/crio-d73565d201bef24919a07bd6860b51b4abf3c1098c75c0ae2b4de752ec420efe WatchSource:0}: Error finding container d73565d201bef24919a07bd6860b51b4abf3c1098c75c0ae2b4de752ec420efe: Status 404 returned error can't find the container with id d73565d201bef24919a07bd6860b51b4abf3c1098c75c0ae2b4de752ec420efe Jul 26 15:10:22.979714 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:22.979678 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5947968457-clxm2" Jul 26 15:10:22.979886 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:22.979857 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5947968457-clxm2" Jul 26 15:10:22.984731 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:22.984701 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ljbtm" event={"ID":"c6ef366b-7f71-4f17-8383-7c1c87a3a1f1","Type":"ContainerStarted","Data":"dec4828d91f30ef7ee4338455737e11c2c3b729e5b52dfd49004e9082e55167a"} Jul 26 15:10:22.985480 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:22.985458 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5947968457-clxm2" Jul 26 15:10:22.987117 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:22.987089 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hbjsw" event={"ID":"a65a53c9-99a1-4097-9686-73738e044085","Type":"ContainerStarted","Data":"362f7668891d50f3323b1b2d73471c059ed4bbe2b9061f4b3050f71399c3ce45"} Jul 26 15:10:22.988976 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:22.988953 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-54pc5" event={"ID":"43ab9a7f-db10-4445-b2b6-b6793c88f75d","Type":"ContainerStarted","Data":"2055827b70dd1fcf8a48283f874cceb5e866d9f9af4807bb6fda2ceca3cdde6d"} Jul 26 15:10:22.989065 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:22.988997 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-54pc5" event={"ID":"43ab9a7f-db10-4445-b2b6-b6793c88f75d","Type":"ContainerStarted","Data":"9d8353ef9d4fe6eb2d82ccab4b6fb4f5aecf8c52625f05a6624d1d5febaed8af"} Jul 26 15:10:22.989065 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:22.989017 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-54pc5" event={"ID":"43ab9a7f-db10-4445-b2b6-b6793c88f75d","Type":"ContainerStarted","Data":"d73565d201bef24919a07bd6860b51b4abf3c1098c75c0ae2b4de752ec420efe"} Jul 26 15:10:23.611730 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:23.611605 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:23.678356 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:23.678299 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:23.775690 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:23.775665 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:23.879610 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:23.879575 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:23.995695 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:23.995661 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-54pc5" event={"ID":"43ab9a7f-db10-4445-b2b6-b6793c88f75d","Type":"ContainerStarted","Data":"0025c8f24aa70e52a83b183c10da1472114b13a756525e48f185079842b0b554"} Jul 26 15:10:23.997707 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:23.997657 2555 generic.go:356] "Generic (PLEG): container finished" podID="c6ef366b-7f71-4f17-8383-7c1c87a3a1f1" containerID="63756c5d940d4a6ce9ab88fbf4dfc726c21e8c5caca9e88b2a07b00ba5ac31e7" exitCode=0 Jul 26 15:10:23.997829 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:23.997699 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ljbtm" event={"ID":"c6ef366b-7f71-4f17-8383-7c1c87a3a1f1","Type":"ContainerDied","Data":"63756c5d940d4a6ce9ab88fbf4dfc726c21e8c5caca9e88b2a07b00ba5ac31e7"} Jul 26 15:10:24.003194 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:24.003175 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5947968457-clxm2" Jul 26 15:10:24.011780 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:24.011734 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-54pc5" podStartSLOduration=2.881990089 podStartE2EDuration="4.011720214s" totalImagesPullingTime="1.129730125s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:10:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:10:24.011691527 +0000 UTC m=+304.435538399" watchObservedRunningTime="2026-07-26 15:10:24.011720214 +0000 UTC m=+304.435567085" Jul 26 15:10:24.169400 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:24.169368 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:24.273408 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:24.273301 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:24.376662 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:24.376627 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:24.911008 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:24.910981 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7fd4475f4c-z475j" Jul 26 15:10:25.007037 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:25.006995 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ljbtm" event={"ID":"c6ef366b-7f71-4f17-8383-7c1c87a3a1f1","Type":"ContainerStarted","Data":"4c6858d8f262f4d2a60a202516255ce1753bcefc638b1811daddac1b12ee6622"} Jul 26 15:10:25.007217 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:25.007062 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ljbtm" event={"ID":"c6ef366b-7f71-4f17-8383-7c1c87a3a1f1","Type":"ContainerStarted","Data":"21a22d8f4a78dddc5103a7eff793f889c40fb594bb54ef01a6f665f35e1fde8a"} Jul 26 15:10:25.024721 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:25.024666 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-ljbtm" podStartSLOduration=3.946385045 podStartE2EDuration="5.024650985s" totalImagesPullingTime="996.413616ms" totalInitContainerRuntime="81.852324ms" isStatefulPod=true podCreationTimestamp="2026-07-26 15:10:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:10:25.024119009 +0000 UTC m=+305.447965881" watchObservedRunningTime="2026-07-26 15:10:25.024650985 +0000 UTC m=+305.448497859" Jul 26 15:10:25.472853 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:25.472474 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:25.526084 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:25.526054 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:25.684969 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:25.684920 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:25.787517 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:25.787276 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:25.872149 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:25.872112 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:26.267988 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:26.267961 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5947968457-clxm2"] Jul 26 15:10:26.853984 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:26.853919 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:26.860959 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:26.860915 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 26 15:10:26.864475 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:26.864458 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:10:26.867952 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:26.867917 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-xwp8l\"" Jul 26 15:10:26.868082 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:26.867952 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 26 15:10:26.868281 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:26.868249 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 26 15:10:26.868399 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:26.868280 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 26 15:10:26.868399 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:26.868315 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 26 15:10:26.868399 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:26.868340 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 26 15:10:26.868399 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:26.868369 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 26 15:10:26.868689 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:26.868670 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 26 15:10:26.868778 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:26.868724 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 26 15:10:26.868778 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:26.868755 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 26 15:10:26.868880 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:26.868806 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 26 15:10:26.868880 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:26.868835 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 26 15:10:26.869104 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:26.869084 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 26 15:10:26.869204 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:26.869106 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 26 15:10:26.869265 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:26.869212 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-cvu6liautt05l\"" Jul 26 15:10:26.871655 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:26.871624 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 26 15:10:26.878489 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:26.876741 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 26 15:10:26.881085 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:26.881067 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 26 15:10:26.923708 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:26.923684 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:26.938631 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:26.938610 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1b26882d-ac13-400d-994f-e0842c802ce6-config-out\") pod \"prometheus-k8s-0\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:10:26.938715 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:26.938636 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1b26882d-ac13-400d-994f-e0842c802ce6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:10:26.938715 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:26.938654 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1b26882d-ac13-400d-994f-e0842c802ce6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:10:26.938715 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:26.938669 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1b26882d-ac13-400d-994f-e0842c802ce6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:10:26.938715 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:26.938684 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1b26882d-ac13-400d-994f-e0842c802ce6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:10:26.938846 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:26.938713 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wtm8\" (UniqueName: \"kubernetes.io/projected/1b26882d-ac13-400d-994f-e0842c802ce6-kube-api-access-8wtm8\") pod \"prometheus-k8s-0\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:10:26.938846 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:26.938762 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1b26882d-ac13-400d-994f-e0842c802ce6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:10:26.938846 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:26.938783 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1b26882d-ac13-400d-994f-e0842c802ce6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:10:26.938846 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:26.938801 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1b26882d-ac13-400d-994f-e0842c802ce6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:10:26.938846 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:26.938823 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b26882d-ac13-400d-994f-e0842c802ce6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:10:26.939030 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:26.938881 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1b26882d-ac13-400d-994f-e0842c802ce6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:10:26.939030 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:26.938902 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1b26882d-ac13-400d-994f-e0842c802ce6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:10:26.939030 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:26.938944 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b26882d-ac13-400d-994f-e0842c802ce6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:10:26.939030 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:26.938974 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b26882d-ac13-400d-994f-e0842c802ce6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:10:26.939030 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:26.939017 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1b26882d-ac13-400d-994f-e0842c802ce6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:10:26.939201 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:26.939042 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b26882d-ac13-400d-994f-e0842c802ce6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:10:26.939201 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:26.939073 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1b26882d-ac13-400d-994f-e0842c802ce6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:10:26.939201 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:26.939119 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1b26882d-ac13-400d-994f-e0842c802ce6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:10:26.939201 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:26.939138 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1b26882d-ac13-400d-994f-e0842c802ce6-config\") pod \"prometheus-k8s-0\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:10:26.939201 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:26.939153 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1b26882d-ac13-400d-994f-e0842c802ce6-web-config\") pod \"prometheus-k8s-0\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:10:27.015462 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:27.015434 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hbjsw" event={"ID":"a65a53c9-99a1-4097-9686-73738e044085","Type":"ContainerStarted","Data":"962333fe60e6afb5215676f770629292d251e9f2ce2aba196a6090a1665e95dd"} Jul 26 15:10:27.030994 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:27.030850 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-hbjsw" podStartSLOduration=33.230241004 podStartE2EDuration="46.030833281s" totalImagesPullingTime="12.800592277s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 15:09:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:10:27.030161766 +0000 UTC m=+307.454008656" watchObservedRunningTime="2026-07-26 15:10:27.030833281 +0000 UTC m=+307.454680153" Jul 26 15:10:27.039755 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:27.039729 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1b26882d-ac13-400d-994f-e0842c802ce6-config\") pod \"prometheus-k8s-0\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:10:27.039755 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:27.039755 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1b26882d-ac13-400d-994f-e0842c802ce6-web-config\") pod \"prometheus-k8s-0\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:10:27.039893 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:27.039771 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1b26882d-ac13-400d-994f-e0842c802ce6-config-out\") pod \"prometheus-k8s-0\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:10:27.039893 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:27.039784 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1b26882d-ac13-400d-994f-e0842c802ce6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:10:27.039893 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:27.039800 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1b26882d-ac13-400d-994f-e0842c802ce6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:10:27.039893 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:27.039824 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1b26882d-ac13-400d-994f-e0842c802ce6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:10:27.040143 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:27.039973 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1b26882d-ac13-400d-994f-e0842c802ce6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:10:27.040143 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:27.039997 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wtm8\" (UniqueName: \"kubernetes.io/projected/1b26882d-ac13-400d-994f-e0842c802ce6-kube-api-access-8wtm8\") pod \"prometheus-k8s-0\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:10:27.040143 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:27.040024 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1b26882d-ac13-400d-994f-e0842c802ce6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:10:27.040143 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:27.040050 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1b26882d-ac13-400d-994f-e0842c802ce6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:10:27.041511 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:27.040248 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1b26882d-ac13-400d-994f-e0842c802ce6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:10:27.041511 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:27.040292 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b26882d-ac13-400d-994f-e0842c802ce6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:10:27.041511 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:27.040319 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1b26882d-ac13-400d-994f-e0842c802ce6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:10:27.041511 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:27.040344 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1b26882d-ac13-400d-994f-e0842c802ce6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:10:27.041511 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:27.040369 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b26882d-ac13-400d-994f-e0842c802ce6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:10:27.041511 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:27.040392 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b26882d-ac13-400d-994f-e0842c802ce6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:10:27.041511 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:27.040486 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1b26882d-ac13-400d-994f-e0842c802ce6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:10:27.041511 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:27.040518 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b26882d-ac13-400d-994f-e0842c802ce6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:10:27.041511 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:27.040545 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1b26882d-ac13-400d-994f-e0842c802ce6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:10:27.041511 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:27.040644 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1b26882d-ac13-400d-994f-e0842c802ce6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:10:27.041511 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:27.041210 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1b26882d-ac13-400d-994f-e0842c802ce6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:10:27.041511 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:27.041380 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b26882d-ac13-400d-994f-e0842c802ce6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:10:27.043537 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:27.042676 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b26882d-ac13-400d-994f-e0842c802ce6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:10:27.043537 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:27.042986 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b26882d-ac13-400d-994f-e0842c802ce6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:10:27.043784 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:27.043758 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1b26882d-ac13-400d-994f-e0842c802ce6-config-out\") pod \"prometheus-k8s-0\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:10:27.043947 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:27.043909 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1b26882d-ac13-400d-994f-e0842c802ce6-web-config\") pod \"prometheus-k8s-0\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:10:27.044023 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:27.043946 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1b26882d-ac13-400d-994f-e0842c802ce6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:10:27.044128 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:27.044107 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b26882d-ac13-400d-994f-e0842c802ce6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:10:27.044286 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:27.044265 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/1b26882d-ac13-400d-994f-e0842c802ce6-config\") pod \"prometheus-k8s-0\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:10:27.044453 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:27.044391 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1b26882d-ac13-400d-994f-e0842c802ce6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:10:27.044453 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:27.044391 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1b26882d-ac13-400d-994f-e0842c802ce6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:10:27.044811 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:27.044767 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1b26882d-ac13-400d-994f-e0842c802ce6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:10:27.045119 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:27.045099 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1b26882d-ac13-400d-994f-e0842c802ce6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:10:27.045292 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:27.045271 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1b26882d-ac13-400d-994f-e0842c802ce6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:10:27.045954 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:27.045914 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1b26882d-ac13-400d-994f-e0842c802ce6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:10:27.046058 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:27.045983 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1b26882d-ac13-400d-994f-e0842c802ce6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:10:27.046121 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:27.046095 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1b26882d-ac13-400d-994f-e0842c802ce6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:10:27.046203 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:27.046184 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1b26882d-ac13-400d-994f-e0842c802ce6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:10:27.046821 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:27.046782 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1b26882d-ac13-400d-994f-e0842c802ce6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:10:27.050909 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:27.050889 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wtm8\" (UniqueName: \"kubernetes.io/projected/1b26882d-ac13-400d-994f-e0842c802ce6-kube-api-access-8wtm8\") pod \"prometheus-k8s-0\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:10:27.174556 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:27.174531 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:10:27.306379 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:27.306347 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 26 15:10:27.309615 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:10:27.309587 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b26882d_ac13_400d_994f_e0842c802ce6.slice/crio-88c96817f02d9c9cb8704ab735b60c4b4ba48d7ffe427bdb81535d437f7cb7e6 WatchSource:0}: Error finding container 88c96817f02d9c9cb8704ab735b60c4b4ba48d7ffe427bdb81535d437f7cb7e6: Status 404 returned error can't find the container with id 88c96817f02d9c9cb8704ab735b60c4b4ba48d7ffe427bdb81535d437f7cb7e6 Jul 26 15:10:27.723572 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:27.723536 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:27.804473 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:27.804429 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:28.020854 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:28.020770 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1b26882d-ac13-400d-994f-e0842c802ce6","Type":"ContainerStarted","Data":"88c96817f02d9c9cb8704ab735b60c4b4ba48d7ffe427bdb81535d437f7cb7e6"} Jul 26 15:10:29.026468 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:29.026438 2555 generic.go:356] "Generic (PLEG): container finished" podID="1b26882d-ac13-400d-994f-e0842c802ce6" containerID="9627574d0ae9b758d8471cb56fcd70cce70e91381e8f55a86ec4fa00e04b958b" exitCode=0 Jul 26 15:10:29.026836 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:29.026518 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1b26882d-ac13-400d-994f-e0842c802ce6","Type":"ContainerDied","Data":"9627574d0ae9b758d8471cb56fcd70cce70e91381e8f55a86ec4fa00e04b958b"} Jul 26 15:10:29.058135 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:29.058100 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlw9m" Jul 26 15:10:29.922237 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:29.922199 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7fd4475f4c-z475j" podUID="3183398e-9b01-4309-a64d-0c37a9c20303" containerName="registry" containerID="cri-o://9b843bd9063c0bd15059b3ef11c5705d53b8970ecd2946290a59a60bbab44ccd" gracePeriod=30 Jul 26 15:10:30.182702 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:30.182645 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fd4475f4c-z475j" Jul 26 15:10:30.263426 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:30.263399 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3183398e-9b01-4309-a64d-0c37a9c20303-installation-pull-secrets\") pod \"3183398e-9b01-4309-a64d-0c37a9c20303\" (UID: \"3183398e-9b01-4309-a64d-0c37a9c20303\") " Jul 26 15:10:30.263543 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:30.263437 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbkhj\" (UniqueName: \"kubernetes.io/projected/3183398e-9b01-4309-a64d-0c37a9c20303-kube-api-access-dbkhj\") pod \"3183398e-9b01-4309-a64d-0c37a9c20303\" (UID: \"3183398e-9b01-4309-a64d-0c37a9c20303\") " Jul 26 15:10:30.263543 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:30.263455 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3183398e-9b01-4309-a64d-0c37a9c20303-bound-sa-token\") pod \"3183398e-9b01-4309-a64d-0c37a9c20303\" (UID: \"3183398e-9b01-4309-a64d-0c37a9c20303\") " Jul 26 15:10:30.263543 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:30.263478 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3183398e-9b01-4309-a64d-0c37a9c20303-image-registry-private-configuration\") pod \"3183398e-9b01-4309-a64d-0c37a9c20303\" (UID: \"3183398e-9b01-4309-a64d-0c37a9c20303\") " Jul 26 15:10:30.263543 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:30.263516 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3183398e-9b01-4309-a64d-0c37a9c20303-registry-tls\") pod \"3183398e-9b01-4309-a64d-0c37a9c20303\" (UID: \"3183398e-9b01-4309-a64d-0c37a9c20303\") " Jul 26 15:10:30.263734 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:30.263565 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3183398e-9b01-4309-a64d-0c37a9c20303-ca-trust-extracted\") pod \"3183398e-9b01-4309-a64d-0c37a9c20303\" (UID: \"3183398e-9b01-4309-a64d-0c37a9c20303\") " Jul 26 15:10:30.263734 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:30.263610 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3183398e-9b01-4309-a64d-0c37a9c20303-trusted-ca\") pod \"3183398e-9b01-4309-a64d-0c37a9c20303\" (UID: \"3183398e-9b01-4309-a64d-0c37a9c20303\") " Jul 26 15:10:30.264133 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:30.263847 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3183398e-9b01-4309-a64d-0c37a9c20303-registry-certificates\") pod \"3183398e-9b01-4309-a64d-0c37a9c20303\" (UID: \"3183398e-9b01-4309-a64d-0c37a9c20303\") " Jul 26 15:10:30.264133 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:30.264014 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3183398e-9b01-4309-a64d-0c37a9c20303-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "3183398e-9b01-4309-a64d-0c37a9c20303" (UID: "3183398e-9b01-4309-a64d-0c37a9c20303"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:10:30.264133 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:30.264105 2555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3183398e-9b01-4309-a64d-0c37a9c20303-trusted-ca\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:10:30.264529 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:30.264492 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3183398e-9b01-4309-a64d-0c37a9c20303-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "3183398e-9b01-4309-a64d-0c37a9c20303" (UID: "3183398e-9b01-4309-a64d-0c37a9c20303"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:10:30.265841 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:30.265790 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3183398e-9b01-4309-a64d-0c37a9c20303-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "3183398e-9b01-4309-a64d-0c37a9c20303" (UID: "3183398e-9b01-4309-a64d-0c37a9c20303"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 15:10:30.265972 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:30.265899 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3183398e-9b01-4309-a64d-0c37a9c20303-kube-api-access-dbkhj" (OuterVolumeSpecName: "kube-api-access-dbkhj") pod "3183398e-9b01-4309-a64d-0c37a9c20303" (UID: "3183398e-9b01-4309-a64d-0c37a9c20303"). InnerVolumeSpecName "kube-api-access-dbkhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:10:30.266040 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:30.266008 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3183398e-9b01-4309-a64d-0c37a9c20303-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "3183398e-9b01-4309-a64d-0c37a9c20303" (UID: "3183398e-9b01-4309-a64d-0c37a9c20303"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:10:30.266165 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:30.266144 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3183398e-9b01-4309-a64d-0c37a9c20303-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "3183398e-9b01-4309-a64d-0c37a9c20303" (UID: "3183398e-9b01-4309-a64d-0c37a9c20303"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:10:30.266253 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:30.266229 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3183398e-9b01-4309-a64d-0c37a9c20303-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "3183398e-9b01-4309-a64d-0c37a9c20303" (UID: "3183398e-9b01-4309-a64d-0c37a9c20303"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 15:10:30.272099 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:30.272076 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3183398e-9b01-4309-a64d-0c37a9c20303-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "3183398e-9b01-4309-a64d-0c37a9c20303" (UID: "3183398e-9b01-4309-a64d-0c37a9c20303"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:10:30.364821 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:30.364790 2555 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3183398e-9b01-4309-a64d-0c37a9c20303-ca-trust-extracted\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:10:30.364821 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:30.364816 2555 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3183398e-9b01-4309-a64d-0c37a9c20303-registry-certificates\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:10:30.364821 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:30.364826 2555 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3183398e-9b01-4309-a64d-0c37a9c20303-installation-pull-secrets\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:10:30.365064 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:30.364835 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbkhj\" (UniqueName: \"kubernetes.io/projected/3183398e-9b01-4309-a64d-0c37a9c20303-kube-api-access-dbkhj\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:10:30.365064 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:30.364844 2555 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3183398e-9b01-4309-a64d-0c37a9c20303-bound-sa-token\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:10:30.365064 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:30.364857 2555 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3183398e-9b01-4309-a64d-0c37a9c20303-image-registry-private-configuration\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:10:30.365064 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:30.364874 2555 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3183398e-9b01-4309-a64d-0c37a9c20303-registry-tls\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:10:31.036002 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:31.035967 2555 generic.go:356] "Generic (PLEG): container finished" podID="3183398e-9b01-4309-a64d-0c37a9c20303" containerID="9b843bd9063c0bd15059b3ef11c5705d53b8970ecd2946290a59a60bbab44ccd" exitCode=0 Jul 26 15:10:31.036158 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:31.036025 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fd4475f4c-z475j" Jul 26 15:10:31.036158 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:31.036025 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7fd4475f4c-z475j" event={"ID":"3183398e-9b01-4309-a64d-0c37a9c20303","Type":"ContainerDied","Data":"9b843bd9063c0bd15059b3ef11c5705d53b8970ecd2946290a59a60bbab44ccd"} Jul 26 15:10:31.036279 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:31.036165 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7fd4475f4c-z475j" event={"ID":"3183398e-9b01-4309-a64d-0c37a9c20303","Type":"ContainerDied","Data":"189fee21f5c03ef7514faa3369422b597234a89fd6c8e7791cce465a02d3db73"} Jul 26 15:10:31.036279 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:31.036203 2555 scope.go:117] "RemoveContainer" containerID="9b843bd9063c0bd15059b3ef11c5705d53b8970ecd2946290a59a60bbab44ccd" Jul 26 15:10:31.045674 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:31.045637 2555 scope.go:117] "RemoveContainer" containerID="9b843bd9063c0bd15059b3ef11c5705d53b8970ecd2946290a59a60bbab44ccd" Jul 26 15:10:31.045953 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:10:31.045917 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b843bd9063c0bd15059b3ef11c5705d53b8970ecd2946290a59a60bbab44ccd\": container with ID starting with 9b843bd9063c0bd15059b3ef11c5705d53b8970ecd2946290a59a60bbab44ccd not found: ID does not exist" containerID="9b843bd9063c0bd15059b3ef11c5705d53b8970ecd2946290a59a60bbab44ccd" Jul 26 15:10:31.046044 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:31.045964 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b843bd9063c0bd15059b3ef11c5705d53b8970ecd2946290a59a60bbab44ccd"} err="failed to get container status \"9b843bd9063c0bd15059b3ef11c5705d53b8970ecd2946290a59a60bbab44ccd\": rpc error: code = NotFound desc = could not find container \"9b843bd9063c0bd15059b3ef11c5705d53b8970ecd2946290a59a60bbab44ccd\": container with ID starting with 9b843bd9063c0bd15059b3ef11c5705d53b8970ecd2946290a59a60bbab44ccd not found: ID does not exist" Jul 26 15:10:31.058679 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:31.058655 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7fd4475f4c-z475j"] Jul 26 15:10:31.062782 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:31.062765 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7fd4475f4c-z475j"] Jul 26 15:10:31.070614 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:31.070591 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:31.128380 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:31.128351 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:31.261821 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:31.261795 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6fd8bd796d-x5gc4"] Jul 26 15:10:31.262211 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:31.262075 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3183398e-9b01-4309-a64d-0c37a9c20303" containerName="registry" Jul 26 15:10:31.262211 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:31.262085 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3183398e-9b01-4309-a64d-0c37a9c20303" containerName="registry" Jul 26 15:10:31.262211 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:31.262141 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3183398e-9b01-4309-a64d-0c37a9c20303" containerName="registry" Jul 26 15:10:31.264757 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:31.264742 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fd8bd796d-x5gc4" Jul 26 15:10:31.271468 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:31.271446 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 26 15:10:31.274169 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:31.274147 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6fd8bd796d-x5gc4"] Jul 26 15:10:31.374120 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:31.374085 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fdc52ca3-7c32-4a07-b286-12e826330cb4-console-serving-cert\") pod \"console-6fd8bd796d-x5gc4\" (UID: \"fdc52ca3-7c32-4a07-b286-12e826330cb4\") " pod="openshift-console/console-6fd8bd796d-x5gc4" Jul 26 15:10:31.374120 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:31.374126 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fdc52ca3-7c32-4a07-b286-12e826330cb4-console-oauth-config\") pod \"console-6fd8bd796d-x5gc4\" (UID: \"fdc52ca3-7c32-4a07-b286-12e826330cb4\") " pod="openshift-console/console-6fd8bd796d-x5gc4" Jul 26 15:10:31.374284 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:31.374149 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fdc52ca3-7c32-4a07-b286-12e826330cb4-service-ca\") pod \"console-6fd8bd796d-x5gc4\" (UID: \"fdc52ca3-7c32-4a07-b286-12e826330cb4\") " pod="openshift-console/console-6fd8bd796d-x5gc4" Jul 26 15:10:31.374284 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:31.374230 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdc52ca3-7c32-4a07-b286-12e826330cb4-trusted-ca-bundle\") pod \"console-6fd8bd796d-x5gc4\" (UID: \"fdc52ca3-7c32-4a07-b286-12e826330cb4\") " pod="openshift-console/console-6fd8bd796d-x5gc4" Jul 26 15:10:31.374284 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:31.374269 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84xcg\" (UniqueName: \"kubernetes.io/projected/fdc52ca3-7c32-4a07-b286-12e826330cb4-kube-api-access-84xcg\") pod \"console-6fd8bd796d-x5gc4\" (UID: \"fdc52ca3-7c32-4a07-b286-12e826330cb4\") " pod="openshift-console/console-6fd8bd796d-x5gc4" Jul 26 15:10:31.374375 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:31.374297 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fdc52ca3-7c32-4a07-b286-12e826330cb4-console-config\") pod \"console-6fd8bd796d-x5gc4\" (UID: \"fdc52ca3-7c32-4a07-b286-12e826330cb4\") " pod="openshift-console/console-6fd8bd796d-x5gc4" Jul 26 15:10:31.374375 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:31.374311 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fdc52ca3-7c32-4a07-b286-12e826330cb4-oauth-serving-cert\") pod \"console-6fd8bd796d-x5gc4\" (UID: \"fdc52ca3-7c32-4a07-b286-12e826330cb4\") " pod="openshift-console/console-6fd8bd796d-x5gc4" Jul 26 15:10:31.475129 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:31.475087 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fdc52ca3-7c32-4a07-b286-12e826330cb4-console-serving-cert\") pod \"console-6fd8bd796d-x5gc4\" (UID: \"fdc52ca3-7c32-4a07-b286-12e826330cb4\") " pod="openshift-console/console-6fd8bd796d-x5gc4" Jul 26 15:10:31.475278 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:31.475133 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fdc52ca3-7c32-4a07-b286-12e826330cb4-console-oauth-config\") pod \"console-6fd8bd796d-x5gc4\" (UID: \"fdc52ca3-7c32-4a07-b286-12e826330cb4\") " pod="openshift-console/console-6fd8bd796d-x5gc4" Jul 26 15:10:31.475278 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:31.475163 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fdc52ca3-7c32-4a07-b286-12e826330cb4-service-ca\") pod \"console-6fd8bd796d-x5gc4\" (UID: \"fdc52ca3-7c32-4a07-b286-12e826330cb4\") " pod="openshift-console/console-6fd8bd796d-x5gc4" Jul 26 15:10:31.475278 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:31.475225 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdc52ca3-7c32-4a07-b286-12e826330cb4-trusted-ca-bundle\") pod \"console-6fd8bd796d-x5gc4\" (UID: \"fdc52ca3-7c32-4a07-b286-12e826330cb4\") " pod="openshift-console/console-6fd8bd796d-x5gc4" Jul 26 15:10:31.475278 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:31.475267 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84xcg\" (UniqueName: \"kubernetes.io/projected/fdc52ca3-7c32-4a07-b286-12e826330cb4-kube-api-access-84xcg\") pod \"console-6fd8bd796d-x5gc4\" (UID: \"fdc52ca3-7c32-4a07-b286-12e826330cb4\") " pod="openshift-console/console-6fd8bd796d-x5gc4" Jul 26 15:10:31.475478 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:31.475330 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fdc52ca3-7c32-4a07-b286-12e826330cb4-console-config\") pod \"console-6fd8bd796d-x5gc4\" (UID: \"fdc52ca3-7c32-4a07-b286-12e826330cb4\") " pod="openshift-console/console-6fd8bd796d-x5gc4" Jul 26 15:10:31.475478 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:31.475371 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fdc52ca3-7c32-4a07-b286-12e826330cb4-oauth-serving-cert\") pod \"console-6fd8bd796d-x5gc4\" (UID: \"fdc52ca3-7c32-4a07-b286-12e826330cb4\") " pod="openshift-console/console-6fd8bd796d-x5gc4" Jul 26 15:10:31.476062 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:31.475986 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fdc52ca3-7c32-4a07-b286-12e826330cb4-service-ca\") pod \"console-6fd8bd796d-x5gc4\" (UID: \"fdc52ca3-7c32-4a07-b286-12e826330cb4\") " pod="openshift-console/console-6fd8bd796d-x5gc4" Jul 26 15:10:31.476062 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:31.476019 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fdc52ca3-7c32-4a07-b286-12e826330cb4-oauth-serving-cert\") pod \"console-6fd8bd796d-x5gc4\" (UID: \"fdc52ca3-7c32-4a07-b286-12e826330cb4\") " pod="openshift-console/console-6fd8bd796d-x5gc4" Jul 26 15:10:31.476254 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:31.476233 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fdc52ca3-7c32-4a07-b286-12e826330cb4-console-config\") pod \"console-6fd8bd796d-x5gc4\" (UID: \"fdc52ca3-7c32-4a07-b286-12e826330cb4\") " pod="openshift-console/console-6fd8bd796d-x5gc4" Jul 26 15:10:31.476365 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:31.476332 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdc52ca3-7c32-4a07-b286-12e826330cb4-trusted-ca-bundle\") pod \"console-6fd8bd796d-x5gc4\" (UID: \"fdc52ca3-7c32-4a07-b286-12e826330cb4\") " pod="openshift-console/console-6fd8bd796d-x5gc4" Jul 26 15:10:31.478046 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:31.478029 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fdc52ca3-7c32-4a07-b286-12e826330cb4-console-oauth-config\") pod \"console-6fd8bd796d-x5gc4\" (UID: \"fdc52ca3-7c32-4a07-b286-12e826330cb4\") " pod="openshift-console/console-6fd8bd796d-x5gc4" Jul 26 15:10:31.478169 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:31.478151 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fdc52ca3-7c32-4a07-b286-12e826330cb4-console-serving-cert\") pod \"console-6fd8bd796d-x5gc4\" (UID: \"fdc52ca3-7c32-4a07-b286-12e826330cb4\") " pod="openshift-console/console-6fd8bd796d-x5gc4" Jul 26 15:10:31.483383 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:31.483366 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84xcg\" (UniqueName: \"kubernetes.io/projected/fdc52ca3-7c32-4a07-b286-12e826330cb4-kube-api-access-84xcg\") pod \"console-6fd8bd796d-x5gc4\" (UID: \"fdc52ca3-7c32-4a07-b286-12e826330cb4\") " pod="openshift-console/console-6fd8bd796d-x5gc4" Jul 26 15:10:31.573892 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:31.573852 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fd8bd796d-x5gc4" Jul 26 15:10:32.063576 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:32.063545 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3183398e-9b01-4309-a64d-0c37a9c20303" path="/var/lib/kubelet/pods/3183398e-9b01-4309-a64d-0c37a9c20303/volumes" Jul 26 15:10:32.536599 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:32.536573 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6fd8bd796d-x5gc4"] Jul 26 15:10:32.538593 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:10:32.538571 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdc52ca3_7c32_4a07_b286_12e826330cb4.slice/crio-9bc121bfc0f6b1b98881a2adaca725ee9baf7633cbb3d3f156189c227a10d89e WatchSource:0}: Error finding container 9bc121bfc0f6b1b98881a2adaca725ee9baf7633cbb3d3f156189c227a10d89e: Status 404 returned error can't find the container with id 9bc121bfc0f6b1b98881a2adaca725ee9baf7633cbb3d3f156189c227a10d89e Jul 26 15:10:32.671796 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:32.671629 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:32.768268 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:32.768195 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:33.048416 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:33.048337 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1b26882d-ac13-400d-994f-e0842c802ce6","Type":"ContainerStarted","Data":"8fa8d6d6eb1829d572a141c32266299561f6f670a78398fc420bd7d3943e7382"} Jul 26 15:10:33.048554 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:33.048456 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1b26882d-ac13-400d-994f-e0842c802ce6","Type":"ContainerStarted","Data":"596937dc730f4594636465eb31af0b45b9f66538e8c96c9b46bf49e7baf77d7b"} Jul 26 15:10:33.050302 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:33.050275 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fd8bd796d-x5gc4" event={"ID":"fdc52ca3-7c32-4a07-b286-12e826330cb4","Type":"ContainerStarted","Data":"69c4a06bf6682fe94d2b1586f3e0731066280e8d15bc6a6509f3653fe32deccf"} Jul 26 15:10:33.050424 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:33.050317 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fd8bd796d-x5gc4" event={"ID":"fdc52ca3-7c32-4a07-b286-12e826330cb4","Type":"ContainerStarted","Data":"9bc121bfc0f6b1b98881a2adaca725ee9baf7633cbb3d3f156189c227a10d89e"} Jul 26 15:10:33.067450 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:33.067390 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6fd8bd796d-x5gc4" podStartSLOduration=2.06737205 podStartE2EDuration="2.06737205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:10:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:10:33.064799126 +0000 UTC m=+313.488645997" watchObservedRunningTime="2026-07-26 15:10:33.06737205 +0000 UTC m=+313.491218923" Jul 26 15:10:33.392560 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:33.392524 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6fd8bd796d-x5gc4"] Jul 26 15:10:33.414363 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:33.414339 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-74b5bd7f57-nj64n"] Jul 26 15:10:33.417788 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:33.417766 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74b5bd7f57-nj64n" Jul 26 15:10:33.427903 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:33.427879 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74b5bd7f57-nj64n"] Jul 26 15:10:33.494606 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:33.494576 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2d56de0a-bf2b-4547-a697-e63aa9b109c7-console-serving-cert\") pod \"console-74b5bd7f57-nj64n\" (UID: \"2d56de0a-bf2b-4547-a697-e63aa9b109c7\") " pod="openshift-console/console-74b5bd7f57-nj64n" Jul 26 15:10:33.494721 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:33.494608 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d56de0a-bf2b-4547-a697-e63aa9b109c7-trusted-ca-bundle\") pod \"console-74b5bd7f57-nj64n\" (UID: \"2d56de0a-bf2b-4547-a697-e63aa9b109c7\") " pod="openshift-console/console-74b5bd7f57-nj64n" Jul 26 15:10:33.494721 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:33.494643 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2d56de0a-bf2b-4547-a697-e63aa9b109c7-console-oauth-config\") pod \"console-74b5bd7f57-nj64n\" (UID: \"2d56de0a-bf2b-4547-a697-e63aa9b109c7\") " pod="openshift-console/console-74b5bd7f57-nj64n" Jul 26 15:10:33.494721 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:33.494685 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2d56de0a-bf2b-4547-a697-e63aa9b109c7-console-config\") pod \"console-74b5bd7f57-nj64n\" (UID: \"2d56de0a-bf2b-4547-a697-e63aa9b109c7\") " pod="openshift-console/console-74b5bd7f57-nj64n" Jul 26 15:10:33.494822 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:33.494768 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2d56de0a-bf2b-4547-a697-e63aa9b109c7-oauth-serving-cert\") pod \"console-74b5bd7f57-nj64n\" (UID: \"2d56de0a-bf2b-4547-a697-e63aa9b109c7\") " pod="openshift-console/console-74b5bd7f57-nj64n" Jul 26 15:10:33.494822 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:33.494805 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwksb\" (UniqueName: \"kubernetes.io/projected/2d56de0a-bf2b-4547-a697-e63aa9b109c7-kube-api-access-xwksb\") pod \"console-74b5bd7f57-nj64n\" (UID: \"2d56de0a-bf2b-4547-a697-e63aa9b109c7\") " pod="openshift-console/console-74b5bd7f57-nj64n" Jul 26 15:10:33.494889 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:33.494828 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2d56de0a-bf2b-4547-a697-e63aa9b109c7-service-ca\") pod \"console-74b5bd7f57-nj64n\" (UID: \"2d56de0a-bf2b-4547-a697-e63aa9b109c7\") " pod="openshift-console/console-74b5bd7f57-nj64n" Jul 26 15:10:33.595463 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:33.595434 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2d56de0a-bf2b-4547-a697-e63aa9b109c7-oauth-serving-cert\") pod \"console-74b5bd7f57-nj64n\" (UID: \"2d56de0a-bf2b-4547-a697-e63aa9b109c7\") " pod="openshift-console/console-74b5bd7f57-nj64n" Jul 26 15:10:33.595836 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:33.595483 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwksb\" (UniqueName: \"kubernetes.io/projected/2d56de0a-bf2b-4547-a697-e63aa9b109c7-kube-api-access-xwksb\") pod \"console-74b5bd7f57-nj64n\" (UID: \"2d56de0a-bf2b-4547-a697-e63aa9b109c7\") " pod="openshift-console/console-74b5bd7f57-nj64n" Jul 26 15:10:33.595836 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:33.595517 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2d56de0a-bf2b-4547-a697-e63aa9b109c7-service-ca\") pod \"console-74b5bd7f57-nj64n\" (UID: \"2d56de0a-bf2b-4547-a697-e63aa9b109c7\") " pod="openshift-console/console-74b5bd7f57-nj64n" Jul 26 15:10:33.595836 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:33.595543 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2d56de0a-bf2b-4547-a697-e63aa9b109c7-console-serving-cert\") pod \"console-74b5bd7f57-nj64n\" (UID: \"2d56de0a-bf2b-4547-a697-e63aa9b109c7\") " pod="openshift-console/console-74b5bd7f57-nj64n" Jul 26 15:10:33.595836 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:33.595558 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d56de0a-bf2b-4547-a697-e63aa9b109c7-trusted-ca-bundle\") pod \"console-74b5bd7f57-nj64n\" (UID: \"2d56de0a-bf2b-4547-a697-e63aa9b109c7\") " pod="openshift-console/console-74b5bd7f57-nj64n" Jul 26 15:10:33.595836 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:33.595590 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2d56de0a-bf2b-4547-a697-e63aa9b109c7-console-oauth-config\") pod \"console-74b5bd7f57-nj64n\" (UID: \"2d56de0a-bf2b-4547-a697-e63aa9b109c7\") " pod="openshift-console/console-74b5bd7f57-nj64n" Jul 26 15:10:33.595836 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:33.595623 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2d56de0a-bf2b-4547-a697-e63aa9b109c7-console-config\") pod \"console-74b5bd7f57-nj64n\" (UID: \"2d56de0a-bf2b-4547-a697-e63aa9b109c7\") " pod="openshift-console/console-74b5bd7f57-nj64n" Jul 26 15:10:33.596405 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:33.596381 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2d56de0a-bf2b-4547-a697-e63aa9b109c7-console-config\") pod \"console-74b5bd7f57-nj64n\" (UID: \"2d56de0a-bf2b-4547-a697-e63aa9b109c7\") " pod="openshift-console/console-74b5bd7f57-nj64n" Jul 26 15:10:33.596502 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:33.596381 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2d56de0a-bf2b-4547-a697-e63aa9b109c7-oauth-serving-cert\") pod \"console-74b5bd7f57-nj64n\" (UID: \"2d56de0a-bf2b-4547-a697-e63aa9b109c7\") " pod="openshift-console/console-74b5bd7f57-nj64n" Jul 26 15:10:33.596558 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:33.596497 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2d56de0a-bf2b-4547-a697-e63aa9b109c7-service-ca\") pod \"console-74b5bd7f57-nj64n\" (UID: \"2d56de0a-bf2b-4547-a697-e63aa9b109c7\") " pod="openshift-console/console-74b5bd7f57-nj64n" Jul 26 15:10:33.596708 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:33.596683 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d56de0a-bf2b-4547-a697-e63aa9b109c7-trusted-ca-bundle\") pod \"console-74b5bd7f57-nj64n\" (UID: \"2d56de0a-bf2b-4547-a697-e63aa9b109c7\") " pod="openshift-console/console-74b5bd7f57-nj64n" Jul 26 15:10:33.598254 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:33.598234 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2d56de0a-bf2b-4547-a697-e63aa9b109c7-console-oauth-config\") pod \"console-74b5bd7f57-nj64n\" (UID: \"2d56de0a-bf2b-4547-a697-e63aa9b109c7\") " pod="openshift-console/console-74b5bd7f57-nj64n" Jul 26 15:10:33.598366 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:33.598344 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2d56de0a-bf2b-4547-a697-e63aa9b109c7-console-serving-cert\") pod \"console-74b5bd7f57-nj64n\" (UID: \"2d56de0a-bf2b-4547-a697-e63aa9b109c7\") " pod="openshift-console/console-74b5bd7f57-nj64n" Jul 26 15:10:33.602459 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:33.602436 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwksb\" (UniqueName: \"kubernetes.io/projected/2d56de0a-bf2b-4547-a697-e63aa9b109c7-kube-api-access-xwksb\") pod \"console-74b5bd7f57-nj64n\" (UID: \"2d56de0a-bf2b-4547-a697-e63aa9b109c7\") " pod="openshift-console/console-74b5bd7f57-nj64n" Jul 26 15:10:33.728539 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:33.728469 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74b5bd7f57-nj64n" Jul 26 15:10:33.880200 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:33.880177 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74b5bd7f57-nj64n"] Jul 26 15:10:34.355066 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:10:34.355035 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d56de0a_bf2b_4547_a697_e63aa9b109c7.slice/crio-02b733230904fd564aad2b34ae1f3c181394f1314c3bf07082a6c2faed3744f2 WatchSource:0}: Error finding container 02b733230904fd564aad2b34ae1f3c181394f1314c3bf07082a6c2faed3744f2: Status 404 returned error can't find the container with id 02b733230904fd564aad2b34ae1f3c181394f1314c3bf07082a6c2faed3744f2 Jul 26 15:10:35.059187 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:35.059152 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74b5bd7f57-nj64n" event={"ID":"2d56de0a-bf2b-4547-a697-e63aa9b109c7","Type":"ContainerStarted","Data":"46861c726ed4d0d19e840d0cc76e7356f9fdf5a1b4422b4dcf75bf1a6503cfc1"} Jul 26 15:10:35.059562 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:35.059196 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74b5bd7f57-nj64n" event={"ID":"2d56de0a-bf2b-4547-a697-e63aa9b109c7","Type":"ContainerStarted","Data":"02b733230904fd564aad2b34ae1f3c181394f1314c3bf07082a6c2faed3744f2"} Jul 26 15:10:35.062282 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:35.062253 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1b26882d-ac13-400d-994f-e0842c802ce6","Type":"ContainerStarted","Data":"ef4c7483f93149eeb0bf8d5cdd7a5dd63434b0fb12c5e0884b39c5f2e0739094"} Jul 26 15:10:35.062419 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:35.062323 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1b26882d-ac13-400d-994f-e0842c802ce6","Type":"ContainerStarted","Data":"dc709ca82a018b8095268adca9e42c5a9f9f6c1716454b219c967ad5889ece3c"} Jul 26 15:10:35.062419 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:35.062366 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1b26882d-ac13-400d-994f-e0842c802ce6","Type":"ContainerStarted","Data":"a4dbcb9bd26556e66f490648d941fe7ad3823fb86ecebe093fc77b00ca6c9078"} Jul 26 15:10:35.062419 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:35.062403 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1b26882d-ac13-400d-994f-e0842c802ce6","Type":"ContainerStarted","Data":"06e84755fc7e613d6358c376545ab50d8008597efeeccc6a70cec6373bd8e963"} Jul 26 15:10:35.075460 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:35.075423 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-74b5bd7f57-nj64n" podStartSLOduration=2.075408974 podStartE2EDuration="2.075408974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:10:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:10:35.074259854 +0000 UTC m=+315.498106725" watchObservedRunningTime="2026-07-26 15:10:35.075408974 +0000 UTC m=+315.499255843" Jul 26 15:10:35.104472 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:35.104420 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.996925274 podStartE2EDuration="9.104402409s" totalImagesPullingTime="6.089685271s" totalInitContainerRuntime="17.791864ms" isStatefulPod=false podCreationTimestamp="2026-07-26 15:10:26 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:10:35.103272181 +0000 UTC m=+315.527119053" watchObservedRunningTime="2026-07-26 15:10:35.104402409 +0000 UTC m=+315.528249280" Jul 26 15:10:35.454374 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:35.454344 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:35.555179 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:35.555147 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:37.175138 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:37.175109 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:10:37.238318 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:37.238290 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:37.339237 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:37.339196 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:40.110018 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:40.109968 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:40.211515 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:40.211483 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:41.574283 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:41.574242 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6fd8bd796d-x5gc4" Jul 26 15:10:42.721552 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:42.721520 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:42.772335 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:42.772301 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:43.729160 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:43.729128 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-74b5bd7f57-nj64n" Jul 26 15:10:43.729548 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:43.729187 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-74b5bd7f57-nj64n" Jul 26 15:10:43.734003 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:43.733982 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-74b5bd7f57-nj64n" Jul 26 15:10:44.100720 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:44.100655 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-74b5bd7f57-nj64n" Jul 26 15:10:44.238166 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:44.238132 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:44.340643 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:44.340611 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:44.395761 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:44.395724 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:51.129840 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:51.129813 2555 generic.go:356] "Generic (PLEG): container finished" podID="750ed7a9-694b-4f9b-afb5-6a3b99fc3834" containerID="cf1f7145b672323c9562091c2464522606232e5c0e6eb4f0337279505ad679eb" exitCode=0 Jul 26 15:10:51.130244 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:51.129881 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wmjfd" event={"ID":"750ed7a9-694b-4f9b-afb5-6a3b99fc3834","Type":"ContainerDied","Data":"cf1f7145b672323c9562091c2464522606232e5c0e6eb4f0337279505ad679eb"} Jul 26 15:10:51.130349 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:51.130334 2555 scope.go:117] "RemoveContainer" containerID="cf1f7145b672323c9562091c2464522606232e5c0e6eb4f0337279505ad679eb" Jul 26 15:10:51.533979 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:51.533887 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:51.588903 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:51.588874 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:10:52.033633 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:52.033595 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5947968457-clxm2" podUID="94b38fca-3d2b-496c-acde-6e3ac99d76fa" containerName="console" containerID="cri-o://807f3bdc59eb71b64fafb56ea89b9e1ff873c319c7ae0eedd31015c37103c542" gracePeriod=15 Jul 26 15:10:52.140136 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:52.140107 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wmjfd" event={"ID":"750ed7a9-694b-4f9b-afb5-6a3b99fc3834","Type":"ContainerStarted","Data":"434c94aa4e2e81272ad5406fd03c610e3ea53f80146c1d5953c8ceb1b8fb3a7d"} Jul 26 15:10:52.325731 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:52.325708 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5947968457-clxm2" Jul 26 15:10:52.450577 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:52.450546 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/94b38fca-3d2b-496c-acde-6e3ac99d76fa-oauth-serving-cert\") pod \"94b38fca-3d2b-496c-acde-6e3ac99d76fa\" (UID: \"94b38fca-3d2b-496c-acde-6e3ac99d76fa\") " Jul 26 15:10:52.450748 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:52.450588 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/94b38fca-3d2b-496c-acde-6e3ac99d76fa-service-ca\") pod \"94b38fca-3d2b-496c-acde-6e3ac99d76fa\" (UID: \"94b38fca-3d2b-496c-acde-6e3ac99d76fa\") " Jul 26 15:10:52.450748 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:52.450607 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/94b38fca-3d2b-496c-acde-6e3ac99d76fa-console-config\") pod \"94b38fca-3d2b-496c-acde-6e3ac99d76fa\" (UID: \"94b38fca-3d2b-496c-acde-6e3ac99d76fa\") " Jul 26 15:10:52.450748 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:52.450648 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5nmtn\" (UniqueName: \"kubernetes.io/projected/94b38fca-3d2b-496c-acde-6e3ac99d76fa-kube-api-access-5nmtn\") pod \"94b38fca-3d2b-496c-acde-6e3ac99d76fa\" (UID: \"94b38fca-3d2b-496c-acde-6e3ac99d76fa\") " Jul 26 15:10:52.450748 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:52.450677 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/94b38fca-3d2b-496c-acde-6e3ac99d76fa-console-oauth-config\") pod \"94b38fca-3d2b-496c-acde-6e3ac99d76fa\" (UID: \"94b38fca-3d2b-496c-acde-6e3ac99d76fa\") " Jul 26 15:10:52.450748 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:52.450695 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/94b38fca-3d2b-496c-acde-6e3ac99d76fa-console-serving-cert\") pod \"94b38fca-3d2b-496c-acde-6e3ac99d76fa\" (UID: \"94b38fca-3d2b-496c-acde-6e3ac99d76fa\") " Jul 26 15:10:52.451035 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:52.450987 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94b38fca-3d2b-496c-acde-6e3ac99d76fa-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "94b38fca-3d2b-496c-acde-6e3ac99d76fa" (UID: "94b38fca-3d2b-496c-acde-6e3ac99d76fa"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:10:52.451090 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:52.451078 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94b38fca-3d2b-496c-acde-6e3ac99d76fa-service-ca" (OuterVolumeSpecName: "service-ca") pod "94b38fca-3d2b-496c-acde-6e3ac99d76fa" (UID: "94b38fca-3d2b-496c-acde-6e3ac99d76fa"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:10:52.451149 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:52.451125 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94b38fca-3d2b-496c-acde-6e3ac99d76fa-console-config" (OuterVolumeSpecName: "console-config") pod "94b38fca-3d2b-496c-acde-6e3ac99d76fa" (UID: "94b38fca-3d2b-496c-acde-6e3ac99d76fa"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:10:52.452944 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:52.452904 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94b38fca-3d2b-496c-acde-6e3ac99d76fa-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "94b38fca-3d2b-496c-acde-6e3ac99d76fa" (UID: "94b38fca-3d2b-496c-acde-6e3ac99d76fa"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 15:10:52.453051 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:52.453004 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94b38fca-3d2b-496c-acde-6e3ac99d76fa-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "94b38fca-3d2b-496c-acde-6e3ac99d76fa" (UID: "94b38fca-3d2b-496c-acde-6e3ac99d76fa"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 15:10:52.453107 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:52.453060 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94b38fca-3d2b-496c-acde-6e3ac99d76fa-kube-api-access-5nmtn" (OuterVolumeSpecName: "kube-api-access-5nmtn") pod "94b38fca-3d2b-496c-acde-6e3ac99d76fa" (UID: "94b38fca-3d2b-496c-acde-6e3ac99d76fa"). InnerVolumeSpecName "kube-api-access-5nmtn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:10:52.551861 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:52.551792 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nmtn\" (UniqueName: \"kubernetes.io/projected/94b38fca-3d2b-496c-acde-6e3ac99d76fa-kube-api-access-5nmtn\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:10:52.551861 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:52.551818 2555 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/94b38fca-3d2b-496c-acde-6e3ac99d76fa-console-oauth-config\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:10:52.551861 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:52.551832 2555 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/94b38fca-3d2b-496c-acde-6e3ac99d76fa-console-serving-cert\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:10:52.551861 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:52.551843 2555 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/94b38fca-3d2b-496c-acde-6e3ac99d76fa-oauth-serving-cert\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:10:52.551861 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:52.551855 2555 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/94b38fca-3d2b-496c-acde-6e3ac99d76fa-service-ca\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:10:52.552116 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:52.551867 2555 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/94b38fca-3d2b-496c-acde-6e3ac99d76fa-console-config\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:10:53.145846 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:53.145812 2555 generic.go:356] "Generic (PLEG): container finished" podID="94b38fca-3d2b-496c-acde-6e3ac99d76fa" containerID="807f3bdc59eb71b64fafb56ea89b9e1ff873c319c7ae0eedd31015c37103c542" exitCode=0 Jul 26 15:10:53.146320 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:53.145860 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5947968457-clxm2" Jul 26 15:10:53.146320 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:53.145896 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5947968457-clxm2" event={"ID":"94b38fca-3d2b-496c-acde-6e3ac99d76fa","Type":"ContainerDied","Data":"807f3bdc59eb71b64fafb56ea89b9e1ff873c319c7ae0eedd31015c37103c542"} Jul 26 15:10:53.146320 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:53.145955 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5947968457-clxm2" event={"ID":"94b38fca-3d2b-496c-acde-6e3ac99d76fa","Type":"ContainerDied","Data":"8e2219e05d056a1ef5b00462f567dfd6bbc09aad1a3ec12abe1f1624f4424d48"} Jul 26 15:10:53.146320 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:53.145975 2555 scope.go:117] "RemoveContainer" containerID="807f3bdc59eb71b64fafb56ea89b9e1ff873c319c7ae0eedd31015c37103c542" Jul 26 15:10:53.155226 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:53.155210 2555 scope.go:117] "RemoveContainer" containerID="807f3bdc59eb71b64fafb56ea89b9e1ff873c319c7ae0eedd31015c37103c542" Jul 26 15:10:53.155479 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:10:53.155461 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"807f3bdc59eb71b64fafb56ea89b9e1ff873c319c7ae0eedd31015c37103c542\": container with ID starting with 807f3bdc59eb71b64fafb56ea89b9e1ff873c319c7ae0eedd31015c37103c542 not found: ID does not exist" containerID="807f3bdc59eb71b64fafb56ea89b9e1ff873c319c7ae0eedd31015c37103c542" Jul 26 15:10:53.155537 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:53.155487 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"807f3bdc59eb71b64fafb56ea89b9e1ff873c319c7ae0eedd31015c37103c542"} err="failed to get container status \"807f3bdc59eb71b64fafb56ea89b9e1ff873c319c7ae0eedd31015c37103c542\": rpc error: code = NotFound desc = could not find container \"807f3bdc59eb71b64fafb56ea89b9e1ff873c319c7ae0eedd31015c37103c542\": container with ID starting with 807f3bdc59eb71b64fafb56ea89b9e1ff873c319c7ae0eedd31015c37103c542 not found: ID does not exist" Jul 26 15:10:53.167896 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:53.167872 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5947968457-clxm2"] Jul 26 15:10:53.171176 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:53.171157 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5947968457-clxm2"] Jul 26 15:10:54.062972 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:54.062921 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94b38fca-3d2b-496c-acde-6e3ac99d76fa" path="/var/lib/kubelet/pods/94b38fca-3d2b-496c-acde-6e3ac99d76fa/volumes" Jul 26 15:10:56.161583 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:56.161554 2555 generic.go:356] "Generic (PLEG): container finished" podID="84fece67-b584-4229-800c-e74f02667c8d" containerID="db38f44262c37140954d48efcc5e815a80425878d71e43a40aaf74f1f8f2602e" exitCode=0 Jul 26 15:10:56.162020 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:56.161602 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-rxf4b" event={"ID":"84fece67-b584-4229-800c-e74f02667c8d","Type":"ContainerDied","Data":"db38f44262c37140954d48efcc5e815a80425878d71e43a40aaf74f1f8f2602e"} Jul 26 15:10:56.162020 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:56.161874 2555 scope.go:117] "RemoveContainer" containerID="db38f44262c37140954d48efcc5e815a80425878d71e43a40aaf74f1f8f2602e" Jul 26 15:10:57.166985 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:10:57.166956 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-rxf4b" event={"ID":"84fece67-b584-4229-800c-e74f02667c8d","Type":"ContainerStarted","Data":"a4be4f129d2fbda0bcfc951c245a8517d009f77a5715d183c517247555ca1bb2"} Jul 26 15:11:00.079665 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:00.079626 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6fd8bd796d-x5gc4" podUID="fdc52ca3-7c32-4a07-b286-12e826330cb4" containerName="console" containerID="cri-o://69c4a06bf6682fe94d2b1586f3e0731066280e8d15bc6a6509f3653fe32deccf" gracePeriod=15 Jul 26 15:11:00.375992 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:00.375971 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fd8bd796d-x5gc4" Jul 26 15:11:00.512438 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:00.512414 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fdc52ca3-7c32-4a07-b286-12e826330cb4-console-serving-cert\") pod \"fdc52ca3-7c32-4a07-b286-12e826330cb4\" (UID: \"fdc52ca3-7c32-4a07-b286-12e826330cb4\") " Jul 26 15:11:00.512617 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:00.512446 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fdc52ca3-7c32-4a07-b286-12e826330cb4-console-oauth-config\") pod \"fdc52ca3-7c32-4a07-b286-12e826330cb4\" (UID: \"fdc52ca3-7c32-4a07-b286-12e826330cb4\") " Jul 26 15:11:00.512617 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:00.512485 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fdc52ca3-7c32-4a07-b286-12e826330cb4-console-config\") pod \"fdc52ca3-7c32-4a07-b286-12e826330cb4\" (UID: \"fdc52ca3-7c32-4a07-b286-12e826330cb4\") " Jul 26 15:11:00.512617 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:00.512516 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fdc52ca3-7c32-4a07-b286-12e826330cb4-service-ca\") pod \"fdc52ca3-7c32-4a07-b286-12e826330cb4\" (UID: \"fdc52ca3-7c32-4a07-b286-12e826330cb4\") " Jul 26 15:11:00.512617 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:00.512531 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdc52ca3-7c32-4a07-b286-12e826330cb4-trusted-ca-bundle\") pod \"fdc52ca3-7c32-4a07-b286-12e826330cb4\" (UID: \"fdc52ca3-7c32-4a07-b286-12e826330cb4\") " Jul 26 15:11:00.512617 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:00.512548 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-84xcg\" (UniqueName: \"kubernetes.io/projected/fdc52ca3-7c32-4a07-b286-12e826330cb4-kube-api-access-84xcg\") pod \"fdc52ca3-7c32-4a07-b286-12e826330cb4\" (UID: \"fdc52ca3-7c32-4a07-b286-12e826330cb4\") " Jul 26 15:11:00.512617 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:00.512567 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fdc52ca3-7c32-4a07-b286-12e826330cb4-oauth-serving-cert\") pod \"fdc52ca3-7c32-4a07-b286-12e826330cb4\" (UID: \"fdc52ca3-7c32-4a07-b286-12e826330cb4\") " Jul 26 15:11:00.513028 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:00.512997 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdc52ca3-7c32-4a07-b286-12e826330cb4-service-ca" (OuterVolumeSpecName: "service-ca") pod "fdc52ca3-7c32-4a07-b286-12e826330cb4" (UID: "fdc52ca3-7c32-4a07-b286-12e826330cb4"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:11:00.513124 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:00.513020 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdc52ca3-7c32-4a07-b286-12e826330cb4-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "fdc52ca3-7c32-4a07-b286-12e826330cb4" (UID: "fdc52ca3-7c32-4a07-b286-12e826330cb4"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:11:00.513124 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:00.513004 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdc52ca3-7c32-4a07-b286-12e826330cb4-console-config" (OuterVolumeSpecName: "console-config") pod "fdc52ca3-7c32-4a07-b286-12e826330cb4" (UID: "fdc52ca3-7c32-4a07-b286-12e826330cb4"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:11:00.513124 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:00.513053 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdc52ca3-7c32-4a07-b286-12e826330cb4-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "fdc52ca3-7c32-4a07-b286-12e826330cb4" (UID: "fdc52ca3-7c32-4a07-b286-12e826330cb4"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:11:00.514552 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:00.514530 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdc52ca3-7c32-4a07-b286-12e826330cb4-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "fdc52ca3-7c32-4a07-b286-12e826330cb4" (UID: "fdc52ca3-7c32-4a07-b286-12e826330cb4"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 15:11:00.514637 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:00.514554 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdc52ca3-7c32-4a07-b286-12e826330cb4-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "fdc52ca3-7c32-4a07-b286-12e826330cb4" (UID: "fdc52ca3-7c32-4a07-b286-12e826330cb4"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 15:11:00.514637 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:00.514568 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdc52ca3-7c32-4a07-b286-12e826330cb4-kube-api-access-84xcg" (OuterVolumeSpecName: "kube-api-access-84xcg") pod "fdc52ca3-7c32-4a07-b286-12e826330cb4" (UID: "fdc52ca3-7c32-4a07-b286-12e826330cb4"). InnerVolumeSpecName "kube-api-access-84xcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:11:00.613662 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:00.613615 2555 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fdc52ca3-7c32-4a07-b286-12e826330cb4-console-serving-cert\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:11:00.613662 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:00.613635 2555 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fdc52ca3-7c32-4a07-b286-12e826330cb4-console-oauth-config\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:11:00.613662 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:00.613644 2555 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fdc52ca3-7c32-4a07-b286-12e826330cb4-console-config\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:11:00.613662 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:00.613652 2555 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fdc52ca3-7c32-4a07-b286-12e826330cb4-service-ca\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:11:00.613662 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:00.613660 2555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdc52ca3-7c32-4a07-b286-12e826330cb4-trusted-ca-bundle\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:11:00.613855 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:00.613667 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84xcg\" (UniqueName: \"kubernetes.io/projected/fdc52ca3-7c32-4a07-b286-12e826330cb4-kube-api-access-84xcg\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:11:00.613855 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:00.613676 2555 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fdc52ca3-7c32-4a07-b286-12e826330cb4-oauth-serving-cert\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:11:01.194900 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:01.194867 2555 generic.go:356] "Generic (PLEG): container finished" podID="fdc52ca3-7c32-4a07-b286-12e826330cb4" containerID="69c4a06bf6682fe94d2b1586f3e0731066280e8d15bc6a6509f3653fe32deccf" exitCode=0 Jul 26 15:11:01.195310 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:01.194906 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fd8bd796d-x5gc4" Jul 26 15:11:01.195310 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:01.194952 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fd8bd796d-x5gc4" event={"ID":"fdc52ca3-7c32-4a07-b286-12e826330cb4","Type":"ContainerDied","Data":"69c4a06bf6682fe94d2b1586f3e0731066280e8d15bc6a6509f3653fe32deccf"} Jul 26 15:11:01.195310 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:01.195004 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fd8bd796d-x5gc4" event={"ID":"fdc52ca3-7c32-4a07-b286-12e826330cb4","Type":"ContainerDied","Data":"9bc121bfc0f6b1b98881a2adaca725ee9baf7633cbb3d3f156189c227a10d89e"} Jul 26 15:11:01.195310 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:01.195025 2555 scope.go:117] "RemoveContainer" containerID="69c4a06bf6682fe94d2b1586f3e0731066280e8d15bc6a6509f3653fe32deccf" Jul 26 15:11:01.203765 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:01.203748 2555 scope.go:117] "RemoveContainer" containerID="69c4a06bf6682fe94d2b1586f3e0731066280e8d15bc6a6509f3653fe32deccf" Jul 26 15:11:01.204041 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:11:01.204021 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69c4a06bf6682fe94d2b1586f3e0731066280e8d15bc6a6509f3653fe32deccf\": container with ID starting with 69c4a06bf6682fe94d2b1586f3e0731066280e8d15bc6a6509f3653fe32deccf not found: ID does not exist" containerID="69c4a06bf6682fe94d2b1586f3e0731066280e8d15bc6a6509f3653fe32deccf" Jul 26 15:11:01.204105 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:01.204051 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69c4a06bf6682fe94d2b1586f3e0731066280e8d15bc6a6509f3653fe32deccf"} err="failed to get container status \"69c4a06bf6682fe94d2b1586f3e0731066280e8d15bc6a6509f3653fe32deccf\": rpc error: code = NotFound desc = could not find container \"69c4a06bf6682fe94d2b1586f3e0731066280e8d15bc6a6509f3653fe32deccf\": container with ID starting with 69c4a06bf6682fe94d2b1586f3e0731066280e8d15bc6a6509f3653fe32deccf not found: ID does not exist" Jul 26 15:11:01.215140 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:01.215121 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6fd8bd796d-x5gc4"] Jul 26 15:11:01.218159 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:01.218137 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6fd8bd796d-x5gc4"] Jul 26 15:11:02.062624 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:02.062590 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdc52ca3-7c32-4a07-b286-12e826330cb4" path="/var/lib/kubelet/pods/fdc52ca3-7c32-4a07-b286-12e826330cb4/volumes" Jul 26 15:11:06.215167 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:06.215140 2555 generic.go:356] "Generic (PLEG): container finished" podID="e5f83acc-8163-48d2-94de-bf52dd9c0e42" containerID="79ae2e51530aa1fa35b435214949599560254f12dcc765c64c8df97f32bce972" exitCode=0 Jul 26 15:11:06.215498 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:06.215208 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-qklsw" event={"ID":"e5f83acc-8163-48d2-94de-bf52dd9c0e42","Type":"ContainerDied","Data":"79ae2e51530aa1fa35b435214949599560254f12dcc765c64c8df97f32bce972"} Jul 26 15:11:06.215498 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:06.215488 2555 scope.go:117] "RemoveContainer" containerID="79ae2e51530aa1fa35b435214949599560254f12dcc765c64c8df97f32bce972" Jul 26 15:11:07.220824 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:07.220792 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-qklsw" event={"ID":"e5f83acc-8163-48d2-94de-bf52dd9c0e42","Type":"ContainerStarted","Data":"cf77dcd3436b686eb3e122b37e0b79bb550f0b64ccbcca715e109e4348109523"} Jul 26 15:11:12.722250 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:12.722166 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:11:12.773520 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:12.773492 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:11:26.574102 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:26.574064 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:11:26.676721 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:26.676688 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:11:27.174981 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:27.174945 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:27.193742 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:27.193717 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:27.308031 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:27.308008 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:33.852315 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:33.852271 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b6101bd-09a1-444f-ae02-50384ba66923-metrics-certs\") pod \"network-metrics-daemon-qlw9m\" (UID: \"1b6101bd-09a1-444f-ae02-50384ba66923\") " pod="openshift-multus/network-metrics-daemon-qlw9m" Jul 26 15:11:33.854511 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:33.854487 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b6101bd-09a1-444f-ae02-50384ba66923-metrics-certs\") pod \"network-metrics-daemon-qlw9m\" (UID: \"1b6101bd-09a1-444f-ae02-50384ba66923\") " pod="openshift-multus/network-metrics-daemon-qlw9m" Jul 26 15:11:33.862063 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:33.862040 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tgnb2\"" Jul 26 15:11:33.869743 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:33.869729 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlw9m" Jul 26 15:11:33.994824 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:33.994793 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qlw9m"] Jul 26 15:11:33.998416 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:11:33.998389 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b6101bd_09a1_444f_ae02_50384ba66923.slice/crio-645a5d6247703ade81800234b12be68395d6beb3e8713782349dfb67e64c90db WatchSource:0}: Error finding container 645a5d6247703ade81800234b12be68395d6beb3e8713782349dfb67e64c90db: Status 404 returned error can't find the container with id 645a5d6247703ade81800234b12be68395d6beb3e8713782349dfb67e64c90db Jul 26 15:11:34.322090 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:34.322056 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qlw9m" event={"ID":"1b6101bd-09a1-444f-ae02-50384ba66923","Type":"ContainerStarted","Data":"645a5d6247703ade81800234b12be68395d6beb3e8713782349dfb67e64c90db"} Jul 26 15:11:35.327550 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:35.327520 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qlw9m" event={"ID":"1b6101bd-09a1-444f-ae02-50384ba66923","Type":"ContainerStarted","Data":"2411aeeb7ded2bc2cbe44be5b40aec7a080ef8a289be2fe176a7cfc9c27a8a49"} Jul 26 15:11:35.327862 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:35.327567 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qlw9m" event={"ID":"1b6101bd-09a1-444f-ae02-50384ba66923","Type":"ContainerStarted","Data":"409d1a791d6b1a384fe4b7c09abc774b5b226ef84e409ccbb38a2a56336b88f6"} Jul 26 15:11:35.345684 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:35.345627 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-qlw9m" podStartSLOduration=374.480792782 podStartE2EDuration="6m15.345608645s" totalImagesPullingTime="864.815863ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:05:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:11:35.344506413 +0000 UTC m=+375.768353284" watchObservedRunningTime="2026-07-26 15:11:35.345608645 +0000 UTC m=+375.769455515" Jul 26 15:11:35.466733 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:35.466696 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:11:35.561198 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:35.561166 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:11:36.753093 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:36.753061 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:11:36.825943 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:36.825902 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:11:41.078315 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:41.078276 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:11:41.178678 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:41.178650 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:11:45.254006 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.253971 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 26 15:11:45.254408 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.254270 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1b26882d-ac13-400d-994f-e0842c802ce6" containerName="prometheus" containerID="cri-o://596937dc730f4594636465eb31af0b45b9f66538e8c96c9b46bf49e7baf77d7b" gracePeriod=600 Jul 26 15:11:45.254408 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.254290 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1b26882d-ac13-400d-994f-e0842c802ce6" containerName="kube-rbac-proxy" containerID="cri-o://dc709ca82a018b8095268adca9e42c5a9f9f6c1716454b219c967ad5889ece3c" gracePeriod=600 Jul 26 15:11:45.254408 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.254325 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1b26882d-ac13-400d-994f-e0842c802ce6" containerName="kube-rbac-proxy-thanos" containerID="cri-o://ef4c7483f93149eeb0bf8d5cdd7a5dd63434b0fb12c5e0884b39c5f2e0739094" gracePeriod=600 Jul 26 15:11:45.254408 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.254338 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1b26882d-ac13-400d-994f-e0842c802ce6" containerName="thanos-sidecar" containerID="cri-o://06e84755fc7e613d6358c376545ab50d8008597efeeccc6a70cec6373bd8e963" gracePeriod=600 Jul 26 15:11:45.254408 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.254349 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1b26882d-ac13-400d-994f-e0842c802ce6" containerName="config-reloader" containerID="cri-o://8fa8d6d6eb1829d572a141c32266299561f6f670a78398fc420bd7d3943e7382" gracePeriod=600 Jul 26 15:11:45.254637 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.254326 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1b26882d-ac13-400d-994f-e0842c802ce6" containerName="kube-rbac-proxy-web" containerID="cri-o://a4dbcb9bd26556e66f490648d941fe7ad3823fb86ecebe093fc77b00ca6c9078" gracePeriod=600 Jul 26 15:11:45.491446 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.491416 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:45.542777 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.542704 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b26882d-ac13-400d-994f-e0842c802ce6-configmap-kubelet-serving-ca-bundle\") pod \"1b26882d-ac13-400d-994f-e0842c802ce6\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " Jul 26 15:11:45.542777 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.542734 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1b26882d-ac13-400d-994f-e0842c802ce6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"1b26882d-ac13-400d-994f-e0842c802ce6\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " Jul 26 15:11:45.542777 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.542755 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1b26882d-ac13-400d-994f-e0842c802ce6-prometheus-k8s-rulefiles-0\") pod \"1b26882d-ac13-400d-994f-e0842c802ce6\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " Jul 26 15:11:45.542777 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.542770 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b26882d-ac13-400d-994f-e0842c802ce6-configmap-metrics-client-ca\") pod \"1b26882d-ac13-400d-994f-e0842c802ce6\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " Jul 26 15:11:45.543077 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.542795 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1b26882d-ac13-400d-994f-e0842c802ce6-prometheus-k8s-rulefiles-2\") pod \"1b26882d-ac13-400d-994f-e0842c802ce6\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " Jul 26 15:11:45.543077 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.542868 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1b26882d-ac13-400d-994f-e0842c802ce6-prometheus-k8s-db\") pod \"1b26882d-ac13-400d-994f-e0842c802ce6\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " Jul 26 15:11:45.543077 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.542919 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1b26882d-ac13-400d-994f-e0842c802ce6-secret-metrics-client-certs\") pod \"1b26882d-ac13-400d-994f-e0842c802ce6\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " Jul 26 15:11:45.543077 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.542956 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1b26882d-ac13-400d-994f-e0842c802ce6-secret-prometheus-k8s-tls\") pod \"1b26882d-ac13-400d-994f-e0842c802ce6\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " Jul 26 15:11:45.543077 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.542990 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1b26882d-ac13-400d-994f-e0842c802ce6-tls-assets\") pod \"1b26882d-ac13-400d-994f-e0842c802ce6\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " Jul 26 15:11:45.543077 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.543013 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1b26882d-ac13-400d-994f-e0842c802ce6-secret-grpc-tls\") pod \"1b26882d-ac13-400d-994f-e0842c802ce6\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " Jul 26 15:11:45.543077 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.543051 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1b26882d-ac13-400d-994f-e0842c802ce6-web-config\") pod \"1b26882d-ac13-400d-994f-e0842c802ce6\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " Jul 26 15:11:45.543410 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.543090 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b26882d-ac13-400d-994f-e0842c802ce6-configmap-serving-certs-ca-bundle\") pod \"1b26882d-ac13-400d-994f-e0842c802ce6\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " Jul 26 15:11:45.543410 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.543128 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b26882d-ac13-400d-994f-e0842c802ce6-prometheus-trusted-ca-bundle\") pod \"1b26882d-ac13-400d-994f-e0842c802ce6\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " Jul 26 15:11:45.543410 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.543136 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b26882d-ac13-400d-994f-e0842c802ce6-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "1b26882d-ac13-400d-994f-e0842c802ce6" (UID: "1b26882d-ac13-400d-994f-e0842c802ce6"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:11:45.543410 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.543159 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8wtm8\" (UniqueName: \"kubernetes.io/projected/1b26882d-ac13-400d-994f-e0842c802ce6-kube-api-access-8wtm8\") pod \"1b26882d-ac13-400d-994f-e0842c802ce6\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " Jul 26 15:11:45.543410 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.543190 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1b26882d-ac13-400d-994f-e0842c802ce6-prometheus-k8s-rulefiles-1\") pod \"1b26882d-ac13-400d-994f-e0842c802ce6\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " Jul 26 15:11:45.543410 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.543219 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1b26882d-ac13-400d-994f-e0842c802ce6-config\") pod \"1b26882d-ac13-400d-994f-e0842c802ce6\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " Jul 26 15:11:45.543410 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.543249 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1b26882d-ac13-400d-994f-e0842c802ce6-thanos-prometheus-http-client-file\") pod \"1b26882d-ac13-400d-994f-e0842c802ce6\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " Jul 26 15:11:45.543410 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.543298 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1b26882d-ac13-400d-994f-e0842c802ce6-config-out\") pod \"1b26882d-ac13-400d-994f-e0842c802ce6\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " Jul 26 15:11:45.543410 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.543322 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1b26882d-ac13-400d-994f-e0842c802ce6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"1b26882d-ac13-400d-994f-e0842c802ce6\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " Jul 26 15:11:45.543410 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.543351 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1b26882d-ac13-400d-994f-e0842c802ce6-secret-kube-rbac-proxy\") pod \"1b26882d-ac13-400d-994f-e0842c802ce6\" (UID: \"1b26882d-ac13-400d-994f-e0842c802ce6\") " Jul 26 15:11:45.545029 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.543186 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b26882d-ac13-400d-994f-e0842c802ce6-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "1b26882d-ac13-400d-994f-e0842c802ce6" (UID: "1b26882d-ac13-400d-994f-e0842c802ce6"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:11:45.545029 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.543199 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b26882d-ac13-400d-994f-e0842c802ce6-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "1b26882d-ac13-400d-994f-e0842c802ce6" (UID: "1b26882d-ac13-400d-994f-e0842c802ce6"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:11:45.545029 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.544025 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b26882d-ac13-400d-994f-e0842c802ce6-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "1b26882d-ac13-400d-994f-e0842c802ce6" (UID: "1b26882d-ac13-400d-994f-e0842c802ce6"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:11:45.545029 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.544833 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b26882d-ac13-400d-994f-e0842c802ce6-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "1b26882d-ac13-400d-994f-e0842c802ce6" (UID: "1b26882d-ac13-400d-994f-e0842c802ce6"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:11:45.545029 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.545021 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b26882d-ac13-400d-994f-e0842c802ce6-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "1b26882d-ac13-400d-994f-e0842c802ce6" (UID: "1b26882d-ac13-400d-994f-e0842c802ce6"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:11:45.545393 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.545334 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b26882d-ac13-400d-994f-e0842c802ce6-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "1b26882d-ac13-400d-994f-e0842c802ce6" (UID: "1b26882d-ac13-400d-994f-e0842c802ce6"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:11:45.545616 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.545594 2555 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b26882d-ac13-400d-994f-e0842c802ce6-prometheus-trusted-ca-bundle\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:11:45.545699 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.545621 2555 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1b26882d-ac13-400d-994f-e0842c802ce6-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:11:45.545699 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.545636 2555 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b26882d-ac13-400d-994f-e0842c802ce6-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:11:45.545699 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.545649 2555 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1b26882d-ac13-400d-994f-e0842c802ce6-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:11:45.545699 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.545662 2555 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b26882d-ac13-400d-994f-e0842c802ce6-configmap-metrics-client-ca\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:11:45.545699 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.545675 2555 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1b26882d-ac13-400d-994f-e0842c802ce6-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:11:45.546096 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.545701 2555 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b26882d-ac13-400d-994f-e0842c802ce6-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:11:45.546096 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.545782 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b26882d-ac13-400d-994f-e0842c802ce6-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "1b26882d-ac13-400d-994f-e0842c802ce6" (UID: "1b26882d-ac13-400d-994f-e0842c802ce6"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 15:11:45.546096 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.546041 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b26882d-ac13-400d-994f-e0842c802ce6-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "1b26882d-ac13-400d-994f-e0842c802ce6" (UID: "1b26882d-ac13-400d-994f-e0842c802ce6"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:11:45.546521 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.546491 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b26882d-ac13-400d-994f-e0842c802ce6-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "1b26882d-ac13-400d-994f-e0842c802ce6" (UID: "1b26882d-ac13-400d-994f-e0842c802ce6"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:11:45.546787 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.546753 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b26882d-ac13-400d-994f-e0842c802ce6-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "1b26882d-ac13-400d-994f-e0842c802ce6" (UID: "1b26882d-ac13-400d-994f-e0842c802ce6"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 15:11:45.547395 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.547319 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b26882d-ac13-400d-994f-e0842c802ce6-config" (OuterVolumeSpecName: "config") pod "1b26882d-ac13-400d-994f-e0842c802ce6" (UID: "1b26882d-ac13-400d-994f-e0842c802ce6"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 15:11:45.547395 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.547370 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b26882d-ac13-400d-994f-e0842c802ce6-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "1b26882d-ac13-400d-994f-e0842c802ce6" (UID: "1b26882d-ac13-400d-994f-e0842c802ce6"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 15:11:45.547537 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.547360 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b26882d-ac13-400d-994f-e0842c802ce6-kube-api-access-8wtm8" (OuterVolumeSpecName: "kube-api-access-8wtm8") pod "1b26882d-ac13-400d-994f-e0842c802ce6" (UID: "1b26882d-ac13-400d-994f-e0842c802ce6"). InnerVolumeSpecName "kube-api-access-8wtm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:11:45.547680 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.547653 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b26882d-ac13-400d-994f-e0842c802ce6-config-out" (OuterVolumeSpecName: "config-out") pod "1b26882d-ac13-400d-994f-e0842c802ce6" (UID: "1b26882d-ac13-400d-994f-e0842c802ce6"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:11:45.547758 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.547719 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b26882d-ac13-400d-994f-e0842c802ce6-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "1b26882d-ac13-400d-994f-e0842c802ce6" (UID: "1b26882d-ac13-400d-994f-e0842c802ce6"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 15:11:45.548405 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.548382 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b26882d-ac13-400d-994f-e0842c802ce6-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "1b26882d-ac13-400d-994f-e0842c802ce6" (UID: "1b26882d-ac13-400d-994f-e0842c802ce6"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 15:11:45.548970 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.548943 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b26882d-ac13-400d-994f-e0842c802ce6-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "1b26882d-ac13-400d-994f-e0842c802ce6" (UID: "1b26882d-ac13-400d-994f-e0842c802ce6"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 15:11:45.549241 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.549221 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b26882d-ac13-400d-994f-e0842c802ce6-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "1b26882d-ac13-400d-994f-e0842c802ce6" (UID: "1b26882d-ac13-400d-994f-e0842c802ce6"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 15:11:45.558039 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.558018 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b26882d-ac13-400d-994f-e0842c802ce6-web-config" (OuterVolumeSpecName: "web-config") pod "1b26882d-ac13-400d-994f-e0842c802ce6" (UID: "1b26882d-ac13-400d-994f-e0842c802ce6"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 15:11:45.646742 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.646719 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wtm8\" (UniqueName: \"kubernetes.io/projected/1b26882d-ac13-400d-994f-e0842c802ce6-kube-api-access-8wtm8\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:11:45.646742 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.646739 2555 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/1b26882d-ac13-400d-994f-e0842c802ce6-config\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:11:45.646892 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.646753 2555 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1b26882d-ac13-400d-994f-e0842c802ce6-thanos-prometheus-http-client-file\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:11:45.646892 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.646766 2555 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1b26882d-ac13-400d-994f-e0842c802ce6-config-out\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:11:45.646892 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.646778 2555 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1b26882d-ac13-400d-994f-e0842c802ce6-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:11:45.646892 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.646791 2555 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1b26882d-ac13-400d-994f-e0842c802ce6-secret-kube-rbac-proxy\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:11:45.646892 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.646805 2555 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1b26882d-ac13-400d-994f-e0842c802ce6-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:11:45.646892 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.646819 2555 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1b26882d-ac13-400d-994f-e0842c802ce6-prometheus-k8s-db\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:11:45.646892 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.646832 2555 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1b26882d-ac13-400d-994f-e0842c802ce6-secret-metrics-client-certs\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:11:45.646892 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.646844 2555 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1b26882d-ac13-400d-994f-e0842c802ce6-secret-prometheus-k8s-tls\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:11:45.646892 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.646858 2555 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1b26882d-ac13-400d-994f-e0842c802ce6-tls-assets\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:11:45.646892 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.646870 2555 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1b26882d-ac13-400d-994f-e0842c802ce6-secret-grpc-tls\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:11:45.646892 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:45.646882 2555 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1b26882d-ac13-400d-994f-e0842c802ce6-web-config\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:11:46.370147 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.370123 2555 generic.go:356] "Generic (PLEG): container finished" podID="1b26882d-ac13-400d-994f-e0842c802ce6" containerID="ef4c7483f93149eeb0bf8d5cdd7a5dd63434b0fb12c5e0884b39c5f2e0739094" exitCode=0 Jul 26 15:11:46.370147 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.370144 2555 generic.go:356] "Generic (PLEG): container finished" podID="1b26882d-ac13-400d-994f-e0842c802ce6" containerID="dc709ca82a018b8095268adca9e42c5a9f9f6c1716454b219c967ad5889ece3c" exitCode=0 Jul 26 15:11:46.370544 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.370156 2555 generic.go:356] "Generic (PLEG): container finished" podID="1b26882d-ac13-400d-994f-e0842c802ce6" containerID="a4dbcb9bd26556e66f490648d941fe7ad3823fb86ecebe093fc77b00ca6c9078" exitCode=0 Jul 26 15:11:46.370544 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.370165 2555 generic.go:356] "Generic (PLEG): container finished" podID="1b26882d-ac13-400d-994f-e0842c802ce6" containerID="06e84755fc7e613d6358c376545ab50d8008597efeeccc6a70cec6373bd8e963" exitCode=0 Jul 26 15:11:46.370544 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.370172 2555 generic.go:356] "Generic (PLEG): container finished" podID="1b26882d-ac13-400d-994f-e0842c802ce6" containerID="8fa8d6d6eb1829d572a141c32266299561f6f670a78398fc420bd7d3943e7382" exitCode=0 Jul 26 15:11:46.370544 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.370179 2555 generic.go:356] "Generic (PLEG): container finished" podID="1b26882d-ac13-400d-994f-e0842c802ce6" containerID="596937dc730f4594636465eb31af0b45b9f66538e8c96c9b46bf49e7baf77d7b" exitCode=0 Jul 26 15:11:46.370544 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.370186 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1b26882d-ac13-400d-994f-e0842c802ce6","Type":"ContainerDied","Data":"ef4c7483f93149eeb0bf8d5cdd7a5dd63434b0fb12c5e0884b39c5f2e0739094"} Jul 26 15:11:46.370544 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.370211 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:46.370544 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.370255 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1b26882d-ac13-400d-994f-e0842c802ce6","Type":"ContainerDied","Data":"dc709ca82a018b8095268adca9e42c5a9f9f6c1716454b219c967ad5889ece3c"} Jul 26 15:11:46.370544 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.370282 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1b26882d-ac13-400d-994f-e0842c802ce6","Type":"ContainerDied","Data":"a4dbcb9bd26556e66f490648d941fe7ad3823fb86ecebe093fc77b00ca6c9078"} Jul 26 15:11:46.370544 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.370307 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1b26882d-ac13-400d-994f-e0842c802ce6","Type":"ContainerDied","Data":"06e84755fc7e613d6358c376545ab50d8008597efeeccc6a70cec6373bd8e963"} Jul 26 15:11:46.370544 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.370330 2555 scope.go:117] "RemoveContainer" containerID="ef4c7483f93149eeb0bf8d5cdd7a5dd63434b0fb12c5e0884b39c5f2e0739094" Jul 26 15:11:46.370544 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.370333 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1b26882d-ac13-400d-994f-e0842c802ce6","Type":"ContainerDied","Data":"8fa8d6d6eb1829d572a141c32266299561f6f670a78398fc420bd7d3943e7382"} Jul 26 15:11:46.370544 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.370438 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1b26882d-ac13-400d-994f-e0842c802ce6","Type":"ContainerDied","Data":"596937dc730f4594636465eb31af0b45b9f66538e8c96c9b46bf49e7baf77d7b"} Jul 26 15:11:46.370544 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.370471 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1b26882d-ac13-400d-994f-e0842c802ce6","Type":"ContainerDied","Data":"88c96817f02d9c9cb8704ab735b60c4b4ba48d7ffe427bdb81535d437f7cb7e6"} Jul 26 15:11:46.377829 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.377813 2555 scope.go:117] "RemoveContainer" containerID="dc709ca82a018b8095268adca9e42c5a9f9f6c1716454b219c967ad5889ece3c" Jul 26 15:11:46.384228 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.384211 2555 scope.go:117] "RemoveContainer" containerID="a4dbcb9bd26556e66f490648d941fe7ad3823fb86ecebe093fc77b00ca6c9078" Jul 26 15:11:46.390312 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.390296 2555 scope.go:117] "RemoveContainer" containerID="06e84755fc7e613d6358c376545ab50d8008597efeeccc6a70cec6373bd8e963" Jul 26 15:11:46.396482 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.396466 2555 scope.go:117] "RemoveContainer" containerID="8fa8d6d6eb1829d572a141c32266299561f6f670a78398fc420bd7d3943e7382" Jul 26 15:11:46.402364 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.402347 2555 scope.go:117] "RemoveContainer" containerID="596937dc730f4594636465eb31af0b45b9f66538e8c96c9b46bf49e7baf77d7b" Jul 26 15:11:46.408881 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.408862 2555 scope.go:117] "RemoveContainer" containerID="9627574d0ae9b758d8471cb56fcd70cce70e91381e8f55a86ec4fa00e04b958b" Jul 26 15:11:46.410543 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.410523 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 26 15:11:46.414810 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.414791 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 26 15:11:46.416019 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.416005 2555 scope.go:117] "RemoveContainer" containerID="ef4c7483f93149eeb0bf8d5cdd7a5dd63434b0fb12c5e0884b39c5f2e0739094" Jul 26 15:11:46.416273 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:11:46.416253 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef4c7483f93149eeb0bf8d5cdd7a5dd63434b0fb12c5e0884b39c5f2e0739094\": container with ID starting with ef4c7483f93149eeb0bf8d5cdd7a5dd63434b0fb12c5e0884b39c5f2e0739094 not found: ID does not exist" containerID="ef4c7483f93149eeb0bf8d5cdd7a5dd63434b0fb12c5e0884b39c5f2e0739094" Jul 26 15:11:46.416318 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.416283 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef4c7483f93149eeb0bf8d5cdd7a5dd63434b0fb12c5e0884b39c5f2e0739094"} err="failed to get container status \"ef4c7483f93149eeb0bf8d5cdd7a5dd63434b0fb12c5e0884b39c5f2e0739094\": rpc error: code = NotFound desc = could not find container \"ef4c7483f93149eeb0bf8d5cdd7a5dd63434b0fb12c5e0884b39c5f2e0739094\": container with ID starting with ef4c7483f93149eeb0bf8d5cdd7a5dd63434b0fb12c5e0884b39c5f2e0739094 not found: ID does not exist" Jul 26 15:11:46.416318 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.416300 2555 scope.go:117] "RemoveContainer" containerID="dc709ca82a018b8095268adca9e42c5a9f9f6c1716454b219c967ad5889ece3c" Jul 26 15:11:46.416574 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:11:46.416553 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc709ca82a018b8095268adca9e42c5a9f9f6c1716454b219c967ad5889ece3c\": container with ID starting with dc709ca82a018b8095268adca9e42c5a9f9f6c1716454b219c967ad5889ece3c not found: ID does not exist" containerID="dc709ca82a018b8095268adca9e42c5a9f9f6c1716454b219c967ad5889ece3c" Jul 26 15:11:46.416640 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.416582 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc709ca82a018b8095268adca9e42c5a9f9f6c1716454b219c967ad5889ece3c"} err="failed to get container status \"dc709ca82a018b8095268adca9e42c5a9f9f6c1716454b219c967ad5889ece3c\": rpc error: code = NotFound desc = could not find container \"dc709ca82a018b8095268adca9e42c5a9f9f6c1716454b219c967ad5889ece3c\": container with ID starting with dc709ca82a018b8095268adca9e42c5a9f9f6c1716454b219c967ad5889ece3c not found: ID does not exist" Jul 26 15:11:46.416640 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.416603 2555 scope.go:117] "RemoveContainer" containerID="a4dbcb9bd26556e66f490648d941fe7ad3823fb86ecebe093fc77b00ca6c9078" Jul 26 15:11:46.416859 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:11:46.416843 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4dbcb9bd26556e66f490648d941fe7ad3823fb86ecebe093fc77b00ca6c9078\": container with ID starting with a4dbcb9bd26556e66f490648d941fe7ad3823fb86ecebe093fc77b00ca6c9078 not found: ID does not exist" containerID="a4dbcb9bd26556e66f490648d941fe7ad3823fb86ecebe093fc77b00ca6c9078" Jul 26 15:11:46.416898 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.416865 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4dbcb9bd26556e66f490648d941fe7ad3823fb86ecebe093fc77b00ca6c9078"} err="failed to get container status \"a4dbcb9bd26556e66f490648d941fe7ad3823fb86ecebe093fc77b00ca6c9078\": rpc error: code = NotFound desc = could not find container \"a4dbcb9bd26556e66f490648d941fe7ad3823fb86ecebe093fc77b00ca6c9078\": container with ID starting with a4dbcb9bd26556e66f490648d941fe7ad3823fb86ecebe093fc77b00ca6c9078 not found: ID does not exist" Jul 26 15:11:46.416898 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.416888 2555 scope.go:117] "RemoveContainer" containerID="06e84755fc7e613d6358c376545ab50d8008597efeeccc6a70cec6373bd8e963" Jul 26 15:11:46.417149 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:11:46.417131 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06e84755fc7e613d6358c376545ab50d8008597efeeccc6a70cec6373bd8e963\": container with ID starting with 06e84755fc7e613d6358c376545ab50d8008597efeeccc6a70cec6373bd8e963 not found: ID does not exist" containerID="06e84755fc7e613d6358c376545ab50d8008597efeeccc6a70cec6373bd8e963" Jul 26 15:11:46.417214 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.417156 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06e84755fc7e613d6358c376545ab50d8008597efeeccc6a70cec6373bd8e963"} err="failed to get container status \"06e84755fc7e613d6358c376545ab50d8008597efeeccc6a70cec6373bd8e963\": rpc error: code = NotFound desc = could not find container \"06e84755fc7e613d6358c376545ab50d8008597efeeccc6a70cec6373bd8e963\": container with ID starting with 06e84755fc7e613d6358c376545ab50d8008597efeeccc6a70cec6373bd8e963 not found: ID does not exist" Jul 26 15:11:46.417214 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.417176 2555 scope.go:117] "RemoveContainer" containerID="8fa8d6d6eb1829d572a141c32266299561f6f670a78398fc420bd7d3943e7382" Jul 26 15:11:46.417429 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:11:46.417413 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fa8d6d6eb1829d572a141c32266299561f6f670a78398fc420bd7d3943e7382\": container with ID starting with 8fa8d6d6eb1829d572a141c32266299561f6f670a78398fc420bd7d3943e7382 not found: ID does not exist" containerID="8fa8d6d6eb1829d572a141c32266299561f6f670a78398fc420bd7d3943e7382" Jul 26 15:11:46.417466 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.417433 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fa8d6d6eb1829d572a141c32266299561f6f670a78398fc420bd7d3943e7382"} err="failed to get container status \"8fa8d6d6eb1829d572a141c32266299561f6f670a78398fc420bd7d3943e7382\": rpc error: code = NotFound desc = could not find container \"8fa8d6d6eb1829d572a141c32266299561f6f670a78398fc420bd7d3943e7382\": container with ID starting with 8fa8d6d6eb1829d572a141c32266299561f6f670a78398fc420bd7d3943e7382 not found: ID does not exist" Jul 26 15:11:46.417466 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.417446 2555 scope.go:117] "RemoveContainer" containerID="596937dc730f4594636465eb31af0b45b9f66538e8c96c9b46bf49e7baf77d7b" Jul 26 15:11:46.417673 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:11:46.417657 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"596937dc730f4594636465eb31af0b45b9f66538e8c96c9b46bf49e7baf77d7b\": container with ID starting with 596937dc730f4594636465eb31af0b45b9f66538e8c96c9b46bf49e7baf77d7b not found: ID does not exist" containerID="596937dc730f4594636465eb31af0b45b9f66538e8c96c9b46bf49e7baf77d7b" Jul 26 15:11:46.417735 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.417680 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"596937dc730f4594636465eb31af0b45b9f66538e8c96c9b46bf49e7baf77d7b"} err="failed to get container status \"596937dc730f4594636465eb31af0b45b9f66538e8c96c9b46bf49e7baf77d7b\": rpc error: code = NotFound desc = could not find container \"596937dc730f4594636465eb31af0b45b9f66538e8c96c9b46bf49e7baf77d7b\": container with ID starting with 596937dc730f4594636465eb31af0b45b9f66538e8c96c9b46bf49e7baf77d7b not found: ID does not exist" Jul 26 15:11:46.417735 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.417698 2555 scope.go:117] "RemoveContainer" containerID="9627574d0ae9b758d8471cb56fcd70cce70e91381e8f55a86ec4fa00e04b958b" Jul 26 15:11:46.417958 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:11:46.417912 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9627574d0ae9b758d8471cb56fcd70cce70e91381e8f55a86ec4fa00e04b958b\": container with ID starting with 9627574d0ae9b758d8471cb56fcd70cce70e91381e8f55a86ec4fa00e04b958b not found: ID does not exist" containerID="9627574d0ae9b758d8471cb56fcd70cce70e91381e8f55a86ec4fa00e04b958b" Jul 26 15:11:46.418027 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.417998 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9627574d0ae9b758d8471cb56fcd70cce70e91381e8f55a86ec4fa00e04b958b"} err="failed to get container status \"9627574d0ae9b758d8471cb56fcd70cce70e91381e8f55a86ec4fa00e04b958b\": rpc error: code = NotFound desc = could not find container \"9627574d0ae9b758d8471cb56fcd70cce70e91381e8f55a86ec4fa00e04b958b\": container with ID starting with 9627574d0ae9b758d8471cb56fcd70cce70e91381e8f55a86ec4fa00e04b958b not found: ID does not exist" Jul 26 15:11:46.418027 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.418019 2555 scope.go:117] "RemoveContainer" containerID="ef4c7483f93149eeb0bf8d5cdd7a5dd63434b0fb12c5e0884b39c5f2e0739094" Jul 26 15:11:46.418272 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.418254 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef4c7483f93149eeb0bf8d5cdd7a5dd63434b0fb12c5e0884b39c5f2e0739094"} err="failed to get container status \"ef4c7483f93149eeb0bf8d5cdd7a5dd63434b0fb12c5e0884b39c5f2e0739094\": rpc error: code = NotFound desc = could not find container \"ef4c7483f93149eeb0bf8d5cdd7a5dd63434b0fb12c5e0884b39c5f2e0739094\": container with ID starting with ef4c7483f93149eeb0bf8d5cdd7a5dd63434b0fb12c5e0884b39c5f2e0739094 not found: ID does not exist" Jul 26 15:11:46.418272 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.418271 2555 scope.go:117] "RemoveContainer" containerID="dc709ca82a018b8095268adca9e42c5a9f9f6c1716454b219c967ad5889ece3c" Jul 26 15:11:46.418474 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.418459 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc709ca82a018b8095268adca9e42c5a9f9f6c1716454b219c967ad5889ece3c"} err="failed to get container status \"dc709ca82a018b8095268adca9e42c5a9f9f6c1716454b219c967ad5889ece3c\": rpc error: code = NotFound desc = could not find container \"dc709ca82a018b8095268adca9e42c5a9f9f6c1716454b219c967ad5889ece3c\": container with ID starting with dc709ca82a018b8095268adca9e42c5a9f9f6c1716454b219c967ad5889ece3c not found: ID does not exist" Jul 26 15:11:46.418519 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.418474 2555 scope.go:117] "RemoveContainer" containerID="a4dbcb9bd26556e66f490648d941fe7ad3823fb86ecebe093fc77b00ca6c9078" Jul 26 15:11:46.418681 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.418662 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4dbcb9bd26556e66f490648d941fe7ad3823fb86ecebe093fc77b00ca6c9078"} err="failed to get container status \"a4dbcb9bd26556e66f490648d941fe7ad3823fb86ecebe093fc77b00ca6c9078\": rpc error: code = NotFound desc = could not find container \"a4dbcb9bd26556e66f490648d941fe7ad3823fb86ecebe093fc77b00ca6c9078\": container with ID starting with a4dbcb9bd26556e66f490648d941fe7ad3823fb86ecebe093fc77b00ca6c9078 not found: ID does not exist" Jul 26 15:11:46.418731 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.418681 2555 scope.go:117] "RemoveContainer" containerID="06e84755fc7e613d6358c376545ab50d8008597efeeccc6a70cec6373bd8e963" Jul 26 15:11:46.418866 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.418850 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06e84755fc7e613d6358c376545ab50d8008597efeeccc6a70cec6373bd8e963"} err="failed to get container status \"06e84755fc7e613d6358c376545ab50d8008597efeeccc6a70cec6373bd8e963\": rpc error: code = NotFound desc = could not find container \"06e84755fc7e613d6358c376545ab50d8008597efeeccc6a70cec6373bd8e963\": container with ID starting with 06e84755fc7e613d6358c376545ab50d8008597efeeccc6a70cec6373bd8e963 not found: ID does not exist" Jul 26 15:11:46.418908 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.418866 2555 scope.go:117] "RemoveContainer" containerID="8fa8d6d6eb1829d572a141c32266299561f6f670a78398fc420bd7d3943e7382" Jul 26 15:11:46.419116 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.419093 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fa8d6d6eb1829d572a141c32266299561f6f670a78398fc420bd7d3943e7382"} err="failed to get container status \"8fa8d6d6eb1829d572a141c32266299561f6f670a78398fc420bd7d3943e7382\": rpc error: code = NotFound desc = could not find container \"8fa8d6d6eb1829d572a141c32266299561f6f670a78398fc420bd7d3943e7382\": container with ID starting with 8fa8d6d6eb1829d572a141c32266299561f6f670a78398fc420bd7d3943e7382 not found: ID does not exist" Jul 26 15:11:46.419116 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.419115 2555 scope.go:117] "RemoveContainer" containerID="596937dc730f4594636465eb31af0b45b9f66538e8c96c9b46bf49e7baf77d7b" Jul 26 15:11:46.419332 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.419315 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"596937dc730f4594636465eb31af0b45b9f66538e8c96c9b46bf49e7baf77d7b"} err="failed to get container status \"596937dc730f4594636465eb31af0b45b9f66538e8c96c9b46bf49e7baf77d7b\": rpc error: code = NotFound desc = could not find container \"596937dc730f4594636465eb31af0b45b9f66538e8c96c9b46bf49e7baf77d7b\": container with ID starting with 596937dc730f4594636465eb31af0b45b9f66538e8c96c9b46bf49e7baf77d7b not found: ID does not exist" Jul 26 15:11:46.419381 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.419333 2555 scope.go:117] "RemoveContainer" containerID="9627574d0ae9b758d8471cb56fcd70cce70e91381e8f55a86ec4fa00e04b958b" Jul 26 15:11:46.419528 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.419511 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9627574d0ae9b758d8471cb56fcd70cce70e91381e8f55a86ec4fa00e04b958b"} err="failed to get container status \"9627574d0ae9b758d8471cb56fcd70cce70e91381e8f55a86ec4fa00e04b958b\": rpc error: code = NotFound desc = could not find container \"9627574d0ae9b758d8471cb56fcd70cce70e91381e8f55a86ec4fa00e04b958b\": container with ID starting with 9627574d0ae9b758d8471cb56fcd70cce70e91381e8f55a86ec4fa00e04b958b not found: ID does not exist" Jul 26 15:11:46.419528 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.419527 2555 scope.go:117] "RemoveContainer" containerID="ef4c7483f93149eeb0bf8d5cdd7a5dd63434b0fb12c5e0884b39c5f2e0739094" Jul 26 15:11:46.419720 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.419705 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef4c7483f93149eeb0bf8d5cdd7a5dd63434b0fb12c5e0884b39c5f2e0739094"} err="failed to get container status \"ef4c7483f93149eeb0bf8d5cdd7a5dd63434b0fb12c5e0884b39c5f2e0739094\": rpc error: code = NotFound desc = could not find container \"ef4c7483f93149eeb0bf8d5cdd7a5dd63434b0fb12c5e0884b39c5f2e0739094\": container with ID starting with ef4c7483f93149eeb0bf8d5cdd7a5dd63434b0fb12c5e0884b39c5f2e0739094 not found: ID does not exist" Jul 26 15:11:46.419720 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.419718 2555 scope.go:117] "RemoveContainer" containerID="dc709ca82a018b8095268adca9e42c5a9f9f6c1716454b219c967ad5889ece3c" Jul 26 15:11:46.419895 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.419879 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc709ca82a018b8095268adca9e42c5a9f9f6c1716454b219c967ad5889ece3c"} err="failed to get container status \"dc709ca82a018b8095268adca9e42c5a9f9f6c1716454b219c967ad5889ece3c\": rpc error: code = NotFound desc = could not find container \"dc709ca82a018b8095268adca9e42c5a9f9f6c1716454b219c967ad5889ece3c\": container with ID starting with dc709ca82a018b8095268adca9e42c5a9f9f6c1716454b219c967ad5889ece3c not found: ID does not exist" Jul 26 15:11:46.419960 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.419895 2555 scope.go:117] "RemoveContainer" containerID="a4dbcb9bd26556e66f490648d941fe7ad3823fb86ecebe093fc77b00ca6c9078" Jul 26 15:11:46.420114 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.420080 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4dbcb9bd26556e66f490648d941fe7ad3823fb86ecebe093fc77b00ca6c9078"} err="failed to get container status \"a4dbcb9bd26556e66f490648d941fe7ad3823fb86ecebe093fc77b00ca6c9078\": rpc error: code = NotFound desc = could not find container \"a4dbcb9bd26556e66f490648d941fe7ad3823fb86ecebe093fc77b00ca6c9078\": container with ID starting with a4dbcb9bd26556e66f490648d941fe7ad3823fb86ecebe093fc77b00ca6c9078 not found: ID does not exist" Jul 26 15:11:46.420114 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.420115 2555 scope.go:117] "RemoveContainer" containerID="06e84755fc7e613d6358c376545ab50d8008597efeeccc6a70cec6373bd8e963" Jul 26 15:11:46.420375 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.420360 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06e84755fc7e613d6358c376545ab50d8008597efeeccc6a70cec6373bd8e963"} err="failed to get container status \"06e84755fc7e613d6358c376545ab50d8008597efeeccc6a70cec6373bd8e963\": rpc error: code = NotFound desc = could not find container \"06e84755fc7e613d6358c376545ab50d8008597efeeccc6a70cec6373bd8e963\": container with ID starting with 06e84755fc7e613d6358c376545ab50d8008597efeeccc6a70cec6373bd8e963 not found: ID does not exist" Jul 26 15:11:46.420413 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.420375 2555 scope.go:117] "RemoveContainer" containerID="8fa8d6d6eb1829d572a141c32266299561f6f670a78398fc420bd7d3943e7382" Jul 26 15:11:46.420556 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.420540 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fa8d6d6eb1829d572a141c32266299561f6f670a78398fc420bd7d3943e7382"} err="failed to get container status \"8fa8d6d6eb1829d572a141c32266299561f6f670a78398fc420bd7d3943e7382\": rpc error: code = NotFound desc = could not find container \"8fa8d6d6eb1829d572a141c32266299561f6f670a78398fc420bd7d3943e7382\": container with ID starting with 8fa8d6d6eb1829d572a141c32266299561f6f670a78398fc420bd7d3943e7382 not found: ID does not exist" Jul 26 15:11:46.420556 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.420555 2555 scope.go:117] "RemoveContainer" containerID="596937dc730f4594636465eb31af0b45b9f66538e8c96c9b46bf49e7baf77d7b" Jul 26 15:11:46.420785 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.420768 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"596937dc730f4594636465eb31af0b45b9f66538e8c96c9b46bf49e7baf77d7b"} err="failed to get container status \"596937dc730f4594636465eb31af0b45b9f66538e8c96c9b46bf49e7baf77d7b\": rpc error: code = NotFound desc = could not find container \"596937dc730f4594636465eb31af0b45b9f66538e8c96c9b46bf49e7baf77d7b\": container with ID starting with 596937dc730f4594636465eb31af0b45b9f66538e8c96c9b46bf49e7baf77d7b not found: ID does not exist" Jul 26 15:11:46.420830 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.420785 2555 scope.go:117] "RemoveContainer" containerID="9627574d0ae9b758d8471cb56fcd70cce70e91381e8f55a86ec4fa00e04b958b" Jul 26 15:11:46.420999 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.420985 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9627574d0ae9b758d8471cb56fcd70cce70e91381e8f55a86ec4fa00e04b958b"} err="failed to get container status \"9627574d0ae9b758d8471cb56fcd70cce70e91381e8f55a86ec4fa00e04b958b\": rpc error: code = NotFound desc = could not find container \"9627574d0ae9b758d8471cb56fcd70cce70e91381e8f55a86ec4fa00e04b958b\": container with ID starting with 9627574d0ae9b758d8471cb56fcd70cce70e91381e8f55a86ec4fa00e04b958b not found: ID does not exist" Jul 26 15:11:46.421052 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.420999 2555 scope.go:117] "RemoveContainer" containerID="ef4c7483f93149eeb0bf8d5cdd7a5dd63434b0fb12c5e0884b39c5f2e0739094" Jul 26 15:11:46.421200 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.421179 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef4c7483f93149eeb0bf8d5cdd7a5dd63434b0fb12c5e0884b39c5f2e0739094"} err="failed to get container status \"ef4c7483f93149eeb0bf8d5cdd7a5dd63434b0fb12c5e0884b39c5f2e0739094\": rpc error: code = NotFound desc = could not find container \"ef4c7483f93149eeb0bf8d5cdd7a5dd63434b0fb12c5e0884b39c5f2e0739094\": container with ID starting with ef4c7483f93149eeb0bf8d5cdd7a5dd63434b0fb12c5e0884b39c5f2e0739094 not found: ID does not exist" Jul 26 15:11:46.421246 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.421200 2555 scope.go:117] "RemoveContainer" containerID="dc709ca82a018b8095268adca9e42c5a9f9f6c1716454b219c967ad5889ece3c" Jul 26 15:11:46.421416 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.421401 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc709ca82a018b8095268adca9e42c5a9f9f6c1716454b219c967ad5889ece3c"} err="failed to get container status \"dc709ca82a018b8095268adca9e42c5a9f9f6c1716454b219c967ad5889ece3c\": rpc error: code = NotFound desc = could not find container \"dc709ca82a018b8095268adca9e42c5a9f9f6c1716454b219c967ad5889ece3c\": container with ID starting with dc709ca82a018b8095268adca9e42c5a9f9f6c1716454b219c967ad5889ece3c not found: ID does not exist" Jul 26 15:11:46.421454 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.421415 2555 scope.go:117] "RemoveContainer" containerID="a4dbcb9bd26556e66f490648d941fe7ad3823fb86ecebe093fc77b00ca6c9078" Jul 26 15:11:46.421615 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.421596 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4dbcb9bd26556e66f490648d941fe7ad3823fb86ecebe093fc77b00ca6c9078"} err="failed to get container status \"a4dbcb9bd26556e66f490648d941fe7ad3823fb86ecebe093fc77b00ca6c9078\": rpc error: code = NotFound desc = could not find container \"a4dbcb9bd26556e66f490648d941fe7ad3823fb86ecebe093fc77b00ca6c9078\": container with ID starting with a4dbcb9bd26556e66f490648d941fe7ad3823fb86ecebe093fc77b00ca6c9078 not found: ID does not exist" Jul 26 15:11:46.421651 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.421616 2555 scope.go:117] "RemoveContainer" containerID="06e84755fc7e613d6358c376545ab50d8008597efeeccc6a70cec6373bd8e963" Jul 26 15:11:46.421828 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.421801 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06e84755fc7e613d6358c376545ab50d8008597efeeccc6a70cec6373bd8e963"} err="failed to get container status \"06e84755fc7e613d6358c376545ab50d8008597efeeccc6a70cec6373bd8e963\": rpc error: code = NotFound desc = could not find container \"06e84755fc7e613d6358c376545ab50d8008597efeeccc6a70cec6373bd8e963\": container with ID starting with 06e84755fc7e613d6358c376545ab50d8008597efeeccc6a70cec6373bd8e963 not found: ID does not exist" Jul 26 15:11:46.421862 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.421832 2555 scope.go:117] "RemoveContainer" containerID="8fa8d6d6eb1829d572a141c32266299561f6f670a78398fc420bd7d3943e7382" Jul 26 15:11:46.422074 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.422056 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fa8d6d6eb1829d572a141c32266299561f6f670a78398fc420bd7d3943e7382"} err="failed to get container status \"8fa8d6d6eb1829d572a141c32266299561f6f670a78398fc420bd7d3943e7382\": rpc error: code = NotFound desc = could not find container \"8fa8d6d6eb1829d572a141c32266299561f6f670a78398fc420bd7d3943e7382\": container with ID starting with 8fa8d6d6eb1829d572a141c32266299561f6f670a78398fc420bd7d3943e7382 not found: ID does not exist" Jul 26 15:11:46.422119 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.422074 2555 scope.go:117] "RemoveContainer" containerID="596937dc730f4594636465eb31af0b45b9f66538e8c96c9b46bf49e7baf77d7b" Jul 26 15:11:46.422298 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.422282 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"596937dc730f4594636465eb31af0b45b9f66538e8c96c9b46bf49e7baf77d7b"} err="failed to get container status \"596937dc730f4594636465eb31af0b45b9f66538e8c96c9b46bf49e7baf77d7b\": rpc error: code = NotFound desc = could not find container \"596937dc730f4594636465eb31af0b45b9f66538e8c96c9b46bf49e7baf77d7b\": container with ID starting with 596937dc730f4594636465eb31af0b45b9f66538e8c96c9b46bf49e7baf77d7b not found: ID does not exist" Jul 26 15:11:46.422338 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.422299 2555 scope.go:117] "RemoveContainer" containerID="9627574d0ae9b758d8471cb56fcd70cce70e91381e8f55a86ec4fa00e04b958b" Jul 26 15:11:46.422505 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.422490 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9627574d0ae9b758d8471cb56fcd70cce70e91381e8f55a86ec4fa00e04b958b"} err="failed to get container status \"9627574d0ae9b758d8471cb56fcd70cce70e91381e8f55a86ec4fa00e04b958b\": rpc error: code = NotFound desc = could not find container \"9627574d0ae9b758d8471cb56fcd70cce70e91381e8f55a86ec4fa00e04b958b\": container with ID starting with 9627574d0ae9b758d8471cb56fcd70cce70e91381e8f55a86ec4fa00e04b958b not found: ID does not exist" Jul 26 15:11:46.422505 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.422504 2555 scope.go:117] "RemoveContainer" containerID="ef4c7483f93149eeb0bf8d5cdd7a5dd63434b0fb12c5e0884b39c5f2e0739094" Jul 26 15:11:46.422700 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.422683 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef4c7483f93149eeb0bf8d5cdd7a5dd63434b0fb12c5e0884b39c5f2e0739094"} err="failed to get container status \"ef4c7483f93149eeb0bf8d5cdd7a5dd63434b0fb12c5e0884b39c5f2e0739094\": rpc error: code = NotFound desc = could not find container \"ef4c7483f93149eeb0bf8d5cdd7a5dd63434b0fb12c5e0884b39c5f2e0739094\": container with ID starting with ef4c7483f93149eeb0bf8d5cdd7a5dd63434b0fb12c5e0884b39c5f2e0739094 not found: ID does not exist" Jul 26 15:11:46.422748 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.422700 2555 scope.go:117] "RemoveContainer" containerID="dc709ca82a018b8095268adca9e42c5a9f9f6c1716454b219c967ad5889ece3c" Jul 26 15:11:46.422889 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.422870 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc709ca82a018b8095268adca9e42c5a9f9f6c1716454b219c967ad5889ece3c"} err="failed to get container status \"dc709ca82a018b8095268adca9e42c5a9f9f6c1716454b219c967ad5889ece3c\": rpc error: code = NotFound desc = could not find container \"dc709ca82a018b8095268adca9e42c5a9f9f6c1716454b219c967ad5889ece3c\": container with ID starting with dc709ca82a018b8095268adca9e42c5a9f9f6c1716454b219c967ad5889ece3c not found: ID does not exist" Jul 26 15:11:46.422970 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.422890 2555 scope.go:117] "RemoveContainer" containerID="a4dbcb9bd26556e66f490648d941fe7ad3823fb86ecebe093fc77b00ca6c9078" Jul 26 15:11:46.423122 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.423106 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4dbcb9bd26556e66f490648d941fe7ad3823fb86ecebe093fc77b00ca6c9078"} err="failed to get container status \"a4dbcb9bd26556e66f490648d941fe7ad3823fb86ecebe093fc77b00ca6c9078\": rpc error: code = NotFound desc = could not find container \"a4dbcb9bd26556e66f490648d941fe7ad3823fb86ecebe093fc77b00ca6c9078\": container with ID starting with a4dbcb9bd26556e66f490648d941fe7ad3823fb86ecebe093fc77b00ca6c9078 not found: ID does not exist" Jul 26 15:11:46.423191 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.423122 2555 scope.go:117] "RemoveContainer" containerID="06e84755fc7e613d6358c376545ab50d8008597efeeccc6a70cec6373bd8e963" Jul 26 15:11:46.423343 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.423322 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06e84755fc7e613d6358c376545ab50d8008597efeeccc6a70cec6373bd8e963"} err="failed to get container status \"06e84755fc7e613d6358c376545ab50d8008597efeeccc6a70cec6373bd8e963\": rpc error: code = NotFound desc = could not find container \"06e84755fc7e613d6358c376545ab50d8008597efeeccc6a70cec6373bd8e963\": container with ID starting with 06e84755fc7e613d6358c376545ab50d8008597efeeccc6a70cec6373bd8e963 not found: ID does not exist" Jul 26 15:11:46.423394 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.423344 2555 scope.go:117] "RemoveContainer" containerID="8fa8d6d6eb1829d572a141c32266299561f6f670a78398fc420bd7d3943e7382" Jul 26 15:11:46.423554 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.423538 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fa8d6d6eb1829d572a141c32266299561f6f670a78398fc420bd7d3943e7382"} err="failed to get container status \"8fa8d6d6eb1829d572a141c32266299561f6f670a78398fc420bd7d3943e7382\": rpc error: code = NotFound desc = could not find container \"8fa8d6d6eb1829d572a141c32266299561f6f670a78398fc420bd7d3943e7382\": container with ID starting with 8fa8d6d6eb1829d572a141c32266299561f6f670a78398fc420bd7d3943e7382 not found: ID does not exist" Jul 26 15:11:46.423611 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.423555 2555 scope.go:117] "RemoveContainer" containerID="596937dc730f4594636465eb31af0b45b9f66538e8c96c9b46bf49e7baf77d7b" Jul 26 15:11:46.423747 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.423732 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"596937dc730f4594636465eb31af0b45b9f66538e8c96c9b46bf49e7baf77d7b"} err="failed to get container status \"596937dc730f4594636465eb31af0b45b9f66538e8c96c9b46bf49e7baf77d7b\": rpc error: code = NotFound desc = could not find container \"596937dc730f4594636465eb31af0b45b9f66538e8c96c9b46bf49e7baf77d7b\": container with ID starting with 596937dc730f4594636465eb31af0b45b9f66538e8c96c9b46bf49e7baf77d7b not found: ID does not exist" Jul 26 15:11:46.423892 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.423747 2555 scope.go:117] "RemoveContainer" containerID="9627574d0ae9b758d8471cb56fcd70cce70e91381e8f55a86ec4fa00e04b958b" Jul 26 15:11:46.423967 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.423924 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9627574d0ae9b758d8471cb56fcd70cce70e91381e8f55a86ec4fa00e04b958b"} err="failed to get container status \"9627574d0ae9b758d8471cb56fcd70cce70e91381e8f55a86ec4fa00e04b958b\": rpc error: code = NotFound desc = could not find container \"9627574d0ae9b758d8471cb56fcd70cce70e91381e8f55a86ec4fa00e04b958b\": container with ID starting with 9627574d0ae9b758d8471cb56fcd70cce70e91381e8f55a86ec4fa00e04b958b not found: ID does not exist" Jul 26 15:11:46.424021 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.423966 2555 scope.go:117] "RemoveContainer" containerID="ef4c7483f93149eeb0bf8d5cdd7a5dd63434b0fb12c5e0884b39c5f2e0739094" Jul 26 15:11:46.424184 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.424162 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef4c7483f93149eeb0bf8d5cdd7a5dd63434b0fb12c5e0884b39c5f2e0739094"} err="failed to get container status \"ef4c7483f93149eeb0bf8d5cdd7a5dd63434b0fb12c5e0884b39c5f2e0739094\": rpc error: code = NotFound desc = could not find container \"ef4c7483f93149eeb0bf8d5cdd7a5dd63434b0fb12c5e0884b39c5f2e0739094\": container with ID starting with ef4c7483f93149eeb0bf8d5cdd7a5dd63434b0fb12c5e0884b39c5f2e0739094 not found: ID does not exist" Jul 26 15:11:46.424234 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.424187 2555 scope.go:117] "RemoveContainer" containerID="dc709ca82a018b8095268adca9e42c5a9f9f6c1716454b219c967ad5889ece3c" Jul 26 15:11:46.424382 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.424360 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc709ca82a018b8095268adca9e42c5a9f9f6c1716454b219c967ad5889ece3c"} err="failed to get container status \"dc709ca82a018b8095268adca9e42c5a9f9f6c1716454b219c967ad5889ece3c\": rpc error: code = NotFound desc = could not find container \"dc709ca82a018b8095268adca9e42c5a9f9f6c1716454b219c967ad5889ece3c\": container with ID starting with dc709ca82a018b8095268adca9e42c5a9f9f6c1716454b219c967ad5889ece3c not found: ID does not exist" Jul 26 15:11:46.424382 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.424381 2555 scope.go:117] "RemoveContainer" containerID="a4dbcb9bd26556e66f490648d941fe7ad3823fb86ecebe093fc77b00ca6c9078" Jul 26 15:11:46.424573 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.424557 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4dbcb9bd26556e66f490648d941fe7ad3823fb86ecebe093fc77b00ca6c9078"} err="failed to get container status \"a4dbcb9bd26556e66f490648d941fe7ad3823fb86ecebe093fc77b00ca6c9078\": rpc error: code = NotFound desc = could not find container \"a4dbcb9bd26556e66f490648d941fe7ad3823fb86ecebe093fc77b00ca6c9078\": container with ID starting with a4dbcb9bd26556e66f490648d941fe7ad3823fb86ecebe093fc77b00ca6c9078 not found: ID does not exist" Jul 26 15:11:46.424612 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.424574 2555 scope.go:117] "RemoveContainer" containerID="06e84755fc7e613d6358c376545ab50d8008597efeeccc6a70cec6373bd8e963" Jul 26 15:11:46.424784 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.424761 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06e84755fc7e613d6358c376545ab50d8008597efeeccc6a70cec6373bd8e963"} err="failed to get container status \"06e84755fc7e613d6358c376545ab50d8008597efeeccc6a70cec6373bd8e963\": rpc error: code = NotFound desc = could not find container \"06e84755fc7e613d6358c376545ab50d8008597efeeccc6a70cec6373bd8e963\": container with ID starting with 06e84755fc7e613d6358c376545ab50d8008597efeeccc6a70cec6373bd8e963 not found: ID does not exist" Jul 26 15:11:46.424841 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.424788 2555 scope.go:117] "RemoveContainer" containerID="8fa8d6d6eb1829d572a141c32266299561f6f670a78398fc420bd7d3943e7382" Jul 26 15:11:46.425051 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.425032 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fa8d6d6eb1829d572a141c32266299561f6f670a78398fc420bd7d3943e7382"} err="failed to get container status \"8fa8d6d6eb1829d572a141c32266299561f6f670a78398fc420bd7d3943e7382\": rpc error: code = NotFound desc = could not find container \"8fa8d6d6eb1829d572a141c32266299561f6f670a78398fc420bd7d3943e7382\": container with ID starting with 8fa8d6d6eb1829d572a141c32266299561f6f670a78398fc420bd7d3943e7382 not found: ID does not exist" Jul 26 15:11:46.425103 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.425051 2555 scope.go:117] "RemoveContainer" containerID="596937dc730f4594636465eb31af0b45b9f66538e8c96c9b46bf49e7baf77d7b" Jul 26 15:11:46.425268 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.425253 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"596937dc730f4594636465eb31af0b45b9f66538e8c96c9b46bf49e7baf77d7b"} err="failed to get container status \"596937dc730f4594636465eb31af0b45b9f66538e8c96c9b46bf49e7baf77d7b\": rpc error: code = NotFound desc = could not find container \"596937dc730f4594636465eb31af0b45b9f66538e8c96c9b46bf49e7baf77d7b\": container with ID starting with 596937dc730f4594636465eb31af0b45b9f66538e8c96c9b46bf49e7baf77d7b not found: ID does not exist" Jul 26 15:11:46.425304 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.425269 2555 scope.go:117] "RemoveContainer" containerID="9627574d0ae9b758d8471cb56fcd70cce70e91381e8f55a86ec4fa00e04b958b" Jul 26 15:11:46.425454 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.425439 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9627574d0ae9b758d8471cb56fcd70cce70e91381e8f55a86ec4fa00e04b958b"} err="failed to get container status \"9627574d0ae9b758d8471cb56fcd70cce70e91381e8f55a86ec4fa00e04b958b\": rpc error: code = NotFound desc = could not find container \"9627574d0ae9b758d8471cb56fcd70cce70e91381e8f55a86ec4fa00e04b958b\": container with ID starting with 9627574d0ae9b758d8471cb56fcd70cce70e91381e8f55a86ec4fa00e04b958b not found: ID does not exist" Jul 26 15:11:46.467245 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.467219 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 26 15:11:46.467494 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.467483 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b26882d-ac13-400d-994f-e0842c802ce6" containerName="init-config-reloader" Jul 26 15:11:46.467529 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.467496 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b26882d-ac13-400d-994f-e0842c802ce6" containerName="init-config-reloader" Jul 26 15:11:46.467529 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.467506 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b26882d-ac13-400d-994f-e0842c802ce6" containerName="thanos-sidecar" Jul 26 15:11:46.467529 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.467511 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b26882d-ac13-400d-994f-e0842c802ce6" containerName="thanos-sidecar" Jul 26 15:11:46.467529 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.467523 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94b38fca-3d2b-496c-acde-6e3ac99d76fa" containerName="console" Jul 26 15:11:46.467529 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.467528 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="94b38fca-3d2b-496c-acde-6e3ac99d76fa" containerName="console" Jul 26 15:11:46.467678 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.467535 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdc52ca3-7c32-4a07-b286-12e826330cb4" containerName="console" Jul 26 15:11:46.467678 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.467540 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdc52ca3-7c32-4a07-b286-12e826330cb4" containerName="console" Jul 26 15:11:46.467678 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.467545 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b26882d-ac13-400d-994f-e0842c802ce6" containerName="config-reloader" Jul 26 15:11:46.467678 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.467550 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b26882d-ac13-400d-994f-e0842c802ce6" containerName="config-reloader" Jul 26 15:11:46.467678 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.467558 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b26882d-ac13-400d-994f-e0842c802ce6" containerName="kube-rbac-proxy-web" Jul 26 15:11:46.467678 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.467564 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b26882d-ac13-400d-994f-e0842c802ce6" containerName="kube-rbac-proxy-web" Jul 26 15:11:46.467678 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.467569 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b26882d-ac13-400d-994f-e0842c802ce6" containerName="kube-rbac-proxy-thanos" Jul 26 15:11:46.467678 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.467574 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b26882d-ac13-400d-994f-e0842c802ce6" containerName="kube-rbac-proxy-thanos" Jul 26 15:11:46.467678 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.467581 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b26882d-ac13-400d-994f-e0842c802ce6" containerName="prometheus" Jul 26 15:11:46.467678 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.467586 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b26882d-ac13-400d-994f-e0842c802ce6" containerName="prometheus" Jul 26 15:11:46.467678 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.467593 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b26882d-ac13-400d-994f-e0842c802ce6" containerName="kube-rbac-proxy" Jul 26 15:11:46.467678 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.467598 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b26882d-ac13-400d-994f-e0842c802ce6" containerName="kube-rbac-proxy" Jul 26 15:11:46.467678 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.467639 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b26882d-ac13-400d-994f-e0842c802ce6" containerName="config-reloader" Jul 26 15:11:46.467678 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.467648 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b26882d-ac13-400d-994f-e0842c802ce6" containerName="kube-rbac-proxy" Jul 26 15:11:46.467678 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.467654 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b26882d-ac13-400d-994f-e0842c802ce6" containerName="kube-rbac-proxy-thanos" Jul 26 15:11:46.467678 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.467661 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b26882d-ac13-400d-994f-e0842c802ce6" containerName="kube-rbac-proxy-web" Jul 26 15:11:46.467678 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.467667 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b26882d-ac13-400d-994f-e0842c802ce6" containerName="prometheus" Jul 26 15:11:46.467678 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.467672 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdc52ca3-7c32-4a07-b286-12e826330cb4" containerName="console" Jul 26 15:11:46.467678 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.467676 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="94b38fca-3d2b-496c-acde-6e3ac99d76fa" containerName="console" Jul 26 15:11:46.467678 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.467681 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b26882d-ac13-400d-994f-e0842c802ce6" containerName="thanos-sidecar" Jul 26 15:11:46.469694 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.469680 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:46.472902 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.472882 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 26 15:11:46.473040 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.472917 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-cvu6liautt05l\"" Jul 26 15:11:46.473227 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.473203 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 26 15:11:46.473333 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.473239 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 26 15:11:46.473333 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.473258 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 26 15:11:46.473488 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.473473 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 26 15:11:46.473556 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.473533 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 26 15:11:46.473637 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.473620 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 26 15:11:46.473715 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.473673 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 26 15:11:46.473986 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.473971 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 26 15:11:46.474310 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.474291 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 26 15:11:46.474387 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.474337 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 26 15:11:46.474445 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.474424 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 26 15:11:46.474487 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.474424 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-xwp8l\"" Jul 26 15:11:46.475059 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.475043 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 26 15:11:46.478546 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.478525 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 26 15:11:46.481765 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.481749 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 26 15:11:46.491842 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.491822 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 26 15:11:46.552812 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.552793 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/082e4a33-5a14-4366-b955-fe623586d569-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"082e4a33-5a14-4366-b955-fe623586d569\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:46.552916 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.552821 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/082e4a33-5a14-4366-b955-fe623586d569-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"082e4a33-5a14-4366-b955-fe623586d569\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:46.552916 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.552837 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/082e4a33-5a14-4366-b955-fe623586d569-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"082e4a33-5a14-4366-b955-fe623586d569\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:46.552916 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.552855 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/082e4a33-5a14-4366-b955-fe623586d569-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"082e4a33-5a14-4366-b955-fe623586d569\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:46.552916 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.552872 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/082e4a33-5a14-4366-b955-fe623586d569-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"082e4a33-5a14-4366-b955-fe623586d569\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:46.552916 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.552888 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/082e4a33-5a14-4366-b955-fe623586d569-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"082e4a33-5a14-4366-b955-fe623586d569\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:46.552916 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.552907 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/082e4a33-5a14-4366-b955-fe623586d569-config\") pod \"prometheus-k8s-0\" (UID: \"082e4a33-5a14-4366-b955-fe623586d569\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:46.553148 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.552921 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/082e4a33-5a14-4366-b955-fe623586d569-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"082e4a33-5a14-4366-b955-fe623586d569\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:46.553148 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.552969 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/082e4a33-5a14-4366-b955-fe623586d569-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"082e4a33-5a14-4366-b955-fe623586d569\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:46.553148 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.553004 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/082e4a33-5a14-4366-b955-fe623586d569-config-out\") pod \"prometheus-k8s-0\" (UID: \"082e4a33-5a14-4366-b955-fe623586d569\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:46.553148 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.553018 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/082e4a33-5a14-4366-b955-fe623586d569-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"082e4a33-5a14-4366-b955-fe623586d569\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:46.553148 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.553072 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/082e4a33-5a14-4366-b955-fe623586d569-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"082e4a33-5a14-4366-b955-fe623586d569\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:46.553148 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.553108 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/082e4a33-5a14-4366-b955-fe623586d569-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"082e4a33-5a14-4366-b955-fe623586d569\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:46.553148 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.553136 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/082e4a33-5a14-4366-b955-fe623586d569-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"082e4a33-5a14-4366-b955-fe623586d569\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:46.553373 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.553154 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/082e4a33-5a14-4366-b955-fe623586d569-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"082e4a33-5a14-4366-b955-fe623586d569\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:46.553373 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.553171 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/082e4a33-5a14-4366-b955-fe623586d569-web-config\") pod \"prometheus-k8s-0\" (UID: \"082e4a33-5a14-4366-b955-fe623586d569\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:46.553373 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.553190 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9l2d\" (UniqueName: \"kubernetes.io/projected/082e4a33-5a14-4366-b955-fe623586d569-kube-api-access-d9l2d\") pod \"prometheus-k8s-0\" (UID: \"082e4a33-5a14-4366-b955-fe623586d569\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:46.553373 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.553212 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/082e4a33-5a14-4366-b955-fe623586d569-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"082e4a33-5a14-4366-b955-fe623586d569\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:46.553373 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.553231 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/082e4a33-5a14-4366-b955-fe623586d569-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"082e4a33-5a14-4366-b955-fe623586d569\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:46.553373 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.553246 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/082e4a33-5a14-4366-b955-fe623586d569-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"082e4a33-5a14-4366-b955-fe623586d569\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:46.653698 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.653646 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/082e4a33-5a14-4366-b955-fe623586d569-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"082e4a33-5a14-4366-b955-fe623586d569\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:46.653698 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.653675 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/082e4a33-5a14-4366-b955-fe623586d569-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"082e4a33-5a14-4366-b955-fe623586d569\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:46.653698 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.653695 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/082e4a33-5a14-4366-b955-fe623586d569-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"082e4a33-5a14-4366-b955-fe623586d569\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:46.653895 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.653709 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/082e4a33-5a14-4366-b955-fe623586d569-web-config\") pod \"prometheus-k8s-0\" (UID: \"082e4a33-5a14-4366-b955-fe623586d569\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:46.653895 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.653760 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9l2d\" (UniqueName: \"kubernetes.io/projected/082e4a33-5a14-4366-b955-fe623586d569-kube-api-access-d9l2d\") pod \"prometheus-k8s-0\" (UID: \"082e4a33-5a14-4366-b955-fe623586d569\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:46.653895 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.653796 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/082e4a33-5a14-4366-b955-fe623586d569-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"082e4a33-5a14-4366-b955-fe623586d569\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:46.654057 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.653911 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/082e4a33-5a14-4366-b955-fe623586d569-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"082e4a33-5a14-4366-b955-fe623586d569\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:46.654057 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.653963 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/082e4a33-5a14-4366-b955-fe623586d569-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"082e4a33-5a14-4366-b955-fe623586d569\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:46.654057 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.654001 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/082e4a33-5a14-4366-b955-fe623586d569-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"082e4a33-5a14-4366-b955-fe623586d569\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:46.654057 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.654020 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/082e4a33-5a14-4366-b955-fe623586d569-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"082e4a33-5a14-4366-b955-fe623586d569\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:46.654057 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.654035 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/082e4a33-5a14-4366-b955-fe623586d569-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"082e4a33-5a14-4366-b955-fe623586d569\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:46.654057 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.654057 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/082e4a33-5a14-4366-b955-fe623586d569-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"082e4a33-5a14-4366-b955-fe623586d569\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:46.654418 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.654389 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/082e4a33-5a14-4366-b955-fe623586d569-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"082e4a33-5a14-4366-b955-fe623586d569\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:46.654534 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.654431 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/082e4a33-5a14-4366-b955-fe623586d569-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"082e4a33-5a14-4366-b955-fe623586d569\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:46.654534 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.654431 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/082e4a33-5a14-4366-b955-fe623586d569-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"082e4a33-5a14-4366-b955-fe623586d569\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:46.654534 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.654457 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/082e4a33-5a14-4366-b955-fe623586d569-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"082e4a33-5a14-4366-b955-fe623586d569\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:46.654534 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.654494 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/082e4a33-5a14-4366-b955-fe623586d569-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"082e4a33-5a14-4366-b955-fe623586d569\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:46.654734 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.654535 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/082e4a33-5a14-4366-b955-fe623586d569-config\") pod \"prometheus-k8s-0\" (UID: \"082e4a33-5a14-4366-b955-fe623586d569\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:46.654734 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.654564 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/082e4a33-5a14-4366-b955-fe623586d569-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"082e4a33-5a14-4366-b955-fe623586d569\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:46.654734 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.654596 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/082e4a33-5a14-4366-b955-fe623586d569-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"082e4a33-5a14-4366-b955-fe623586d569\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:46.654734 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.654633 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/082e4a33-5a14-4366-b955-fe623586d569-config-out\") pod \"prometheus-k8s-0\" (UID: \"082e4a33-5a14-4366-b955-fe623586d569\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:46.654734 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.654659 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/082e4a33-5a14-4366-b955-fe623586d569-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"082e4a33-5a14-4366-b955-fe623586d569\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:46.654734 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.654697 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/082e4a33-5a14-4366-b955-fe623586d569-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"082e4a33-5a14-4366-b955-fe623586d569\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:46.655380 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.655060 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/082e4a33-5a14-4366-b955-fe623586d569-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"082e4a33-5a14-4366-b955-fe623586d569\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:46.655380 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.655285 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/082e4a33-5a14-4366-b955-fe623586d569-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"082e4a33-5a14-4366-b955-fe623586d569\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:46.655550 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.655441 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/082e4a33-5a14-4366-b955-fe623586d569-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"082e4a33-5a14-4366-b955-fe623586d569\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:46.656115 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.656091 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/082e4a33-5a14-4366-b955-fe623586d569-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"082e4a33-5a14-4366-b955-fe623586d569\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:46.657079 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.657035 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/082e4a33-5a14-4366-b955-fe623586d569-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"082e4a33-5a14-4366-b955-fe623586d569\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:46.657427 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.657405 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/082e4a33-5a14-4366-b955-fe623586d569-web-config\") pod \"prometheus-k8s-0\" (UID: \"082e4a33-5a14-4366-b955-fe623586d569\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:46.658286 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.658257 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/082e4a33-5a14-4366-b955-fe623586d569-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"082e4a33-5a14-4366-b955-fe623586d569\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:46.658458 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.658431 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/082e4a33-5a14-4366-b955-fe623586d569-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"082e4a33-5a14-4366-b955-fe623586d569\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:46.658557 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.658533 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/082e4a33-5a14-4366-b955-fe623586d569-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"082e4a33-5a14-4366-b955-fe623586d569\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:46.658753 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.658716 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/082e4a33-5a14-4366-b955-fe623586d569-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"082e4a33-5a14-4366-b955-fe623586d569\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:46.658907 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.658882 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/082e4a33-5a14-4366-b955-fe623586d569-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"082e4a33-5a14-4366-b955-fe623586d569\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:46.659103 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.659085 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/082e4a33-5a14-4366-b955-fe623586d569-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"082e4a33-5a14-4366-b955-fe623586d569\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:46.659501 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.659481 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/082e4a33-5a14-4366-b955-fe623586d569-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"082e4a33-5a14-4366-b955-fe623586d569\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:46.659639 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.659622 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/082e4a33-5a14-4366-b955-fe623586d569-config\") pod \"prometheus-k8s-0\" (UID: \"082e4a33-5a14-4366-b955-fe623586d569\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:46.659742 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.659721 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/082e4a33-5a14-4366-b955-fe623586d569-config-out\") pod \"prometheus-k8s-0\" (UID: \"082e4a33-5a14-4366-b955-fe623586d569\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:46.660388 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.660365 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/082e4a33-5a14-4366-b955-fe623586d569-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"082e4a33-5a14-4366-b955-fe623586d569\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:46.661843 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.661824 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9l2d\" (UniqueName: \"kubernetes.io/projected/082e4a33-5a14-4366-b955-fe623586d569-kube-api-access-d9l2d\") pod \"prometheus-k8s-0\" (UID: \"082e4a33-5a14-4366-b955-fe623586d569\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:46.779158 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.779132 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:46.912403 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:46.912216 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 26 15:11:46.917110 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:11:46.915243 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod082e4a33_5a14_4366_b955_fe623586d569.slice/crio-4b1766d1c56547c0a19c787d87193d1eadcad277e85155078e390dc16c80fe4c WatchSource:0}: Error finding container 4b1766d1c56547c0a19c787d87193d1eadcad277e85155078e390dc16c80fe4c: Status 404 returned error can't find the container with id 4b1766d1c56547c0a19c787d87193d1eadcad277e85155078e390dc16c80fe4c Jul 26 15:11:47.375895 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:47.375862 2555 generic.go:356] "Generic (PLEG): container finished" podID="082e4a33-5a14-4366-b955-fe623586d569" containerID="79cfed56b234b4ba6d0a11c5550d6d70a435b8b3cd8a179e74cbe7c44e38ae4a" exitCode=0 Jul 26 15:11:47.376337 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:47.375921 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"082e4a33-5a14-4366-b955-fe623586d569","Type":"ContainerDied","Data":"79cfed56b234b4ba6d0a11c5550d6d70a435b8b3cd8a179e74cbe7c44e38ae4a"} Jul 26 15:11:47.376337 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:47.376007 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"082e4a33-5a14-4366-b955-fe623586d569","Type":"ContainerStarted","Data":"4b1766d1c56547c0a19c787d87193d1eadcad277e85155078e390dc16c80fe4c"} Jul 26 15:11:48.063422 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:48.063348 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b26882d-ac13-400d-994f-e0842c802ce6" path="/var/lib/kubelet/pods/1b26882d-ac13-400d-994f-e0842c802ce6/volumes" Jul 26 15:11:48.383756 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:48.383727 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"082e4a33-5a14-4366-b955-fe623586d569","Type":"ContainerStarted","Data":"27e91cb666f1109aab9f8f1b727bff8c9dd1ae54f73b54c9793826a11a4b968e"} Jul 26 15:11:48.384134 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:48.383789 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"082e4a33-5a14-4366-b955-fe623586d569","Type":"ContainerStarted","Data":"ec3b3ea8d1ec11b1b7fd5196cc9d1fd37e961d2673a7e4e596b58dfae7d3467a"} Jul 26 15:11:48.384134 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:48.383815 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"082e4a33-5a14-4366-b955-fe623586d569","Type":"ContainerStarted","Data":"8716ae75a83a400e09cf412eb4ca533f40a628ca1c70e8acc2f2957dc9f1076b"} Jul 26 15:11:48.384134 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:48.383839 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"082e4a33-5a14-4366-b955-fe623586d569","Type":"ContainerStarted","Data":"be1a85bbf2319028b19b1a7bbaa8376e810423129fccaaabf3a86ec10df87ad9"} Jul 26 15:11:48.384134 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:48.383862 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"082e4a33-5a14-4366-b955-fe623586d569","Type":"ContainerStarted","Data":"b9b8aaa7db99fe457e7e729fff25a9856016de58690bd833de30d30a3600367d"} Jul 26 15:11:48.384134 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:48.383883 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"082e4a33-5a14-4366-b955-fe623586d569","Type":"ContainerStarted","Data":"1990a726637b3f4263dc143c2d433d6a051384a7265ff33c1f44eece61071058"} Jul 26 15:11:48.419413 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:48.419357 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.404146147 podStartE2EDuration="2.419341123s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.194976ms" isStatefulPod=false podCreationTimestamp="2026-07-26 15:11:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:11:48.414050977 +0000 UTC m=+388.837897833" watchObservedRunningTime="2026-07-26 15:11:48.419341123 +0000 UTC m=+388.843187993" Jul 26 15:11:48.598690 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:48.598655 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:11:48.670290 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:48.670214 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:11:49.781179 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:49.781140 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:11:49.849807 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:49.849780 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:11:51.779521 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:51.779496 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:11:57.736551 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:57.736512 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:11:57.839349 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:11:57.839321 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:12:00.669950 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:12:00.669902 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74b5bd7f57-nj64n"] Jul 26 15:12:00.784199 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:12:00.784172 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:12:00.846644 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:12:00.846615 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:12:00.902402 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:12:00.902373 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:12:25.688277 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:12:25.688233 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-74b5bd7f57-nj64n" podUID="2d56de0a-bf2b-4547-a697-e63aa9b109c7" containerName="console" containerID="cri-o://46861c726ed4d0d19e840d0cc76e7356f9fdf5a1b4422b4dcf75bf1a6503cfc1" gracePeriod=15 Jul 26 15:12:25.920285 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:12:25.920263 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74b5bd7f57-nj64n" Jul 26 15:12:26.050210 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:12:26.050132 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xwksb\" (UniqueName: \"kubernetes.io/projected/2d56de0a-bf2b-4547-a697-e63aa9b109c7-kube-api-access-xwksb\") pod \"2d56de0a-bf2b-4547-a697-e63aa9b109c7\" (UID: \"2d56de0a-bf2b-4547-a697-e63aa9b109c7\") " Jul 26 15:12:26.050210 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:12:26.050190 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2d56de0a-bf2b-4547-a697-e63aa9b109c7-service-ca\") pod \"2d56de0a-bf2b-4547-a697-e63aa9b109c7\" (UID: \"2d56de0a-bf2b-4547-a697-e63aa9b109c7\") " Jul 26 15:12:26.050423 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:12:26.050222 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d56de0a-bf2b-4547-a697-e63aa9b109c7-trusted-ca-bundle\") pod \"2d56de0a-bf2b-4547-a697-e63aa9b109c7\" (UID: \"2d56de0a-bf2b-4547-a697-e63aa9b109c7\") " Jul 26 15:12:26.050423 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:12:26.050269 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2d56de0a-bf2b-4547-a697-e63aa9b109c7-console-serving-cert\") pod \"2d56de0a-bf2b-4547-a697-e63aa9b109c7\" (UID: \"2d56de0a-bf2b-4547-a697-e63aa9b109c7\") " Jul 26 15:12:26.050423 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:12:26.050291 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2d56de0a-bf2b-4547-a697-e63aa9b109c7-console-config\") pod \"2d56de0a-bf2b-4547-a697-e63aa9b109c7\" (UID: \"2d56de0a-bf2b-4547-a697-e63aa9b109c7\") " Jul 26 15:12:26.050423 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:12:26.050311 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2d56de0a-bf2b-4547-a697-e63aa9b109c7-oauth-serving-cert\") pod \"2d56de0a-bf2b-4547-a697-e63aa9b109c7\" (UID: \"2d56de0a-bf2b-4547-a697-e63aa9b109c7\") " Jul 26 15:12:26.050423 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:12:26.050337 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2d56de0a-bf2b-4547-a697-e63aa9b109c7-console-oauth-config\") pod \"2d56de0a-bf2b-4547-a697-e63aa9b109c7\" (UID: \"2d56de0a-bf2b-4547-a697-e63aa9b109c7\") " Jul 26 15:12:26.050705 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:12:26.050678 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d56de0a-bf2b-4547-a697-e63aa9b109c7-service-ca" (OuterVolumeSpecName: "service-ca") pod "2d56de0a-bf2b-4547-a697-e63aa9b109c7" (UID: "2d56de0a-bf2b-4547-a697-e63aa9b109c7"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:12:26.050705 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:12:26.050688 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d56de0a-bf2b-4547-a697-e63aa9b109c7-console-config" (OuterVolumeSpecName: "console-config") pod "2d56de0a-bf2b-4547-a697-e63aa9b109c7" (UID: "2d56de0a-bf2b-4547-a697-e63aa9b109c7"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:12:26.050817 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:12:26.050682 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d56de0a-bf2b-4547-a697-e63aa9b109c7-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "2d56de0a-bf2b-4547-a697-e63aa9b109c7" (UID: "2d56de0a-bf2b-4547-a697-e63aa9b109c7"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:12:26.050817 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:12:26.050726 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d56de0a-bf2b-4547-a697-e63aa9b109c7-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2d56de0a-bf2b-4547-a697-e63aa9b109c7" (UID: "2d56de0a-bf2b-4547-a697-e63aa9b109c7"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:12:26.052439 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:12:26.052415 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d56de0a-bf2b-4547-a697-e63aa9b109c7-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2d56de0a-bf2b-4547-a697-e63aa9b109c7" (UID: "2d56de0a-bf2b-4547-a697-e63aa9b109c7"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 15:12:26.052717 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:12:26.052694 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d56de0a-bf2b-4547-a697-e63aa9b109c7-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2d56de0a-bf2b-4547-a697-e63aa9b109c7" (UID: "2d56de0a-bf2b-4547-a697-e63aa9b109c7"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 15:12:26.052814 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:12:26.052724 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d56de0a-bf2b-4547-a697-e63aa9b109c7-kube-api-access-xwksb" (OuterVolumeSpecName: "kube-api-access-xwksb") pod "2d56de0a-bf2b-4547-a697-e63aa9b109c7" (UID: "2d56de0a-bf2b-4547-a697-e63aa9b109c7"). InnerVolumeSpecName "kube-api-access-xwksb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:12:26.151507 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:12:26.151486 2555 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2d56de0a-bf2b-4547-a697-e63aa9b109c7-console-serving-cert\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:12:26.151507 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:12:26.151506 2555 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2d56de0a-bf2b-4547-a697-e63aa9b109c7-console-config\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:12:26.151630 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:12:26.151516 2555 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2d56de0a-bf2b-4547-a697-e63aa9b109c7-oauth-serving-cert\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:12:26.151630 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:12:26.151526 2555 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2d56de0a-bf2b-4547-a697-e63aa9b109c7-console-oauth-config\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:12:26.151630 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:12:26.151534 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwksb\" (UniqueName: \"kubernetes.io/projected/2d56de0a-bf2b-4547-a697-e63aa9b109c7-kube-api-access-xwksb\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:12:26.151630 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:12:26.151546 2555 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2d56de0a-bf2b-4547-a697-e63aa9b109c7-service-ca\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:12:26.151630 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:12:26.151558 2555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d56de0a-bf2b-4547-a697-e63aa9b109c7-trusted-ca-bundle\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:12:26.530867 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:12:26.530838 2555 generic.go:356] "Generic (PLEG): container finished" podID="2d56de0a-bf2b-4547-a697-e63aa9b109c7" containerID="46861c726ed4d0d19e840d0cc76e7356f9fdf5a1b4422b4dcf75bf1a6503cfc1" exitCode=0 Jul 26 15:12:26.531023 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:12:26.530888 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74b5bd7f57-nj64n" Jul 26 15:12:26.531023 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:12:26.530919 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74b5bd7f57-nj64n" event={"ID":"2d56de0a-bf2b-4547-a697-e63aa9b109c7","Type":"ContainerDied","Data":"46861c726ed4d0d19e840d0cc76e7356f9fdf5a1b4422b4dcf75bf1a6503cfc1"} Jul 26 15:12:26.531023 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:12:26.530995 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74b5bd7f57-nj64n" event={"ID":"2d56de0a-bf2b-4547-a697-e63aa9b109c7","Type":"ContainerDied","Data":"02b733230904fd564aad2b34ae1f3c181394f1314c3bf07082a6c2faed3744f2"} Jul 26 15:12:26.531023 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:12:26.531018 2555 scope.go:117] "RemoveContainer" containerID="46861c726ed4d0d19e840d0cc76e7356f9fdf5a1b4422b4dcf75bf1a6503cfc1" Jul 26 15:12:26.539389 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:12:26.539369 2555 scope.go:117] "RemoveContainer" containerID="46861c726ed4d0d19e840d0cc76e7356f9fdf5a1b4422b4dcf75bf1a6503cfc1" Jul 26 15:12:26.539686 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:12:26.539661 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46861c726ed4d0d19e840d0cc76e7356f9fdf5a1b4422b4dcf75bf1a6503cfc1\": container with ID starting with 46861c726ed4d0d19e840d0cc76e7356f9fdf5a1b4422b4dcf75bf1a6503cfc1 not found: ID does not exist" containerID="46861c726ed4d0d19e840d0cc76e7356f9fdf5a1b4422b4dcf75bf1a6503cfc1" Jul 26 15:12:26.539783 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:12:26.539691 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46861c726ed4d0d19e840d0cc76e7356f9fdf5a1b4422b4dcf75bf1a6503cfc1"} err="failed to get container status \"46861c726ed4d0d19e840d0cc76e7356f9fdf5a1b4422b4dcf75bf1a6503cfc1\": rpc error: code = NotFound desc = could not find container \"46861c726ed4d0d19e840d0cc76e7356f9fdf5a1b4422b4dcf75bf1a6503cfc1\": container with ID starting with 46861c726ed4d0d19e840d0cc76e7356f9fdf5a1b4422b4dcf75bf1a6503cfc1 not found: ID does not exist" Jul 26 15:12:26.548031 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:12:26.548009 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74b5bd7f57-nj64n"] Jul 26 15:12:26.553051 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:12:26.553032 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-74b5bd7f57-nj64n"] Jul 26 15:12:27.444186 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:12:27.444155 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:12:27.544337 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:12:27.544310 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:12:28.062690 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:12:28.062661 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d56de0a-bf2b-4547-a697-e63aa9b109c7" path="/var/lib/kubelet/pods/2d56de0a-bf2b-4547-a697-e63aa9b109c7/volumes" Jul 26 15:12:30.198156 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:12:30.198122 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:12:46.780149 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:12:46.780063 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:12:46.795071 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:12:46.795045 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:12:47.621909 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:12:47.621881 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 26 15:12:49.299611 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:12:49.299573 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:12:50.095448 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:12:50.095412 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:12:52.941102 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:12:52.941070 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:12:53.034965 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:12:53.034914 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:12:53.405947 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:12:53.405897 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:12:53.503529 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:12:53.503498 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:12:55.216777 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:12:55.216737 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:12:55.331024 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:12:55.330993 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:12:55.450599 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:12:55.450565 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:12:55.547608 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:12:55.547541 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:12:57.808353 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:12:57.808324 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:12:58.197785 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:12:58.197750 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:12:59.495271 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:12:59.495227 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:12:59.595773 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:12:59.595743 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:13:00.537060 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:00.537024 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:13:04.990911 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:04.990875 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:13:05.410314 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:05.410280 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:13:09.909924 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:09.909884 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:13:10.007842 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:10.007810 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:13:15.132110 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:15.132070 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:13:15.232229 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:15.232187 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:13:17.423822 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:17.423784 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:13:31.512366 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:31.512330 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:13:31.608653 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:31.608623 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:13:32.122060 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:32.122026 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:13:32.333255 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:32.333185 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:13:32.434734 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:32.434667 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:13:32.479328 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:32.479305 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5c55b8954b-vbslq"] Jul 26 15:13:32.479567 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:32.479555 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d56de0a-bf2b-4547-a697-e63aa9b109c7" containerName="console" Jul 26 15:13:32.479606 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:32.479568 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d56de0a-bf2b-4547-a697-e63aa9b109c7" containerName="console" Jul 26 15:13:32.479642 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:32.479629 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d56de0a-bf2b-4547-a697-e63aa9b109c7" containerName="console" Jul 26 15:13:32.482804 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:32.482788 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5c55b8954b-vbslq" Jul 26 15:13:32.486507 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:32.486487 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 26 15:13:32.486805 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:32.486792 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-7bw26\"" Jul 26 15:13:32.487451 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:32.487436 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 26 15:13:32.487498 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:32.487471 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 26 15:13:32.487498 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:32.487482 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 26 15:13:32.499675 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:32.499656 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5c55b8954b-vbslq"] Jul 26 15:13:32.602865 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:32.602840 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:13:32.627118 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:32.627098 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g5k4\" (UniqueName: \"kubernetes.io/projected/08867a8f-2cc2-493d-b5b0-2c06659d50db-kube-api-access-5g5k4\") pod \"opendatahub-operator-controller-manager-5c55b8954b-vbslq\" (UID: \"08867a8f-2cc2-493d-b5b0-2c06659d50db\") " pod="opendatahub/opendatahub-operator-controller-manager-5c55b8954b-vbslq" Jul 26 15:13:32.627204 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:32.627126 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08867a8f-2cc2-493d-b5b0-2c06659d50db-tmp\") pod \"opendatahub-operator-controller-manager-5c55b8954b-vbslq\" (UID: \"08867a8f-2cc2-493d-b5b0-2c06659d50db\") " pod="opendatahub/opendatahub-operator-controller-manager-5c55b8954b-vbslq" Jul 26 15:13:32.627204 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:32.627153 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/08867a8f-2cc2-493d-b5b0-2c06659d50db-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c55b8954b-vbslq\" (UID: \"08867a8f-2cc2-493d-b5b0-2c06659d50db\") " pod="opendatahub/opendatahub-operator-controller-manager-5c55b8954b-vbslq" Jul 26 15:13:32.627204 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:32.627178 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/08867a8f-2cc2-493d-b5b0-2c06659d50db-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c55b8954b-vbslq\" (UID: \"08867a8f-2cc2-493d-b5b0-2c06659d50db\") " pod="opendatahub/opendatahub-operator-controller-manager-5c55b8954b-vbslq" Jul 26 15:13:32.627312 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:32.627227 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/08867a8f-2cc2-493d-b5b0-2c06659d50db-manifests\") pod \"opendatahub-operator-controller-manager-5c55b8954b-vbslq\" (UID: \"08867a8f-2cc2-493d-b5b0-2c06659d50db\") " pod="opendatahub/opendatahub-operator-controller-manager-5c55b8954b-vbslq" Jul 26 15:13:32.700737 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:32.700683 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:13:32.727632 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:32.727609 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g5k4\" (UniqueName: \"kubernetes.io/projected/08867a8f-2cc2-493d-b5b0-2c06659d50db-kube-api-access-5g5k4\") pod \"opendatahub-operator-controller-manager-5c55b8954b-vbslq\" (UID: \"08867a8f-2cc2-493d-b5b0-2c06659d50db\") " pod="opendatahub/opendatahub-operator-controller-manager-5c55b8954b-vbslq" Jul 26 15:13:32.727699 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:32.727636 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08867a8f-2cc2-493d-b5b0-2c06659d50db-tmp\") pod \"opendatahub-operator-controller-manager-5c55b8954b-vbslq\" (UID: \"08867a8f-2cc2-493d-b5b0-2c06659d50db\") " pod="opendatahub/opendatahub-operator-controller-manager-5c55b8954b-vbslq" Jul 26 15:13:32.727699 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:32.727662 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/08867a8f-2cc2-493d-b5b0-2c06659d50db-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c55b8954b-vbslq\" (UID: \"08867a8f-2cc2-493d-b5b0-2c06659d50db\") " pod="opendatahub/opendatahub-operator-controller-manager-5c55b8954b-vbslq" Jul 26 15:13:32.727699 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:32.727679 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/08867a8f-2cc2-493d-b5b0-2c06659d50db-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c55b8954b-vbslq\" (UID: \"08867a8f-2cc2-493d-b5b0-2c06659d50db\") " pod="opendatahub/opendatahub-operator-controller-manager-5c55b8954b-vbslq" Jul 26 15:13:32.727816 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:32.727701 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/08867a8f-2cc2-493d-b5b0-2c06659d50db-manifests\") pod \"opendatahub-operator-controller-manager-5c55b8954b-vbslq\" (UID: \"08867a8f-2cc2-493d-b5b0-2c06659d50db\") " pod="opendatahub/opendatahub-operator-controller-manager-5c55b8954b-vbslq" Jul 26 15:13:32.728150 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:32.728124 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/08867a8f-2cc2-493d-b5b0-2c06659d50db-manifests\") pod \"opendatahub-operator-controller-manager-5c55b8954b-vbslq\" (UID: \"08867a8f-2cc2-493d-b5b0-2c06659d50db\") " pod="opendatahub/opendatahub-operator-controller-manager-5c55b8954b-vbslq" Jul 26 15:13:32.729924 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:32.729900 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08867a8f-2cc2-493d-b5b0-2c06659d50db-tmp\") pod \"opendatahub-operator-controller-manager-5c55b8954b-vbslq\" (UID: \"08867a8f-2cc2-493d-b5b0-2c06659d50db\") " pod="opendatahub/opendatahub-operator-controller-manager-5c55b8954b-vbslq" Jul 26 15:13:32.730015 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:32.729999 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/08867a8f-2cc2-493d-b5b0-2c06659d50db-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c55b8954b-vbslq\" (UID: \"08867a8f-2cc2-493d-b5b0-2c06659d50db\") " pod="opendatahub/opendatahub-operator-controller-manager-5c55b8954b-vbslq" Jul 26 15:13:32.730090 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:32.730075 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/08867a8f-2cc2-493d-b5b0-2c06659d50db-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c55b8954b-vbslq\" (UID: \"08867a8f-2cc2-493d-b5b0-2c06659d50db\") " pod="opendatahub/opendatahub-operator-controller-manager-5c55b8954b-vbslq" Jul 26 15:13:32.735691 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:32.735669 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g5k4\" (UniqueName: \"kubernetes.io/projected/08867a8f-2cc2-493d-b5b0-2c06659d50db-kube-api-access-5g5k4\") pod \"opendatahub-operator-controller-manager-5c55b8954b-vbslq\" (UID: \"08867a8f-2cc2-493d-b5b0-2c06659d50db\") " pod="opendatahub/opendatahub-operator-controller-manager-5c55b8954b-vbslq" Jul 26 15:13:32.792418 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:32.792400 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5c55b8954b-vbslq" Jul 26 15:13:32.797457 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:32.797430 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:13:32.898718 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:32.898688 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:13:32.927798 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:32.927778 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5c55b8954b-vbslq"] Jul 26 15:13:32.930550 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:13:32.930521 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08867a8f_2cc2_493d_b5b0_2c06659d50db.slice/crio-70e314cea096dfa5a4d2358cf87d5e0e9b81e3e4d03775569c43273a3291e5ae WatchSource:0}: Error finding container 70e314cea096dfa5a4d2358cf87d5e0e9b81e3e4d03775569c43273a3291e5ae: Status 404 returned error can't find the container with id 70e314cea096dfa5a4d2358cf87d5e0e9b81e3e4d03775569c43273a3291e5ae Jul 26 15:13:32.998867 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:32.998817 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:13:33.099790 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:33.099765 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:13:33.775016 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:33.774966 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c55b8954b-vbslq" event={"ID":"08867a8f-2cc2-493d-b5b0-2c06659d50db","Type":"ContainerStarted","Data":"70e314cea096dfa5a4d2358cf87d5e0e9b81e3e4d03775569c43273a3291e5ae"} Jul 26 15:13:36.790228 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:36.790196 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c55b8954b-vbslq" event={"ID":"08867a8f-2cc2-493d-b5b0-2c06659d50db","Type":"ContainerStarted","Data":"4ba8097ec2579d4eb16ffd9c07ff5e39942366517b46a907c291e8ef13985aed"} Jul 26 15:13:37.795695 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:37.795663 2555 generic.go:356] "Generic (PLEG): container finished" podID="08867a8f-2cc2-493d-b5b0-2c06659d50db" containerID="4ba8097ec2579d4eb16ffd9c07ff5e39942366517b46a907c291e8ef13985aed" exitCode=0 Jul 26 15:13:37.795695 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:37.795699 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c55b8954b-vbslq" event={"ID":"08867a8f-2cc2-493d-b5b0-2c06659d50db","Type":"ContainerDied","Data":"4ba8097ec2579d4eb16ffd9c07ff5e39942366517b46a907c291e8ef13985aed"} Jul 26 15:13:38.801772 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:38.801741 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c55b8954b-vbslq" event={"ID":"08867a8f-2cc2-493d-b5b0-2c06659d50db","Type":"ContainerStarted","Data":"14a4c45317d2282f7bc16592e5addec4781174ca6de9ff4034f2de1baf0982c1"} Jul 26 15:13:38.802161 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:38.801856 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5c55b8954b-vbslq" Jul 26 15:13:38.822414 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:38.822375 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5c55b8954b-vbslq" podStartSLOduration=2.365361223 podStartE2EDuration="6.822363197s" totalImagesPullingTime="3.481647271s" totalInitContainerRuntime="975.354703ms" isStatefulPod=false podCreationTimestamp="2026-07-26 15:13:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:13:38.821990698 +0000 UTC m=+499.245837571" watchObservedRunningTime="2026-07-26 15:13:38.822363197 +0000 UTC m=+499.246210067" Jul 26 15:13:41.654830 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:41.654794 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:13:41.756838 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:41.756804 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:13:43.593446 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:43.593408 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:13:43.649826 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:43.649796 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:13:49.807766 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:49.807733 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5c55b8954b-vbslq" Jul 26 15:13:49.927849 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:49.927816 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-694b8c44c9-bq52h"] Jul 26 15:13:49.933683 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:49.933660 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-694b8c44c9-bq52h" Jul 26 15:13:49.935850 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:49.935826 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 26 15:13:49.935850 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:49.935840 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 26 15:13:49.936044 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:49.935894 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 26 15:13:49.936044 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:49.935922 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 26 15:13:49.936044 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:49.935983 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-8nqg9\"" Jul 26 15:13:49.936241 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:49.936222 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 26 15:13:49.941163 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:49.941144 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-694b8c44c9-bq52h"] Jul 26 15:13:50.057684 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:50.057653 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/daf5b316-e8be-40ca-a73c-6580b46f079b-metrics-cert\") pod \"lws-controller-manager-694b8c44c9-bq52h\" (UID: \"daf5b316-e8be-40ca-a73c-6580b46f079b\") " pod="openshift-lws-operator/lws-controller-manager-694b8c44c9-bq52h" Jul 26 15:13:50.057885 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:50.057710 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8mgq\" (UniqueName: \"kubernetes.io/projected/daf5b316-e8be-40ca-a73c-6580b46f079b-kube-api-access-q8mgq\") pod \"lws-controller-manager-694b8c44c9-bq52h\" (UID: \"daf5b316-e8be-40ca-a73c-6580b46f079b\") " pod="openshift-lws-operator/lws-controller-manager-694b8c44c9-bq52h" Jul 26 15:13:50.057885 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:50.057737 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/daf5b316-e8be-40ca-a73c-6580b46f079b-manager-config\") pod \"lws-controller-manager-694b8c44c9-bq52h\" (UID: \"daf5b316-e8be-40ca-a73c-6580b46f079b\") " pod="openshift-lws-operator/lws-controller-manager-694b8c44c9-bq52h" Jul 26 15:13:50.057885 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:50.057762 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/daf5b316-e8be-40ca-a73c-6580b46f079b-cert\") pod \"lws-controller-manager-694b8c44c9-bq52h\" (UID: \"daf5b316-e8be-40ca-a73c-6580b46f079b\") " pod="openshift-lws-operator/lws-controller-manager-694b8c44c9-bq52h" Jul 26 15:13:50.158342 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:50.158296 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8mgq\" (UniqueName: \"kubernetes.io/projected/daf5b316-e8be-40ca-a73c-6580b46f079b-kube-api-access-q8mgq\") pod \"lws-controller-manager-694b8c44c9-bq52h\" (UID: \"daf5b316-e8be-40ca-a73c-6580b46f079b\") " pod="openshift-lws-operator/lws-controller-manager-694b8c44c9-bq52h" Jul 26 15:13:50.158342 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:50.158345 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/daf5b316-e8be-40ca-a73c-6580b46f079b-manager-config\") pod \"lws-controller-manager-694b8c44c9-bq52h\" (UID: \"daf5b316-e8be-40ca-a73c-6580b46f079b\") " pod="openshift-lws-operator/lws-controller-manager-694b8c44c9-bq52h" Jul 26 15:13:50.158584 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:50.158364 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/daf5b316-e8be-40ca-a73c-6580b46f079b-cert\") pod \"lws-controller-manager-694b8c44c9-bq52h\" (UID: \"daf5b316-e8be-40ca-a73c-6580b46f079b\") " pod="openshift-lws-operator/lws-controller-manager-694b8c44c9-bq52h" Jul 26 15:13:50.158584 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:50.158400 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/daf5b316-e8be-40ca-a73c-6580b46f079b-metrics-cert\") pod \"lws-controller-manager-694b8c44c9-bq52h\" (UID: \"daf5b316-e8be-40ca-a73c-6580b46f079b\") " pod="openshift-lws-operator/lws-controller-manager-694b8c44c9-bq52h" Jul 26 15:13:50.159066 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:50.159041 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/daf5b316-e8be-40ca-a73c-6580b46f079b-manager-config\") pod \"lws-controller-manager-694b8c44c9-bq52h\" (UID: \"daf5b316-e8be-40ca-a73c-6580b46f079b\") " pod="openshift-lws-operator/lws-controller-manager-694b8c44c9-bq52h" Jul 26 15:13:50.160720 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:50.160702 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/daf5b316-e8be-40ca-a73c-6580b46f079b-metrics-cert\") pod \"lws-controller-manager-694b8c44c9-bq52h\" (UID: \"daf5b316-e8be-40ca-a73c-6580b46f079b\") " pod="openshift-lws-operator/lws-controller-manager-694b8c44c9-bq52h" Jul 26 15:13:50.160832 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:50.160817 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/daf5b316-e8be-40ca-a73c-6580b46f079b-cert\") pod \"lws-controller-manager-694b8c44c9-bq52h\" (UID: \"daf5b316-e8be-40ca-a73c-6580b46f079b\") " pod="openshift-lws-operator/lws-controller-manager-694b8c44c9-bq52h" Jul 26 15:13:50.165408 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:50.165384 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8mgq\" (UniqueName: \"kubernetes.io/projected/daf5b316-e8be-40ca-a73c-6580b46f079b-kube-api-access-q8mgq\") pod \"lws-controller-manager-694b8c44c9-bq52h\" (UID: \"daf5b316-e8be-40ca-a73c-6580b46f079b\") " pod="openshift-lws-operator/lws-controller-manager-694b8c44c9-bq52h" Jul 26 15:13:50.242650 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:50.242623 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-694b8c44c9-bq52h" Jul 26 15:13:50.361299 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:50.361272 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-694b8c44c9-bq52h"] Jul 26 15:13:50.363597 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:13:50.363572 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaf5b316_e8be_40ca_a73c_6580b46f079b.slice/crio-ed51596cdc313681fd237f06f345fd4bd3f56cba0ef5f3e18b1a9ab32d68b5b5 WatchSource:0}: Error finding container ed51596cdc313681fd237f06f345fd4bd3f56cba0ef5f3e18b1a9ab32d68b5b5: Status 404 returned error can't find the container with id ed51596cdc313681fd237f06f345fd4bd3f56cba0ef5f3e18b1a9ab32d68b5b5 Jul 26 15:13:50.849710 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:50.849676 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-694b8c44c9-bq52h" event={"ID":"daf5b316-e8be-40ca-a73c-6580b46f079b","Type":"ContainerStarted","Data":"ed51596cdc313681fd237f06f345fd4bd3f56cba0ef5f3e18b1a9ab32d68b5b5"} Jul 26 15:13:51.508469 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:51.508384 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:13:51.686901 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:51.686864 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:13:51.925615 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:51.925555 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:13:52.064719 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:52.064692 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-69f589b649-xvn52"] Jul 26 15:13:52.074526 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:52.074499 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-69f589b649-xvn52" Jul 26 15:13:52.075028 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:52.074996 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-69f589b649-xvn52"] Jul 26 15:13:52.076728 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:52.076703 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 26 15:13:52.076868 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:52.076731 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 26 15:13:52.077455 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:52.077438 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 26 15:13:52.077575 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:52.077488 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 26 15:13:52.077643 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:52.077438 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-9ldvq\"" Jul 26 15:13:52.086569 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:52.086546 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:13:52.175277 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:52.175245 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/781bfb24-f7d7-45a3-bf9d-b8adc44aa6c2-tls-certs\") pod \"kube-auth-proxy-69f589b649-xvn52\" (UID: \"781bfb24-f7d7-45a3-bf9d-b8adc44aa6c2\") " pod="openshift-ingress/kube-auth-proxy-69f589b649-xvn52" Jul 26 15:13:52.175441 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:52.175353 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/781bfb24-f7d7-45a3-bf9d-b8adc44aa6c2-tmp\") pod \"kube-auth-proxy-69f589b649-xvn52\" (UID: \"781bfb24-f7d7-45a3-bf9d-b8adc44aa6c2\") " pod="openshift-ingress/kube-auth-proxy-69f589b649-xvn52" Jul 26 15:13:52.175441 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:52.175410 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzpd7\" (UniqueName: \"kubernetes.io/projected/781bfb24-f7d7-45a3-bf9d-b8adc44aa6c2-kube-api-access-kzpd7\") pod \"kube-auth-proxy-69f589b649-xvn52\" (UID: \"781bfb24-f7d7-45a3-bf9d-b8adc44aa6c2\") " pod="openshift-ingress/kube-auth-proxy-69f589b649-xvn52" Jul 26 15:13:52.276269 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:52.276208 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzpd7\" (UniqueName: \"kubernetes.io/projected/781bfb24-f7d7-45a3-bf9d-b8adc44aa6c2-kube-api-access-kzpd7\") pod \"kube-auth-proxy-69f589b649-xvn52\" (UID: \"781bfb24-f7d7-45a3-bf9d-b8adc44aa6c2\") " pod="openshift-ingress/kube-auth-proxy-69f589b649-xvn52" Jul 26 15:13:52.276269 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:52.276258 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/781bfb24-f7d7-45a3-bf9d-b8adc44aa6c2-tls-certs\") pod \"kube-auth-proxy-69f589b649-xvn52\" (UID: \"781bfb24-f7d7-45a3-bf9d-b8adc44aa6c2\") " pod="openshift-ingress/kube-auth-proxy-69f589b649-xvn52" Jul 26 15:13:52.276472 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:52.276298 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/781bfb24-f7d7-45a3-bf9d-b8adc44aa6c2-tmp\") pod \"kube-auth-proxy-69f589b649-xvn52\" (UID: \"781bfb24-f7d7-45a3-bf9d-b8adc44aa6c2\") " pod="openshift-ingress/kube-auth-proxy-69f589b649-xvn52" Jul 26 15:13:52.278339 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:52.278315 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/781bfb24-f7d7-45a3-bf9d-b8adc44aa6c2-tmp\") pod \"kube-auth-proxy-69f589b649-xvn52\" (UID: \"781bfb24-f7d7-45a3-bf9d-b8adc44aa6c2\") " pod="openshift-ingress/kube-auth-proxy-69f589b649-xvn52" Jul 26 15:13:52.278578 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:52.278557 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/781bfb24-f7d7-45a3-bf9d-b8adc44aa6c2-tls-certs\") pod \"kube-auth-proxy-69f589b649-xvn52\" (UID: \"781bfb24-f7d7-45a3-bf9d-b8adc44aa6c2\") " pod="openshift-ingress/kube-auth-proxy-69f589b649-xvn52" Jul 26 15:13:52.286993 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:52.286969 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzpd7\" (UniqueName: \"kubernetes.io/projected/781bfb24-f7d7-45a3-bf9d-b8adc44aa6c2-kube-api-access-kzpd7\") pod \"kube-auth-proxy-69f589b649-xvn52\" (UID: \"781bfb24-f7d7-45a3-bf9d-b8adc44aa6c2\") " pod="openshift-ingress/kube-auth-proxy-69f589b649-xvn52" Jul 26 15:13:52.386452 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:52.386419 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-69f589b649-xvn52" Jul 26 15:13:52.391424 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:52.391387 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:13:52.503742 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:52.503709 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:13:52.603108 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:52.603041 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:13:52.693024 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:52.692868 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:13:52.748509 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:52.748487 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-69f589b649-xvn52"] Jul 26 15:13:52.751368 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:13:52.751347 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod781bfb24_f7d7_45a3_bf9d_b8adc44aa6c2.slice/crio-283d2c1ea46dcfef018ad5cfba8b5f17c2ab574bbaf4e5f33993a12c0c19142b WatchSource:0}: Error finding container 283d2c1ea46dcfef018ad5cfba8b5f17c2ab574bbaf4e5f33993a12c0c19142b: Status 404 returned error can't find the container with id 283d2c1ea46dcfef018ad5cfba8b5f17c2ab574bbaf4e5f33993a12c0c19142b Jul 26 15:13:52.860658 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:52.860585 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-694b8c44c9-bq52h" event={"ID":"daf5b316-e8be-40ca-a73c-6580b46f079b","Type":"ContainerStarted","Data":"5e843645afc34f38995aa48f8d65e25f65aa7a40bc41a24eb60b05affad9c633"} Jul 26 15:13:52.860789 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:52.860683 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-694b8c44c9-bq52h" Jul 26 15:13:52.861716 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:52.861693 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-69f589b649-xvn52" event={"ID":"781bfb24-f7d7-45a3-bf9d-b8adc44aa6c2","Type":"ContainerStarted","Data":"283d2c1ea46dcfef018ad5cfba8b5f17c2ab574bbaf4e5f33993a12c0c19142b"} Jul 26 15:13:52.873924 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:52.873902 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:13:52.878485 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:52.878452 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-694b8c44c9-bq52h" podStartSLOduration=1.573108377 podStartE2EDuration="3.878441539s" totalImagesPullingTime="2.305333162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:13:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:13:52.878330073 +0000 UTC m=+513.302176958" watchObservedRunningTime="2026-07-26 15:13:52.878441539 +0000 UTC m=+513.302288409" Jul 26 15:13:53.266746 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:53.266708 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:13:53.358893 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:53.358859 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:13:53.428368 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:53.428324 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:13:54.448073 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:54.448038 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:13:55.878466 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:55.878428 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-69f589b649-xvn52" event={"ID":"781bfb24-f7d7-45a3-bf9d-b8adc44aa6c2","Type":"ContainerStarted","Data":"7fb2eaea7950e88f18f49896169de506ea34d39bfde365f31752012e395c1c8b"} Jul 26 15:13:55.893906 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:55.893744 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-69f589b649-xvn52" podStartSLOduration=1.340337891 podStartE2EDuration="3.893729267s" totalImagesPullingTime="2.553391376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:13:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:13:55.893155505 +0000 UTC m=+516.317002379" watchObservedRunningTime="2026-07-26 15:13:55.893729267 +0000 UTC m=+516.317576136" Jul 26 15:13:59.438374 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:59.438339 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:13:59.537376 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:13:59.537341 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:14:00.170535 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:14:00.170504 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:14:00.371630 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:14:00.371595 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:14:00.496088 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:14:00.496010 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:14:03.392446 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:14:03.392412 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:14:03.487974 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:14:03.487922 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:14:03.854700 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:14:03.854625 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:14:03.868458 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:14:03.868432 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-694b8c44c9-bq52h" Jul 26 15:14:03.954189 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:14:03.954153 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:14:04.775818 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:14:04.775785 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:14:04.876685 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:14:04.876658 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:14:04.977466 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:14:04.977436 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:14:06.609284 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:14:06.609219 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:14:09.207775 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:14:09.207739 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:14:09.274660 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:14:09.274630 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:14:46.100739 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:14:46.100706 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:14:46.158671 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:14:46.158644 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:14:48.046146 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:14:48.046111 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:14:48.097964 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:14:48.097910 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:14:53.040138 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:14:53.040106 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:14:53.133027 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:14:53.132995 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:14:53.221185 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:14:53.221154 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:15:15.807317 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:15.807272 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:15:17.007144 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:17.007104 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:15:17.638533 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:17.638502 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:15:19.985968 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:19.985919 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9f857_1fbafda6-6242-400e-b2d4-24584608e5f5/ovn-acl-logging/0.log" Jul 26 15:15:19.990129 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:19.990106 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9f857_1fbafda6-6242-400e-b2d4-24584608e5f5/ovn-acl-logging/0.log" Jul 26 15:15:20.843841 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:20.843802 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:15:23.232575 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:23.232539 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:15:23.457651 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:23.457622 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:15:25.740012 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:25.739973 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:15:25.838382 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:25.838350 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:15:28.510892 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:28.510850 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:15:30.613329 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:30.613294 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:15:30.720650 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:30.720619 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:15:31.023226 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:31.023186 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:15:31.124835 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:31.124802 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:15:32.256717 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:32.256687 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:15:34.603171 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:34.603133 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-754ff4fd64-zkr5f"] Jul 26 15:15:34.606556 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:34.606534 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-754ff4fd64-zkr5f" Jul 26 15:15:34.608975 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:34.608922 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 26 15:15:34.609275 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:34.609252 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 26 15:15:34.609586 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:34.609567 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 26 15:15:34.609713 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:34.609625 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-r8ctd\"" Jul 26 15:15:34.609713 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:34.609675 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 26 15:15:34.609713 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:34.609635 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 26 15:15:34.609713 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:34.609625 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 26 15:15:34.610044 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:34.610014 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 26 15:15:34.610203 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:34.610175 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/35dbf057-b87d-4846-91af-4445273f3272-console-oauth-config\") pod \"console-754ff4fd64-zkr5f\" (UID: \"35dbf057-b87d-4846-91af-4445273f3272\") " pod="openshift-console/console-754ff4fd64-zkr5f" Jul 26 15:15:34.610302 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:34.610229 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/35dbf057-b87d-4846-91af-4445273f3272-console-serving-cert\") pod \"console-754ff4fd64-zkr5f\" (UID: \"35dbf057-b87d-4846-91af-4445273f3272\") " pod="openshift-console/console-754ff4fd64-zkr5f" Jul 26 15:15:34.610302 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:34.610271 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/35dbf057-b87d-4846-91af-4445273f3272-service-ca\") pod \"console-754ff4fd64-zkr5f\" (UID: \"35dbf057-b87d-4846-91af-4445273f3272\") " pod="openshift-console/console-754ff4fd64-zkr5f" Jul 26 15:15:34.610414 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:34.610377 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/35dbf057-b87d-4846-91af-4445273f3272-console-config\") pod \"console-754ff4fd64-zkr5f\" (UID: \"35dbf057-b87d-4846-91af-4445273f3272\") " pod="openshift-console/console-754ff4fd64-zkr5f" Jul 26 15:15:34.610473 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:34.610412 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/35dbf057-b87d-4846-91af-4445273f3272-oauth-serving-cert\") pod \"console-754ff4fd64-zkr5f\" (UID: \"35dbf057-b87d-4846-91af-4445273f3272\") " pod="openshift-console/console-754ff4fd64-zkr5f" Jul 26 15:15:34.610530 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:34.610480 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44vkd\" (UniqueName: \"kubernetes.io/projected/35dbf057-b87d-4846-91af-4445273f3272-kube-api-access-44vkd\") pod \"console-754ff4fd64-zkr5f\" (UID: \"35dbf057-b87d-4846-91af-4445273f3272\") " pod="openshift-console/console-754ff4fd64-zkr5f" Jul 26 15:15:34.610530 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:34.610511 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35dbf057-b87d-4846-91af-4445273f3272-trusted-ca-bundle\") pod \"console-754ff4fd64-zkr5f\" (UID: \"35dbf057-b87d-4846-91af-4445273f3272\") " pod="openshift-console/console-754ff4fd64-zkr5f" Jul 26 15:15:34.616030 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:34.616009 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 26 15:15:34.622168 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:34.622139 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-754ff4fd64-zkr5f"] Jul 26 15:15:34.670464 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:34.670430 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:15:34.711441 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:34.711413 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/35dbf057-b87d-4846-91af-4445273f3272-console-oauth-config\") pod \"console-754ff4fd64-zkr5f\" (UID: \"35dbf057-b87d-4846-91af-4445273f3272\") " pod="openshift-console/console-754ff4fd64-zkr5f" Jul 26 15:15:34.711561 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:34.711444 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/35dbf057-b87d-4846-91af-4445273f3272-console-serving-cert\") pod \"console-754ff4fd64-zkr5f\" (UID: \"35dbf057-b87d-4846-91af-4445273f3272\") " pod="openshift-console/console-754ff4fd64-zkr5f" Jul 26 15:15:34.711561 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:34.711469 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/35dbf057-b87d-4846-91af-4445273f3272-service-ca\") pod \"console-754ff4fd64-zkr5f\" (UID: \"35dbf057-b87d-4846-91af-4445273f3272\") " pod="openshift-console/console-754ff4fd64-zkr5f" Jul 26 15:15:34.711561 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:34.711487 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/35dbf057-b87d-4846-91af-4445273f3272-console-config\") pod \"console-754ff4fd64-zkr5f\" (UID: \"35dbf057-b87d-4846-91af-4445273f3272\") " pod="openshift-console/console-754ff4fd64-zkr5f" Jul 26 15:15:34.711561 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:34.711504 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/35dbf057-b87d-4846-91af-4445273f3272-oauth-serving-cert\") pod \"console-754ff4fd64-zkr5f\" (UID: \"35dbf057-b87d-4846-91af-4445273f3272\") " pod="openshift-console/console-754ff4fd64-zkr5f" Jul 26 15:15:34.711561 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:34.711547 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44vkd\" (UniqueName: \"kubernetes.io/projected/35dbf057-b87d-4846-91af-4445273f3272-kube-api-access-44vkd\") pod \"console-754ff4fd64-zkr5f\" (UID: \"35dbf057-b87d-4846-91af-4445273f3272\") " pod="openshift-console/console-754ff4fd64-zkr5f" Jul 26 15:15:34.711799 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:34.711565 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35dbf057-b87d-4846-91af-4445273f3272-trusted-ca-bundle\") pod \"console-754ff4fd64-zkr5f\" (UID: \"35dbf057-b87d-4846-91af-4445273f3272\") " pod="openshift-console/console-754ff4fd64-zkr5f" Jul 26 15:15:34.712221 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:34.712198 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/35dbf057-b87d-4846-91af-4445273f3272-service-ca\") pod \"console-754ff4fd64-zkr5f\" (UID: \"35dbf057-b87d-4846-91af-4445273f3272\") " pod="openshift-console/console-754ff4fd64-zkr5f" Jul 26 15:15:34.712315 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:34.712301 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/35dbf057-b87d-4846-91af-4445273f3272-console-config\") pod \"console-754ff4fd64-zkr5f\" (UID: \"35dbf057-b87d-4846-91af-4445273f3272\") " pod="openshift-console/console-754ff4fd64-zkr5f" Jul 26 15:15:34.712395 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:34.712378 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/35dbf057-b87d-4846-91af-4445273f3272-oauth-serving-cert\") pod \"console-754ff4fd64-zkr5f\" (UID: \"35dbf057-b87d-4846-91af-4445273f3272\") " pod="openshift-console/console-754ff4fd64-zkr5f" Jul 26 15:15:34.712462 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:34.712442 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35dbf057-b87d-4846-91af-4445273f3272-trusted-ca-bundle\") pod \"console-754ff4fd64-zkr5f\" (UID: \"35dbf057-b87d-4846-91af-4445273f3272\") " pod="openshift-console/console-754ff4fd64-zkr5f" Jul 26 15:15:34.713801 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:34.713781 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/35dbf057-b87d-4846-91af-4445273f3272-console-oauth-config\") pod \"console-754ff4fd64-zkr5f\" (UID: \"35dbf057-b87d-4846-91af-4445273f3272\") " pod="openshift-console/console-754ff4fd64-zkr5f" Jul 26 15:15:34.713872 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:34.713851 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/35dbf057-b87d-4846-91af-4445273f3272-console-serving-cert\") pod \"console-754ff4fd64-zkr5f\" (UID: \"35dbf057-b87d-4846-91af-4445273f3272\") " pod="openshift-console/console-754ff4fd64-zkr5f" Jul 26 15:15:34.724245 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:34.724220 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44vkd\" (UniqueName: \"kubernetes.io/projected/35dbf057-b87d-4846-91af-4445273f3272-kube-api-access-44vkd\") pod \"console-754ff4fd64-zkr5f\" (UID: \"35dbf057-b87d-4846-91af-4445273f3272\") " pod="openshift-console/console-754ff4fd64-zkr5f" Jul 26 15:15:34.791558 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:34.791530 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:15:34.918192 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:34.918170 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-754ff4fd64-zkr5f" Jul 26 15:15:35.042225 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:35.042203 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-754ff4fd64-zkr5f"] Jul 26 15:15:35.287133 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:35.287038 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-754ff4fd64-zkr5f" event={"ID":"35dbf057-b87d-4846-91af-4445273f3272","Type":"ContainerStarted","Data":"58a5ee7f61f7c22f825e394a43fec37dc0b382ba2598b0352af150089c02ee05"} Jul 26 15:15:35.287300 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:35.287139 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-754ff4fd64-zkr5f" event={"ID":"35dbf057-b87d-4846-91af-4445273f3272","Type":"ContainerStarted","Data":"fcc6a2746c62161c8e0893c1f1f547bdd754d738e2dd44ba7dbc251c81bc8091"} Jul 26 15:15:35.304319 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:35.304276 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-754ff4fd64-zkr5f" podStartSLOduration=1.304261693 podStartE2EDuration="1.304261693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:15:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:15:35.302410836 +0000 UTC m=+615.726257719" watchObservedRunningTime="2026-07-26 15:15:35.304261693 +0000 UTC m=+615.728108563" Jul 26 15:15:38.100854 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:38.100819 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-w7kd4"] Jul 26 15:15:38.104132 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:38.104117 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-w7kd4" Jul 26 15:15:38.106017 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:38.105999 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 26 15:15:38.106017 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:38.106007 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 26 15:15:38.106192 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:38.106074 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 26 15:15:38.106328 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:38.106313 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 26 15:15:38.106585 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:38.106571 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-6r862\"" Jul 26 15:15:38.112882 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:38.112860 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-w7kd4"] Jul 26 15:15:38.113175 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:38.113115 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:15:38.135714 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:38.135694 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9c09ff1b-831c-4351-a943-71edd17a8845-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-w7kd4\" (UID: \"9c09ff1b-831c-4351-a943-71edd17a8845\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-w7kd4" Jul 26 15:15:38.135805 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:38.135736 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2s76\" (UniqueName: \"kubernetes.io/projected/9c09ff1b-831c-4351-a943-71edd17a8845-kube-api-access-p2s76\") pod \"kuadrant-console-plugin-66b5774985-w7kd4\" (UID: \"9c09ff1b-831c-4351-a943-71edd17a8845\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-w7kd4" Jul 26 15:15:38.135857 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:38.135801 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9c09ff1b-831c-4351-a943-71edd17a8845-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-w7kd4\" (UID: \"9c09ff1b-831c-4351-a943-71edd17a8845\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-w7kd4" Jul 26 15:15:38.237174 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:38.237148 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2s76\" (UniqueName: \"kubernetes.io/projected/9c09ff1b-831c-4351-a943-71edd17a8845-kube-api-access-p2s76\") pod \"kuadrant-console-plugin-66b5774985-w7kd4\" (UID: \"9c09ff1b-831c-4351-a943-71edd17a8845\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-w7kd4" Jul 26 15:15:38.237273 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:38.237180 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9c09ff1b-831c-4351-a943-71edd17a8845-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-w7kd4\" (UID: \"9c09ff1b-831c-4351-a943-71edd17a8845\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-w7kd4" Jul 26 15:15:38.237273 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:38.237264 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9c09ff1b-831c-4351-a943-71edd17a8845-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-w7kd4\" (UID: \"9c09ff1b-831c-4351-a943-71edd17a8845\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-w7kd4" Jul 26 15:15:38.237359 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:15:38.237342 2555 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 26 15:15:38.237404 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:15:38.237396 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c09ff1b-831c-4351-a943-71edd17a8845-plugin-serving-cert podName:9c09ff1b-831c-4351-a943-71edd17a8845 nodeName:}" failed. No retries permitted until 2026-07-26 15:15:38.737381312 +0000 UTC m=+619.161228161 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/9c09ff1b-831c-4351-a943-71edd17a8845-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-w7kd4" (UID: "9c09ff1b-831c-4351-a943-71edd17a8845") : secret "plugin-serving-cert" not found Jul 26 15:15:38.237817 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:38.237801 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9c09ff1b-831c-4351-a943-71edd17a8845-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-w7kd4\" (UID: \"9c09ff1b-831c-4351-a943-71edd17a8845\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-w7kd4" Jul 26 15:15:38.247973 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:38.247949 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2s76\" (UniqueName: \"kubernetes.io/projected/9c09ff1b-831c-4351-a943-71edd17a8845-kube-api-access-p2s76\") pod \"kuadrant-console-plugin-66b5774985-w7kd4\" (UID: \"9c09ff1b-831c-4351-a943-71edd17a8845\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-w7kd4" Jul 26 15:15:38.632193 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:38.632160 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:15:38.732585 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:38.732554 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:15:38.741633 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:38.741609 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9c09ff1b-831c-4351-a943-71edd17a8845-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-w7kd4\" (UID: \"9c09ff1b-831c-4351-a943-71edd17a8845\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-w7kd4" Jul 26 15:15:38.744396 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:38.744374 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9c09ff1b-831c-4351-a943-71edd17a8845-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-w7kd4\" (UID: \"9c09ff1b-831c-4351-a943-71edd17a8845\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-w7kd4" Jul 26 15:15:39.013810 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:39.013741 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-w7kd4" Jul 26 15:15:39.134338 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:39.134312 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-w7kd4"] Jul 26 15:15:39.136833 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:15:39.136806 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c09ff1b_831c_4351_a943_71edd17a8845.slice/crio-bf0d9e4c7c4749d8ccbfe08b59b26ccbaf8963538279b1711edda43c53b310f1 WatchSource:0}: Error finding container bf0d9e4c7c4749d8ccbfe08b59b26ccbaf8963538279b1711edda43c53b310f1: Status 404 returned error can't find the container with id bf0d9e4c7c4749d8ccbfe08b59b26ccbaf8963538279b1711edda43c53b310f1 Jul 26 15:15:39.138255 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:39.138239 2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 26 15:15:39.307455 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:39.307379 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-w7kd4" event={"ID":"9c09ff1b-831c-4351-a943-71edd17a8845","Type":"ContainerStarted","Data":"bf0d9e4c7c4749d8ccbfe08b59b26ccbaf8963538279b1711edda43c53b310f1"} Jul 26 15:15:42.141765 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:42.141573 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:15:42.242331 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:42.242293 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:15:44.918679 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:44.918644 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-754ff4fd64-zkr5f" Jul 26 15:15:44.919158 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:44.918710 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-754ff4fd64-zkr5f" Jul 26 15:15:44.924129 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:44.924106 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-754ff4fd64-zkr5f" Jul 26 15:15:45.339128 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:45.339043 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-754ff4fd64-zkr5f" Jul 26 15:15:45.482318 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:45.482270 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:15:45.576272 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:45.576241 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:15:45.627809 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:45.627778 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:15:48.855911 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:48.855878 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r9hq7"] Jul 26 15:15:48.876098 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:48.876065 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r9hq7"] Jul 26 15:15:48.876271 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:48.876186 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r9hq7" Jul 26 15:15:48.879255 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:48.879230 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-46msh\"" Jul 26 15:15:48.933173 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:48.933140 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcw69\" (UniqueName: \"kubernetes.io/projected/51d0e3e9-db56-494a-85ce-12179457089d-kube-api-access-hcw69\") pod \"limitador-operator-controller-manager-859d7b78bd-r9hq7\" (UID: \"51d0e3e9-db56-494a-85ce-12179457089d\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r9hq7" Jul 26 15:15:49.034436 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:49.034394 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcw69\" (UniqueName: \"kubernetes.io/projected/51d0e3e9-db56-494a-85ce-12179457089d-kube-api-access-hcw69\") pod \"limitador-operator-controller-manager-859d7b78bd-r9hq7\" (UID: \"51d0e3e9-db56-494a-85ce-12179457089d\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r9hq7" Jul 26 15:15:49.063589 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:49.063556 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcw69\" (UniqueName: \"kubernetes.io/projected/51d0e3e9-db56-494a-85ce-12179457089d-kube-api-access-hcw69\") pod \"limitador-operator-controller-manager-859d7b78bd-r9hq7\" (UID: \"51d0e3e9-db56-494a-85ce-12179457089d\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r9hq7" Jul 26 15:15:49.188461 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:49.188427 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r9hq7" Jul 26 15:15:49.984125 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:49.984057 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:15:50.086259 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:50.086219 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:15:51.434764 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:51.434726 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:15:51.542642 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:51.542598 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:15:55.989845 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:55.989804 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:15:56.053772 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:56.053737 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:15:56.580488 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:56.580443 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:15:56.681356 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:56.681319 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:15:59.779342 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:59.779285 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:15:59.879249 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:15:59.879216 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:16:00.369002 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:00.368969 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r9hq7"] Jul 26 15:16:00.376120 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:16:00.376093 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51d0e3e9_db56_494a_85ce_12179457089d.slice/crio-974a7f6ed41d76a0403209c83626c4f3de595de36134219b11c3c04a5ed53c0d WatchSource:0}: Error finding container 974a7f6ed41d76a0403209c83626c4f3de595de36134219b11c3c04a5ed53c0d: Status 404 returned error can't find the container with id 974a7f6ed41d76a0403209c83626c4f3de595de36134219b11c3c04a5ed53c0d Jul 26 15:16:00.409252 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:00.409220 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-w7kd4" event={"ID":"9c09ff1b-831c-4351-a943-71edd17a8845","Type":"ContainerStarted","Data":"1c4ce11a46c171b3c5019a5e7476f440a998cb8eff3c37af5cff3528e6b5e060"} Jul 26 15:16:00.410511 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:00.410487 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r9hq7" event={"ID":"51d0e3e9-db56-494a-85ce-12179457089d","Type":"ContainerStarted","Data":"974a7f6ed41d76a0403209c83626c4f3de595de36134219b11c3c04a5ed53c0d"} Jul 26 15:16:00.424585 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:00.424544 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-w7kd4" podStartSLOduration=1.22489224 podStartE2EDuration="22.42453098s" totalImagesPullingTime="21.19963874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:15:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:16:00.422910018 +0000 UTC m=+640.846756887" watchObservedRunningTime="2026-07-26 15:16:00.42453098 +0000 UTC m=+640.848377850" Jul 26 15:16:02.422016 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:02.421982 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r9hq7" event={"ID":"51d0e3e9-db56-494a-85ce-12179457089d","Type":"ContainerStarted","Data":"0d2c7e6b4fbdac7f1fdc141222e359bedecd5a8e6a6252eb31a93bcfbf23765f"} Jul 26 15:16:02.422385 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:02.422067 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r9hq7" Jul 26 15:16:02.436754 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:02.436701 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r9hq7" podStartSLOduration=12.855036897 podStartE2EDuration="14.436681239s" totalImagesPullingTime="1.581644342s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:15:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:16:02.436201651 +0000 UTC m=+642.860048522" watchObservedRunningTime="2026-07-26 15:16:02.436681239 +0000 UTC m=+642.860528111" Jul 26 15:16:03.775069 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:03.775038 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:16:03.884713 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:03.884685 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:16:13.427841 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:13.427808 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r9hq7" Jul 26 15:16:17.691962 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:17.691912 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:16:18.007021 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:18.006920 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:16:22.313381 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:22.313344 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:16:22.419441 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:22.419411 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:16:22.521493 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:22.521440 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:16:23.095909 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:23.095874 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-9qmvs"] Jul 26 15:16:23.130711 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:23.130678 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:16:23.213212 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:23.213176 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:16:23.230580 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:23.230558 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-9qmvs"] Jul 26 15:16:23.230718 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:23.230590 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-v7tw2"] Jul 26 15:16:23.230764 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:23.230722 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-9qmvs" Jul 26 15:16:23.232969 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:23.232948 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-jtr64\"" Jul 26 15:16:23.247729 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:23.247710 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-v7tw2"] Jul 26 15:16:23.247815 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:23.247788 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-v7tw2" Jul 26 15:16:23.318731 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:23.318708 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d7dt\" (UniqueName: \"kubernetes.io/projected/5372dd13-f256-435c-b5b9-8ef285417f13-kube-api-access-7d7dt\") pod \"authorino-58cd955675-9qmvs\" (UID: \"5372dd13-f256-435c-b5b9-8ef285417f13\") " pod="kuadrant-system/authorino-58cd955675-9qmvs" Jul 26 15:16:23.420017 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:23.419990 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ql5h\" (UniqueName: \"kubernetes.io/projected/aebc49b0-bcad-4188-b814-110ab786e6a2-kube-api-access-6ql5h\") pod \"authorino-75d556bd77-v7tw2\" (UID: \"aebc49b0-bcad-4188-b814-110ab786e6a2\") " pod="kuadrant-system/authorino-75d556bd77-v7tw2" Jul 26 15:16:23.420126 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:23.420049 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d7dt\" (UniqueName: \"kubernetes.io/projected/5372dd13-f256-435c-b5b9-8ef285417f13-kube-api-access-7d7dt\") pod \"authorino-58cd955675-9qmvs\" (UID: \"5372dd13-f256-435c-b5b9-8ef285417f13\") " pod="kuadrant-system/authorino-58cd955675-9qmvs" Jul 26 15:16:23.427283 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:23.427265 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d7dt\" (UniqueName: \"kubernetes.io/projected/5372dd13-f256-435c-b5b9-8ef285417f13-kube-api-access-7d7dt\") pod \"authorino-58cd955675-9qmvs\" (UID: \"5372dd13-f256-435c-b5b9-8ef285417f13\") " pod="kuadrant-system/authorino-58cd955675-9qmvs" Jul 26 15:16:23.520600 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:23.520578 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ql5h\" (UniqueName: \"kubernetes.io/projected/aebc49b0-bcad-4188-b814-110ab786e6a2-kube-api-access-6ql5h\") pod \"authorino-75d556bd77-v7tw2\" (UID: \"aebc49b0-bcad-4188-b814-110ab786e6a2\") " pod="kuadrant-system/authorino-75d556bd77-v7tw2" Jul 26 15:16:23.528070 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:23.528051 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ql5h\" (UniqueName: \"kubernetes.io/projected/aebc49b0-bcad-4188-b814-110ab786e6a2-kube-api-access-6ql5h\") pod \"authorino-75d556bd77-v7tw2\" (UID: \"aebc49b0-bcad-4188-b814-110ab786e6a2\") " pod="kuadrant-system/authorino-75d556bd77-v7tw2" Jul 26 15:16:23.539811 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:23.539790 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-9qmvs" Jul 26 15:16:23.555936 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:23.555917 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-v7tw2" Jul 26 15:16:23.703436 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:23.703405 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-9qmvs"] Jul 26 15:16:23.706511 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:16:23.706486 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5372dd13_f256_435c_b5b9_8ef285417f13.slice/crio-a3b3a8b764d4a3cc2cf112c69474c9d0b556fe5ddcd03f4f36030f5a86b2bb81 WatchSource:0}: Error finding container a3b3a8b764d4a3cc2cf112c69474c9d0b556fe5ddcd03f4f36030f5a86b2bb81: Status 404 returned error can't find the container with id a3b3a8b764d4a3cc2cf112c69474c9d0b556fe5ddcd03f4f36030f5a86b2bb81 Jul 26 15:16:23.721652 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:23.721557 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-v7tw2"] Jul 26 15:16:23.723996 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:16:23.723975 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaebc49b0_bcad_4188_b814_110ab786e6a2.slice/crio-2c6eb27d97e6438708b711a877731893ce3aee1946caa7f73b1b9aa0223fc28f WatchSource:0}: Error finding container 2c6eb27d97e6438708b711a877731893ce3aee1946caa7f73b1b9aa0223fc28f: Status 404 returned error can't find the container with id 2c6eb27d97e6438708b711a877731893ce3aee1946caa7f73b1b9aa0223fc28f Jul 26 15:16:24.521861 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:24.521828 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-v7tw2" event={"ID":"aebc49b0-bcad-4188-b814-110ab786e6a2","Type":"ContainerStarted","Data":"2c6eb27d97e6438708b711a877731893ce3aee1946caa7f73b1b9aa0223fc28f"} Jul 26 15:16:24.523815 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:24.523270 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-9qmvs" event={"ID":"5372dd13-f256-435c-b5b9-8ef285417f13","Type":"ContainerStarted","Data":"a3b3a8b764d4a3cc2cf112c69474c9d0b556fe5ddcd03f4f36030f5a86b2bb81"} Jul 26 15:16:28.544947 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:28.544896 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-v7tw2" event={"ID":"aebc49b0-bcad-4188-b814-110ab786e6a2","Type":"ContainerStarted","Data":"cdfe8edc86fdd970c0fb7c14104ae94527a641d09d9be4df60c74c75a323be26"} Jul 26 15:16:28.546433 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:28.546405 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-9qmvs" event={"ID":"5372dd13-f256-435c-b5b9-8ef285417f13","Type":"ContainerStarted","Data":"5eb5ae0f5962ea259fbb7c52379858af45ff6c1b197078ed94a80d49f1713c88"} Jul 26 15:16:28.558671 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:28.558630 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-v7tw2" podStartSLOduration=1.745383786 podStartE2EDuration="5.558617039s" totalImagesPullingTime="3.813233253s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:16:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:16:28.557652493 +0000 UTC m=+668.981499364" watchObservedRunningTime="2026-07-26 15:16:28.558617039 +0000 UTC m=+668.982463909" Jul 26 15:16:28.571413 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:28.571370 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-9qmvs" podStartSLOduration=1.759839383 podStartE2EDuration="5.571357109s" totalImagesPullingTime="3.811517726s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:16:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:16:28.569381745 +0000 UTC m=+668.993228619" watchObservedRunningTime="2026-07-26 15:16:28.571357109 +0000 UTC m=+668.995203980" Jul 26 15:16:28.594391 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:28.594361 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-9qmvs"] Jul 26 15:16:30.554498 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:30.554454 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-9qmvs" podUID="5372dd13-f256-435c-b5b9-8ef285417f13" containerName="authorino" containerID="cri-o://5eb5ae0f5962ea259fbb7c52379858af45ff6c1b197078ed94a80d49f1713c88" gracePeriod=30 Jul 26 15:16:31.430919 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:31.430889 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-9qmvs" Jul 26 15:16:31.483596 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:31.483568 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7d7dt\" (UniqueName: \"kubernetes.io/projected/5372dd13-f256-435c-b5b9-8ef285417f13-kube-api-access-7d7dt\") pod \"5372dd13-f256-435c-b5b9-8ef285417f13\" (UID: \"5372dd13-f256-435c-b5b9-8ef285417f13\") " Jul 26 15:16:31.485572 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:31.485540 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5372dd13-f256-435c-b5b9-8ef285417f13-kube-api-access-7d7dt" (OuterVolumeSpecName: "kube-api-access-7d7dt") pod "5372dd13-f256-435c-b5b9-8ef285417f13" (UID: "5372dd13-f256-435c-b5b9-8ef285417f13"). InnerVolumeSpecName "kube-api-access-7d7dt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:16:31.560178 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:31.560127 2555 generic.go:356] "Generic (PLEG): container finished" podID="5372dd13-f256-435c-b5b9-8ef285417f13" containerID="5eb5ae0f5962ea259fbb7c52379858af45ff6c1b197078ed94a80d49f1713c88" exitCode=0 Jul 26 15:16:31.560178 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:31.560167 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-9qmvs" Jul 26 15:16:31.560597 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:31.560212 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-9qmvs" event={"ID":"5372dd13-f256-435c-b5b9-8ef285417f13","Type":"ContainerDied","Data":"5eb5ae0f5962ea259fbb7c52379858af45ff6c1b197078ed94a80d49f1713c88"} Jul 26 15:16:31.560597 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:31.560255 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-9qmvs" event={"ID":"5372dd13-f256-435c-b5b9-8ef285417f13","Type":"ContainerDied","Data":"a3b3a8b764d4a3cc2cf112c69474c9d0b556fe5ddcd03f4f36030f5a86b2bb81"} Jul 26 15:16:31.560597 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:31.560283 2555 scope.go:117] "RemoveContainer" containerID="5eb5ae0f5962ea259fbb7c52379858af45ff6c1b197078ed94a80d49f1713c88" Jul 26 15:16:31.568747 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:31.568731 2555 scope.go:117] "RemoveContainer" containerID="5eb5ae0f5962ea259fbb7c52379858af45ff6c1b197078ed94a80d49f1713c88" Jul 26 15:16:31.569018 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:16:31.569000 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5eb5ae0f5962ea259fbb7c52379858af45ff6c1b197078ed94a80d49f1713c88\": container with ID starting with 5eb5ae0f5962ea259fbb7c52379858af45ff6c1b197078ed94a80d49f1713c88 not found: ID does not exist" containerID="5eb5ae0f5962ea259fbb7c52379858af45ff6c1b197078ed94a80d49f1713c88" Jul 26 15:16:31.569070 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:31.569026 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5eb5ae0f5962ea259fbb7c52379858af45ff6c1b197078ed94a80d49f1713c88"} err="failed to get container status \"5eb5ae0f5962ea259fbb7c52379858af45ff6c1b197078ed94a80d49f1713c88\": rpc error: code = NotFound desc = could not find container \"5eb5ae0f5962ea259fbb7c52379858af45ff6c1b197078ed94a80d49f1713c88\": container with ID starting with 5eb5ae0f5962ea259fbb7c52379858af45ff6c1b197078ed94a80d49f1713c88 not found: ID does not exist" Jul 26 15:16:31.579065 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:31.579045 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-9qmvs"] Jul 26 15:16:31.580506 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:31.580484 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-9qmvs"] Jul 26 15:16:31.584408 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:31.584392 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7d7dt\" (UniqueName: \"kubernetes.io/projected/5372dd13-f256-435c-b5b9-8ef285417f13-kube-api-access-7d7dt\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:16:32.066456 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:32.066427 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5372dd13-f256-435c-b5b9-8ef285417f13" path="/var/lib/kubelet/pods/5372dd13-f256-435c-b5b9-8ef285417f13/volumes" Jul 26 15:16:38.445677 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:38.445641 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:16:38.505466 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:38.505439 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:16:38.919725 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:38.919694 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:16:44.825503 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:44.825469 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:16:50.828423 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:50.828391 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:16:50.954089 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:50.954052 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:16:51.054287 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:51.054260 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-2xq94"] Jul 26 15:16:51.054585 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:51.054574 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5372dd13-f256-435c-b5b9-8ef285417f13" containerName="authorino" Jul 26 15:16:51.054632 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:51.054587 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5372dd13-f256-435c-b5b9-8ef285417f13" containerName="authorino" Jul 26 15:16:51.054666 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:51.054650 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5372dd13-f256-435c-b5b9-8ef285417f13" containerName="authorino" Jul 26 15:16:51.057082 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:51.057065 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-2xq94" Jul 26 15:16:51.063973 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:51.063950 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-2xq94"] Jul 26 15:16:51.117548 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:51.117518 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5sfm\" (UniqueName: \"kubernetes.io/projected/2c2f1419-b763-4853-add4-6857aba9d34b-kube-api-access-p5sfm\") pod \"authorino-5d4b67c446-2xq94\" (UID: \"2c2f1419-b763-4853-add4-6857aba9d34b\") " pod="kuadrant-system/authorino-5d4b67c446-2xq94" Jul 26 15:16:51.218367 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:51.218342 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5sfm\" (UniqueName: \"kubernetes.io/projected/2c2f1419-b763-4853-add4-6857aba9d34b-kube-api-access-p5sfm\") pod \"authorino-5d4b67c446-2xq94\" (UID: \"2c2f1419-b763-4853-add4-6857aba9d34b\") " pod="kuadrant-system/authorino-5d4b67c446-2xq94" Jul 26 15:16:51.226301 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:51.226277 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5sfm\" (UniqueName: \"kubernetes.io/projected/2c2f1419-b763-4853-add4-6857aba9d34b-kube-api-access-p5sfm\") pod \"authorino-5d4b67c446-2xq94\" (UID: \"2c2f1419-b763-4853-add4-6857aba9d34b\") " pod="kuadrant-system/authorino-5d4b67c446-2xq94" Jul 26 15:16:51.232048 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:51.232025 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-2xq94"] Jul 26 15:16:51.232276 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:51.232263 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-2xq94" Jul 26 15:16:51.264510 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:51.264479 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-76877994-2kxsg"] Jul 26 15:16:51.267945 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:51.267909 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-2kxsg" Jul 26 15:16:51.269969 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:51.269944 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 26 15:16:51.274959 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:51.274919 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76877994-2kxsg"] Jul 26 15:16:51.313802 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:51.313486 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-2kxsg"] Jul 26 15:16:51.314003 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:16:51.313805 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-wdrx8 tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-76877994-2kxsg" podUID="2dba0739-dc3b-492e-a9b5-2ef7b16871bd" Jul 26 15:16:51.319506 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:51.319475 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdrx8\" (UniqueName: \"kubernetes.io/projected/2dba0739-dc3b-492e-a9b5-2ef7b16871bd-kube-api-access-wdrx8\") pod \"authorino-76877994-2kxsg\" (UID: \"2dba0739-dc3b-492e-a9b5-2ef7b16871bd\") " pod="kuadrant-system/authorino-76877994-2kxsg" Jul 26 15:16:51.319624 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:51.319547 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2dba0739-dc3b-492e-a9b5-2ef7b16871bd-tls-cert\") pod \"authorino-76877994-2kxsg\" (UID: \"2dba0739-dc3b-492e-a9b5-2ef7b16871bd\") " pod="kuadrant-system/authorino-76877994-2kxsg" Jul 26 15:16:51.342825 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:51.342800 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5fdd89c4b9-wcwqg"] Jul 26 15:16:51.345760 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:51.345741 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5fdd89c4b9-wcwqg" Jul 26 15:16:51.354208 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:51.354186 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5fdd89c4b9-wcwqg"] Jul 26 15:16:51.361724 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:51.361705 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-2xq94"] Jul 26 15:16:51.364304 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:16:51.364279 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2c2f1419_b763_4853_add4_6857aba9d34b.slice/crio-57f8729fe173a9e5668a70304513c2a8ead31f6428e95fb657187a47f5612032 WatchSource:0}: Error finding container 57f8729fe173a9e5668a70304513c2a8ead31f6428e95fb657187a47f5612032: Status 404 returned error can't find the container with id 57f8729fe173a9e5668a70304513c2a8ead31f6428e95fb657187a47f5612032 Jul 26 15:16:51.419954 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:51.419925 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvrxv\" (UniqueName: \"kubernetes.io/projected/7d575e37-bb9f-427e-86a1-9df7fa029043-kube-api-access-jvrxv\") pod \"authorino-5fdd89c4b9-wcwqg\" (UID: \"7d575e37-bb9f-427e-86a1-9df7fa029043\") " pod="kuadrant-system/authorino-5fdd89c4b9-wcwqg" Jul 26 15:16:51.420048 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:51.420018 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdrx8\" (UniqueName: \"kubernetes.io/projected/2dba0739-dc3b-492e-a9b5-2ef7b16871bd-kube-api-access-wdrx8\") pod \"authorino-76877994-2kxsg\" (UID: \"2dba0739-dc3b-492e-a9b5-2ef7b16871bd\") " pod="kuadrant-system/authorino-76877994-2kxsg" Jul 26 15:16:51.420093 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:51.420071 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2dba0739-dc3b-492e-a9b5-2ef7b16871bd-tls-cert\") pod \"authorino-76877994-2kxsg\" (UID: \"2dba0739-dc3b-492e-a9b5-2ef7b16871bd\") " pod="kuadrant-system/authorino-76877994-2kxsg" Jul 26 15:16:51.420127 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:51.420106 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7d575e37-bb9f-427e-86a1-9df7fa029043-tls-cert\") pod \"authorino-5fdd89c4b9-wcwqg\" (UID: \"7d575e37-bb9f-427e-86a1-9df7fa029043\") " pod="kuadrant-system/authorino-5fdd89c4b9-wcwqg" Jul 26 15:16:51.422319 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:51.422303 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2dba0739-dc3b-492e-a9b5-2ef7b16871bd-tls-cert\") pod \"authorino-76877994-2kxsg\" (UID: \"2dba0739-dc3b-492e-a9b5-2ef7b16871bd\") " pod="kuadrant-system/authorino-76877994-2kxsg" Jul 26 15:16:51.426651 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:51.426625 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdrx8\" (UniqueName: \"kubernetes.io/projected/2dba0739-dc3b-492e-a9b5-2ef7b16871bd-kube-api-access-wdrx8\") pod \"authorino-76877994-2kxsg\" (UID: \"2dba0739-dc3b-492e-a9b5-2ef7b16871bd\") " pod="kuadrant-system/authorino-76877994-2kxsg" Jul 26 15:16:51.520995 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:51.520970 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7d575e37-bb9f-427e-86a1-9df7fa029043-tls-cert\") pod \"authorino-5fdd89c4b9-wcwqg\" (UID: \"7d575e37-bb9f-427e-86a1-9df7fa029043\") " pod="kuadrant-system/authorino-5fdd89c4b9-wcwqg" Jul 26 15:16:51.521081 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:51.521019 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvrxv\" (UniqueName: \"kubernetes.io/projected/7d575e37-bb9f-427e-86a1-9df7fa029043-kube-api-access-jvrxv\") pod \"authorino-5fdd89c4b9-wcwqg\" (UID: \"7d575e37-bb9f-427e-86a1-9df7fa029043\") " pod="kuadrant-system/authorino-5fdd89c4b9-wcwqg" Jul 26 15:16:51.522972 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:51.522957 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7d575e37-bb9f-427e-86a1-9df7fa029043-tls-cert\") pod \"authorino-5fdd89c4b9-wcwqg\" (UID: \"7d575e37-bb9f-427e-86a1-9df7fa029043\") " pod="kuadrant-system/authorino-5fdd89c4b9-wcwqg" Jul 26 15:16:51.527802 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:51.527779 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvrxv\" (UniqueName: \"kubernetes.io/projected/7d575e37-bb9f-427e-86a1-9df7fa029043-kube-api-access-jvrxv\") pod \"authorino-5fdd89c4b9-wcwqg\" (UID: \"7d575e37-bb9f-427e-86a1-9df7fa029043\") " pod="kuadrant-system/authorino-5fdd89c4b9-wcwqg" Jul 26 15:16:51.649641 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:51.649577 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-2kxsg" Jul 26 15:16:51.649641 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:51.649595 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-2xq94" event={"ID":"2c2f1419-b763-4853-add4-6857aba9d34b","Type":"ContainerStarted","Data":"57f8729fe173a9e5668a70304513c2a8ead31f6428e95fb657187a47f5612032"} Jul 26 15:16:51.655219 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:51.655200 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5fdd89c4b9-wcwqg" Jul 26 15:16:51.656836 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:51.656818 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-2kxsg" Jul 26 15:16:51.722419 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:51.722393 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wdrx8\" (UniqueName: \"kubernetes.io/projected/2dba0739-dc3b-492e-a9b5-2ef7b16871bd-kube-api-access-wdrx8\") pod \"2dba0739-dc3b-492e-a9b5-2ef7b16871bd\" (UID: \"2dba0739-dc3b-492e-a9b5-2ef7b16871bd\") " Jul 26 15:16:51.722559 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:51.722511 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2dba0739-dc3b-492e-a9b5-2ef7b16871bd-tls-cert\") pod \"2dba0739-dc3b-492e-a9b5-2ef7b16871bd\" (UID: \"2dba0739-dc3b-492e-a9b5-2ef7b16871bd\") " Jul 26 15:16:51.724324 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:51.724294 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dba0739-dc3b-492e-a9b5-2ef7b16871bd-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "2dba0739-dc3b-492e-a9b5-2ef7b16871bd" (UID: "2dba0739-dc3b-492e-a9b5-2ef7b16871bd"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 15:16:51.724443 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:51.724372 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dba0739-dc3b-492e-a9b5-2ef7b16871bd-kube-api-access-wdrx8" (OuterVolumeSpecName: "kube-api-access-wdrx8") pod "2dba0739-dc3b-492e-a9b5-2ef7b16871bd" (UID: "2dba0739-dc3b-492e-a9b5-2ef7b16871bd"). InnerVolumeSpecName "kube-api-access-wdrx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:16:51.781843 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:51.781817 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5fdd89c4b9-wcwqg"] Jul 26 15:16:51.784111 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:16:51.784085 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7d575e37_bb9f_427e_86a1_9df7fa029043.slice/crio-cae87d4c93c948e621086c79e952db640f84f66c778cbf0c7ac4aeb995cc1993 WatchSource:0}: Error finding container cae87d4c93c948e621086c79e952db640f84f66c778cbf0c7ac4aeb995cc1993: Status 404 returned error can't find the container with id cae87d4c93c948e621086c79e952db640f84f66c778cbf0c7ac4aeb995cc1993 Jul 26 15:16:51.823644 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:51.823620 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdrx8\" (UniqueName: \"kubernetes.io/projected/2dba0739-dc3b-492e-a9b5-2ef7b16871bd-kube-api-access-wdrx8\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:16:51.823644 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:51.823644 2555 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2dba0739-dc3b-492e-a9b5-2ef7b16871bd-tls-cert\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:16:52.660565 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:52.660526 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5fdd89c4b9-wcwqg" event={"ID":"7d575e37-bb9f-427e-86a1-9df7fa029043","Type":"ContainerStarted","Data":"65ab3f2921e49f1c30fcdc0695c236e06307a5e27153f46f316b1fec33860061"} Jul 26 15:16:52.661007 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:52.660573 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5fdd89c4b9-wcwqg" event={"ID":"7d575e37-bb9f-427e-86a1-9df7fa029043","Type":"ContainerStarted","Data":"cae87d4c93c948e621086c79e952db640f84f66c778cbf0c7ac4aeb995cc1993"} Jul 26 15:16:52.662152 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:52.662132 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-2kxsg" Jul 26 15:16:52.662251 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:52.662164 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-2xq94" event={"ID":"2c2f1419-b763-4853-add4-6857aba9d34b","Type":"ContainerStarted","Data":"36131fef825fe321358df90b3d8ac1cbe0ec936c0124c1c8cee85539a9bb4ace"} Jul 26 15:16:52.662251 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:52.662209 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-2xq94" podUID="2c2f1419-b763-4853-add4-6857aba9d34b" containerName="authorino" containerID="cri-o://36131fef825fe321358df90b3d8ac1cbe0ec936c0124c1c8cee85539a9bb4ace" gracePeriod=30 Jul 26 15:16:52.676553 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:52.676516 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5fdd89c4b9-wcwqg" podStartSLOduration=1.354248284 podStartE2EDuration="1.676502753s" totalImagesPullingTime="322.254469ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:16:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:16:52.67418604 +0000 UTC m=+693.098032972" watchObservedRunningTime="2026-07-26 15:16:52.676502753 +0000 UTC m=+693.100349623" Jul 26 15:16:52.694263 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:52.694234 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-2kxsg"] Jul 26 15:16:52.699774 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:52.699751 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-76877994-2kxsg"] Jul 26 15:16:52.705406 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:52.705383 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-v7tw2"] Jul 26 15:16:52.705557 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:52.705539 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-v7tw2" podUID="aebc49b0-bcad-4188-b814-110ab786e6a2" containerName="authorino" containerID="cri-o://cdfe8edc86fdd970c0fb7c14104ae94527a641d09d9be4df60c74c75a323be26" gracePeriod=30 Jul 26 15:16:52.715605 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:52.715565 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-2xq94" podStartSLOduration=1.350754599 podStartE2EDuration="1.715549206s" totalImagesPullingTime="364.794607ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:16:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:16:52.714056377 +0000 UTC m=+693.137903251" watchObservedRunningTime="2026-07-26 15:16:52.715549206 +0000 UTC m=+693.139396076" Jul 26 15:16:52.964578 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:52.964556 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-2xq94" Jul 26 15:16:52.967261 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:52.967238 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-v7tw2" Jul 26 15:16:53.033626 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:53.033599 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6ql5h\" (UniqueName: \"kubernetes.io/projected/aebc49b0-bcad-4188-b814-110ab786e6a2-kube-api-access-6ql5h\") pod \"aebc49b0-bcad-4188-b814-110ab786e6a2\" (UID: \"aebc49b0-bcad-4188-b814-110ab786e6a2\") " Jul 26 15:16:53.033752 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:53.033654 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p5sfm\" (UniqueName: \"kubernetes.io/projected/2c2f1419-b763-4853-add4-6857aba9d34b-kube-api-access-p5sfm\") pod \"2c2f1419-b763-4853-add4-6857aba9d34b\" (UID: \"2c2f1419-b763-4853-add4-6857aba9d34b\") " Jul 26 15:16:53.035598 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:53.035572 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c2f1419-b763-4853-add4-6857aba9d34b-kube-api-access-p5sfm" (OuterVolumeSpecName: "kube-api-access-p5sfm") pod "2c2f1419-b763-4853-add4-6857aba9d34b" (UID: "2c2f1419-b763-4853-add4-6857aba9d34b"). InnerVolumeSpecName "kube-api-access-p5sfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:16:53.035598 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:53.035586 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aebc49b0-bcad-4188-b814-110ab786e6a2-kube-api-access-6ql5h" (OuterVolumeSpecName: "kube-api-access-6ql5h") pod "aebc49b0-bcad-4188-b814-110ab786e6a2" (UID: "aebc49b0-bcad-4188-b814-110ab786e6a2"). InnerVolumeSpecName "kube-api-access-6ql5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:16:53.134576 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:53.134553 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ql5h\" (UniqueName: \"kubernetes.io/projected/aebc49b0-bcad-4188-b814-110ab786e6a2-kube-api-access-6ql5h\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:16:53.134576 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:53.134570 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5sfm\" (UniqueName: \"kubernetes.io/projected/2c2f1419-b763-4853-add4-6857aba9d34b-kube-api-access-p5sfm\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:16:53.670812 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:53.670778 2555 generic.go:356] "Generic (PLEG): container finished" podID="aebc49b0-bcad-4188-b814-110ab786e6a2" containerID="cdfe8edc86fdd970c0fb7c14104ae94527a641d09d9be4df60c74c75a323be26" exitCode=0 Jul 26 15:16:53.671241 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:53.670821 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-v7tw2" Jul 26 15:16:53.671241 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:53.670822 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-v7tw2" event={"ID":"aebc49b0-bcad-4188-b814-110ab786e6a2","Type":"ContainerDied","Data":"cdfe8edc86fdd970c0fb7c14104ae94527a641d09d9be4df60c74c75a323be26"} Jul 26 15:16:53.671241 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:53.670891 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-v7tw2" event={"ID":"aebc49b0-bcad-4188-b814-110ab786e6a2","Type":"ContainerDied","Data":"2c6eb27d97e6438708b711a877731893ce3aee1946caa7f73b1b9aa0223fc28f"} Jul 26 15:16:53.671241 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:53.670917 2555 scope.go:117] "RemoveContainer" containerID="cdfe8edc86fdd970c0fb7c14104ae94527a641d09d9be4df60c74c75a323be26" Jul 26 15:16:53.672653 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:53.672628 2555 generic.go:356] "Generic (PLEG): container finished" podID="2c2f1419-b763-4853-add4-6857aba9d34b" containerID="36131fef825fe321358df90b3d8ac1cbe0ec936c0124c1c8cee85539a9bb4ace" exitCode=0 Jul 26 15:16:53.672771 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:53.672684 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-2xq94" event={"ID":"2c2f1419-b763-4853-add4-6857aba9d34b","Type":"ContainerDied","Data":"36131fef825fe321358df90b3d8ac1cbe0ec936c0124c1c8cee85539a9bb4ace"} Jul 26 15:16:53.672771 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:53.672729 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-2xq94" event={"ID":"2c2f1419-b763-4853-add4-6857aba9d34b","Type":"ContainerDied","Data":"57f8729fe173a9e5668a70304513c2a8ead31f6428e95fb657187a47f5612032"} Jul 26 15:16:53.672771 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:53.672688 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-2xq94" Jul 26 15:16:53.680048 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:53.679916 2555 scope.go:117] "RemoveContainer" containerID="cdfe8edc86fdd970c0fb7c14104ae94527a641d09d9be4df60c74c75a323be26" Jul 26 15:16:53.680318 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:16:53.680214 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdfe8edc86fdd970c0fb7c14104ae94527a641d09d9be4df60c74c75a323be26\": container with ID starting with cdfe8edc86fdd970c0fb7c14104ae94527a641d09d9be4df60c74c75a323be26 not found: ID does not exist" containerID="cdfe8edc86fdd970c0fb7c14104ae94527a641d09d9be4df60c74c75a323be26" Jul 26 15:16:53.680318 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:53.680253 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdfe8edc86fdd970c0fb7c14104ae94527a641d09d9be4df60c74c75a323be26"} err="failed to get container status \"cdfe8edc86fdd970c0fb7c14104ae94527a641d09d9be4df60c74c75a323be26\": rpc error: code = NotFound desc = could not find container \"cdfe8edc86fdd970c0fb7c14104ae94527a641d09d9be4df60c74c75a323be26\": container with ID starting with cdfe8edc86fdd970c0fb7c14104ae94527a641d09d9be4df60c74c75a323be26 not found: ID does not exist" Jul 26 15:16:53.680318 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:53.680272 2555 scope.go:117] "RemoveContainer" containerID="36131fef825fe321358df90b3d8ac1cbe0ec936c0124c1c8cee85539a9bb4ace" Jul 26 15:16:53.687767 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:53.687753 2555 scope.go:117] "RemoveContainer" containerID="36131fef825fe321358df90b3d8ac1cbe0ec936c0124c1c8cee85539a9bb4ace" Jul 26 15:16:53.688043 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:16:53.688026 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36131fef825fe321358df90b3d8ac1cbe0ec936c0124c1c8cee85539a9bb4ace\": container with ID starting with 36131fef825fe321358df90b3d8ac1cbe0ec936c0124c1c8cee85539a9bb4ace not found: ID does not exist" containerID="36131fef825fe321358df90b3d8ac1cbe0ec936c0124c1c8cee85539a9bb4ace" Jul 26 15:16:53.688106 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:53.688048 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36131fef825fe321358df90b3d8ac1cbe0ec936c0124c1c8cee85539a9bb4ace"} err="failed to get container status \"36131fef825fe321358df90b3d8ac1cbe0ec936c0124c1c8cee85539a9bb4ace\": rpc error: code = NotFound desc = could not find container \"36131fef825fe321358df90b3d8ac1cbe0ec936c0124c1c8cee85539a9bb4ace\": container with ID starting with 36131fef825fe321358df90b3d8ac1cbe0ec936c0124c1c8cee85539a9bb4ace not found: ID does not exist" Jul 26 15:16:53.691407 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:53.691327 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-v7tw2"] Jul 26 15:16:53.693207 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:53.693185 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-v7tw2"] Jul 26 15:16:53.701896 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:53.701876 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-2xq94"] Jul 26 15:16:53.705181 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:53.705160 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-2xq94"] Jul 26 15:16:54.063835 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:54.063765 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c2f1419-b763-4853-add4-6857aba9d34b" path="/var/lib/kubelet/pods/2c2f1419-b763-4853-add4-6857aba9d34b/volumes" Jul 26 15:16:54.064107 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:54.064094 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2dba0739-dc3b-492e-a9b5-2ef7b16871bd" path="/var/lib/kubelet/pods/2dba0739-dc3b-492e-a9b5-2ef7b16871bd/volumes" Jul 26 15:16:54.064303 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:54.064291 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aebc49b0-bcad-4188-b814-110ab786e6a2" path="/var/lib/kubelet/pods/aebc49b0-bcad-4188-b814-110ab786e6a2/volumes" Jul 26 15:16:57.356258 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:57.356226 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:16:57.369212 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:57.369189 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7fc4ffc667-cnbkt"] Jul 26 15:16:57.369524 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:57.369511 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aebc49b0-bcad-4188-b814-110ab786e6a2" containerName="authorino" Jul 26 15:16:57.369570 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:57.369526 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="aebc49b0-bcad-4188-b814-110ab786e6a2" containerName="authorino" Jul 26 15:16:57.369570 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:57.369547 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c2f1419-b763-4853-add4-6857aba9d34b" containerName="authorino" Jul 26 15:16:57.369570 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:57.369553 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c2f1419-b763-4853-add4-6857aba9d34b" containerName="authorino" Jul 26 15:16:57.369654 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:57.369612 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c2f1419-b763-4853-add4-6857aba9d34b" containerName="authorino" Jul 26 15:16:57.369654 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:57.369623 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="aebc49b0-bcad-4188-b814-110ab786e6a2" containerName="authorino" Jul 26 15:16:57.374180 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:57.374163 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7fc4ffc667-cnbkt" Jul 26 15:16:57.376270 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:57.376251 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ccgw4\"" Jul 26 15:16:57.376486 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:57.376470 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 26 15:16:57.376703 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:57.376686 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 26 15:16:57.385153 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:57.385132 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7fc4ffc667-cnbkt"] Jul 26 15:16:57.456117 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:57.456084 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:16:57.464111 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:57.464091 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssxk5\" (UniqueName: \"kubernetes.io/projected/76ccc809-4977-44c4-b910-66568fd64ad8-kube-api-access-ssxk5\") pod \"maas-controller-7fc4ffc667-cnbkt\" (UID: \"76ccc809-4977-44c4-b910-66568fd64ad8\") " pod="opendatahub/maas-controller-7fc4ffc667-cnbkt" Jul 26 15:16:57.464192 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:57.464177 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/76ccc809-4977-44c4-b910-66568fd64ad8-webhook-cert\") pod \"maas-controller-7fc4ffc667-cnbkt\" (UID: \"76ccc809-4977-44c4-b910-66568fd64ad8\") " pod="opendatahub/maas-controller-7fc4ffc667-cnbkt" Jul 26 15:16:57.553683 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:57.553658 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:16:57.564771 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:57.564741 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/76ccc809-4977-44c4-b910-66568fd64ad8-webhook-cert\") pod \"maas-controller-7fc4ffc667-cnbkt\" (UID: \"76ccc809-4977-44c4-b910-66568fd64ad8\") " pod="opendatahub/maas-controller-7fc4ffc667-cnbkt" Jul 26 15:16:57.564880 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:57.564799 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssxk5\" (UniqueName: \"kubernetes.io/projected/76ccc809-4977-44c4-b910-66568fd64ad8-kube-api-access-ssxk5\") pod \"maas-controller-7fc4ffc667-cnbkt\" (UID: \"76ccc809-4977-44c4-b910-66568fd64ad8\") " pod="opendatahub/maas-controller-7fc4ffc667-cnbkt" Jul 26 15:16:57.564963 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:16:57.564875 2555 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 26 15:16:57.564963 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:16:57.564941 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76ccc809-4977-44c4-b910-66568fd64ad8-webhook-cert podName:76ccc809-4977-44c4-b910-66568fd64ad8 nodeName:}" failed. No retries permitted until 2026-07-26 15:16:58.064913605 +0000 UTC m=+698.488760455 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/76ccc809-4977-44c4-b910-66568fd64ad8-webhook-cert") pod "maas-controller-7fc4ffc667-cnbkt" (UID: "76ccc809-4977-44c4-b910-66568fd64ad8") : secret "maas-controller-webhook-cert" not found Jul 26 15:16:57.574016 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:57.573997 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssxk5\" (UniqueName: \"kubernetes.io/projected/76ccc809-4977-44c4-b910-66568fd64ad8-kube-api-access-ssxk5\") pod \"maas-controller-7fc4ffc667-cnbkt\" (UID: \"76ccc809-4977-44c4-b910-66568fd64ad8\") " pod="opendatahub/maas-controller-7fc4ffc667-cnbkt" Jul 26 15:16:57.742117 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:57.742086 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:16:58.067729 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:58.067652 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/76ccc809-4977-44c4-b910-66568fd64ad8-webhook-cert\") pod \"maas-controller-7fc4ffc667-cnbkt\" (UID: \"76ccc809-4977-44c4-b910-66568fd64ad8\") " pod="opendatahub/maas-controller-7fc4ffc667-cnbkt" Jul 26 15:16:58.070046 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:58.070024 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/76ccc809-4977-44c4-b910-66568fd64ad8-webhook-cert\") pod \"maas-controller-7fc4ffc667-cnbkt\" (UID: \"76ccc809-4977-44c4-b910-66568fd64ad8\") " pod="opendatahub/maas-controller-7fc4ffc667-cnbkt" Jul 26 15:16:58.284389 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:58.284362 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7fc4ffc667-cnbkt" Jul 26 15:16:58.404702 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:58.404681 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7fc4ffc667-cnbkt"] Jul 26 15:16:58.406718 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:16:58.406689 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76ccc809_4977_44c4_b910_66568fd64ad8.slice/crio-fc7de194b3c4718236bf261af2690b8cf7359c98a8359f623764f1a68e7a902e WatchSource:0}: Error finding container fc7de194b3c4718236bf261af2690b8cf7359c98a8359f623764f1a68e7a902e: Status 404 returned error can't find the container with id fc7de194b3c4718236bf261af2690b8cf7359c98a8359f623764f1a68e7a902e Jul 26 15:16:58.698177 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:16:58.698141 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7fc4ffc667-cnbkt" event={"ID":"76ccc809-4977-44c4-b910-66568fd64ad8","Type":"ContainerStarted","Data":"fc7de194b3c4718236bf261af2690b8cf7359c98a8359f623764f1a68e7a902e"} Jul 26 15:17:00.336737 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:17:00.336691 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:17:00.439474 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:17:00.439438 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:17:00.709720 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:17:00.709684 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7fc4ffc667-cnbkt" event={"ID":"76ccc809-4977-44c4-b910-66568fd64ad8","Type":"ContainerStarted","Data":"fa3a6b8d5113b84b08c2b115bf5f8d33570cb38e63c5dad9787d19a7f5424276"} Jul 26 15:17:00.709879 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:17:00.709864 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7fc4ffc667-cnbkt" Jul 26 15:17:00.727004 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:17:00.726957 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7fc4ffc667-cnbkt" podStartSLOduration=1.52131262 podStartE2EDuration="3.726919737s" totalImagesPullingTime="2.205607117s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:16:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:17:00.726872429 +0000 UTC m=+701.150719322" watchObservedRunningTime="2026-07-26 15:17:00.726919737 +0000 UTC m=+701.150766607" Jul 26 15:17:11.716944 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:17:11.716903 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7fc4ffc667-cnbkt" Jul 26 15:17:11.973416 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:17:11.970290 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7fc4ffc667-cnbkt"] Jul 26 15:17:11.973416 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:17:11.970597 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7fc4ffc667-cnbkt" podUID="76ccc809-4977-44c4-b910-66568fd64ad8" containerName="manager" containerID="cri-o://fa3a6b8d5113b84b08c2b115bf5f8d33570cb38e63c5dad9787d19a7f5424276" gracePeriod=30 Jul 26 15:17:12.212746 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:17:12.212724 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7fc4ffc667-cnbkt" Jul 26 15:17:12.286422 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:17:12.286348 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ssxk5\" (UniqueName: \"kubernetes.io/projected/76ccc809-4977-44c4-b910-66568fd64ad8-kube-api-access-ssxk5\") pod \"76ccc809-4977-44c4-b910-66568fd64ad8\" (UID: \"76ccc809-4977-44c4-b910-66568fd64ad8\") " Jul 26 15:17:12.286422 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:17:12.286382 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/76ccc809-4977-44c4-b910-66568fd64ad8-webhook-cert\") pod \"76ccc809-4977-44c4-b910-66568fd64ad8\" (UID: \"76ccc809-4977-44c4-b910-66568fd64ad8\") " Jul 26 15:17:12.288454 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:17:12.288419 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76ccc809-4977-44c4-b910-66568fd64ad8-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "76ccc809-4977-44c4-b910-66568fd64ad8" (UID: "76ccc809-4977-44c4-b910-66568fd64ad8"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 15:17:12.288454 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:17:12.288419 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76ccc809-4977-44c4-b910-66568fd64ad8-kube-api-access-ssxk5" (OuterVolumeSpecName: "kube-api-access-ssxk5") pod "76ccc809-4977-44c4-b910-66568fd64ad8" (UID: "76ccc809-4977-44c4-b910-66568fd64ad8"). InnerVolumeSpecName "kube-api-access-ssxk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:17:12.386925 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:17:12.386898 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssxk5\" (UniqueName: \"kubernetes.io/projected/76ccc809-4977-44c4-b910-66568fd64ad8-kube-api-access-ssxk5\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:17:12.386925 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:17:12.386921 2555 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/76ccc809-4977-44c4-b910-66568fd64ad8-webhook-cert\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:17:12.764643 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:17:12.764615 2555 generic.go:356] "Generic (PLEG): container finished" podID="76ccc809-4977-44c4-b910-66568fd64ad8" containerID="fa3a6b8d5113b84b08c2b115bf5f8d33570cb38e63c5dad9787d19a7f5424276" exitCode=0 Jul 26 15:17:12.764986 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:17:12.764664 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7fc4ffc667-cnbkt" Jul 26 15:17:12.764986 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:17:12.764666 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7fc4ffc667-cnbkt" event={"ID":"76ccc809-4977-44c4-b910-66568fd64ad8","Type":"ContainerDied","Data":"fa3a6b8d5113b84b08c2b115bf5f8d33570cb38e63c5dad9787d19a7f5424276"} Jul 26 15:17:12.764986 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:17:12.764726 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7fc4ffc667-cnbkt" event={"ID":"76ccc809-4977-44c4-b910-66568fd64ad8","Type":"ContainerDied","Data":"fc7de194b3c4718236bf261af2690b8cf7359c98a8359f623764f1a68e7a902e"} Jul 26 15:17:12.764986 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:17:12.764756 2555 scope.go:117] "RemoveContainer" containerID="fa3a6b8d5113b84b08c2b115bf5f8d33570cb38e63c5dad9787d19a7f5424276" Jul 26 15:17:12.773528 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:17:12.773510 2555 scope.go:117] "RemoveContainer" containerID="fa3a6b8d5113b84b08c2b115bf5f8d33570cb38e63c5dad9787d19a7f5424276" Jul 26 15:17:12.773753 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:17:12.773734 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa3a6b8d5113b84b08c2b115bf5f8d33570cb38e63c5dad9787d19a7f5424276\": container with ID starting with fa3a6b8d5113b84b08c2b115bf5f8d33570cb38e63c5dad9787d19a7f5424276 not found: ID does not exist" containerID="fa3a6b8d5113b84b08c2b115bf5f8d33570cb38e63c5dad9787d19a7f5424276" Jul 26 15:17:12.773802 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:17:12.773760 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa3a6b8d5113b84b08c2b115bf5f8d33570cb38e63c5dad9787d19a7f5424276"} err="failed to get container status \"fa3a6b8d5113b84b08c2b115bf5f8d33570cb38e63c5dad9787d19a7f5424276\": rpc error: code = NotFound desc = could not find container \"fa3a6b8d5113b84b08c2b115bf5f8d33570cb38e63c5dad9787d19a7f5424276\": container with ID starting with fa3a6b8d5113b84b08c2b115bf5f8d33570cb38e63c5dad9787d19a7f5424276 not found: ID does not exist" Jul 26 15:17:12.784140 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:17:12.784117 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7fc4ffc667-cnbkt"] Jul 26 15:17:12.788342 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:17:12.788318 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7fc4ffc667-cnbkt"] Jul 26 15:17:14.064538 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:17:14.064496 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76ccc809-4977-44c4-b910-66568fd64ad8" path="/var/lib/kubelet/pods/76ccc809-4977-44c4-b910-66568fd64ad8/volumes" Jul 26 15:17:29.819780 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:17:29.819747 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:17:29.920079 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:17:29.920050 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:17:30.445751 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:17:30.445714 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:17:30.637915 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:17:30.637885 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:17:30.878812 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:17:30.878774 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:17:30.979980 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:17:30.979945 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:17:40.074625 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:17:40.074570 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:17:40.173462 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:17:40.173436 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:17:42.128137 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:17:42.128097 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:17:58.970021 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:17:58.969976 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:18:04.328913 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:04.328879 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:18:22.830859 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:22.830830 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-f6bg2"] Jul 26 15:18:22.831378 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:22.831166 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76ccc809-4977-44c4-b910-66568fd64ad8" containerName="manager" Jul 26 15:18:22.831378 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:22.831179 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="76ccc809-4977-44c4-b910-66568fd64ad8" containerName="manager" Jul 26 15:18:22.831378 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:22.831258 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="76ccc809-4977-44c4-b910-66568fd64ad8" containerName="manager" Jul 26 15:18:22.834217 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:22.834198 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-f6bg2" Jul 26 15:18:22.836242 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:22.836216 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 26 15:18:22.836348 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:22.836304 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 26 15:18:22.836687 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:22.836665 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-ks67c\"" Jul 26 15:18:22.836764 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:22.836708 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 26 15:18:22.842571 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:22.842549 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-f6bg2"] Jul 26 15:18:22.867166 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:22.867139 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:18:22.921845 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:22.921817 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee63dd78-4e6c-41dc-a11e-8f1b6ad335ab-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-f6bg2\" (UID: \"ee63dd78-4e6c-41dc-a11e-8f1b6ad335ab\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-f6bg2" Jul 26 15:18:22.921965 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:22.921851 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee63dd78-4e6c-41dc-a11e-8f1b6ad335ab-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-f6bg2\" (UID: \"ee63dd78-4e6c-41dc-a11e-8f1b6ad335ab\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-f6bg2" Jul 26 15:18:22.921965 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:22.921870 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee63dd78-4e6c-41dc-a11e-8f1b6ad335ab-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-f6bg2\" (UID: \"ee63dd78-4e6c-41dc-a11e-8f1b6ad335ab\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-f6bg2" Jul 26 15:18:22.922062 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:22.921967 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee63dd78-4e6c-41dc-a11e-8f1b6ad335ab-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-f6bg2\" (UID: \"ee63dd78-4e6c-41dc-a11e-8f1b6ad335ab\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-f6bg2" Jul 26 15:18:22.922062 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:22.922023 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee63dd78-4e6c-41dc-a11e-8f1b6ad335ab-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-f6bg2\" (UID: \"ee63dd78-4e6c-41dc-a11e-8f1b6ad335ab\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-f6bg2" Jul 26 15:18:22.922139 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:22.922097 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwkxx\" (UniqueName: \"kubernetes.io/projected/ee63dd78-4e6c-41dc-a11e-8f1b6ad335ab-kube-api-access-jwkxx\") pod \"e2e-trlp-test-simulated-kserve-598466548d-f6bg2\" (UID: \"ee63dd78-4e6c-41dc-a11e-8f1b6ad335ab\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-f6bg2" Jul 26 15:18:23.022441 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:23.022416 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwkxx\" (UniqueName: \"kubernetes.io/projected/ee63dd78-4e6c-41dc-a11e-8f1b6ad335ab-kube-api-access-jwkxx\") pod \"e2e-trlp-test-simulated-kserve-598466548d-f6bg2\" (UID: \"ee63dd78-4e6c-41dc-a11e-8f1b6ad335ab\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-f6bg2" Jul 26 15:18:23.022547 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:23.022446 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee63dd78-4e6c-41dc-a11e-8f1b6ad335ab-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-f6bg2\" (UID: \"ee63dd78-4e6c-41dc-a11e-8f1b6ad335ab\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-f6bg2" Jul 26 15:18:23.022547 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:23.022467 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee63dd78-4e6c-41dc-a11e-8f1b6ad335ab-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-f6bg2\" (UID: \"ee63dd78-4e6c-41dc-a11e-8f1b6ad335ab\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-f6bg2" Jul 26 15:18:23.022547 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:23.022484 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee63dd78-4e6c-41dc-a11e-8f1b6ad335ab-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-f6bg2\" (UID: \"ee63dd78-4e6c-41dc-a11e-8f1b6ad335ab\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-f6bg2" Jul 26 15:18:23.022547 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:23.022528 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee63dd78-4e6c-41dc-a11e-8f1b6ad335ab-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-f6bg2\" (UID: \"ee63dd78-4e6c-41dc-a11e-8f1b6ad335ab\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-f6bg2" Jul 26 15:18:23.022757 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:23.022588 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee63dd78-4e6c-41dc-a11e-8f1b6ad335ab-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-f6bg2\" (UID: \"ee63dd78-4e6c-41dc-a11e-8f1b6ad335ab\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-f6bg2" Jul 26 15:18:23.022852 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:23.022833 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee63dd78-4e6c-41dc-a11e-8f1b6ad335ab-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-f6bg2\" (UID: \"ee63dd78-4e6c-41dc-a11e-8f1b6ad335ab\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-f6bg2" Jul 26 15:18:23.022914 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:23.022890 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee63dd78-4e6c-41dc-a11e-8f1b6ad335ab-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-f6bg2\" (UID: \"ee63dd78-4e6c-41dc-a11e-8f1b6ad335ab\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-f6bg2" Jul 26 15:18:23.022988 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:23.022949 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee63dd78-4e6c-41dc-a11e-8f1b6ad335ab-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-f6bg2\" (UID: \"ee63dd78-4e6c-41dc-a11e-8f1b6ad335ab\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-f6bg2" Jul 26 15:18:23.024724 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:23.024701 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee63dd78-4e6c-41dc-a11e-8f1b6ad335ab-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-f6bg2\" (UID: \"ee63dd78-4e6c-41dc-a11e-8f1b6ad335ab\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-f6bg2" Jul 26 15:18:23.025109 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:23.025090 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee63dd78-4e6c-41dc-a11e-8f1b6ad335ab-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-f6bg2\" (UID: \"ee63dd78-4e6c-41dc-a11e-8f1b6ad335ab\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-f6bg2" Jul 26 15:18:23.030553 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:23.030535 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwkxx\" (UniqueName: \"kubernetes.io/projected/ee63dd78-4e6c-41dc-a11e-8f1b6ad335ab-kube-api-access-jwkxx\") pod \"e2e-trlp-test-simulated-kserve-598466548d-f6bg2\" (UID: \"ee63dd78-4e6c-41dc-a11e-8f1b6ad335ab\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-f6bg2" Jul 26 15:18:23.145458 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:23.145436 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-f6bg2" Jul 26 15:18:23.268801 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:23.268752 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-f6bg2"] Jul 26 15:18:23.271019 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:18:23.270985 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee63dd78_4e6c_41dc_a11e_8f1b6ad335ab.slice/crio-c3ca435caa2d2491430aef2bbe482baddbaf3bf47ca5e879913ac519eca0ba7b WatchSource:0}: Error finding container c3ca435caa2d2491430aef2bbe482baddbaf3bf47ca5e879913ac519eca0ba7b: Status 404 returned error can't find the container with id c3ca435caa2d2491430aef2bbe482baddbaf3bf47ca5e879913ac519eca0ba7b Jul 26 15:18:24.079335 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:24.079292 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-f6bg2" event={"ID":"ee63dd78-4e6c-41dc-a11e-8f1b6ad335ab","Type":"ContainerStarted","Data":"c3ca435caa2d2491430aef2bbe482baddbaf3bf47ca5e879913ac519eca0ba7b"} Jul 26 15:18:25.085878 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:25.085839 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-f6bg2" event={"ID":"ee63dd78-4e6c-41dc-a11e-8f1b6ad335ab","Type":"ContainerStarted","Data":"9db70222f8495bcd8ef5c38bcfdb5d9ef1c9dc66da959e3440a0e1c8a10ddb53"} Jul 26 15:18:25.101464 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:25.101420 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-f6bg2" podStartSLOduration=2.097387186 podStartE2EDuration="3.101405163s" totalImagesPullingTime="1.004017977s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:18:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:18:25.100249803 +0000 UTC m=+785.524096685" watchObservedRunningTime="2026-07-26 15:18:25.101405163 +0000 UTC m=+785.525252033" Jul 26 15:18:33.146578 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:33.146535 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-f6bg2" Jul 26 15:18:33.146973 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:33.146620 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-f6bg2" Jul 26 15:18:33.159159 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:33.159136 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-f6bg2" Jul 26 15:18:34.138066 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:34.138031 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-f6bg2" Jul 26 15:18:35.331138 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:35.331107 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrck9"] Jul 26 15:18:35.334453 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:35.334437 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrck9" Jul 26 15:18:35.336348 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:35.336320 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 26 15:18:35.344556 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:35.344537 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrck9"] Jul 26 15:18:35.355495 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:35.355451 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:18:35.433215 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:35.433196 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56c46022-da23-42c1-aa80-b3e0229ce1a2-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrck9\" (UID: \"56c46022-da23-42c1-aa80-b3e0229ce1a2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrck9" Jul 26 15:18:35.433330 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:35.433225 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k827q\" (UniqueName: \"kubernetes.io/projected/56c46022-da23-42c1-aa80-b3e0229ce1a2-kube-api-access-k827q\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrck9\" (UID: \"56c46022-da23-42c1-aa80-b3e0229ce1a2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrck9" Jul 26 15:18:35.433330 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:35.433246 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56c46022-da23-42c1-aa80-b3e0229ce1a2-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrck9\" (UID: \"56c46022-da23-42c1-aa80-b3e0229ce1a2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrck9" Jul 26 15:18:35.433330 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:35.433265 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56c46022-da23-42c1-aa80-b3e0229ce1a2-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrck9\" (UID: \"56c46022-da23-42c1-aa80-b3e0229ce1a2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrck9" Jul 26 15:18:35.433469 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:35.433379 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56c46022-da23-42c1-aa80-b3e0229ce1a2-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrck9\" (UID: \"56c46022-da23-42c1-aa80-b3e0229ce1a2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrck9" Jul 26 15:18:35.433469 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:35.433411 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56c46022-da23-42c1-aa80-b3e0229ce1a2-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrck9\" (UID: \"56c46022-da23-42c1-aa80-b3e0229ce1a2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrck9" Jul 26 15:18:35.534324 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:35.534297 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56c46022-da23-42c1-aa80-b3e0229ce1a2-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrck9\" (UID: \"56c46022-da23-42c1-aa80-b3e0229ce1a2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrck9" Jul 26 15:18:35.534441 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:35.534336 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k827q\" (UniqueName: \"kubernetes.io/projected/56c46022-da23-42c1-aa80-b3e0229ce1a2-kube-api-access-k827q\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrck9\" (UID: \"56c46022-da23-42c1-aa80-b3e0229ce1a2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrck9" Jul 26 15:18:35.534441 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:35.534363 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56c46022-da23-42c1-aa80-b3e0229ce1a2-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrck9\" (UID: \"56c46022-da23-42c1-aa80-b3e0229ce1a2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrck9" Jul 26 15:18:35.534441 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:35.534389 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56c46022-da23-42c1-aa80-b3e0229ce1a2-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrck9\" (UID: \"56c46022-da23-42c1-aa80-b3e0229ce1a2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrck9" Jul 26 15:18:35.534588 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:35.534457 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56c46022-da23-42c1-aa80-b3e0229ce1a2-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrck9\" (UID: \"56c46022-da23-42c1-aa80-b3e0229ce1a2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrck9" Jul 26 15:18:35.534645 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:35.534586 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56c46022-da23-42c1-aa80-b3e0229ce1a2-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrck9\" (UID: \"56c46022-da23-42c1-aa80-b3e0229ce1a2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrck9" Jul 26 15:18:35.534698 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:35.534671 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56c46022-da23-42c1-aa80-b3e0229ce1a2-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrck9\" (UID: \"56c46022-da23-42c1-aa80-b3e0229ce1a2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrck9" Jul 26 15:18:35.534751 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:35.534715 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56c46022-da23-42c1-aa80-b3e0229ce1a2-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrck9\" (UID: \"56c46022-da23-42c1-aa80-b3e0229ce1a2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrck9" Jul 26 15:18:35.534751 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:35.534744 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56c46022-da23-42c1-aa80-b3e0229ce1a2-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrck9\" (UID: \"56c46022-da23-42c1-aa80-b3e0229ce1a2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrck9" Jul 26 15:18:35.536620 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:35.536592 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56c46022-da23-42c1-aa80-b3e0229ce1a2-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrck9\" (UID: \"56c46022-da23-42c1-aa80-b3e0229ce1a2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrck9" Jul 26 15:18:35.536765 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:35.536747 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56c46022-da23-42c1-aa80-b3e0229ce1a2-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrck9\" (UID: \"56c46022-da23-42c1-aa80-b3e0229ce1a2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrck9" Jul 26 15:18:35.540832 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:35.540809 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k827q\" (UniqueName: \"kubernetes.io/projected/56c46022-da23-42c1-aa80-b3e0229ce1a2-kube-api-access-k827q\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrck9\" (UID: \"56c46022-da23-42c1-aa80-b3e0229ce1a2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrck9" Jul 26 15:18:35.644971 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:35.644947 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrck9" Jul 26 15:18:35.765893 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:35.765870 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrck9"] Jul 26 15:18:35.767696 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:18:35.767664 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56c46022_da23_42c1_aa80_b3e0229ce1a2.slice/crio-bf03ddb6566236c198ad84c90f8b5dad1ee0b04b36bea3c5f3c37384da2a988f WatchSource:0}: Error finding container bf03ddb6566236c198ad84c90f8b5dad1ee0b04b36bea3c5f3c37384da2a988f: Status 404 returned error can't find the container with id bf03ddb6566236c198ad84c90f8b5dad1ee0b04b36bea3c5f3c37384da2a988f Jul 26 15:18:36.136349 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:36.136312 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrck9" event={"ID":"56c46022-da23-42c1-aa80-b3e0229ce1a2","Type":"ContainerStarted","Data":"9a17548f40ebc5c788fef44e35d55feba7ad79a9c3a6d6f337fca58611006e10"} Jul 26 15:18:36.136349 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:36.136357 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrck9" event={"ID":"56c46022-da23-42c1-aa80-b3e0229ce1a2","Type":"ContainerStarted","Data":"bf03ddb6566236c198ad84c90f8b5dad1ee0b04b36bea3c5f3c37384da2a988f"} Jul 26 15:18:36.152391 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:36.152350 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrck9" podStartSLOduration=0.984404455 podStartE2EDuration="1.152337065s" totalImagesPullingTime="167.93261ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:18:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:18:36.151546086 +0000 UTC m=+796.575392957" watchObservedRunningTime="2026-07-26 15:18:36.152337065 +0000 UTC m=+796.576183952" Jul 26 15:18:45.645307 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:45.645219 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrck9" Jul 26 15:18:45.645768 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:45.645325 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrck9" Jul 26 15:18:45.657195 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:45.657172 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrck9" Jul 26 15:18:46.196916 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:46.196887 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrck9" Jul 26 15:18:57.138465 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:18:57.138426 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:19:10.426098 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:19:10.426065 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-6dp5m"] Jul 26 15:19:10.429821 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:19:10.429795 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6dp5m" Jul 26 15:19:10.431781 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:19:10.431763 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 26 15:19:10.437720 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:19:10.437701 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-6dp5m"] Jul 26 15:19:10.462963 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:19:10.462912 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:19:10.511226 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:19:10.511200 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8073ae4b-3724-4254-b3c6-0a4507caae4b-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6dp5m\" (UID: \"8073ae4b-3724-4254-b3c6-0a4507caae4b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6dp5m" Jul 26 15:19:10.511350 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:19:10.511235 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjdzv\" (UniqueName: \"kubernetes.io/projected/8073ae4b-3724-4254-b3c6-0a4507caae4b-kube-api-access-mjdzv\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6dp5m\" (UID: \"8073ae4b-3724-4254-b3c6-0a4507caae4b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6dp5m" Jul 26 15:19:10.511350 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:19:10.511259 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8073ae4b-3724-4254-b3c6-0a4507caae4b-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6dp5m\" (UID: \"8073ae4b-3724-4254-b3c6-0a4507caae4b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6dp5m" Jul 26 15:19:10.511350 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:19:10.511324 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8073ae4b-3724-4254-b3c6-0a4507caae4b-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6dp5m\" (UID: \"8073ae4b-3724-4254-b3c6-0a4507caae4b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6dp5m" Jul 26 15:19:10.511492 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:19:10.511376 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8073ae4b-3724-4254-b3c6-0a4507caae4b-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6dp5m\" (UID: \"8073ae4b-3724-4254-b3c6-0a4507caae4b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6dp5m" Jul 26 15:19:10.511492 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:19:10.511404 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8073ae4b-3724-4254-b3c6-0a4507caae4b-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6dp5m\" (UID: \"8073ae4b-3724-4254-b3c6-0a4507caae4b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6dp5m" Jul 26 15:19:10.613371 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:19:10.612645 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8073ae4b-3724-4254-b3c6-0a4507caae4b-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6dp5m\" (UID: \"8073ae4b-3724-4254-b3c6-0a4507caae4b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6dp5m" Jul 26 15:19:10.613371 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:19:10.612742 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjdzv\" (UniqueName: \"kubernetes.io/projected/8073ae4b-3724-4254-b3c6-0a4507caae4b-kube-api-access-mjdzv\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6dp5m\" (UID: \"8073ae4b-3724-4254-b3c6-0a4507caae4b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6dp5m" Jul 26 15:19:10.613371 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:19:10.612794 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8073ae4b-3724-4254-b3c6-0a4507caae4b-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6dp5m\" (UID: \"8073ae4b-3724-4254-b3c6-0a4507caae4b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6dp5m" Jul 26 15:19:10.613371 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:19:10.612829 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8073ae4b-3724-4254-b3c6-0a4507caae4b-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6dp5m\" (UID: \"8073ae4b-3724-4254-b3c6-0a4507caae4b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6dp5m" Jul 26 15:19:10.613371 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:19:10.612875 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8073ae4b-3724-4254-b3c6-0a4507caae4b-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6dp5m\" (UID: \"8073ae4b-3724-4254-b3c6-0a4507caae4b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6dp5m" Jul 26 15:19:10.613371 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:19:10.612903 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8073ae4b-3724-4254-b3c6-0a4507caae4b-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6dp5m\" (UID: \"8073ae4b-3724-4254-b3c6-0a4507caae4b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6dp5m" Jul 26 15:19:10.613371 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:19:10.613056 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8073ae4b-3724-4254-b3c6-0a4507caae4b-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6dp5m\" (UID: \"8073ae4b-3724-4254-b3c6-0a4507caae4b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6dp5m" Jul 26 15:19:10.613371 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:19:10.613309 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8073ae4b-3724-4254-b3c6-0a4507caae4b-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6dp5m\" (UID: \"8073ae4b-3724-4254-b3c6-0a4507caae4b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6dp5m" Jul 26 15:19:10.613856 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:19:10.613789 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8073ae4b-3724-4254-b3c6-0a4507caae4b-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6dp5m\" (UID: \"8073ae4b-3724-4254-b3c6-0a4507caae4b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6dp5m" Jul 26 15:19:10.617951 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:19:10.615951 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8073ae4b-3724-4254-b3c6-0a4507caae4b-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6dp5m\" (UID: \"8073ae4b-3724-4254-b3c6-0a4507caae4b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6dp5m" Jul 26 15:19:10.619836 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:19:10.619357 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8073ae4b-3724-4254-b3c6-0a4507caae4b-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6dp5m\" (UID: \"8073ae4b-3724-4254-b3c6-0a4507caae4b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6dp5m" Jul 26 15:19:10.620647 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:19:10.620627 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjdzv\" (UniqueName: \"kubernetes.io/projected/8073ae4b-3724-4254-b3c6-0a4507caae4b-kube-api-access-mjdzv\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6dp5m\" (UID: \"8073ae4b-3724-4254-b3c6-0a4507caae4b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6dp5m" Jul 26 15:19:10.739645 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:19:10.739586 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6dp5m" Jul 26 15:19:10.862075 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:19:10.862020 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-6dp5m"] Jul 26 15:19:10.864255 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:19:10.864231 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8073ae4b_3724_4254_b3c6_0a4507caae4b.slice/crio-23c86b2d49c1df46b20e005bc6b575208051514f8a76837132100cef70a1bbf8 WatchSource:0}: Error finding container 23c86b2d49c1df46b20e005bc6b575208051514f8a76837132100cef70a1bbf8: Status 404 returned error can't find the container with id 23c86b2d49c1df46b20e005bc6b575208051514f8a76837132100cef70a1bbf8 Jul 26 15:19:11.297419 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:19:11.297338 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6dp5m" event={"ID":"8073ae4b-3724-4254-b3c6-0a4507caae4b","Type":"ContainerStarted","Data":"83a434d7557eb6d5f70aef76d866128ce47600d7aa34043483e1e4af0f459bd0"} Jul 26 15:19:11.297419 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:19:11.297376 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6dp5m" event={"ID":"8073ae4b-3724-4254-b3c6-0a4507caae4b","Type":"ContainerStarted","Data":"23c86b2d49c1df46b20e005bc6b575208051514f8a76837132100cef70a1bbf8"} Jul 26 15:19:11.316000 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:19:11.315953 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6dp5m" podStartSLOduration=1.153671645 podStartE2EDuration="1.315915802s" totalImagesPullingTime="162.244157ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:19:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:19:11.315708887 +0000 UTC m=+831.739555763" watchObservedRunningTime="2026-07-26 15:19:11.315915802 +0000 UTC m=+831.739762674" Jul 26 15:19:20.739695 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:19:20.739650 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6dp5m" Jul 26 15:19:20.739695 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:19:20.739703 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6dp5m" Jul 26 15:19:20.751921 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:19:20.751892 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6dp5m" Jul 26 15:19:21.352349 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:19:21.352319 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6dp5m" Jul 26 15:20:19.248400 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:20:19.248368 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:20:20.016785 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:20:20.016756 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9f857_1fbafda6-6242-400e-b2d4-24584608e5f5/ovn-acl-logging/0.log" Jul 26 15:20:20.016992 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:20:20.016853 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9f857_1fbafda6-6242-400e-b2d4-24584608e5f5/ovn-acl-logging/0.log" Jul 26 15:20:20.699375 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:20:20.699342 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:20:25.860925 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:20:25.860894 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7c6bc86c6d-74nmr"] Jul 26 15:20:25.864096 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:20:25.864080 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c6bc86c6d-74nmr" Jul 26 15:20:25.872565 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:20:25.872544 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7c6bc86c6d-74nmr"] Jul 26 15:20:25.995675 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:20:25.995652 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpssr\" (UniqueName: \"kubernetes.io/projected/4096ea8f-3158-4364-9858-346b97b711a7-kube-api-access-fpssr\") pod \"authorino-7c6bc86c6d-74nmr\" (UID: \"4096ea8f-3158-4364-9858-346b97b711a7\") " pod="kuadrant-system/authorino-7c6bc86c6d-74nmr" Jul 26 15:20:25.995819 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:20:25.995691 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4096ea8f-3158-4364-9858-346b97b711a7-tls-cert\") pod \"authorino-7c6bc86c6d-74nmr\" (UID: \"4096ea8f-3158-4364-9858-346b97b711a7\") " pod="kuadrant-system/authorino-7c6bc86c6d-74nmr" Jul 26 15:20:26.096726 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:20:26.096700 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpssr\" (UniqueName: \"kubernetes.io/projected/4096ea8f-3158-4364-9858-346b97b711a7-kube-api-access-fpssr\") pod \"authorino-7c6bc86c6d-74nmr\" (UID: \"4096ea8f-3158-4364-9858-346b97b711a7\") " pod="kuadrant-system/authorino-7c6bc86c6d-74nmr" Jul 26 15:20:26.096861 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:20:26.096741 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4096ea8f-3158-4364-9858-346b97b711a7-tls-cert\") pod \"authorino-7c6bc86c6d-74nmr\" (UID: \"4096ea8f-3158-4364-9858-346b97b711a7\") " pod="kuadrant-system/authorino-7c6bc86c6d-74nmr" Jul 26 15:20:26.098976 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:20:26.098956 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4096ea8f-3158-4364-9858-346b97b711a7-tls-cert\") pod \"authorino-7c6bc86c6d-74nmr\" (UID: \"4096ea8f-3158-4364-9858-346b97b711a7\") " pod="kuadrant-system/authorino-7c6bc86c6d-74nmr" Jul 26 15:20:26.112109 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:20:26.112056 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpssr\" (UniqueName: \"kubernetes.io/projected/4096ea8f-3158-4364-9858-346b97b711a7-kube-api-access-fpssr\") pod \"authorino-7c6bc86c6d-74nmr\" (UID: \"4096ea8f-3158-4364-9858-346b97b711a7\") " pod="kuadrant-system/authorino-7c6bc86c6d-74nmr" Jul 26 15:20:26.126411 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:20:26.126379 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:20:26.173501 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:20:26.173480 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c6bc86c6d-74nmr" Jul 26 15:20:26.224944 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:20:26.223959 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:20:26.303437 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:20:26.303412 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7c6bc86c6d-74nmr"] Jul 26 15:20:26.305751 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:20:26.305724 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4096ea8f_3158_4364_9858_346b97b711a7.slice/crio-e4a83bd07bfc402be016ed581e66815f4c5e1c5bb73c8270c61e1e1fc753d9c0 WatchSource:0}: Error finding container e4a83bd07bfc402be016ed581e66815f4c5e1c5bb73c8270c61e1e1fc753d9c0: Status 404 returned error can't find the container with id e4a83bd07bfc402be016ed581e66815f4c5e1c5bb73c8270c61e1e1fc753d9c0 Jul 26 15:20:26.625359 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:20:26.625324 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c6bc86c6d-74nmr" event={"ID":"4096ea8f-3158-4364-9858-346b97b711a7","Type":"ContainerStarted","Data":"e4a83bd07bfc402be016ed581e66815f4c5e1c5bb73c8270c61e1e1fc753d9c0"} Jul 26 15:20:27.262757 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:20:27.262715 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:20:27.354116 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:20:27.354077 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:20:27.632418 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:20:27.632384 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c6bc86c6d-74nmr" event={"ID":"4096ea8f-3158-4364-9858-346b97b711a7","Type":"ContainerStarted","Data":"d6c53e3ff756c879193238b7d12c181dd7c6d6f74802f2cb42fa5971b465a626"} Jul 26 15:20:27.647186 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:20:27.647127 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7c6bc86c6d-74nmr" podStartSLOduration=2.221800471 podStartE2EDuration="2.647108072s" totalImagesPullingTime="425.307601ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:20:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:20:27.646523243 +0000 UTC m=+908.070370113" watchObservedRunningTime="2026-07-26 15:20:27.647108072 +0000 UTC m=+908.070954945" Jul 26 15:20:27.674351 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:20:27.674317 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5fdd89c4b9-wcwqg"] Jul 26 15:20:27.674577 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:20:27.674542 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5fdd89c4b9-wcwqg" podUID="7d575e37-bb9f-427e-86a1-9df7fa029043" containerName="authorino" containerID="cri-o://65ab3f2921e49f1c30fcdc0695c236e06307a5e27153f46f316b1fec33860061" gracePeriod=30 Jul 26 15:20:27.919249 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:20:27.919227 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5fdd89c4b9-wcwqg" Jul 26 15:20:28.014555 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:20:28.014524 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvrxv\" (UniqueName: \"kubernetes.io/projected/7d575e37-bb9f-427e-86a1-9df7fa029043-kube-api-access-jvrxv\") pod \"7d575e37-bb9f-427e-86a1-9df7fa029043\" (UID: \"7d575e37-bb9f-427e-86a1-9df7fa029043\") " Jul 26 15:20:28.014555 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:20:28.014554 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7d575e37-bb9f-427e-86a1-9df7fa029043-tls-cert\") pod \"7d575e37-bb9f-427e-86a1-9df7fa029043\" (UID: \"7d575e37-bb9f-427e-86a1-9df7fa029043\") " Jul 26 15:20:28.016555 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:20:28.016521 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d575e37-bb9f-427e-86a1-9df7fa029043-kube-api-access-jvrxv" (OuterVolumeSpecName: "kube-api-access-jvrxv") pod "7d575e37-bb9f-427e-86a1-9df7fa029043" (UID: "7d575e37-bb9f-427e-86a1-9df7fa029043"). InnerVolumeSpecName "kube-api-access-jvrxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:20:28.023755 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:20:28.023730 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d575e37-bb9f-427e-86a1-9df7fa029043-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "7d575e37-bb9f-427e-86a1-9df7fa029043" (UID: "7d575e37-bb9f-427e-86a1-9df7fa029043"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 15:20:28.116152 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:20:28.116120 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvrxv\" (UniqueName: \"kubernetes.io/projected/7d575e37-bb9f-427e-86a1-9df7fa029043-kube-api-access-jvrxv\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:20:28.116152 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:20:28.116150 2555 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7d575e37-bb9f-427e-86a1-9df7fa029043-tls-cert\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:20:28.638083 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:20:28.638044 2555 generic.go:356] "Generic (PLEG): container finished" podID="7d575e37-bb9f-427e-86a1-9df7fa029043" containerID="65ab3f2921e49f1c30fcdc0695c236e06307a5e27153f46f316b1fec33860061" exitCode=0 Jul 26 15:20:28.638083 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:20:28.638081 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5fdd89c4b9-wcwqg" Jul 26 15:20:28.638591 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:20:28.638131 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5fdd89c4b9-wcwqg" event={"ID":"7d575e37-bb9f-427e-86a1-9df7fa029043","Type":"ContainerDied","Data":"65ab3f2921e49f1c30fcdc0695c236e06307a5e27153f46f316b1fec33860061"} Jul 26 15:20:28.638591 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:20:28.638179 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5fdd89c4b9-wcwqg" event={"ID":"7d575e37-bb9f-427e-86a1-9df7fa029043","Type":"ContainerDied","Data":"cae87d4c93c948e621086c79e952db640f84f66c778cbf0c7ac4aeb995cc1993"} Jul 26 15:20:28.638591 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:20:28.638203 2555 scope.go:117] "RemoveContainer" containerID="65ab3f2921e49f1c30fcdc0695c236e06307a5e27153f46f316b1fec33860061" Jul 26 15:20:28.647222 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:20:28.647196 2555 scope.go:117] "RemoveContainer" containerID="65ab3f2921e49f1c30fcdc0695c236e06307a5e27153f46f316b1fec33860061" Jul 26 15:20:28.647494 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:20:28.647475 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65ab3f2921e49f1c30fcdc0695c236e06307a5e27153f46f316b1fec33860061\": container with ID starting with 65ab3f2921e49f1c30fcdc0695c236e06307a5e27153f46f316b1fec33860061 not found: ID does not exist" containerID="65ab3f2921e49f1c30fcdc0695c236e06307a5e27153f46f316b1fec33860061" Jul 26 15:20:28.647544 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:20:28.647503 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65ab3f2921e49f1c30fcdc0695c236e06307a5e27153f46f316b1fec33860061"} err="failed to get container status \"65ab3f2921e49f1c30fcdc0695c236e06307a5e27153f46f316b1fec33860061\": rpc error: code = NotFound desc = could not find container \"65ab3f2921e49f1c30fcdc0695c236e06307a5e27153f46f316b1fec33860061\": container with ID starting with 65ab3f2921e49f1c30fcdc0695c236e06307a5e27153f46f316b1fec33860061 not found: ID does not exist" Jul 26 15:20:28.653063 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:20:28.653038 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5fdd89c4b9-wcwqg"] Jul 26 15:20:28.657503 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:20:28.657476 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5fdd89c4b9-wcwqg"] Jul 26 15:20:30.065204 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:20:30.065170 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d575e37-bb9f-427e-86a1-9df7fa029043" path="/var/lib/kubelet/pods/7d575e37-bb9f-427e-86a1-9df7fa029043/volumes" Jul 26 15:21:00.543460 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:21:00.543422 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:22:03.661153 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:03.661102 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:22:03.714007 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:03.713968 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:22:14.917718 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:14.917685 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-766dd44bbb-7qgnw"] Jul 26 15:22:14.918097 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:14.918033 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d575e37-bb9f-427e-86a1-9df7fa029043" containerName="authorino" Jul 26 15:22:14.918097 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:14.918045 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d575e37-bb9f-427e-86a1-9df7fa029043" containerName="authorino" Jul 26 15:22:14.918097 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:14.918098 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d575e37-bb9f-427e-86a1-9df7fa029043" containerName="authorino" Jul 26 15:22:14.921200 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:14.921172 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-766dd44bbb-7qgnw" Jul 26 15:22:14.923718 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:14.923702 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 26 15:22:14.923840 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:14.923722 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ccgw4\"" Jul 26 15:22:14.923840 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:14.923703 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 26 15:22:14.930401 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:14.929145 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-766dd44bbb-7qgnw"] Jul 26 15:22:14.984617 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:14.984594 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6m4x\" (UniqueName: \"kubernetes.io/projected/6bb28251-70d3-4aab-88ff-00ac2102c2de-kube-api-access-m6m4x\") pod \"maas-controller-766dd44bbb-7qgnw\" (UID: \"6bb28251-70d3-4aab-88ff-00ac2102c2de\") " pod="opendatahub/maas-controller-766dd44bbb-7qgnw" Jul 26 15:22:14.984747 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:14.984672 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6bb28251-70d3-4aab-88ff-00ac2102c2de-webhook-cert\") pod \"maas-controller-766dd44bbb-7qgnw\" (UID: \"6bb28251-70d3-4aab-88ff-00ac2102c2de\") " pod="opendatahub/maas-controller-766dd44bbb-7qgnw" Jul 26 15:22:15.085302 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:15.085277 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6bb28251-70d3-4aab-88ff-00ac2102c2de-webhook-cert\") pod \"maas-controller-766dd44bbb-7qgnw\" (UID: \"6bb28251-70d3-4aab-88ff-00ac2102c2de\") " pod="opendatahub/maas-controller-766dd44bbb-7qgnw" Jul 26 15:22:15.085410 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:15.085314 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6m4x\" (UniqueName: \"kubernetes.io/projected/6bb28251-70d3-4aab-88ff-00ac2102c2de-kube-api-access-m6m4x\") pod \"maas-controller-766dd44bbb-7qgnw\" (UID: \"6bb28251-70d3-4aab-88ff-00ac2102c2de\") " pod="opendatahub/maas-controller-766dd44bbb-7qgnw" Jul 26 15:22:15.087619 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:15.087601 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6bb28251-70d3-4aab-88ff-00ac2102c2de-webhook-cert\") pod \"maas-controller-766dd44bbb-7qgnw\" (UID: \"6bb28251-70d3-4aab-88ff-00ac2102c2de\") " pod="opendatahub/maas-controller-766dd44bbb-7qgnw" Jul 26 15:22:15.092788 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:15.092757 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6m4x\" (UniqueName: \"kubernetes.io/projected/6bb28251-70d3-4aab-88ff-00ac2102c2de-kube-api-access-m6m4x\") pod \"maas-controller-766dd44bbb-7qgnw\" (UID: \"6bb28251-70d3-4aab-88ff-00ac2102c2de\") " pod="opendatahub/maas-controller-766dd44bbb-7qgnw" Jul 26 15:22:15.235970 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:15.235895 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-766dd44bbb-7qgnw" Jul 26 15:22:15.362671 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:15.362647 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-766dd44bbb-7qgnw"] Jul 26 15:22:15.364599 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:22:15.364574 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bb28251_70d3_4aab_88ff_00ac2102c2de.slice/crio-6627d3712dc1ac9f62d48c455b7687a30272d1bd0ec797d8171f43828863b4a1 WatchSource:0}: Error finding container 6627d3712dc1ac9f62d48c455b7687a30272d1bd0ec797d8171f43828863b4a1: Status 404 returned error can't find the container with id 6627d3712dc1ac9f62d48c455b7687a30272d1bd0ec797d8171f43828863b4a1 Jul 26 15:22:15.365866 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:15.365849 2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 26 15:22:16.126966 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:16.126872 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-766dd44bbb-7qgnw" event={"ID":"6bb28251-70d3-4aab-88ff-00ac2102c2de","Type":"ContainerStarted","Data":"9c8d735b23aa28e1db79509cb32cedd153644aaaad0790ea6fe3111bc03a1af0"} Jul 26 15:22:16.126966 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:16.126921 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-766dd44bbb-7qgnw" event={"ID":"6bb28251-70d3-4aab-88ff-00ac2102c2de","Type":"ContainerStarted","Data":"6627d3712dc1ac9f62d48c455b7687a30272d1bd0ec797d8171f43828863b4a1"} Jul 26 15:22:16.127425 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:16.127002 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-766dd44bbb-7qgnw" Jul 26 15:22:16.154535 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:16.154477 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-766dd44bbb-7qgnw" podStartSLOduration=1.6539374869999999 podStartE2EDuration="2.154463673s" totalImagesPullingTime="500.526186ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:22:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:22:16.152050582 +0000 UTC m=+1016.575897452" watchObservedRunningTime="2026-07-26 15:22:16.154463673 +0000 UTC m=+1016.578310543" Jul 26 15:22:16.405008 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:16.404900 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:22:27.133296 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:27.133263 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-766dd44bbb-7qgnw" Jul 26 15:22:31.666703 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:31.666664 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:22:31.766308 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:31.766278 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-766dd44bbb-7qgnw"] Jul 26 15:22:31.766520 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:31.766483 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-766dd44bbb-7qgnw" podUID="6bb28251-70d3-4aab-88ff-00ac2102c2de" containerName="manager" containerID="cri-o://9c8d735b23aa28e1db79509cb32cedd153644aaaad0790ea6fe3111bc03a1af0" gracePeriod=30 Jul 26 15:22:32.001052 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:32.001029 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-766dd44bbb-7qgnw" Jul 26 15:22:32.117714 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:32.117681 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6m4x\" (UniqueName: \"kubernetes.io/projected/6bb28251-70d3-4aab-88ff-00ac2102c2de-kube-api-access-m6m4x\") pod \"6bb28251-70d3-4aab-88ff-00ac2102c2de\" (UID: \"6bb28251-70d3-4aab-88ff-00ac2102c2de\") " Jul 26 15:22:32.117862 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:32.117727 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6bb28251-70d3-4aab-88ff-00ac2102c2de-webhook-cert\") pod \"6bb28251-70d3-4aab-88ff-00ac2102c2de\" (UID: \"6bb28251-70d3-4aab-88ff-00ac2102c2de\") " Jul 26 15:22:32.119620 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:32.119590 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bb28251-70d3-4aab-88ff-00ac2102c2de-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "6bb28251-70d3-4aab-88ff-00ac2102c2de" (UID: "6bb28251-70d3-4aab-88ff-00ac2102c2de"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 15:22:32.119716 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:32.119600 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bb28251-70d3-4aab-88ff-00ac2102c2de-kube-api-access-m6m4x" (OuterVolumeSpecName: "kube-api-access-m6m4x") pod "6bb28251-70d3-4aab-88ff-00ac2102c2de" (UID: "6bb28251-70d3-4aab-88ff-00ac2102c2de"). InnerVolumeSpecName "kube-api-access-m6m4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:22:32.198347 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:32.198282 2555 generic.go:356] "Generic (PLEG): container finished" podID="6bb28251-70d3-4aab-88ff-00ac2102c2de" containerID="9c8d735b23aa28e1db79509cb32cedd153644aaaad0790ea6fe3111bc03a1af0" exitCode=0 Jul 26 15:22:32.198347 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:32.198326 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-766dd44bbb-7qgnw" Jul 26 15:22:32.198470 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:32.198361 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-766dd44bbb-7qgnw" event={"ID":"6bb28251-70d3-4aab-88ff-00ac2102c2de","Type":"ContainerDied","Data":"9c8d735b23aa28e1db79509cb32cedd153644aaaad0790ea6fe3111bc03a1af0"} Jul 26 15:22:32.198470 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:32.198412 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-766dd44bbb-7qgnw" event={"ID":"6bb28251-70d3-4aab-88ff-00ac2102c2de","Type":"ContainerDied","Data":"6627d3712dc1ac9f62d48c455b7687a30272d1bd0ec797d8171f43828863b4a1"} Jul 26 15:22:32.198470 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:32.198433 2555 scope.go:117] "RemoveContainer" containerID="9c8d735b23aa28e1db79509cb32cedd153644aaaad0790ea6fe3111bc03a1af0" Jul 26 15:22:32.208339 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:32.208316 2555 scope.go:117] "RemoveContainer" containerID="9c8d735b23aa28e1db79509cb32cedd153644aaaad0790ea6fe3111bc03a1af0" Jul 26 15:22:32.208757 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:22:32.208736 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c8d735b23aa28e1db79509cb32cedd153644aaaad0790ea6fe3111bc03a1af0\": container with ID starting with 9c8d735b23aa28e1db79509cb32cedd153644aaaad0790ea6fe3111bc03a1af0 not found: ID does not exist" containerID="9c8d735b23aa28e1db79509cb32cedd153644aaaad0790ea6fe3111bc03a1af0" Jul 26 15:22:32.208817 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:32.208763 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c8d735b23aa28e1db79509cb32cedd153644aaaad0790ea6fe3111bc03a1af0"} err="failed to get container status \"9c8d735b23aa28e1db79509cb32cedd153644aaaad0790ea6fe3111bc03a1af0\": rpc error: code = NotFound desc = could not find container \"9c8d735b23aa28e1db79509cb32cedd153644aaaad0790ea6fe3111bc03a1af0\": container with ID starting with 9c8d735b23aa28e1db79509cb32cedd153644aaaad0790ea6fe3111bc03a1af0 not found: ID does not exist" Jul 26 15:22:32.218994 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:32.218969 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6m4x\" (UniqueName: \"kubernetes.io/projected/6bb28251-70d3-4aab-88ff-00ac2102c2de-kube-api-access-m6m4x\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:22:32.219222 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:32.219000 2555 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6bb28251-70d3-4aab-88ff-00ac2102c2de-webhook-cert\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:22:32.220889 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:32.220870 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-766dd44bbb-7qgnw"] Jul 26 15:22:32.224660 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:32.224638 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-766dd44bbb-7qgnw"] Jul 26 15:22:32.894445 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:32.894413 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:22:32.999970 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:32.999944 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-766dd44bbb-2dzqh"] Jul 26 15:22:33.000264 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:33.000253 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bb28251-70d3-4aab-88ff-00ac2102c2de" containerName="manager" Jul 26 15:22:33.000310 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:33.000266 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bb28251-70d3-4aab-88ff-00ac2102c2de" containerName="manager" Jul 26 15:22:33.000343 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:33.000326 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bb28251-70d3-4aab-88ff-00ac2102c2de" containerName="manager" Jul 26 15:22:33.004514 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:33.004498 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-766dd44bbb-2dzqh" Jul 26 15:22:33.006971 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:33.006949 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 26 15:22:33.007094 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:33.006953 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ccgw4\"" Jul 26 15:22:33.007094 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:33.006954 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 26 15:22:33.011611 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:33.011558 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-766dd44bbb-2dzqh"] Jul 26 15:22:33.126076 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:33.126044 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xms89\" (UniqueName: \"kubernetes.io/projected/c5dceda4-6cc2-47f0-ade7-574acf5bd536-kube-api-access-xms89\") pod \"maas-controller-766dd44bbb-2dzqh\" (UID: \"c5dceda4-6cc2-47f0-ade7-574acf5bd536\") " pod="opendatahub/maas-controller-766dd44bbb-2dzqh" Jul 26 15:22:33.126197 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:33.126145 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c5dceda4-6cc2-47f0-ade7-574acf5bd536-webhook-cert\") pod \"maas-controller-766dd44bbb-2dzqh\" (UID: \"c5dceda4-6cc2-47f0-ade7-574acf5bd536\") " pod="opendatahub/maas-controller-766dd44bbb-2dzqh" Jul 26 15:22:33.227351 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:33.227277 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c5dceda4-6cc2-47f0-ade7-574acf5bd536-webhook-cert\") pod \"maas-controller-766dd44bbb-2dzqh\" (UID: \"c5dceda4-6cc2-47f0-ade7-574acf5bd536\") " pod="opendatahub/maas-controller-766dd44bbb-2dzqh" Jul 26 15:22:33.227351 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:33.227343 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xms89\" (UniqueName: \"kubernetes.io/projected/c5dceda4-6cc2-47f0-ade7-574acf5bd536-kube-api-access-xms89\") pod \"maas-controller-766dd44bbb-2dzqh\" (UID: \"c5dceda4-6cc2-47f0-ade7-574acf5bd536\") " pod="opendatahub/maas-controller-766dd44bbb-2dzqh" Jul 26 15:22:33.229690 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:33.229660 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c5dceda4-6cc2-47f0-ade7-574acf5bd536-webhook-cert\") pod \"maas-controller-766dd44bbb-2dzqh\" (UID: \"c5dceda4-6cc2-47f0-ade7-574acf5bd536\") " pod="opendatahub/maas-controller-766dd44bbb-2dzqh" Jul 26 15:22:33.235602 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:33.235573 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xms89\" (UniqueName: \"kubernetes.io/projected/c5dceda4-6cc2-47f0-ade7-574acf5bd536-kube-api-access-xms89\") pod \"maas-controller-766dd44bbb-2dzqh\" (UID: \"c5dceda4-6cc2-47f0-ade7-574acf5bd536\") " pod="opendatahub/maas-controller-766dd44bbb-2dzqh" Jul 26 15:22:33.315152 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:33.315133 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-766dd44bbb-2dzqh" Jul 26 15:22:33.437330 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:33.437304 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-766dd44bbb-2dzqh"] Jul 26 15:22:33.439624 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:22:33.439602 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5dceda4_6cc2_47f0_ade7_574acf5bd536.slice/crio-7de0317a2d0a52192e177280f861195a48ecf2235599f4072e091cc7e99314cc WatchSource:0}: Error finding container 7de0317a2d0a52192e177280f861195a48ecf2235599f4072e091cc7e99314cc: Status 404 returned error can't find the container with id 7de0317a2d0a52192e177280f861195a48ecf2235599f4072e091cc7e99314cc Jul 26 15:22:34.066622 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:34.066540 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bb28251-70d3-4aab-88ff-00ac2102c2de" path="/var/lib/kubelet/pods/6bb28251-70d3-4aab-88ff-00ac2102c2de/volumes" Jul 26 15:22:34.210650 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:34.210619 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-766dd44bbb-2dzqh" event={"ID":"c5dceda4-6cc2-47f0-ade7-574acf5bd536","Type":"ContainerStarted","Data":"8f50cf8125fbeed22d3eebbd99830d03e1e971508224711465acf022667e91a0"} Jul 26 15:22:34.210893 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:34.210675 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-766dd44bbb-2dzqh" Jul 26 15:22:34.210893 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:34.210692 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-766dd44bbb-2dzqh" event={"ID":"c5dceda4-6cc2-47f0-ade7-574acf5bd536","Type":"ContainerStarted","Data":"7de0317a2d0a52192e177280f861195a48ecf2235599f4072e091cc7e99314cc"} Jul 26 15:22:34.228857 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:34.228811 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-766dd44bbb-2dzqh" podStartSLOduration=1.900368289 podStartE2EDuration="2.228796826s" totalImagesPullingTime="328.428537ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:22:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:22:34.227337165 +0000 UTC m=+1034.651184056" watchObservedRunningTime="2026-07-26 15:22:34.228796826 +0000 UTC m=+1034.652643696" Jul 26 15:22:45.217847 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:22:45.217818 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-766dd44bbb-2dzqh" Jul 26 15:23:08.434090 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:23:08.434048 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:23:08.530640 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:23:08.530599 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:23:08.630722 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:23:08.630693 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:23:14.458071 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:23:14.457982 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:23:14.560190 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:23:14.560149 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:23:14.659044 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:23:14.659014 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:23:14.842497 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:23:14.842396 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:23:20.722235 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:23:20.722190 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:23:20.821939 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:23:20.821894 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:25:10.101203 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:25:10.101163 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:25:10.191771 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:25:10.191742 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:25:20.042361 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:25:20.042332 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9f857_1fbafda6-6242-400e-b2d4-24584608e5f5/ovn-acl-logging/0.log" Jul 26 15:25:20.044693 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:25:20.044675 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9f857_1fbafda6-6242-400e-b2d4-24584608e5f5/ovn-acl-logging/0.log" Jul 26 15:26:37.084597 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:26:37.084560 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:30:20.070248 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:30:20.070218 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9f857_1fbafda6-6242-400e-b2d4-24584608e5f5/ovn-acl-logging/0.log" Jul 26 15:30:20.073401 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:30:20.073380 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9f857_1fbafda6-6242-400e-b2d4-24584608e5f5/ovn-acl-logging/0.log" Jul 26 15:30:51.235427 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:30:51.235334 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:31:03.227227 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:03.227195 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-de82b40d-kserve-94b9d5dc9-rrstt"] Jul 26 15:31:03.230398 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:03.230380 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-de82b40d-kserve-94b9d5dc9-rrstt" Jul 26 15:31:03.232591 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:03.232568 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-de82b40d-kserve-self-signed-certs\"" Jul 26 15:31:03.241548 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:03.241528 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-de82b40d-kserve-94b9d5dc9-rrstt"] Jul 26 15:31:03.253079 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:03.253055 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04293771-ff15-4ab0-b1d5-0aa80d229076-home\") pod \"e2e-same-id-a-de82b40d-kserve-94b9d5dc9-rrstt\" (UID: \"04293771-ff15-4ab0-b1d5-0aa80d229076\") " pod="llm/e2e-same-id-a-de82b40d-kserve-94b9d5dc9-rrstt" Jul 26 15:31:03.253192 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:03.253102 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04293771-ff15-4ab0-b1d5-0aa80d229076-dshm\") pod \"e2e-same-id-a-de82b40d-kserve-94b9d5dc9-rrstt\" (UID: \"04293771-ff15-4ab0-b1d5-0aa80d229076\") " pod="llm/e2e-same-id-a-de82b40d-kserve-94b9d5dc9-rrstt" Jul 26 15:31:03.253192 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:03.253121 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04293771-ff15-4ab0-b1d5-0aa80d229076-model-cache\") pod \"e2e-same-id-a-de82b40d-kserve-94b9d5dc9-rrstt\" (UID: \"04293771-ff15-4ab0-b1d5-0aa80d229076\") " pod="llm/e2e-same-id-a-de82b40d-kserve-94b9d5dc9-rrstt" Jul 26 15:31:03.253192 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:03.253152 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04293771-ff15-4ab0-b1d5-0aa80d229076-tls-certs\") pod \"e2e-same-id-a-de82b40d-kserve-94b9d5dc9-rrstt\" (UID: \"04293771-ff15-4ab0-b1d5-0aa80d229076\") " pod="llm/e2e-same-id-a-de82b40d-kserve-94b9d5dc9-rrstt" Jul 26 15:31:03.253192 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:03.253180 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04293771-ff15-4ab0-b1d5-0aa80d229076-tmp-dir\") pod \"e2e-same-id-a-de82b40d-kserve-94b9d5dc9-rrstt\" (UID: \"04293771-ff15-4ab0-b1d5-0aa80d229076\") " pod="llm/e2e-same-id-a-de82b40d-kserve-94b9d5dc9-rrstt" Jul 26 15:31:03.253414 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:03.253221 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rftds\" (UniqueName: \"kubernetes.io/projected/04293771-ff15-4ab0-b1d5-0aa80d229076-kube-api-access-rftds\") pod \"e2e-same-id-a-de82b40d-kserve-94b9d5dc9-rrstt\" (UID: \"04293771-ff15-4ab0-b1d5-0aa80d229076\") " pod="llm/e2e-same-id-a-de82b40d-kserve-94b9d5dc9-rrstt" Jul 26 15:31:03.267518 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:03.267493 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:31:03.354397 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:03.354374 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04293771-ff15-4ab0-b1d5-0aa80d229076-dshm\") pod \"e2e-same-id-a-de82b40d-kserve-94b9d5dc9-rrstt\" (UID: \"04293771-ff15-4ab0-b1d5-0aa80d229076\") " pod="llm/e2e-same-id-a-de82b40d-kserve-94b9d5dc9-rrstt" Jul 26 15:31:03.354507 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:03.354409 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04293771-ff15-4ab0-b1d5-0aa80d229076-model-cache\") pod \"e2e-same-id-a-de82b40d-kserve-94b9d5dc9-rrstt\" (UID: \"04293771-ff15-4ab0-b1d5-0aa80d229076\") " pod="llm/e2e-same-id-a-de82b40d-kserve-94b9d5dc9-rrstt" Jul 26 15:31:03.354507 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:03.354438 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04293771-ff15-4ab0-b1d5-0aa80d229076-tls-certs\") pod \"e2e-same-id-a-de82b40d-kserve-94b9d5dc9-rrstt\" (UID: \"04293771-ff15-4ab0-b1d5-0aa80d229076\") " pod="llm/e2e-same-id-a-de82b40d-kserve-94b9d5dc9-rrstt" Jul 26 15:31:03.354507 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:03.354472 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04293771-ff15-4ab0-b1d5-0aa80d229076-tmp-dir\") pod \"e2e-same-id-a-de82b40d-kserve-94b9d5dc9-rrstt\" (UID: \"04293771-ff15-4ab0-b1d5-0aa80d229076\") " pod="llm/e2e-same-id-a-de82b40d-kserve-94b9d5dc9-rrstt" Jul 26 15:31:03.354705 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:03.354678 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rftds\" (UniqueName: \"kubernetes.io/projected/04293771-ff15-4ab0-b1d5-0aa80d229076-kube-api-access-rftds\") pod \"e2e-same-id-a-de82b40d-kserve-94b9d5dc9-rrstt\" (UID: \"04293771-ff15-4ab0-b1d5-0aa80d229076\") " pod="llm/e2e-same-id-a-de82b40d-kserve-94b9d5dc9-rrstt" Jul 26 15:31:03.354844 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:03.354787 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04293771-ff15-4ab0-b1d5-0aa80d229076-home\") pod \"e2e-same-id-a-de82b40d-kserve-94b9d5dc9-rrstt\" (UID: \"04293771-ff15-4ab0-b1d5-0aa80d229076\") " pod="llm/e2e-same-id-a-de82b40d-kserve-94b9d5dc9-rrstt" Jul 26 15:31:03.354844 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:03.354805 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04293771-ff15-4ab0-b1d5-0aa80d229076-model-cache\") pod \"e2e-same-id-a-de82b40d-kserve-94b9d5dc9-rrstt\" (UID: \"04293771-ff15-4ab0-b1d5-0aa80d229076\") " pod="llm/e2e-same-id-a-de82b40d-kserve-94b9d5dc9-rrstt" Jul 26 15:31:03.355012 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:03.354911 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04293771-ff15-4ab0-b1d5-0aa80d229076-tmp-dir\") pod \"e2e-same-id-a-de82b40d-kserve-94b9d5dc9-rrstt\" (UID: \"04293771-ff15-4ab0-b1d5-0aa80d229076\") " pod="llm/e2e-same-id-a-de82b40d-kserve-94b9d5dc9-rrstt" Jul 26 15:31:03.355153 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:03.355129 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04293771-ff15-4ab0-b1d5-0aa80d229076-home\") pod \"e2e-same-id-a-de82b40d-kserve-94b9d5dc9-rrstt\" (UID: \"04293771-ff15-4ab0-b1d5-0aa80d229076\") " pod="llm/e2e-same-id-a-de82b40d-kserve-94b9d5dc9-rrstt" Jul 26 15:31:03.356702 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:03.356682 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04293771-ff15-4ab0-b1d5-0aa80d229076-dshm\") pod \"e2e-same-id-a-de82b40d-kserve-94b9d5dc9-rrstt\" (UID: \"04293771-ff15-4ab0-b1d5-0aa80d229076\") " pod="llm/e2e-same-id-a-de82b40d-kserve-94b9d5dc9-rrstt" Jul 26 15:31:03.356985 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:03.356965 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04293771-ff15-4ab0-b1d5-0aa80d229076-tls-certs\") pod \"e2e-same-id-a-de82b40d-kserve-94b9d5dc9-rrstt\" (UID: \"04293771-ff15-4ab0-b1d5-0aa80d229076\") " pod="llm/e2e-same-id-a-de82b40d-kserve-94b9d5dc9-rrstt" Jul 26 15:31:03.361733 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:03.361715 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rftds\" (UniqueName: \"kubernetes.io/projected/04293771-ff15-4ab0-b1d5-0aa80d229076-kube-api-access-rftds\") pod \"e2e-same-id-a-de82b40d-kserve-94b9d5dc9-rrstt\" (UID: \"04293771-ff15-4ab0-b1d5-0aa80d229076\") " pod="llm/e2e-same-id-a-de82b40d-kserve-94b9d5dc9-rrstt" Jul 26 15:31:03.539571 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:03.539518 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-de82b40d-kserve-94b9d5dc9-rrstt" Jul 26 15:31:03.664580 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:03.664556 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-de82b40d-kserve-94b9d5dc9-rrstt"] Jul 26 15:31:03.667382 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:31:03.667354 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod04293771_ff15_4ab0_b1d5_0aa80d229076.slice/crio-00a76f14cbc24b6b34942c02b63c1c8f76ddda39826131137615bbb9bdb0d17c WatchSource:0}: Error finding container 00a76f14cbc24b6b34942c02b63c1c8f76ddda39826131137615bbb9bdb0d17c: Status 404 returned error can't find the container with id 00a76f14cbc24b6b34942c02b63c1c8f76ddda39826131137615bbb9bdb0d17c Jul 26 15:31:04.518740 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:04.518706 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-de82b40d-kserve-94b9d5dc9-rrstt" event={"ID":"04293771-ff15-4ab0-b1d5-0aa80d229076","Type":"ContainerStarted","Data":"cbe543d46e2d0af7edf1a402ebdf038f3900438fa10b88e1626759d5b9be633d"} Jul 26 15:31:04.518740 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:04.518747 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-de82b40d-kserve-94b9d5dc9-rrstt" event={"ID":"04293771-ff15-4ab0-b1d5-0aa80d229076","Type":"ContainerStarted","Data":"00a76f14cbc24b6b34942c02b63c1c8f76ddda39826131137615bbb9bdb0d17c"} Jul 26 15:31:04.533765 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:04.533714 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-de82b40d-kserve-94b9d5dc9-rrstt" podStartSLOduration=1.533699039 podStartE2EDuration="1.533699039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:31:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:31:04.532619033 +0000 UTC m=+1544.956465901" watchObservedRunningTime="2026-07-26 15:31:04.533699039 +0000 UTC m=+1544.957545909" Jul 26 15:31:13.334405 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:13.334374 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-de82b40d-kserve-79c94d5f87-98lwx"] Jul 26 15:31:13.338183 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:13.338162 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-de82b40d-kserve-79c94d5f87-98lwx" Jul 26 15:31:13.340113 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:13.340093 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-de82b40d-kserve-self-signed-certs\"" Jul 26 15:31:13.351298 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:13.351270 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-de82b40d-kserve-79c94d5f87-98lwx"] Jul 26 15:31:13.372288 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:13.372265 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:31:13.438640 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:13.438620 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2f13508-c58f-4441-b817-858b8b073d32-home\") pod \"e2e-same-id-b-de82b40d-kserve-79c94d5f87-98lwx\" (UID: \"a2f13508-c58f-4441-b817-858b8b073d32\") " pod="llm/e2e-same-id-b-de82b40d-kserve-79c94d5f87-98lwx" Jul 26 15:31:13.438741 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:13.438657 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2f13508-c58f-4441-b817-858b8b073d32-tmp-dir\") pod \"e2e-same-id-b-de82b40d-kserve-79c94d5f87-98lwx\" (UID: \"a2f13508-c58f-4441-b817-858b8b073d32\") " pod="llm/e2e-same-id-b-de82b40d-kserve-79c94d5f87-98lwx" Jul 26 15:31:13.438741 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:13.438721 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2f13508-c58f-4441-b817-858b8b073d32-tls-certs\") pod \"e2e-same-id-b-de82b40d-kserve-79c94d5f87-98lwx\" (UID: \"a2f13508-c58f-4441-b817-858b8b073d32\") " pod="llm/e2e-same-id-b-de82b40d-kserve-79c94d5f87-98lwx" Jul 26 15:31:13.438852 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:13.438834 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8ndz\" (UniqueName: \"kubernetes.io/projected/a2f13508-c58f-4441-b817-858b8b073d32-kube-api-access-h8ndz\") pod \"e2e-same-id-b-de82b40d-kserve-79c94d5f87-98lwx\" (UID: \"a2f13508-c58f-4441-b817-858b8b073d32\") " pod="llm/e2e-same-id-b-de82b40d-kserve-79c94d5f87-98lwx" Jul 26 15:31:13.438890 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:13.438868 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2f13508-c58f-4441-b817-858b8b073d32-model-cache\") pod \"e2e-same-id-b-de82b40d-kserve-79c94d5f87-98lwx\" (UID: \"a2f13508-c58f-4441-b817-858b8b073d32\") " pod="llm/e2e-same-id-b-de82b40d-kserve-79c94d5f87-98lwx" Jul 26 15:31:13.438950 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:13.438892 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2f13508-c58f-4441-b817-858b8b073d32-dshm\") pod \"e2e-same-id-b-de82b40d-kserve-79c94d5f87-98lwx\" (UID: \"a2f13508-c58f-4441-b817-858b8b073d32\") " pod="llm/e2e-same-id-b-de82b40d-kserve-79c94d5f87-98lwx" Jul 26 15:31:13.539842 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:13.539818 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-de82b40d-kserve-94b9d5dc9-rrstt" Jul 26 15:31:13.539968 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:13.539877 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-de82b40d-kserve-94b9d5dc9-rrstt" Jul 26 15:31:13.540114 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:13.540090 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8ndz\" (UniqueName: \"kubernetes.io/projected/a2f13508-c58f-4441-b817-858b8b073d32-kube-api-access-h8ndz\") pod \"e2e-same-id-b-de82b40d-kserve-79c94d5f87-98lwx\" (UID: \"a2f13508-c58f-4441-b817-858b8b073d32\") " pod="llm/e2e-same-id-b-de82b40d-kserve-79c94d5f87-98lwx" Jul 26 15:31:13.540194 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:13.540139 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2f13508-c58f-4441-b817-858b8b073d32-model-cache\") pod \"e2e-same-id-b-de82b40d-kserve-79c94d5f87-98lwx\" (UID: \"a2f13508-c58f-4441-b817-858b8b073d32\") " pod="llm/e2e-same-id-b-de82b40d-kserve-79c94d5f87-98lwx" Jul 26 15:31:13.540194 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:13.540172 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2f13508-c58f-4441-b817-858b8b073d32-dshm\") pod \"e2e-same-id-b-de82b40d-kserve-79c94d5f87-98lwx\" (UID: \"a2f13508-c58f-4441-b817-858b8b073d32\") " pod="llm/e2e-same-id-b-de82b40d-kserve-79c94d5f87-98lwx" Jul 26 15:31:13.540296 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:13.540210 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2f13508-c58f-4441-b817-858b8b073d32-home\") pod \"e2e-same-id-b-de82b40d-kserve-79c94d5f87-98lwx\" (UID: \"a2f13508-c58f-4441-b817-858b8b073d32\") " pod="llm/e2e-same-id-b-de82b40d-kserve-79c94d5f87-98lwx" Jul 26 15:31:13.540296 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:13.540239 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2f13508-c58f-4441-b817-858b8b073d32-tmp-dir\") pod \"e2e-same-id-b-de82b40d-kserve-79c94d5f87-98lwx\" (UID: \"a2f13508-c58f-4441-b817-858b8b073d32\") " pod="llm/e2e-same-id-b-de82b40d-kserve-79c94d5f87-98lwx" Jul 26 15:31:13.540296 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:13.540282 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2f13508-c58f-4441-b817-858b8b073d32-tls-certs\") pod \"e2e-same-id-b-de82b40d-kserve-79c94d5f87-98lwx\" (UID: \"a2f13508-c58f-4441-b817-858b8b073d32\") " pod="llm/e2e-same-id-b-de82b40d-kserve-79c94d5f87-98lwx" Jul 26 15:31:13.540605 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:13.540581 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2f13508-c58f-4441-b817-858b8b073d32-model-cache\") pod \"e2e-same-id-b-de82b40d-kserve-79c94d5f87-98lwx\" (UID: \"a2f13508-c58f-4441-b817-858b8b073d32\") " pod="llm/e2e-same-id-b-de82b40d-kserve-79c94d5f87-98lwx" Jul 26 15:31:13.540605 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:13.540602 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2f13508-c58f-4441-b817-858b8b073d32-home\") pod \"e2e-same-id-b-de82b40d-kserve-79c94d5f87-98lwx\" (UID: \"a2f13508-c58f-4441-b817-858b8b073d32\") " pod="llm/e2e-same-id-b-de82b40d-kserve-79c94d5f87-98lwx" Jul 26 15:31:13.540718 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:13.540702 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2f13508-c58f-4441-b817-858b8b073d32-tmp-dir\") pod \"e2e-same-id-b-de82b40d-kserve-79c94d5f87-98lwx\" (UID: \"a2f13508-c58f-4441-b817-858b8b073d32\") " pod="llm/e2e-same-id-b-de82b40d-kserve-79c94d5f87-98lwx" Jul 26 15:31:13.542603 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:13.542583 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2f13508-c58f-4441-b817-858b8b073d32-dshm\") pod \"e2e-same-id-b-de82b40d-kserve-79c94d5f87-98lwx\" (UID: \"a2f13508-c58f-4441-b817-858b8b073d32\") " pod="llm/e2e-same-id-b-de82b40d-kserve-79c94d5f87-98lwx" Jul 26 15:31:13.542708 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:13.542696 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2f13508-c58f-4441-b817-858b8b073d32-tls-certs\") pod \"e2e-same-id-b-de82b40d-kserve-79c94d5f87-98lwx\" (UID: \"a2f13508-c58f-4441-b817-858b8b073d32\") " pod="llm/e2e-same-id-b-de82b40d-kserve-79c94d5f87-98lwx" Jul 26 15:31:13.546461 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:13.546443 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8ndz\" (UniqueName: \"kubernetes.io/projected/a2f13508-c58f-4441-b817-858b8b073d32-kube-api-access-h8ndz\") pod \"e2e-same-id-b-de82b40d-kserve-79c94d5f87-98lwx\" (UID: \"a2f13508-c58f-4441-b817-858b8b073d32\") " pod="llm/e2e-same-id-b-de82b40d-kserve-79c94d5f87-98lwx" Jul 26 15:31:13.552203 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:13.552183 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-de82b40d-kserve-94b9d5dc9-rrstt" Jul 26 15:31:13.571380 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:13.571358 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-de82b40d-kserve-94b9d5dc9-rrstt" Jul 26 15:31:13.647879 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:13.647848 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-de82b40d-kserve-79c94d5f87-98lwx" Jul 26 15:31:13.778529 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:13.778507 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-de82b40d-kserve-79c94d5f87-98lwx"] Jul 26 15:31:13.780433 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:31:13.780400 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda2f13508_c58f_4441_b817_858b8b073d32.slice/crio-ec718c6b09c313618ec6894ad010a23b1c16a351c672d8673ae02eb4ec876340 WatchSource:0}: Error finding container ec718c6b09c313618ec6894ad010a23b1c16a351c672d8673ae02eb4ec876340: Status 404 returned error can't find the container with id ec718c6b09c313618ec6894ad010a23b1c16a351c672d8673ae02eb4ec876340 Jul 26 15:31:14.565552 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:14.565515 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-de82b40d-kserve-79c94d5f87-98lwx" event={"ID":"a2f13508-c58f-4441-b817-858b8b073d32","Type":"ContainerStarted","Data":"875fc81676875890dc1632bdc0de523c415f28fe915ad206888fe34254854a9f"} Jul 26 15:31:14.566015 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:14.565564 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-de82b40d-kserve-79c94d5f87-98lwx" event={"ID":"a2f13508-c58f-4441-b817-858b8b073d32","Type":"ContainerStarted","Data":"ec718c6b09c313618ec6894ad010a23b1c16a351c672d8673ae02eb4ec876340"} Jul 26 15:31:14.584112 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:14.584063 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-de82b40d-kserve-79c94d5f87-98lwx" podStartSLOduration=1.584047419 podStartE2EDuration="1.584047419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:31:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:31:14.582619418 +0000 UTC m=+1555.006466294" watchObservedRunningTime="2026-07-26 15:31:14.584047419 +0000 UTC m=+1555.007894289" Jul 26 15:31:16.669036 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:16.669005 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:31:16.769952 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:16.769897 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:31:23.648499 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:23.648467 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-de82b40d-kserve-79c94d5f87-98lwx" Jul 26 15:31:23.648499 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:23.648510 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-de82b40d-kserve-79c94d5f87-98lwx" Jul 26 15:31:23.660214 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:23.660191 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-de82b40d-kserve-79c94d5f87-98lwx" Jul 26 15:31:24.622503 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:24.622471 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-de82b40d-kserve-79c94d5f87-98lwx" Jul 26 15:31:39.977086 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:39.977053 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-de82b40d-kserve-94b9d5dc9-rrstt"] Jul 26 15:31:40.003198 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:40.003170 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:31:40.818698 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:40.818668 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-de82b40d-kserve-79c94d5f87-98lwx"] Jul 26 15:31:41.084704 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:41.084620 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:31:54.995798 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:54.995741 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-de82b40d-kserve-94b9d5dc9-rrstt" podUID="04293771-ff15-4ab0-b1d5-0aa80d229076" containerName="main" containerID="cri-o://cbe543d46e2d0af7edf1a402ebdf038f3900438fa10b88e1626759d5b9be633d" gracePeriod=45 Jul 26 15:31:55.244765 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:55.244741 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-de82b40d-kserve-94b9d5dc9-rrstt" Jul 26 15:31:55.290896 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:55.290841 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04293771-ff15-4ab0-b1d5-0aa80d229076-dshm\") pod \"04293771-ff15-4ab0-b1d5-0aa80d229076\" (UID: \"04293771-ff15-4ab0-b1d5-0aa80d229076\") " Jul 26 15:31:55.290896 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:55.290879 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04293771-ff15-4ab0-b1d5-0aa80d229076-tls-certs\") pod \"04293771-ff15-4ab0-b1d5-0aa80d229076\" (UID: \"04293771-ff15-4ab0-b1d5-0aa80d229076\") " Jul 26 15:31:55.291056 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:55.290902 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rftds\" (UniqueName: \"kubernetes.io/projected/04293771-ff15-4ab0-b1d5-0aa80d229076-kube-api-access-rftds\") pod \"04293771-ff15-4ab0-b1d5-0aa80d229076\" (UID: \"04293771-ff15-4ab0-b1d5-0aa80d229076\") " Jul 26 15:31:55.291056 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:55.290952 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04293771-ff15-4ab0-b1d5-0aa80d229076-model-cache\") pod \"04293771-ff15-4ab0-b1d5-0aa80d229076\" (UID: \"04293771-ff15-4ab0-b1d5-0aa80d229076\") " Jul 26 15:31:55.291056 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:55.290972 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04293771-ff15-4ab0-b1d5-0aa80d229076-tmp-dir\") pod \"04293771-ff15-4ab0-b1d5-0aa80d229076\" (UID: \"04293771-ff15-4ab0-b1d5-0aa80d229076\") " Jul 26 15:31:55.291056 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:55.291047 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04293771-ff15-4ab0-b1d5-0aa80d229076-home\") pod \"04293771-ff15-4ab0-b1d5-0aa80d229076\" (UID: \"04293771-ff15-4ab0-b1d5-0aa80d229076\") " Jul 26 15:31:55.291203 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:55.291174 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04293771-ff15-4ab0-b1d5-0aa80d229076-model-cache" (OuterVolumeSpecName: "model-cache") pod "04293771-ff15-4ab0-b1d5-0aa80d229076" (UID: "04293771-ff15-4ab0-b1d5-0aa80d229076"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:31:55.291287 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:55.291256 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04293771-ff15-4ab0-b1d5-0aa80d229076-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "04293771-ff15-4ab0-b1d5-0aa80d229076" (UID: "04293771-ff15-4ab0-b1d5-0aa80d229076"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:31:55.291420 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:55.291341 2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04293771-ff15-4ab0-b1d5-0aa80d229076-model-cache\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:31:55.291420 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:55.291352 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04293771-ff15-4ab0-b1d5-0aa80d229076-home" (OuterVolumeSpecName: "home") pod "04293771-ff15-4ab0-b1d5-0aa80d229076" (UID: "04293771-ff15-4ab0-b1d5-0aa80d229076"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:31:55.293096 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:55.293070 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04293771-ff15-4ab0-b1d5-0aa80d229076-dshm" (OuterVolumeSpecName: "dshm") pod "04293771-ff15-4ab0-b1d5-0aa80d229076" (UID: "04293771-ff15-4ab0-b1d5-0aa80d229076"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:31:55.293192 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:55.293125 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04293771-ff15-4ab0-b1d5-0aa80d229076-kube-api-access-rftds" (OuterVolumeSpecName: "kube-api-access-rftds") pod "04293771-ff15-4ab0-b1d5-0aa80d229076" (UID: "04293771-ff15-4ab0-b1d5-0aa80d229076"). InnerVolumeSpecName "kube-api-access-rftds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:31:55.293192 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:55.293171 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04293771-ff15-4ab0-b1d5-0aa80d229076-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "04293771-ff15-4ab0-b1d5-0aa80d229076" (UID: "04293771-ff15-4ab0-b1d5-0aa80d229076"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 15:31:55.392587 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:55.392566 2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04293771-ff15-4ab0-b1d5-0aa80d229076-home\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:31:55.392587 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:55.392586 2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04293771-ff15-4ab0-b1d5-0aa80d229076-dshm\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:31:55.392725 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:55.392595 2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04293771-ff15-4ab0-b1d5-0aa80d229076-tls-certs\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:31:55.392725 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:55.392605 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rftds\" (UniqueName: \"kubernetes.io/projected/04293771-ff15-4ab0-b1d5-0aa80d229076-kube-api-access-rftds\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:31:55.392725 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:55.392613 2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04293771-ff15-4ab0-b1d5-0aa80d229076-tmp-dir\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:31:55.766132 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:55.766095 2555 generic.go:356] "Generic (PLEG): container finished" podID="04293771-ff15-4ab0-b1d5-0aa80d229076" containerID="cbe543d46e2d0af7edf1a402ebdf038f3900438fa10b88e1626759d5b9be633d" exitCode=0 Jul 26 15:31:55.766289 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:55.766141 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-de82b40d-kserve-94b9d5dc9-rrstt" Jul 26 15:31:55.766289 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:55.766158 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-de82b40d-kserve-94b9d5dc9-rrstt" event={"ID":"04293771-ff15-4ab0-b1d5-0aa80d229076","Type":"ContainerDied","Data":"cbe543d46e2d0af7edf1a402ebdf038f3900438fa10b88e1626759d5b9be633d"} Jul 26 15:31:55.766289 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:55.766193 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-de82b40d-kserve-94b9d5dc9-rrstt" event={"ID":"04293771-ff15-4ab0-b1d5-0aa80d229076","Type":"ContainerDied","Data":"00a76f14cbc24b6b34942c02b63c1c8f76ddda39826131137615bbb9bdb0d17c"} Jul 26 15:31:55.766289 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:55.766210 2555 scope.go:117] "RemoveContainer" containerID="cbe543d46e2d0af7edf1a402ebdf038f3900438fa10b88e1626759d5b9be633d" Jul 26 15:31:55.775191 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:55.775174 2555 scope.go:117] "RemoveContainer" containerID="cbe543d46e2d0af7edf1a402ebdf038f3900438fa10b88e1626759d5b9be633d" Jul 26 15:31:55.775431 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:31:55.775412 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbe543d46e2d0af7edf1a402ebdf038f3900438fa10b88e1626759d5b9be633d\": container with ID starting with cbe543d46e2d0af7edf1a402ebdf038f3900438fa10b88e1626759d5b9be633d not found: ID does not exist" containerID="cbe543d46e2d0af7edf1a402ebdf038f3900438fa10b88e1626759d5b9be633d" Jul 26 15:31:55.775515 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:55.775443 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbe543d46e2d0af7edf1a402ebdf038f3900438fa10b88e1626759d5b9be633d"} err="failed to get container status \"cbe543d46e2d0af7edf1a402ebdf038f3900438fa10b88e1626759d5b9be633d\": rpc error: code = NotFound desc = could not find container \"cbe543d46e2d0af7edf1a402ebdf038f3900438fa10b88e1626759d5b9be633d\": container with ID starting with cbe543d46e2d0af7edf1a402ebdf038f3900438fa10b88e1626759d5b9be633d not found: ID does not exist" Jul 26 15:31:55.786250 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:55.786230 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-de82b40d-kserve-94b9d5dc9-rrstt"] Jul 26 15:31:55.793039 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:55.789150 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-de82b40d-kserve-94b9d5dc9-rrstt"] Jul 26 15:31:55.837411 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:55.837375 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-de82b40d-kserve-79c94d5f87-98lwx" podUID="a2f13508-c58f-4441-b817-858b8b073d32" containerName="main" containerID="cri-o://875fc81676875890dc1632bdc0de523c415f28fe915ad206888fe34254854a9f" gracePeriod=45 Jul 26 15:31:56.065613 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:56.065542 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04293771-ff15-4ab0-b1d5-0aa80d229076" path="/var/lib/kubelet/pods/04293771-ff15-4ab0-b1d5-0aa80d229076/volumes" Jul 26 15:31:56.088545 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:56.088523 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-de82b40d-kserve-79c94d5f87-98lwx" Jul 26 15:31:56.198377 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:56.198352 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2f13508-c58f-4441-b817-858b8b073d32-home\") pod \"a2f13508-c58f-4441-b817-858b8b073d32\" (UID: \"a2f13508-c58f-4441-b817-858b8b073d32\") " Jul 26 15:31:56.198521 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:56.198382 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2f13508-c58f-4441-b817-858b8b073d32-tls-certs\") pod \"a2f13508-c58f-4441-b817-858b8b073d32\" (UID: \"a2f13508-c58f-4441-b817-858b8b073d32\") " Jul 26 15:31:56.198521 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:56.198414 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2f13508-c58f-4441-b817-858b8b073d32-tmp-dir\") pod \"a2f13508-c58f-4441-b817-858b8b073d32\" (UID: \"a2f13508-c58f-4441-b817-858b8b073d32\") " Jul 26 15:31:56.198521 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:56.198433 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2f13508-c58f-4441-b817-858b8b073d32-dshm\") pod \"a2f13508-c58f-4441-b817-858b8b073d32\" (UID: \"a2f13508-c58f-4441-b817-858b8b073d32\") " Jul 26 15:31:56.198521 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:56.198489 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8ndz\" (UniqueName: \"kubernetes.io/projected/a2f13508-c58f-4441-b817-858b8b073d32-kube-api-access-h8ndz\") pod \"a2f13508-c58f-4441-b817-858b8b073d32\" (UID: \"a2f13508-c58f-4441-b817-858b8b073d32\") " Jul 26 15:31:56.198740 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:56.198562 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2f13508-c58f-4441-b817-858b8b073d32-model-cache\") pod \"a2f13508-c58f-4441-b817-858b8b073d32\" (UID: \"a2f13508-c58f-4441-b817-858b8b073d32\") " Jul 26 15:31:56.198740 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:56.198666 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2f13508-c58f-4441-b817-858b8b073d32-home" (OuterVolumeSpecName: "home") pod "a2f13508-c58f-4441-b817-858b8b073d32" (UID: "a2f13508-c58f-4441-b817-858b8b073d32"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:31:56.198740 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:56.198680 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2f13508-c58f-4441-b817-858b8b073d32-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a2f13508-c58f-4441-b817-858b8b073d32" (UID: "a2f13508-c58f-4441-b817-858b8b073d32"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:31:56.198900 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:56.198874 2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2f13508-c58f-4441-b817-858b8b073d32-home\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:31:56.198900 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:56.198884 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2f13508-c58f-4441-b817-858b8b073d32-model-cache" (OuterVolumeSpecName: "model-cache") pod "a2f13508-c58f-4441-b817-858b8b073d32" (UID: "a2f13508-c58f-4441-b817-858b8b073d32"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:31:56.199037 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:56.198893 2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2f13508-c58f-4441-b817-858b8b073d32-tmp-dir\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:31:56.200476 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:56.200452 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2f13508-c58f-4441-b817-858b8b073d32-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a2f13508-c58f-4441-b817-858b8b073d32" (UID: "a2f13508-c58f-4441-b817-858b8b073d32"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 15:31:56.200589 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:56.200501 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2f13508-c58f-4441-b817-858b8b073d32-dshm" (OuterVolumeSpecName: "dshm") pod "a2f13508-c58f-4441-b817-858b8b073d32" (UID: "a2f13508-c58f-4441-b817-858b8b073d32"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:31:56.200589 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:56.200504 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2f13508-c58f-4441-b817-858b8b073d32-kube-api-access-h8ndz" (OuterVolumeSpecName: "kube-api-access-h8ndz") pod "a2f13508-c58f-4441-b817-858b8b073d32" (UID: "a2f13508-c58f-4441-b817-858b8b073d32"). InnerVolumeSpecName "kube-api-access-h8ndz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:31:56.299942 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:56.299915 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8ndz\" (UniqueName: \"kubernetes.io/projected/a2f13508-c58f-4441-b817-858b8b073d32-kube-api-access-h8ndz\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:31:56.300047 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:56.299947 2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2f13508-c58f-4441-b817-858b8b073d32-model-cache\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:31:56.300047 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:56.299960 2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2f13508-c58f-4441-b817-858b8b073d32-tls-certs\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:31:56.300047 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:56.299967 2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2f13508-c58f-4441-b817-858b8b073d32-dshm\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:31:56.775083 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:56.775049 2555 generic.go:356] "Generic (PLEG): container finished" podID="a2f13508-c58f-4441-b817-858b8b073d32" containerID="875fc81676875890dc1632bdc0de523c415f28fe915ad206888fe34254854a9f" exitCode=0 Jul 26 15:31:56.775227 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:56.775097 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-de82b40d-kserve-79c94d5f87-98lwx" event={"ID":"a2f13508-c58f-4441-b817-858b8b073d32","Type":"ContainerDied","Data":"875fc81676875890dc1632bdc0de523c415f28fe915ad206888fe34254854a9f"} Jul 26 15:31:56.775227 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:56.775124 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-de82b40d-kserve-79c94d5f87-98lwx" event={"ID":"a2f13508-c58f-4441-b817-858b8b073d32","Type":"ContainerDied","Data":"ec718c6b09c313618ec6894ad010a23b1c16a351c672d8673ae02eb4ec876340"} Jul 26 15:31:56.775227 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:56.775140 2555 scope.go:117] "RemoveContainer" containerID="875fc81676875890dc1632bdc0de523c415f28fe915ad206888fe34254854a9f" Jul 26 15:31:56.775227 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:56.775100 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-de82b40d-kserve-79c94d5f87-98lwx" Jul 26 15:31:56.783815 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:56.783797 2555 scope.go:117] "RemoveContainer" containerID="875fc81676875890dc1632bdc0de523c415f28fe915ad206888fe34254854a9f" Jul 26 15:31:56.784101 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:31:56.784075 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"875fc81676875890dc1632bdc0de523c415f28fe915ad206888fe34254854a9f\": container with ID starting with 875fc81676875890dc1632bdc0de523c415f28fe915ad206888fe34254854a9f not found: ID does not exist" containerID="875fc81676875890dc1632bdc0de523c415f28fe915ad206888fe34254854a9f" Jul 26 15:31:56.784255 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:56.784104 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"875fc81676875890dc1632bdc0de523c415f28fe915ad206888fe34254854a9f"} err="failed to get container status \"875fc81676875890dc1632bdc0de523c415f28fe915ad206888fe34254854a9f\": rpc error: code = NotFound desc = could not find container \"875fc81676875890dc1632bdc0de523c415f28fe915ad206888fe34254854a9f\": container with ID starting with 875fc81676875890dc1632bdc0de523c415f28fe915ad206888fe34254854a9f not found: ID does not exist" Jul 26 15:31:56.797613 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:56.797592 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-de82b40d-kserve-79c94d5f87-98lwx"] Jul 26 15:31:56.799117 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:56.799099 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-de82b40d-kserve-79c94d5f87-98lwx"] Jul 26 15:31:58.063510 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:31:58.063474 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2f13508-c58f-4441-b817-858b8b073d32" path="/var/lib/kubelet/pods/a2f13508-c58f-4441-b817-858b8b073d32/volumes" Jul 26 15:33:19.903492 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:33:19.903457 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:33:33.118888 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:33:33.118847 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:34:32.130311 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:34:32.130274 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:34:32.185157 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:34:32.185131 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:34:56.745977 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:34:56.745937 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:35:20.096922 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:35:20.096892 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9f857_1fbafda6-6242-400e-b2d4-24584608e5f5/ovn-acl-logging/0.log" Jul 26 15:35:20.101548 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:35:20.101526 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9f857_1fbafda6-6242-400e-b2d4-24584608e5f5/ovn-acl-logging/0.log" Jul 26 15:35:20.269271 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:35:20.269221 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:35:23.288515 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:35:23.288475 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:35:23.386283 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:35:23.386232 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:35:23.717698 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:35:23.717662 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:35:29.231522 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:35:29.231483 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:35:29.289519 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:35:29.289491 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:35:29.458725 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:35:29.458698 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:35:29.780866 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:35:29.780833 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:35:29.983097 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:35:29.983065 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:35:31.889246 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:35:31.889219 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:35:31.989266 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:35:31.989230 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:35:32.051744 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:35:32.051697 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:35:32.379738 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:35:32.379689 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:35:32.561448 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:35:32.561413 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:35:32.662993 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:35:32.662888 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:36:07.896061 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:36:07.896013 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:36:07.997330 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:36:07.997299 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:36:08.097038 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:36:08.097007 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:36:12.542270 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:36:12.542233 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:36:12.696209 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:36:12.696174 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:36:12.934177 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:36:12.934118 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:36:13.189338 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:36:13.189265 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:36:13.290122 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:36:13.290092 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:36:13.389792 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:36:13.389765 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:36:13.765762 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:36:13.765722 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:36:14.060755 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:36:14.060659 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:36:16.704501 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:36:16.704456 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:36:16.811731 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:36:16.811697 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:36:16.913757 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:36:16.913723 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:36:17.343411 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:36:17.343367 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:36:17.451789 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:36:17.451746 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:36:52.277443 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:36:52.277362 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:36:53.337340 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:36:53.337303 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:36:56.755649 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:36:56.755607 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:36:56.904862 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:36:56.904824 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:36:57.133852 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:36:57.133816 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:36:57.420800 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:36:57.420727 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:36:57.689437 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:36:57.689361 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:36:57.985289 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:36:57.985209 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:37:00.445309 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:37:00.445278 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:37:00.523862 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:37:00.523825 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:37:00.587488 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:37:00.587464 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:37:01.025093 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:37:01.025057 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:37:01.129357 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:37:01.129326 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:37:01.247432 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:37:01.247400 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:37:36.732188 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:37:36.732144 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:37:36.802404 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:37:36.802371 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:37:36.864455 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:37:36.864422 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:37:41.302647 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:37:41.302587 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:37:42.522452 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:37:42.522419 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:37:42.746617 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:37:42.746587 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:37:43.032429 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:37:43.032401 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:37:43.304073 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:37:43.303976 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:37:43.404998 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:37:43.404964 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:37:45.714119 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:37:45.714080 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:37:45.803156 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:37:45.803125 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:37:45.864863 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:37:45.864833 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:37:46.245583 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:37:46.245553 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:37:46.420032 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:37:46.419996 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:37:46.517572 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:37:46.517486 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:21.149113 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:38:21.149080 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:21.216402 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:38:21.216369 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:21.276827 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:38:21.276795 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:25.643617 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:38:25.643577 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:25.811057 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:38:25.811018 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:26.040254 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:38:26.040169 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:26.513144 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:38:26.513113 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:26.614869 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:38:26.614835 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:26.715038 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:38:26.715011 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:28.119797 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:38:28.119763 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:33.812281 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:38:33.812241 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:33.878180 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:38:33.878137 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:33.960380 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:38:33.960348 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:34.060039 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:38:34.059997 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:34.160707 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:38:34.160675 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:38:34.520150 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:38:34.520073 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:00.483263 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:39:00.483205 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:02.314547 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:39:02.314512 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:06.778346 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:39:06.778310 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:06.906477 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:39:06.906443 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:07.099518 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:39:07.099433 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:22.867013 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:39:22.866971 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:28.566807 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:39:28.566774 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:28.671349 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:39:28.671313 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:28.766855 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:39:28.766821 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:39.276800 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:39:39.276758 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:40.064590 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:39:40.064558 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:46.452060 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:39:46.451971 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:46.560870 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:39:46.560828 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:46.653813 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:39:46.653776 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:46.790420 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:39:46.790335 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:47.517060 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:39:47.517019 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:47.615743 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:39:47.615707 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:47.666100 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:39:47.666065 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:49.340586 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:39:49.340552 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:55.822471 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:39:55.822431 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:55.929791 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:39:55.929757 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:39:56.505695 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:39:56.505659 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:40:02.301412 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:40:02.301374 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:40:02.409469 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:40:02.409433 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:40:02.558925 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:40:02.558849 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:40:02.831975 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:40:02.831874 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:40:02.929896 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:40:02.929864 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:40:06.150241 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:40:06.150207 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:40:06.247181 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:40:06.247144 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:40:06.319181 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:40:06.319107 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:40:06.800903 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:40:06.800837 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:40:07.030292 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:40:07.030260 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:40:07.133864 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:40:07.133831 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:40:10.643124 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:40:10.643091 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:40:11.285776 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:40:11.285740 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:40:11.384556 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:40:11.384529 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:40:11.711688 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:40:11.711659 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:40:17.469766 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:40:17.469724 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:40:17.570411 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:40:17.570371 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:40:20.124132 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:40:20.124108 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9f857_1fbafda6-6242-400e-b2d4-24584608e5f5/ovn-acl-logging/0.log" Jul 26 15:40:20.130882 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:40:20.130862 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9f857_1fbafda6-6242-400e-b2d4-24584608e5f5/ovn-acl-logging/0.log" Jul 26 15:40:28.155521 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:40:28.155491 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:40:28.256558 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:40:28.256519 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:40:28.357378 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:40:28.357343 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:40:32.895887 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:40:32.895855 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:40:33.267439 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:40:33.267350 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:40:38.917278 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:40:38.917242 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:40:38.984360 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:40:38.984329 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:40:39.184041 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:40:39.183956 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:40:39.406429 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:40:39.406390 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:40:39.538260 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:40:39.538186 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:40:39.639033 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:40:39.639001 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:40:39.751338 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:40:39.751307 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:40:45.541200 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:40:45.541160 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:40:45.655574 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:40:45.655527 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:40:45.742137 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:40:45.742110 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:40:45.893668 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:40:45.893630 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:40:51.419219 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:40:51.419185 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:40:51.519677 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:40:51.519639 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:40:58.179788 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:40:58.179748 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:40:58.473175 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:40:58.473093 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:40:58.806260 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:40:58.806176 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:41:01.424639 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:41:01.424600 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:41:01.506215 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:41:01.506181 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:41:01.570773 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:41:01.570747 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:41:01.753287 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:41:01.753210 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:41:02.096740 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:41:02.096657 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:41:02.200738 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:41:02.200705 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:41:03.960890 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:41:03.960852 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:41:05.899372 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:41:05.899335 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:41:25.623406 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:41:25.623368 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:41:50.108967 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:41:50.108919 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:41:50.209862 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:41:50.209826 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:41:55.585411 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:41:55.585378 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-52d355e1-fcdjp"] Jul 26 15:41:55.587764 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:41:55.585741 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2f13508-c58f-4441-b817-858b8b073d32" containerName="main" Jul 26 15:41:55.587764 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:41:55.585755 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2f13508-c58f-4441-b817-858b8b073d32" containerName="main" Jul 26 15:41:55.587764 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:41:55.585766 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04293771-ff15-4ab0-b1d5-0aa80d229076" containerName="main" Jul 26 15:41:55.587764 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:41:55.585772 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="04293771-ff15-4ab0-b1d5-0aa80d229076" containerName="main" Jul 26 15:41:55.587764 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:41:55.585855 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2f13508-c58f-4441-b817-858b8b073d32" containerName="main" Jul 26 15:41:55.587764 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:41:55.585865 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="04293771-ff15-4ab0-b1d5-0aa80d229076" containerName="main" Jul 26 15:41:55.588618 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:41:55.588603 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-52d355e1-fcdjp" Jul 26 15:41:55.590764 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:41:55.590738 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 26 15:41:55.590879 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:41:55.590769 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d665x\"" Jul 26 15:41:55.594781 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:41:55.594750 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-52d355e1-fcdjp"] Jul 26 15:41:55.720916 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:41:55.720884 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d75826d-2b8d-4648-a0c7-1d0f4c4575f9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-52d355e1-fcdjp\" (UID: \"8d75826d-2b8d-4648-a0c7-1d0f4c4575f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-52d355e1-fcdjp" Jul 26 15:41:55.822209 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:41:55.822178 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d75826d-2b8d-4648-a0c7-1d0f4c4575f9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-52d355e1-fcdjp\" (UID: \"8d75826d-2b8d-4648-a0c7-1d0f4c4575f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-52d355e1-fcdjp" Jul 26 15:41:55.822717 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:41:55.822698 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d75826d-2b8d-4648-a0c7-1d0f4c4575f9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-52d355e1-fcdjp\" (UID: \"8d75826d-2b8d-4648-a0c7-1d0f4c4575f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-52d355e1-fcdjp" Jul 26 15:41:55.907491 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:41:55.907469 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-52d355e1-fcdjp" Jul 26 15:41:56.034060 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:41:56.034030 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-52d355e1-fcdjp"] Jul 26 15:41:56.037295 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:41:56.037266 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d75826d_2b8d_4648_a0c7_1d0f4c4575f9.slice/crio-4f42adfffe13ce5addaa54d437d1d59afa06d7e5a446272914d491d43284c607 WatchSource:0}: Error finding container 4f42adfffe13ce5addaa54d437d1d59afa06d7e5a446272914d491d43284c607: Status 404 returned error can't find the container with id 4f42adfffe13ce5addaa54d437d1d59afa06d7e5a446272914d491d43284c607 Jul 26 15:41:56.039090 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:41:56.039070 2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 26 15:41:56.491981 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:41:56.491919 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-52d355e1-fcdjp" event={"ID":"8d75826d-2b8d-4648-a0c7-1d0f4c4575f9","Type":"ContainerStarted","Data":"4f42adfffe13ce5addaa54d437d1d59afa06d7e5a446272914d491d43284c607"} Jul 26 15:41:58.502677 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:41:58.502640 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-52d355e1-fcdjp" event={"ID":"8d75826d-2b8d-4648-a0c7-1d0f4c4575f9","Type":"ContainerStarted","Data":"43e099a8dd7993657511c01563a522dd05cd75607f5251eb876e07455b76b936"} Jul 26 15:41:58.516609 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:41:58.516568 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-52d355e1-fcdjp" podStartSLOduration=1.48155032 podStartE2EDuration="3.516554436s" totalImagesPullingTime="2.035004116s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:41:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:41:58.515315656 +0000 UTC m=+2198.939162527" watchObservedRunningTime="2026-07-26 15:41:58.516554436 +0000 UTC m=+2198.940401306" Jul 26 15:41:59.508689 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:41:59.508654 2555 generic.go:356] "Generic (PLEG): container finished" podID="8d75826d-2b8d-4648-a0c7-1d0f4c4575f9" containerID="43e099a8dd7993657511c01563a522dd05cd75607f5251eb876e07455b76b936" exitCode=0 Jul 26 15:41:59.509143 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:41:59.508743 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-52d355e1-fcdjp" event={"ID":"8d75826d-2b8d-4648-a0c7-1d0f4c4575f9","Type":"ContainerDied","Data":"43e099a8dd7993657511c01563a522dd05cd75607f5251eb876e07455b76b936"} Jul 26 15:42:00.639635 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:42:00.639613 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-52d355e1-fcdjp" Jul 26 15:42:00.763648 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:42:00.763583 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d75826d-2b8d-4648-a0c7-1d0f4c4575f9-maas-api-service-ca\") pod \"8d75826d-2b8d-4648-a0c7-1d0f4c4575f9\" (UID: \"8d75826d-2b8d-4648-a0c7-1d0f4c4575f9\") " Jul 26 15:42:00.763914 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:42:00.763893 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d75826d-2b8d-4648-a0c7-1d0f4c4575f9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8d75826d-2b8d-4648-a0c7-1d0f4c4575f9" (UID: "8d75826d-2b8d-4648-a0c7-1d0f4c4575f9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:42:00.864365 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:42:00.864332 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d75826d-2b8d-4648-a0c7-1d0f4c4575f9-maas-api-service-ca\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:42:01.520138 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:42:01.520103 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-52d355e1-fcdjp" event={"ID":"8d75826d-2b8d-4648-a0c7-1d0f4c4575f9","Type":"ContainerDied","Data":"4f42adfffe13ce5addaa54d437d1d59afa06d7e5a446272914d491d43284c607"} Jul 26 15:42:01.520138 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:42:01.520126 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-52d355e1-fcdjp" Jul 26 15:42:01.520352 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:42:01.520161 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f42adfffe13ce5addaa54d437d1d59afa06d7e5a446272914d491d43284c607" Jul 26 15:42:11.123756 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:42:11.123714 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:42:12.282594 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:42:12.282557 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:42:15.664392 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:42:15.664353 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-52d355e1-fcdjp"] Jul 26 15:42:15.666307 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:42:15.666273 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:42:15.666503 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:42:15.666484 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-52d355e1-fcdjp"] Jul 26 15:42:16.063593 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:42:16.063515 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d75826d-2b8d-4648-a0c7-1d0f4c4575f9" path="/var/lib/kubelet/pods/8d75826d-2b8d-4648-a0c7-1d0f4c4575f9/volumes" Jul 26 15:42:18.903605 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:42:18.903565 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:42:23.927330 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:42:23.927283 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:43:00.620232 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:43:00.620199 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:43:00.933740 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:43:00.933663 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:43:06.486598 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:43:06.486564 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:43:12.088217 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:43:12.088175 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:43:12.188341 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:43:12.188313 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:43:12.363190 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:43:12.363108 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:43:12.607414 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:43:12.607379 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:43:13.647945 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:43:13.647894 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:43:13.746721 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:43:13.746687 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:43:13.847467 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:43:13.847433 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:43:14.571795 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:43:14.571756 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:43:14.669760 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:43:14.669723 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:43:14.769572 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:43:14.769538 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:43:16.164541 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:43:16.164506 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:43:21.857995 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:43:21.857956 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:43:21.957958 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:43:21.957894 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:43:22.519207 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:43:22.519173 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:43:28.321408 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:43:28.321365 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:43:28.386755 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:43:28.386722 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:43:28.480240 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:43:28.480213 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:43:28.573859 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:43:28.573775 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:43:39.036722 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:43:39.036683 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:43:39.398092 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:43:39.398048 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:44:00.341111 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:44:00.341066 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:44:06.423433 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:44:06.423391 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:44:06.524672 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:44:06.524636 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:44:06.774190 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:44:06.774103 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:44:07.086436 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:44:07.086341 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:44:07.552097 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:44:07.552055 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:44:10.538624 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:44:10.538575 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:44:10.634909 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:44:10.634870 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:44:10.764282 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:44:10.764247 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:44:11.005219 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:44:11.005184 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:44:11.342765 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:44:11.342687 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:44:11.441519 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:44:11.441492 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:44:15.027715 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:44:15.027643 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:44:15.360860 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:44:15.360785 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:44:15.658943 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:44:15.658896 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:44:15.925376 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:44:15.925287 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:44:18.596192 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:44:18.596132 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:44:18.674808 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:44:18.674780 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:44:18.752727 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:44:18.752694 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:44:18.915174 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:44:18.915145 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:44:19.293035 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:44:19.292959 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:44:19.384007 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:44:19.383973 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:44:23.092667 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:44:23.092633 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:44:41.531414 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:44:41.531381 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-9e47ce/test-model-9e47ce-kserve-799c7cb847-5f9wm"] Jul 26 15:44:41.531996 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:44:41.531877 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d75826d-2b8d-4648-a0c7-1d0f4c4575f9" containerName="revoke-keys" Jul 26 15:44:41.531996 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:44:41.531897 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d75826d-2b8d-4648-a0c7-1d0f4c4575f9" containerName="revoke-keys" Jul 26 15:44:41.532126 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:44:41.532035 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d75826d-2b8d-4648-a0c7-1d0f4c4575f9" containerName="revoke-keys" Jul 26 15:44:41.534861 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:44:41.534841 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9e47ce/test-model-9e47ce-kserve-799c7cb847-5f9wm" Jul 26 15:44:41.536980 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:44:41.536957 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-9e47ce\"/\"kube-root-ca.crt\"" Jul 26 15:44:41.537502 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:44:41.537482 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-9e47ce\"/\"default-dockercfg-b2ghx\"" Jul 26 15:44:41.537551 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:44:41.537501 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-9e47ce\"/\"openshift-service-ca.crt\"" Jul 26 15:44:41.537551 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:44:41.537537 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-9e47ce\"/\"test-model-9e47ce-kserve-self-signed-certs\"" Jul 26 15:44:41.544236 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:44:41.544216 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-9e47ce/test-model-9e47ce-kserve-799c7cb847-5f9wm"] Jul 26 15:44:41.555247 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:44:41.555218 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:44:41.636659 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:44:41.636635 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a359ecb-f307-4abe-9558-d3239c7cd0b8-home\") pod \"test-model-9e47ce-kserve-799c7cb847-5f9wm\" (UID: \"3a359ecb-f307-4abe-9558-d3239c7cd0b8\") " pod="ai-tenant-e2e-inf-a-9e47ce/test-model-9e47ce-kserve-799c7cb847-5f9wm" Jul 26 15:44:41.636659 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:44:41.636660 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a359ecb-f307-4abe-9558-d3239c7cd0b8-model-cache\") pod \"test-model-9e47ce-kserve-799c7cb847-5f9wm\" (UID: \"3a359ecb-f307-4abe-9558-d3239c7cd0b8\") " pod="ai-tenant-e2e-inf-a-9e47ce/test-model-9e47ce-kserve-799c7cb847-5f9wm" Jul 26 15:44:41.636813 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:44:41.636684 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a359ecb-f307-4abe-9558-d3239c7cd0b8-tls-certs\") pod \"test-model-9e47ce-kserve-799c7cb847-5f9wm\" (UID: \"3a359ecb-f307-4abe-9558-d3239c7cd0b8\") " pod="ai-tenant-e2e-inf-a-9e47ce/test-model-9e47ce-kserve-799c7cb847-5f9wm" Jul 26 15:44:41.636813 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:44:41.636711 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a359ecb-f307-4abe-9558-d3239c7cd0b8-dshm\") pod \"test-model-9e47ce-kserve-799c7cb847-5f9wm\" (UID: \"3a359ecb-f307-4abe-9558-d3239c7cd0b8\") " pod="ai-tenant-e2e-inf-a-9e47ce/test-model-9e47ce-kserve-799c7cb847-5f9wm" Jul 26 15:44:41.636813 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:44:41.636779 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a359ecb-f307-4abe-9558-d3239c7cd0b8-tmp-dir\") pod \"test-model-9e47ce-kserve-799c7cb847-5f9wm\" (UID: \"3a359ecb-f307-4abe-9558-d3239c7cd0b8\") " pod="ai-tenant-e2e-inf-a-9e47ce/test-model-9e47ce-kserve-799c7cb847-5f9wm" Jul 26 15:44:41.636813 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:44:41.636796 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5qj9\" (UniqueName: \"kubernetes.io/projected/3a359ecb-f307-4abe-9558-d3239c7cd0b8-kube-api-access-q5qj9\") pod \"test-model-9e47ce-kserve-799c7cb847-5f9wm\" (UID: \"3a359ecb-f307-4abe-9558-d3239c7cd0b8\") " pod="ai-tenant-e2e-inf-a-9e47ce/test-model-9e47ce-kserve-799c7cb847-5f9wm" Jul 26 15:44:41.737891 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:44:41.737865 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a359ecb-f307-4abe-9558-d3239c7cd0b8-home\") pod \"test-model-9e47ce-kserve-799c7cb847-5f9wm\" (UID: \"3a359ecb-f307-4abe-9558-d3239c7cd0b8\") " pod="ai-tenant-e2e-inf-a-9e47ce/test-model-9e47ce-kserve-799c7cb847-5f9wm" Jul 26 15:44:41.738025 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:44:41.737895 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a359ecb-f307-4abe-9558-d3239c7cd0b8-model-cache\") pod \"test-model-9e47ce-kserve-799c7cb847-5f9wm\" (UID: \"3a359ecb-f307-4abe-9558-d3239c7cd0b8\") " pod="ai-tenant-e2e-inf-a-9e47ce/test-model-9e47ce-kserve-799c7cb847-5f9wm" Jul 26 15:44:41.738025 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:44:41.737913 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a359ecb-f307-4abe-9558-d3239c7cd0b8-tls-certs\") pod \"test-model-9e47ce-kserve-799c7cb847-5f9wm\" (UID: \"3a359ecb-f307-4abe-9558-d3239c7cd0b8\") " pod="ai-tenant-e2e-inf-a-9e47ce/test-model-9e47ce-kserve-799c7cb847-5f9wm" Jul 26 15:44:41.738025 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:44:41.737958 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a359ecb-f307-4abe-9558-d3239c7cd0b8-dshm\") pod \"test-model-9e47ce-kserve-799c7cb847-5f9wm\" (UID: \"3a359ecb-f307-4abe-9558-d3239c7cd0b8\") " pod="ai-tenant-e2e-inf-a-9e47ce/test-model-9e47ce-kserve-799c7cb847-5f9wm" Jul 26 15:44:41.738025 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:44:41.737997 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a359ecb-f307-4abe-9558-d3239c7cd0b8-tmp-dir\") pod \"test-model-9e47ce-kserve-799c7cb847-5f9wm\" (UID: \"3a359ecb-f307-4abe-9558-d3239c7cd0b8\") " pod="ai-tenant-e2e-inf-a-9e47ce/test-model-9e47ce-kserve-799c7cb847-5f9wm" Jul 26 15:44:41.738025 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:44:41.738023 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5qj9\" (UniqueName: \"kubernetes.io/projected/3a359ecb-f307-4abe-9558-d3239c7cd0b8-kube-api-access-q5qj9\") pod \"test-model-9e47ce-kserve-799c7cb847-5f9wm\" (UID: \"3a359ecb-f307-4abe-9558-d3239c7cd0b8\") " pod="ai-tenant-e2e-inf-a-9e47ce/test-model-9e47ce-kserve-799c7cb847-5f9wm" Jul 26 15:44:41.738286 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:44:41.738270 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a359ecb-f307-4abe-9558-d3239c7cd0b8-home\") pod \"test-model-9e47ce-kserve-799c7cb847-5f9wm\" (UID: \"3a359ecb-f307-4abe-9558-d3239c7cd0b8\") " pod="ai-tenant-e2e-inf-a-9e47ce/test-model-9e47ce-kserve-799c7cb847-5f9wm" Jul 26 15:44:41.738343 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:44:41.738324 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a359ecb-f307-4abe-9558-d3239c7cd0b8-model-cache\") pod \"test-model-9e47ce-kserve-799c7cb847-5f9wm\" (UID: \"3a359ecb-f307-4abe-9558-d3239c7cd0b8\") " pod="ai-tenant-e2e-inf-a-9e47ce/test-model-9e47ce-kserve-799c7cb847-5f9wm" Jul 26 15:44:41.738424 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:44:41.738406 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a359ecb-f307-4abe-9558-d3239c7cd0b8-tmp-dir\") pod \"test-model-9e47ce-kserve-799c7cb847-5f9wm\" (UID: \"3a359ecb-f307-4abe-9558-d3239c7cd0b8\") " pod="ai-tenant-e2e-inf-a-9e47ce/test-model-9e47ce-kserve-799c7cb847-5f9wm" Jul 26 15:44:41.740196 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:44:41.740177 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a359ecb-f307-4abe-9558-d3239c7cd0b8-dshm\") pod \"test-model-9e47ce-kserve-799c7cb847-5f9wm\" (UID: \"3a359ecb-f307-4abe-9558-d3239c7cd0b8\") " pod="ai-tenant-e2e-inf-a-9e47ce/test-model-9e47ce-kserve-799c7cb847-5f9wm" Jul 26 15:44:41.740549 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:44:41.740529 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a359ecb-f307-4abe-9558-d3239c7cd0b8-tls-certs\") pod \"test-model-9e47ce-kserve-799c7cb847-5f9wm\" (UID: \"3a359ecb-f307-4abe-9558-d3239c7cd0b8\") " pod="ai-tenant-e2e-inf-a-9e47ce/test-model-9e47ce-kserve-799c7cb847-5f9wm" Jul 26 15:44:41.745033 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:44:41.745011 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5qj9\" (UniqueName: \"kubernetes.io/projected/3a359ecb-f307-4abe-9558-d3239c7cd0b8-kube-api-access-q5qj9\") pod \"test-model-9e47ce-kserve-799c7cb847-5f9wm\" (UID: \"3a359ecb-f307-4abe-9558-d3239c7cd0b8\") " pod="ai-tenant-e2e-inf-a-9e47ce/test-model-9e47ce-kserve-799c7cb847-5f9wm" Jul 26 15:44:41.844786 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:44:41.844719 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9e47ce/test-model-9e47ce-kserve-799c7cb847-5f9wm" Jul 26 15:44:41.967433 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:44:41.967411 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-9e47ce/test-model-9e47ce-kserve-799c7cb847-5f9wm"] Jul 26 15:44:41.969572 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:44:41.969541 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a359ecb_f307_4abe_9558_d3239c7cd0b8.slice/crio-fe457b095955d373efbe37d8aa3a30ae1e2d80a7d29332a4e6f8dec4a8b11b91 WatchSource:0}: Error finding container fe457b095955d373efbe37d8aa3a30ae1e2d80a7d29332a4e6f8dec4a8b11b91: Status 404 returned error can't find the container with id fe457b095955d373efbe37d8aa3a30ae1e2d80a7d29332a4e6f8dec4a8b11b91 Jul 26 15:44:42.248763 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:44:42.248730 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9e47ce/test-model-9e47ce-kserve-799c7cb847-5f9wm" event={"ID":"3a359ecb-f307-4abe-9558-d3239c7cd0b8","Type":"ContainerStarted","Data":"bc9b461bcf270dd2a6d4efbcfbdb089ab82ba9a888b47dd7c434a5d07498446f"} Jul 26 15:44:42.248954 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:44:42.248773 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9e47ce/test-model-9e47ce-kserve-799c7cb847-5f9wm" event={"ID":"3a359ecb-f307-4abe-9558-d3239c7cd0b8","Type":"ContainerStarted","Data":"fe457b095955d373efbe37d8aa3a30ae1e2d80a7d29332a4e6f8dec4a8b11b91"} Jul 26 15:44:42.264031 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:44:42.263990 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-9e47ce/test-model-9e47ce-kserve-799c7cb847-5f9wm" podStartSLOduration=1.263973846 podStartE2EDuration="1.263973846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:44:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:44:42.262682268 +0000 UTC m=+2362.686529138" watchObservedRunningTime="2026-07-26 15:44:42.263973846 +0000 UTC m=+2362.687820732" Jul 26 15:44:51.845048 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:44:51.845010 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-9e47ce/test-model-9e47ce-kserve-799c7cb847-5f9wm" Jul 26 15:44:51.845435 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:44:51.845066 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-9e47ce/test-model-9e47ce-kserve-799c7cb847-5f9wm" Jul 26 15:44:51.856987 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:44:51.856966 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-9e47ce/test-model-9e47ce-kserve-799c7cb847-5f9wm" Jul 26 15:44:52.314481 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:44:52.314450 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-9e47ce/test-model-9e47ce-kserve-799c7cb847-5f9wm" Jul 26 15:45:00.128284 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:00.128251 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9e47ce-29751345-mkdjj"] Jul 26 15:45:00.130385 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:00.130368 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9e47ce-29751345-mkdjj" Jul 26 15:45:00.132372 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:00.132355 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d665x\"" Jul 26 15:45:00.134915 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:00.134896 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29751345-2t29j"] Jul 26 15:45:00.137169 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:00.137151 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29751345-2t29j" Jul 26 15:45:00.137419 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:00.137403 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-28fa20-29751345-bpcpq"] Jul 26 15:45:00.140033 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:00.140012 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9e47ce-29751345-mkdjj"] Jul 26 15:45:00.140129 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:00.140096 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-28fa20-29751345-bpcpq" Jul 26 15:45:00.144353 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:00.144331 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-28fa20-29751345-bpcpq"] Jul 26 15:45:00.144859 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:00.144837 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9e47ce-29751345-mkdjj" Jul 26 15:45:00.148305 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:00.147107 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29751345-2t29j"] Jul 26 15:45:00.171105 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:00.171085 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29751345-2t29j" Jul 26 15:45:00.177266 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:00.177245 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-28fa20-29751345-bpcpq" Jul 26 15:45:00.296516 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:00.296423 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9e47ce-29751345-mkdjj"] Jul 26 15:45:00.297743 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:45:00.297714 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc76a4299_c2df_42f3_908d_d465d5e3faad.slice/crio-6414533b35caaa5450ecfc666dcb00cfbea462a259aea823d17e42d922bbc685 WatchSource:0}: Error finding container 6414533b35caaa5450ecfc666dcb00cfbea462a259aea823d17e42d922bbc685: Status 404 returned error can't find the container with id 6414533b35caaa5450ecfc666dcb00cfbea462a259aea823d17e42d922bbc685 Jul 26 15:45:00.335522 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:00.335496 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9e47ce-29751345-mkdjj" event={"ID":"c76a4299-c2df-42f3-908d-d465d5e3faad","Type":"ContainerStarted","Data":"6414533b35caaa5450ecfc666dcb00cfbea462a259aea823d17e42d922bbc685"} Jul 26 15:45:00.531672 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:00.531646 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-28fa20-29751345-bpcpq"] Jul 26 15:45:00.532755 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:45:00.532728 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4276fdf7_2019_4b10_a7b1_8d7b13f153b5.slice/crio-d0797631a4effa655e78509ba2a04f8fdb055899a9b292fe6e1960cf6e383503 WatchSource:0}: Error finding container d0797631a4effa655e78509ba2a04f8fdb055899a9b292fe6e1960cf6e383503: Status 404 returned error can't find the container with id d0797631a4effa655e78509ba2a04f8fdb055899a9b292fe6e1960cf6e383503 Jul 26 15:45:00.533382 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:00.533364 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29751345-2t29j"] Jul 26 15:45:00.534972 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:45:00.534899 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fe785cd_5fb9_4e0b_a888_ece9ab03d6eb.slice/crio-484ac9a12dcdc5f55078b6976ff05a8d6e1a575eec187a3ef2351447a41f74b5 WatchSource:0}: Error finding container 484ac9a12dcdc5f55078b6976ff05a8d6e1a575eec187a3ef2351447a41f74b5: Status 404 returned error can't find the container with id 484ac9a12dcdc5f55078b6976ff05a8d6e1a575eec187a3ef2351447a41f74b5 Jul 26 15:45:01.341703 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:01.341662 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-28fa20-29751345-bpcpq" event={"ID":"4276fdf7-2019-4b10-a7b1-8d7b13f153b5","Type":"ContainerStarted","Data":"ecee411fdf3eccba5d10f0306d628518e932f02838ac676a022ec39d6d301494"} Jul 26 15:45:01.342149 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:01.341723 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-28fa20-29751345-bpcpq" event={"ID":"4276fdf7-2019-4b10-a7b1-8d7b13f153b5","Type":"ContainerStarted","Data":"d0797631a4effa655e78509ba2a04f8fdb055899a9b292fe6e1960cf6e383503"} Jul 26 15:45:01.343407 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:01.343379 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29751345-2t29j" event={"ID":"1fe785cd-5fb9-4e0b-a888-ece9ab03d6eb","Type":"ContainerStarted","Data":"1c9ea722ed6b16496ff2936cbf54bc5d6786e4ea58c9b35c4ed5a256fc30d3b7"} Jul 26 15:45:01.343538 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:01.343432 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29751345-2t29j" event={"ID":"1fe785cd-5fb9-4e0b-a888-ece9ab03d6eb","Type":"ContainerStarted","Data":"484ac9a12dcdc5f55078b6976ff05a8d6e1a575eec187a3ef2351447a41f74b5"} Jul 26 15:45:01.345028 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:01.345006 2555 generic.go:356] "Generic (PLEG): container finished" podID="c76a4299-c2df-42f3-908d-d465d5e3faad" containerID="4a762115a6da09577cbc2c18ebd12c736a05a27acef02f80f0d68b561dfb5aa6" exitCode=0 Jul 26 15:45:01.345123 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:01.345048 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9e47ce-29751345-mkdjj" event={"ID":"c76a4299-c2df-42f3-908d-d465d5e3faad","Type":"ContainerDied","Data":"4a762115a6da09577cbc2c18ebd12c736a05a27acef02f80f0d68b561dfb5aa6"} Jul 26 15:45:01.356260 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:01.356215 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-28fa20-29751345-bpcpq" podStartSLOduration=1.35620358 podStartE2EDuration="1.35620358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:45:01.354881337 +0000 UTC m=+2381.778728220" watchObservedRunningTime="2026-07-26 15:45:01.35620358 +0000 UTC m=+2381.780050450" Jul 26 15:45:01.368728 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:01.368689 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29751345-2t29j" podStartSLOduration=1.36867705 podStartE2EDuration="1.36867705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:45:01.367465114 +0000 UTC m=+2381.791311979" watchObservedRunningTime="2026-07-26 15:45:01.36867705 +0000 UTC m=+2381.792523920" Jul 26 15:45:02.356322 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:02.356288 2555 generic.go:356] "Generic (PLEG): container finished" podID="4276fdf7-2019-4b10-a7b1-8d7b13f153b5" containerID="ecee411fdf3eccba5d10f0306d628518e932f02838ac676a022ec39d6d301494" exitCode=0 Jul 26 15:45:02.356701 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:02.356372 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-28fa20-29751345-bpcpq" event={"ID":"4276fdf7-2019-4b10-a7b1-8d7b13f153b5","Type":"ContainerDied","Data":"ecee411fdf3eccba5d10f0306d628518e932f02838ac676a022ec39d6d301494"} Jul 26 15:45:02.357872 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:02.357850 2555 generic.go:356] "Generic (PLEG): container finished" podID="1fe785cd-5fb9-4e0b-a888-ece9ab03d6eb" containerID="1c9ea722ed6b16496ff2936cbf54bc5d6786e4ea58c9b35c4ed5a256fc30d3b7" exitCode=0 Jul 26 15:45:02.358023 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:02.357907 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29751345-2t29j" event={"ID":"1fe785cd-5fb9-4e0b-a888-ece9ab03d6eb","Type":"ContainerDied","Data":"1c9ea722ed6b16496ff2936cbf54bc5d6786e4ea58c9b35c4ed5a256fc30d3b7"} Jul 26 15:45:02.480879 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:02.480858 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9e47ce-29751345-mkdjj" Jul 26 15:45:02.830179 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:02.830141 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:45:03.364684 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:03.364649 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9e47ce-29751345-mkdjj" Jul 26 15:45:03.364684 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:03.364676 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9e47ce-29751345-mkdjj" event={"ID":"c76a4299-c2df-42f3-908d-d465d5e3faad","Type":"ContainerDied","Data":"6414533b35caaa5450ecfc666dcb00cfbea462a259aea823d17e42d922bbc685"} Jul 26 15:45:03.365229 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:03.364731 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6414533b35caaa5450ecfc666dcb00cfbea462a259aea823d17e42d922bbc685" Jul 26 15:45:03.504429 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:03.504404 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29751345-2t29j" Jul 26 15:45:03.507230 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:03.507212 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-28fa20-29751345-bpcpq" Jul 26 15:45:04.371678 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:04.371650 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-28fa20-29751345-bpcpq" Jul 26 15:45:04.372103 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:04.371692 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-28fa20-29751345-bpcpq" event={"ID":"4276fdf7-2019-4b10-a7b1-8d7b13f153b5","Type":"ContainerDied","Data":"d0797631a4effa655e78509ba2a04f8fdb055899a9b292fe6e1960cf6e383503"} Jul 26 15:45:04.372103 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:04.371746 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0797631a4effa655e78509ba2a04f8fdb055899a9b292fe6e1960cf6e383503" Jul 26 15:45:04.373243 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:04.373219 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29751345-2t29j" event={"ID":"1fe785cd-5fb9-4e0b-a888-ece9ab03d6eb","Type":"ContainerDied","Data":"484ac9a12dcdc5f55078b6976ff05a8d6e1a575eec187a3ef2351447a41f74b5"} Jul 26 15:45:04.373356 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:04.373248 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29751345-2t29j" Jul 26 15:45:04.373356 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:04.373265 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="484ac9a12dcdc5f55078b6976ff05a8d6e1a575eec187a3ef2351447a41f74b5" Jul 26 15:45:09.891453 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:09.891420 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:45:19.064222 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:19.064187 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:45:20.153033 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:20.152996 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9f857_1fbafda6-6242-400e-b2d4-24584608e5f5/ovn-acl-logging/0.log" Jul 26 15:45:20.160155 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:20.160137 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9f857_1fbafda6-6242-400e-b2d4-24584608e5f5/ovn-acl-logging/0.log" Jul 26 15:45:52.823658 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:52.823582 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-9e47ce/test-model-9e47ce-kserve-799c7cb847-5f9wm"] Jul 26 15:45:52.842445 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:52.842411 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:45:53.831507 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:53.831471 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:45:53.932525 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:53.932495 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e47ce-mkddz"] Jul 26 15:45:53.932842 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:53.932830 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c76a4299-c2df-42f3-908d-d465d5e3faad" containerName="cleanup" Jul 26 15:45:53.932885 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:53.932843 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c76a4299-c2df-42f3-908d-d465d5e3faad" containerName="cleanup" Jul 26 15:45:53.932885 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:53.932854 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4276fdf7-2019-4b10-a7b1-8d7b13f153b5" containerName="cleanup" Jul 26 15:45:53.932885 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:53.932859 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4276fdf7-2019-4b10-a7b1-8d7b13f153b5" containerName="cleanup" Jul 26 15:45:53.932885 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:53.932866 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fe785cd-5fb9-4e0b-a888-ece9ab03d6eb" containerName="cleanup" Jul 26 15:45:53.932885 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:53.932871 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fe785cd-5fb9-4e0b-a888-ece9ab03d6eb" containerName="cleanup" Jul 26 15:45:53.933049 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:53.932940 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4276fdf7-2019-4b10-a7b1-8d7b13f153b5" containerName="cleanup" Jul 26 15:45:53.933049 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:53.932950 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c76a4299-c2df-42f3-908d-d465d5e3faad" containerName="cleanup" Jul 26 15:45:53.933049 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:53.932955 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fe785cd-5fb9-4e0b-a888-ece9ab03d6eb" containerName="cleanup" Jul 26 15:45:53.936071 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:53.936057 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e47ce-mkddz" Jul 26 15:45:53.937895 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:53.937878 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 26 15:45:53.938072 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:53.938053 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d665x\"" Jul 26 15:45:53.942699 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:53.942677 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e47ce-mkddz"] Jul 26 15:45:54.015537 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:54.015502 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c9e0c97-1038-4535-8a78-303f1612db2a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9e47ce-mkddz\" (UID: \"3c9e0c97-1038-4535-8a78-303f1612db2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e47ce-mkddz" Jul 26 15:45:54.116299 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:54.116259 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c9e0c97-1038-4535-8a78-303f1612db2a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9e47ce-mkddz\" (UID: \"3c9e0c97-1038-4535-8a78-303f1612db2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e47ce-mkddz" Jul 26 15:45:54.116858 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:54.116838 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c9e0c97-1038-4535-8a78-303f1612db2a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9e47ce-mkddz\" (UID: \"3c9e0c97-1038-4535-8a78-303f1612db2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e47ce-mkddz" Jul 26 15:45:54.250646 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:54.250611 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e47ce-mkddz" Jul 26 15:45:54.373146 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:54.373119 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e47ce-mkddz"] Jul 26 15:45:54.374561 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:45:54.374536 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c9e0c97_1038_4535_8a78_303f1612db2a.slice/crio-62525518f897df98e5392a1b9b9f3a3894050ed32923f6ccfc318d74c6d1f741 WatchSource:0}: Error finding container 62525518f897df98e5392a1b9b9f3a3894050ed32923f6ccfc318d74c6d1f741: Status 404 returned error can't find the container with id 62525518f897df98e5392a1b9b9f3a3894050ed32923f6ccfc318d74c6d1f741 Jul 26 15:45:54.608427 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:54.608404 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e47ce-mkddz" event={"ID":"3c9e0c97-1038-4535-8a78-303f1612db2a","Type":"ContainerStarted","Data":"ce27ba97198e23a0a2e4c09575b997cd9b8a5fb2b862bf48c3edf7b0b5882cea"} Jul 26 15:45:54.608548 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:54.608450 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e47ce-mkddz" event={"ID":"3c9e0c97-1038-4535-8a78-303f1612db2a","Type":"ContainerStarted","Data":"62525518f897df98e5392a1b9b9f3a3894050ed32923f6ccfc318d74c6d1f741"} Jul 26 15:45:54.625290 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:54.625211 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e47ce-mkddz" podStartSLOduration=1.625196602 podStartE2EDuration="1.625196602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:45:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:45:54.622740752 +0000 UTC m=+2435.046587623" watchObservedRunningTime="2026-07-26 15:45:54.625196602 +0000 UTC m=+2435.049043472" Jul 26 15:45:55.207906 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:55.207875 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:45:55.614127 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:55.614091 2555 generic.go:356] "Generic (PLEG): container finished" podID="3c9e0c97-1038-4535-8a78-303f1612db2a" containerID="ce27ba97198e23a0a2e4c09575b997cd9b8a5fb2b862bf48c3edf7b0b5882cea" exitCode=0 Jul 26 15:45:55.614127 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:55.614128 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e47ce-mkddz" event={"ID":"3c9e0c97-1038-4535-8a78-303f1612db2a","Type":"ContainerDied","Data":"ce27ba97198e23a0a2e4c09575b997cd9b8a5fb2b862bf48c3edf7b0b5882cea"} Jul 26 15:45:56.745475 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:56.745447 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e47ce-mkddz" Jul 26 15:45:56.835977 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:56.835954 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c9e0c97-1038-4535-8a78-303f1612db2a-maas-api-service-ca\") pod \"3c9e0c97-1038-4535-8a78-303f1612db2a\" (UID: \"3c9e0c97-1038-4535-8a78-303f1612db2a\") " Jul 26 15:45:56.836302 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:56.836278 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c9e0c97-1038-4535-8a78-303f1612db2a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3c9e0c97-1038-4535-8a78-303f1612db2a" (UID: "3c9e0c97-1038-4535-8a78-303f1612db2a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:45:56.937168 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:56.937109 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c9e0c97-1038-4535-8a78-303f1612db2a-maas-api-service-ca\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:45:57.625508 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:57.625482 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e47ce-mkddz" Jul 26 15:45:57.625508 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:57.625490 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e47ce-mkddz" event={"ID":"3c9e0c97-1038-4535-8a78-303f1612db2a","Type":"ContainerDied","Data":"62525518f897df98e5392a1b9b9f3a3894050ed32923f6ccfc318d74c6d1f741"} Jul 26 15:45:57.625749 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:45:57.625541 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62525518f897df98e5392a1b9b9f3a3894050ed32923f6ccfc318d74c6d1f741" Jul 26 15:46:04.471608 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:04.471575 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9e47ce-29751345-mkdjj"] Jul 26 15:46:04.477086 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:04.477061 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9e47ce-29751345-mkdjj"] Jul 26 15:46:04.480832 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:04.480809 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:46:04.547568 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:04.547536 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:46:04.648963 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:04.648915 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:46:06.064059 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:06.064027 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c76a4299-c2df-42f3-908d-d465d5e3faad" path="/var/lib/kubelet/pods/c76a4299-c2df-42f3-908d-d465d5e3faad/volumes" Jul 26 15:46:07.841952 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:07.841892 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-9e47ce/test-model-9e47ce-kserve-799c7cb847-5f9wm" podUID="3a359ecb-f307-4abe-9558-d3239c7cd0b8" containerName="main" containerID="cri-o://bc9b461bcf270dd2a6d4efbcfbdb089ab82ba9a888b47dd7c434a5d07498446f" gracePeriod=45 Jul 26 15:46:08.081200 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:08.081176 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9e47ce/test-model-9e47ce-kserve-799c7cb847-5f9wm" Jul 26 15:46:08.121883 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:08.121860 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a359ecb-f307-4abe-9558-d3239c7cd0b8-model-cache\") pod \"3a359ecb-f307-4abe-9558-d3239c7cd0b8\" (UID: \"3a359ecb-f307-4abe-9558-d3239c7cd0b8\") " Jul 26 15:46:08.122000 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:08.121905 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a359ecb-f307-4abe-9558-d3239c7cd0b8-home\") pod \"3a359ecb-f307-4abe-9558-d3239c7cd0b8\" (UID: \"3a359ecb-f307-4abe-9558-d3239c7cd0b8\") " Jul 26 15:46:08.122000 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:08.121925 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q5qj9\" (UniqueName: \"kubernetes.io/projected/3a359ecb-f307-4abe-9558-d3239c7cd0b8-kube-api-access-q5qj9\") pod \"3a359ecb-f307-4abe-9558-d3239c7cd0b8\" (UID: \"3a359ecb-f307-4abe-9558-d3239c7cd0b8\") " Jul 26 15:46:08.122000 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:08.121963 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a359ecb-f307-4abe-9558-d3239c7cd0b8-dshm\") pod \"3a359ecb-f307-4abe-9558-d3239c7cd0b8\" (UID: \"3a359ecb-f307-4abe-9558-d3239c7cd0b8\") " Jul 26 15:46:08.122176 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:08.122024 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a359ecb-f307-4abe-9558-d3239c7cd0b8-tls-certs\") pod \"3a359ecb-f307-4abe-9558-d3239c7cd0b8\" (UID: \"3a359ecb-f307-4abe-9558-d3239c7cd0b8\") " Jul 26 15:46:08.122176 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:08.122046 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a359ecb-f307-4abe-9558-d3239c7cd0b8-tmp-dir\") pod \"3a359ecb-f307-4abe-9558-d3239c7cd0b8\" (UID: \"3a359ecb-f307-4abe-9558-d3239c7cd0b8\") " Jul 26 15:46:08.122176 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:08.122113 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a359ecb-f307-4abe-9558-d3239c7cd0b8-model-cache" (OuterVolumeSpecName: "model-cache") pod "3a359ecb-f307-4abe-9558-d3239c7cd0b8" (UID: "3a359ecb-f307-4abe-9558-d3239c7cd0b8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:46:08.122176 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:08.122126 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a359ecb-f307-4abe-9558-d3239c7cd0b8-home" (OuterVolumeSpecName: "home") pod "3a359ecb-f307-4abe-9558-d3239c7cd0b8" (UID: "3a359ecb-f307-4abe-9558-d3239c7cd0b8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:46:08.122405 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:08.122380 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a359ecb-f307-4abe-9558-d3239c7cd0b8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3a359ecb-f307-4abe-9558-d3239c7cd0b8" (UID: "3a359ecb-f307-4abe-9558-d3239c7cd0b8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:46:08.122541 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:08.122493 2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a359ecb-f307-4abe-9558-d3239c7cd0b8-model-cache\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:46:08.122541 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:08.122514 2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a359ecb-f307-4abe-9558-d3239c7cd0b8-home\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:46:08.122541 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:08.122528 2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a359ecb-f307-4abe-9558-d3239c7cd0b8-tmp-dir\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:46:08.124220 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:08.124198 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a359ecb-f307-4abe-9558-d3239c7cd0b8-dshm" (OuterVolumeSpecName: "dshm") pod "3a359ecb-f307-4abe-9558-d3239c7cd0b8" (UID: "3a359ecb-f307-4abe-9558-d3239c7cd0b8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:46:08.124296 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:08.124222 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a359ecb-f307-4abe-9558-d3239c7cd0b8-kube-api-access-q5qj9" (OuterVolumeSpecName: "kube-api-access-q5qj9") pod "3a359ecb-f307-4abe-9558-d3239c7cd0b8" (UID: "3a359ecb-f307-4abe-9558-d3239c7cd0b8"). InnerVolumeSpecName "kube-api-access-q5qj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:46:08.124296 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:08.124227 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a359ecb-f307-4abe-9558-d3239c7cd0b8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3a359ecb-f307-4abe-9558-d3239c7cd0b8" (UID: "3a359ecb-f307-4abe-9558-d3239c7cd0b8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 15:46:08.223639 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:08.223619 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5qj9\" (UniqueName: \"kubernetes.io/projected/3a359ecb-f307-4abe-9558-d3239c7cd0b8-kube-api-access-q5qj9\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:46:08.223639 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:08.223639 2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a359ecb-f307-4abe-9558-d3239c7cd0b8-dshm\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:46:08.223751 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:08.223648 2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a359ecb-f307-4abe-9558-d3239c7cd0b8-tls-certs\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:46:08.679718 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:08.679688 2555 generic.go:356] "Generic (PLEG): container finished" podID="3a359ecb-f307-4abe-9558-d3239c7cd0b8" containerID="bc9b461bcf270dd2a6d4efbcfbdb089ab82ba9a888b47dd7c434a5d07498446f" exitCode=0 Jul 26 15:46:08.679846 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:08.679735 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9e47ce/test-model-9e47ce-kserve-799c7cb847-5f9wm" event={"ID":"3a359ecb-f307-4abe-9558-d3239c7cd0b8","Type":"ContainerDied","Data":"bc9b461bcf270dd2a6d4efbcfbdb089ab82ba9a888b47dd7c434a5d07498446f"} Jul 26 15:46:08.679846 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:08.679736 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9e47ce/test-model-9e47ce-kserve-799c7cb847-5f9wm" Jul 26 15:46:08.679846 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:08.679769 2555 scope.go:117] "RemoveContainer" containerID="bc9b461bcf270dd2a6d4efbcfbdb089ab82ba9a888b47dd7c434a5d07498446f" Jul 26 15:46:08.679989 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:08.679760 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9e47ce/test-model-9e47ce-kserve-799c7cb847-5f9wm" event={"ID":"3a359ecb-f307-4abe-9558-d3239c7cd0b8","Type":"ContainerDied","Data":"fe457b095955d373efbe37d8aa3a30ae1e2d80a7d29332a4e6f8dec4a8b11b91"} Jul 26 15:46:08.689452 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:08.689438 2555 scope.go:117] "RemoveContainer" containerID="bc9b461bcf270dd2a6d4efbcfbdb089ab82ba9a888b47dd7c434a5d07498446f" Jul 26 15:46:08.689674 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:46:08.689655 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc9b461bcf270dd2a6d4efbcfbdb089ab82ba9a888b47dd7c434a5d07498446f\": container with ID starting with bc9b461bcf270dd2a6d4efbcfbdb089ab82ba9a888b47dd7c434a5d07498446f not found: ID does not exist" containerID="bc9b461bcf270dd2a6d4efbcfbdb089ab82ba9a888b47dd7c434a5d07498446f" Jul 26 15:46:08.689733 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:08.689680 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc9b461bcf270dd2a6d4efbcfbdb089ab82ba9a888b47dd7c434a5d07498446f"} err="failed to get container status \"bc9b461bcf270dd2a6d4efbcfbdb089ab82ba9a888b47dd7c434a5d07498446f\": rpc error: code = NotFound desc = could not find container \"bc9b461bcf270dd2a6d4efbcfbdb089ab82ba9a888b47dd7c434a5d07498446f\": container with ID starting with bc9b461bcf270dd2a6d4efbcfbdb089ab82ba9a888b47dd7c434a5d07498446f not found: ID does not exist" Jul 26 15:46:08.699941 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:08.699910 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-9e47ce/test-model-9e47ce-kserve-799c7cb847-5f9wm"] Jul 26 15:46:08.703608 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:08.703586 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-9e47ce/test-model-9e47ce-kserve-799c7cb847-5f9wm"] Jul 26 15:46:09.009387 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:09.009309 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:46:09.024760 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:09.024739 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e47ce-mkddz"] Jul 26 15:46:09.026831 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:09.026808 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e47ce-mkddz"] Jul 26 15:46:09.393626 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:09.393593 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:46:10.063571 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:10.063538 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a359ecb-f307-4abe-9558-d3239c7cd0b8" path="/var/lib/kubelet/pods/3a359ecb-f307-4abe-9558-d3239c7cd0b8/volumes" Jul 26 15:46:10.063947 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:10.063850 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c9e0c97-1038-4535-8a78-303f1612db2a" path="/var/lib/kubelet/pods/3c9e0c97-1038-4535-8a78-303f1612db2a/volumes" Jul 26 15:46:15.116314 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:15.116278 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:46:15.183688 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:15.183657 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:46:15.356426 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:15.356393 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:46:15.591378 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:15.591299 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:46:15.713164 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:15.713130 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-28fa20-pr9l5"] Jul 26 15:46:15.713465 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:15.713451 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c9e0c97-1038-4535-8a78-303f1612db2a" containerName="revoke-keys" Jul 26 15:46:15.713465 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:15.713464 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c9e0c97-1038-4535-8a78-303f1612db2a" containerName="revoke-keys" Jul 26 15:46:15.713595 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:15.713481 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a359ecb-f307-4abe-9558-d3239c7cd0b8" containerName="main" Jul 26 15:46:15.713595 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:15.713487 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a359ecb-f307-4abe-9558-d3239c7cd0b8" containerName="main" Jul 26 15:46:15.713595 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:15.713556 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c9e0c97-1038-4535-8a78-303f1612db2a" containerName="revoke-keys" Jul 26 15:46:15.713595 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:15.713570 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a359ecb-f307-4abe-9558-d3239c7cd0b8" containerName="main" Jul 26 15:46:15.717588 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:15.717566 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-28fa20-pr9l5" Jul 26 15:46:15.719521 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:15.719499 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 26 15:46:15.719615 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:15.719528 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d665x\"" Jul 26 15:46:15.722904 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:15.722876 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-28fa20-pr9l5"] Jul 26 15:46:15.788880 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:15.788853 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca76fcf6-61da-4f48-8f75-1e0e24c89638-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-28fa20-pr9l5\" (UID: \"ca76fcf6-61da-4f48-8f75-1e0e24c89638\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-28fa20-pr9l5" Jul 26 15:46:15.890268 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:15.890242 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca76fcf6-61da-4f48-8f75-1e0e24c89638-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-28fa20-pr9l5\" (UID: \"ca76fcf6-61da-4f48-8f75-1e0e24c89638\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-28fa20-pr9l5" Jul 26 15:46:15.890778 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:15.890761 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca76fcf6-61da-4f48-8f75-1e0e24c89638-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-28fa20-pr9l5\" (UID: \"ca76fcf6-61da-4f48-8f75-1e0e24c89638\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-28fa20-pr9l5" Jul 26 15:46:16.029515 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:16.029490 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-28fa20-pr9l5" Jul 26 15:46:16.155886 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:16.155861 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-28fa20-pr9l5"] Jul 26 15:46:16.157636 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:46:16.157599 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca76fcf6_61da_4f48_8f75_1e0e24c89638.slice/crio-de6335e71d4b354f13a5fdad6acc1a592a589b493af298f8bf21974979e10a48 WatchSource:0}: Error finding container de6335e71d4b354f13a5fdad6acc1a592a589b493af298f8bf21974979e10a48: Status 404 returned error can't find the container with id de6335e71d4b354f13a5fdad6acc1a592a589b493af298f8bf21974979e10a48 Jul 26 15:46:16.719224 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:16.719145 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-28fa20-pr9l5" event={"ID":"ca76fcf6-61da-4f48-8f75-1e0e24c89638","Type":"ContainerStarted","Data":"ad656bd5d26b834f2bad4602711f1578d24528662f3014ef804cfa41dceeb25e"} Jul 26 15:46:16.719224 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:16.719216 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-28fa20-pr9l5" event={"ID":"ca76fcf6-61da-4f48-8f75-1e0e24c89638","Type":"ContainerStarted","Data":"de6335e71d4b354f13a5fdad6acc1a592a589b493af298f8bf21974979e10a48"} Jul 26 15:46:16.733692 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:16.733646 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-28fa20-pr9l5" podStartSLOduration=1.733633292 podStartE2EDuration="1.733633292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:46:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:46:16.732039757 +0000 UTC m=+2457.155886630" watchObservedRunningTime="2026-07-26 15:46:16.733633292 +0000 UTC m=+2457.157480162" Jul 26 15:46:17.724820 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:17.724787 2555 generic.go:356] "Generic (PLEG): container finished" podID="ca76fcf6-61da-4f48-8f75-1e0e24c89638" containerID="ad656bd5d26b834f2bad4602711f1578d24528662f3014ef804cfa41dceeb25e" exitCode=0 Jul 26 15:46:17.725259 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:17.724844 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-28fa20-pr9l5" event={"ID":"ca76fcf6-61da-4f48-8f75-1e0e24c89638","Type":"ContainerDied","Data":"ad656bd5d26b834f2bad4602711f1578d24528662f3014ef804cfa41dceeb25e"} Jul 26 15:46:18.856348 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:18.856326 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-28fa20-pr9l5" Jul 26 15:46:18.914503 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:18.914478 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca76fcf6-61da-4f48-8f75-1e0e24c89638-maas-api-service-ca\") pod \"ca76fcf6-61da-4f48-8f75-1e0e24c89638\" (UID: \"ca76fcf6-61da-4f48-8f75-1e0e24c89638\") " Jul 26 15:46:18.914813 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:18.914787 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca76fcf6-61da-4f48-8f75-1e0e24c89638-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ca76fcf6-61da-4f48-8f75-1e0e24c89638" (UID: "ca76fcf6-61da-4f48-8f75-1e0e24c89638"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:46:19.015195 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:19.015136 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca76fcf6-61da-4f48-8f75-1e0e24c89638-maas-api-service-ca\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:46:19.736648 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:19.736619 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-28fa20-pr9l5" Jul 26 15:46:19.736812 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:19.736610 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-28fa20-pr9l5" event={"ID":"ca76fcf6-61da-4f48-8f75-1e0e24c89638","Type":"ContainerDied","Data":"de6335e71d4b354f13a5fdad6acc1a592a589b493af298f8bf21974979e10a48"} Jul 26 15:46:19.736812 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:19.736746 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de6335e71d4b354f13a5fdad6acc1a592a589b493af298f8bf21974979e10a48" Jul 26 15:46:26.155039 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:26.155002 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-28fa20-29751345-bpcpq"] Jul 26 15:46:26.158250 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:26.158223 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-28fa20-29751345-bpcpq"] Jul 26 15:46:26.159259 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:26.159236 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:46:26.224805 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:26.224771 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:46:26.284384 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:26.284349 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:46:26.866759 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:26.866721 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:46:26.919324 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:26.919293 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:46:28.064191 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:28.064160 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4276fdf7-2019-4b10-a7b1-8d7b13f153b5" path="/var/lib/kubelet/pods/4276fdf7-2019-4b10-a7b1-8d7b13f153b5/volumes" Jul 26 15:46:30.785889 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:30.785860 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-28fa20-pr9l5"] Jul 26 15:46:30.788531 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:30.788475 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:46:30.788692 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:30.788495 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-28fa20-pr9l5"] Jul 26 15:46:31.159029 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:31.158999 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:46:32.063799 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:32.063768 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca76fcf6-61da-4f48-8f75-1e0e24c89638" path="/var/lib/kubelet/pods/ca76fcf6-61da-4f48-8f75-1e0e24c89638/volumes" Jul 26 15:46:36.866471 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:36.866436 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:46:36.929669 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:36.929625 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:46:37.131156 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:37.131077 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:46:37.362738 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:37.362699 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:46:37.687544 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:37.687510 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:46:37.995971 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:37.995882 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:46:38.095253 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:38.095221 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:46:40.469083 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:40.469029 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:46:40.569185 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:40.569151 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:46:40.713709 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:40.713674 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:46:40.941414 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:40.941380 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:46:41.262387 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:41.262298 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:46:41.361957 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:41.361910 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:46:44.959910 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:44.959872 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:46:45.410194 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:45.410155 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:46:45.715881 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:45.715758 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:46:46.069219 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:46.069120 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:46:48.519727 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:48.519696 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:46:48.614088 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:48.614052 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:46:48.683525 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:48.683487 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:46:48.796542 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:48.796449 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:46:49.316151 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:49.316119 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:46:49.407149 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:49.407114 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:46:53.010835 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:46:53.010796 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:47:01.002557 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:01.002522 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ad2c2-rfr22"] Jul 26 15:47:01.003109 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:01.002902 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca76fcf6-61da-4f48-8f75-1e0e24c89638" containerName="revoke-keys" Jul 26 15:47:01.003109 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:01.002914 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca76fcf6-61da-4f48-8f75-1e0e24c89638" containerName="revoke-keys" Jul 26 15:47:01.003109 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:01.002992 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca76fcf6-61da-4f48-8f75-1e0e24c89638" containerName="revoke-keys" Jul 26 15:47:01.005755 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:01.005732 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ad2c2-rfr22" Jul 26 15:47:01.007815 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:01.007791 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 26 15:47:01.007970 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:01.007826 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d665x\"" Jul 26 15:47:01.011629 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:01.011607 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ad2c2-rfr22"] Jul 26 15:47:01.043017 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:01.042990 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/023170ee-b45b-4fa7-b9e9-98ca31fb0e6b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2ad2c2-rfr22\" (UID: \"023170ee-b45b-4fa7-b9e9-98ca31fb0e6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ad2c2-rfr22" Jul 26 15:47:01.143423 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:01.143395 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/023170ee-b45b-4fa7-b9e9-98ca31fb0e6b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2ad2c2-rfr22\" (UID: \"023170ee-b45b-4fa7-b9e9-98ca31fb0e6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ad2c2-rfr22" Jul 26 15:47:01.144061 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:01.144042 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/023170ee-b45b-4fa7-b9e9-98ca31fb0e6b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2ad2c2-rfr22\" (UID: \"023170ee-b45b-4fa7-b9e9-98ca31fb0e6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ad2c2-rfr22" Jul 26 15:47:01.319413 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:01.319343 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ad2c2-rfr22" Jul 26 15:47:01.443125 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:01.443096 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ad2c2-rfr22"] Jul 26 15:47:01.446679 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:47:01.446646 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod023170ee_b45b_4fa7_b9e9_98ca31fb0e6b.slice/crio-049dcf059ec850b53fcd5a2ceb3ce079a9119e862d430f7eb967e69c9e1c40b0 WatchSource:0}: Error finding container 049dcf059ec850b53fcd5a2ceb3ce079a9119e862d430f7eb967e69c9e1c40b0: Status 404 returned error can't find the container with id 049dcf059ec850b53fcd5a2ceb3ce079a9119e862d430f7eb967e69c9e1c40b0 Jul 26 15:47:01.943426 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:01.943392 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ad2c2-rfr22" event={"ID":"023170ee-b45b-4fa7-b9e9-98ca31fb0e6b","Type":"ContainerStarted","Data":"78fd068c487642b3da3d4bdcbd64c027193cbb8dd69f7caad14295ba32a75faa"} Jul 26 15:47:01.943607 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:01.943444 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ad2c2-rfr22" event={"ID":"023170ee-b45b-4fa7-b9e9-98ca31fb0e6b","Type":"ContainerStarted","Data":"049dcf059ec850b53fcd5a2ceb3ce079a9119e862d430f7eb967e69c9e1c40b0"} Jul 26 15:47:01.956770 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:01.956727 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ad2c2-rfr22" podStartSLOduration=1.9567116439999999 podStartE2EDuration="1.956711644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:47:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:47:01.955465653 +0000 UTC m=+2502.379312548" watchObservedRunningTime="2026-07-26 15:47:01.956711644 +0000 UTC m=+2502.380558525" Jul 26 15:47:02.949741 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:02.949707 2555 generic.go:356] "Generic (PLEG): container finished" podID="023170ee-b45b-4fa7-b9e9-98ca31fb0e6b" containerID="78fd068c487642b3da3d4bdcbd64c027193cbb8dd69f7caad14295ba32a75faa" exitCode=0 Jul 26 15:47:02.950178 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:02.949780 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ad2c2-rfr22" event={"ID":"023170ee-b45b-4fa7-b9e9-98ca31fb0e6b","Type":"ContainerDied","Data":"78fd068c487642b3da3d4bdcbd64c027193cbb8dd69f7caad14295ba32a75faa"} Jul 26 15:47:04.082082 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:04.082058 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ad2c2-rfr22" Jul 26 15:47:04.169694 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:04.169669 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/023170ee-b45b-4fa7-b9e9-98ca31fb0e6b-maas-api-service-ca\") pod \"023170ee-b45b-4fa7-b9e9-98ca31fb0e6b\" (UID: \"023170ee-b45b-4fa7-b9e9-98ca31fb0e6b\") " Jul 26 15:47:04.170007 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:04.169986 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/023170ee-b45b-4fa7-b9e9-98ca31fb0e6b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "023170ee-b45b-4fa7-b9e9-98ca31fb0e6b" (UID: "023170ee-b45b-4fa7-b9e9-98ca31fb0e6b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:47:04.271234 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:04.271176 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/023170ee-b45b-4fa7-b9e9-98ca31fb0e6b-maas-api-service-ca\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:47:04.960906 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:04.960874 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ad2c2-rfr22" Jul 26 15:47:04.961095 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:04.960875 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ad2c2-rfr22" event={"ID":"023170ee-b45b-4fa7-b9e9-98ca31fb0e6b","Type":"ContainerDied","Data":"049dcf059ec850b53fcd5a2ceb3ce079a9119e862d430f7eb967e69c9e1c40b0"} Jul 26 15:47:04.961095 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:04.961012 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="049dcf059ec850b53fcd5a2ceb3ce079a9119e862d430f7eb967e69c9e1c40b0" Jul 26 15:47:16.680979 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:16.680890 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:47:17.603121 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:17.603090 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:47:21.087924 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:21.087886 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:47:21.088898 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:21.088875 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ad2c2-rfr22"] Jul 26 15:47:21.091861 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:21.091836 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ad2c2-rfr22"] Jul 26 15:47:21.438173 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:21.438143 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:47:22.064148 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:22.064121 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="023170ee-b45b-4fa7-b9e9-98ca31fb0e6b" path="/var/lib/kubelet/pods/023170ee-b45b-4fa7-b9e9-98ca31fb0e6b/volumes" Jul 26 15:47:26.979756 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:26.979717 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:47:27.042947 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:27.042902 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:47:27.312174 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:27.312092 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:47:27.477449 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:27.477423 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:47:27.594831 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:27.594761 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14ba5c-bbnt8"] Jul 26 15:47:27.595102 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:27.595088 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="023170ee-b45b-4fa7-b9e9-98ca31fb0e6b" containerName="revoke-keys" Jul 26 15:47:27.595153 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:27.595103 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="023170ee-b45b-4fa7-b9e9-98ca31fb0e6b" containerName="revoke-keys" Jul 26 15:47:27.595188 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:27.595176 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="023170ee-b45b-4fa7-b9e9-98ca31fb0e6b" containerName="revoke-keys" Jul 26 15:47:27.598031 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:27.598013 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14ba5c-bbnt8" Jul 26 15:47:27.599955 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:27.599920 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 26 15:47:27.600089 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:27.599974 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d665x\"" Jul 26 15:47:27.606002 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:27.605981 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14ba5c-bbnt8"] Jul 26 15:47:27.656851 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:27.656826 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5f6aaa9-1938-4539-9562-0cb3486af138-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-14ba5c-bbnt8\" (UID: \"c5f6aaa9-1938-4539-9562-0cb3486af138\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14ba5c-bbnt8" Jul 26 15:47:27.758110 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:27.758084 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5f6aaa9-1938-4539-9562-0cb3486af138-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-14ba5c-bbnt8\" (UID: \"c5f6aaa9-1938-4539-9562-0cb3486af138\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14ba5c-bbnt8" Jul 26 15:47:27.758693 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:27.758668 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5f6aaa9-1938-4539-9562-0cb3486af138-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-14ba5c-bbnt8\" (UID: \"c5f6aaa9-1938-4539-9562-0cb3486af138\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14ba5c-bbnt8" Jul 26 15:47:27.908435 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:27.908409 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14ba5c-bbnt8" Jul 26 15:47:28.238906 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:28.238812 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14ba5c-bbnt8"] Jul 26 15:47:28.241769 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:47:28.241740 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5f6aaa9_1938_4539_9562_0cb3486af138.slice/crio-b535a4dad951dfef4e8b77deefac2687ccf9d8dd6421ed90d1e262a59a4a9f4a WatchSource:0}: Error finding container b535a4dad951dfef4e8b77deefac2687ccf9d8dd6421ed90d1e262a59a4a9f4a: Status 404 returned error can't find the container with id b535a4dad951dfef4e8b77deefac2687ccf9d8dd6421ed90d1e262a59a4a9f4a Jul 26 15:47:29.079854 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:29.079822 2555 generic.go:356] "Generic (PLEG): container finished" podID="c5f6aaa9-1938-4539-9562-0cb3486af138" containerID="1a72c514acdeac067dfd903849fe1d22f20a95b947b269a440c8463ee80dc23a" exitCode=0 Jul 26 15:47:29.080049 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:29.079868 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14ba5c-bbnt8" event={"ID":"c5f6aaa9-1938-4539-9562-0cb3486af138","Type":"ContainerDied","Data":"1a72c514acdeac067dfd903849fe1d22f20a95b947b269a440c8463ee80dc23a"} Jul 26 15:47:29.080049 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:29.079965 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14ba5c-bbnt8" event={"ID":"c5f6aaa9-1938-4539-9562-0cb3486af138","Type":"ContainerStarted","Data":"b535a4dad951dfef4e8b77deefac2687ccf9d8dd6421ed90d1e262a59a4a9f4a"} Jul 26 15:47:30.210650 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:30.210626 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14ba5c-bbnt8" Jul 26 15:47:30.277093 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:30.277065 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5f6aaa9-1938-4539-9562-0cb3486af138-maas-api-service-ca\") pod \"c5f6aaa9-1938-4539-9562-0cb3486af138\" (UID: \"c5f6aaa9-1938-4539-9562-0cb3486af138\") " Jul 26 15:47:30.277364 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:30.277343 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5f6aaa9-1938-4539-9562-0cb3486af138-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c5f6aaa9-1938-4539-9562-0cb3486af138" (UID: "c5f6aaa9-1938-4539-9562-0cb3486af138"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:47:30.377897 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:30.377874 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5f6aaa9-1938-4539-9562-0cb3486af138-maas-api-service-ca\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:47:31.090561 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:31.090529 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14ba5c-bbnt8" event={"ID":"c5f6aaa9-1938-4539-9562-0cb3486af138","Type":"ContainerDied","Data":"b535a4dad951dfef4e8b77deefac2687ccf9d8dd6421ed90d1e262a59a4a9f4a"} Jul 26 15:47:31.090711 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:31.090564 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14ba5c-bbnt8" Jul 26 15:47:31.090711 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:31.090571 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b535a4dad951dfef4e8b77deefac2687ccf9d8dd6421ed90d1e262a59a4a9f4a" Jul 26 15:47:43.047100 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:43.047058 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:47:43.128377 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:43.128345 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:47:43.201093 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:43.201063 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:47:47.676283 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:47.676246 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14ba5c-bbnt8"] Jul 26 15:47:47.678977 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:47.678918 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14ba5c-bbnt8"] Jul 26 15:47:47.679471 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:47.679442 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:47:48.049702 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:48.049615 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:47:48.063942 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:48.063904 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5f6aaa9-1938-4539-9562-0cb3486af138" path="/var/lib/kubelet/pods/c5f6aaa9-1938-4539-9562-0cb3486af138/volumes" Jul 26 15:47:53.670207 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:53.670172 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:47:53.728721 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:53.728696 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:47:53.929268 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:53.929190 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:47:54.158583 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:54.158554 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:47:54.497209 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:54.497178 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:47:54.805036 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:54.804964 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:47:55.082020 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:55.081925 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:47:57.939983 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:57.939944 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:47:58.034396 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:58.034360 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:47:58.108972 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:58.108925 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:47:58.632187 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:58.632154 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:47:58.741993 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:58.741903 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:47:58.840349 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:47:58.840316 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:48:02.439344 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:48:02.439306 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:48:02.764797 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:48:02.764713 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:48:03.057770 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:48:03.057683 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:48:03.272763 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:48:03.272731 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:48:06.013501 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:48:06.013466 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:48:06.099183 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:48:06.099145 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:48:06.159195 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:48:06.159164 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:48:06.690125 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:48:06.690081 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:48:06.792726 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:48:06.792692 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:48:06.864937 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:48:06.864901 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:48:10.499205 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:48:10.499168 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:48:20.227691 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:48:20.227660 2555 scope.go:117] "RemoveContainer" containerID="43e099a8dd7993657511c01563a522dd05cd75607f5251eb876e07455b76b936" Jul 26 15:48:29.665250 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:48:29.665213 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:48:48.240486 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:48:48.240410 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-60c814/auth-test-model-60c814-kserve-7fd5c84b75-862s9"] Jul 26 15:48:48.240906 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:48:48.240758 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5f6aaa9-1938-4539-9562-0cb3486af138" containerName="revoke-keys" Jul 26 15:48:48.240906 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:48:48.240768 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5f6aaa9-1938-4539-9562-0cb3486af138" containerName="revoke-keys" Jul 26 15:48:48.240906 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:48:48.240830 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5f6aaa9-1938-4539-9562-0cb3486af138" containerName="revoke-keys" Jul 26 15:48:48.244050 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:48:48.244029 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-60c814/auth-test-model-60c814-kserve-7fd5c84b75-862s9" Jul 26 15:48:48.247235 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:48:48.247212 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-60c814\"/\"kube-root-ca.crt\"" Jul 26 15:48:48.247874 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:48:48.247849 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-60c814\"/\"auth-test-model-60c814-kserve-self-signed-certs\"" Jul 26 15:48:48.248014 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:48:48.247851 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-60c814\"/\"openshift-service-ca.crt\"" Jul 26 15:48:48.248014 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:48:48.247856 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-60c814\"/\"default-dockercfg-vkltq\"" Jul 26 15:48:48.252371 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:48:48.252347 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-60c814/auth-test-model-60c814-kserve-7fd5c84b75-862s9"] Jul 26 15:48:48.261403 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:48:48.261379 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:48:48.409834 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:48:48.409805 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e0e6c17-de15-41b1-b824-3501782d76a6-tmp-dir\") pod \"auth-test-model-60c814-kserve-7fd5c84b75-862s9\" (UID: \"7e0e6c17-de15-41b1-b824-3501782d76a6\") " pod="ai-tenant-e2e-shared-b-60c814/auth-test-model-60c814-kserve-7fd5c84b75-862s9" Jul 26 15:48:48.410002 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:48:48.409845 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e0e6c17-de15-41b1-b824-3501782d76a6-model-cache\") pod \"auth-test-model-60c814-kserve-7fd5c84b75-862s9\" (UID: \"7e0e6c17-de15-41b1-b824-3501782d76a6\") " pod="ai-tenant-e2e-shared-b-60c814/auth-test-model-60c814-kserve-7fd5c84b75-862s9" Jul 26 15:48:48.410002 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:48:48.409863 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e0e6c17-de15-41b1-b824-3501782d76a6-tls-certs\") pod \"auth-test-model-60c814-kserve-7fd5c84b75-862s9\" (UID: \"7e0e6c17-de15-41b1-b824-3501782d76a6\") " pod="ai-tenant-e2e-shared-b-60c814/auth-test-model-60c814-kserve-7fd5c84b75-862s9" Jul 26 15:48:48.410088 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:48:48.409990 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzrrd\" (UniqueName: \"kubernetes.io/projected/7e0e6c17-de15-41b1-b824-3501782d76a6-kube-api-access-lzrrd\") pod \"auth-test-model-60c814-kserve-7fd5c84b75-862s9\" (UID: \"7e0e6c17-de15-41b1-b824-3501782d76a6\") " pod="ai-tenant-e2e-shared-b-60c814/auth-test-model-60c814-kserve-7fd5c84b75-862s9" Jul 26 15:48:48.410088 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:48:48.410057 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e0e6c17-de15-41b1-b824-3501782d76a6-home\") pod \"auth-test-model-60c814-kserve-7fd5c84b75-862s9\" (UID: \"7e0e6c17-de15-41b1-b824-3501782d76a6\") " pod="ai-tenant-e2e-shared-b-60c814/auth-test-model-60c814-kserve-7fd5c84b75-862s9" Jul 26 15:48:48.410151 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:48:48.410104 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e0e6c17-de15-41b1-b824-3501782d76a6-dshm\") pod \"auth-test-model-60c814-kserve-7fd5c84b75-862s9\" (UID: \"7e0e6c17-de15-41b1-b824-3501782d76a6\") " pod="ai-tenant-e2e-shared-b-60c814/auth-test-model-60c814-kserve-7fd5c84b75-862s9" Jul 26 15:48:48.511089 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:48:48.511024 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e0e6c17-de15-41b1-b824-3501782d76a6-dshm\") pod \"auth-test-model-60c814-kserve-7fd5c84b75-862s9\" (UID: \"7e0e6c17-de15-41b1-b824-3501782d76a6\") " pod="ai-tenant-e2e-shared-b-60c814/auth-test-model-60c814-kserve-7fd5c84b75-862s9" Jul 26 15:48:48.511089 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:48:48.511057 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e0e6c17-de15-41b1-b824-3501782d76a6-tmp-dir\") pod \"auth-test-model-60c814-kserve-7fd5c84b75-862s9\" (UID: \"7e0e6c17-de15-41b1-b824-3501782d76a6\") " pod="ai-tenant-e2e-shared-b-60c814/auth-test-model-60c814-kserve-7fd5c84b75-862s9" Jul 26 15:48:48.511089 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:48:48.511082 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e0e6c17-de15-41b1-b824-3501782d76a6-model-cache\") pod \"auth-test-model-60c814-kserve-7fd5c84b75-862s9\" (UID: \"7e0e6c17-de15-41b1-b824-3501782d76a6\") " pod="ai-tenant-e2e-shared-b-60c814/auth-test-model-60c814-kserve-7fd5c84b75-862s9" Jul 26 15:48:48.511320 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:48:48.511101 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e0e6c17-de15-41b1-b824-3501782d76a6-tls-certs\") pod \"auth-test-model-60c814-kserve-7fd5c84b75-862s9\" (UID: \"7e0e6c17-de15-41b1-b824-3501782d76a6\") " pod="ai-tenant-e2e-shared-b-60c814/auth-test-model-60c814-kserve-7fd5c84b75-862s9" Jul 26 15:48:48.511320 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:48:48.511156 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzrrd\" (UniqueName: \"kubernetes.io/projected/7e0e6c17-de15-41b1-b824-3501782d76a6-kube-api-access-lzrrd\") pod \"auth-test-model-60c814-kserve-7fd5c84b75-862s9\" (UID: \"7e0e6c17-de15-41b1-b824-3501782d76a6\") " pod="ai-tenant-e2e-shared-b-60c814/auth-test-model-60c814-kserve-7fd5c84b75-862s9" Jul 26 15:48:48.511320 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:48:48.511206 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e0e6c17-de15-41b1-b824-3501782d76a6-home\") pod \"auth-test-model-60c814-kserve-7fd5c84b75-862s9\" (UID: \"7e0e6c17-de15-41b1-b824-3501782d76a6\") " pod="ai-tenant-e2e-shared-b-60c814/auth-test-model-60c814-kserve-7fd5c84b75-862s9" Jul 26 15:48:48.511522 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:48:48.511501 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e0e6c17-de15-41b1-b824-3501782d76a6-tmp-dir\") pod \"auth-test-model-60c814-kserve-7fd5c84b75-862s9\" (UID: \"7e0e6c17-de15-41b1-b824-3501782d76a6\") " pod="ai-tenant-e2e-shared-b-60c814/auth-test-model-60c814-kserve-7fd5c84b75-862s9" Jul 26 15:48:48.511595 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:48:48.511546 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e0e6c17-de15-41b1-b824-3501782d76a6-model-cache\") pod \"auth-test-model-60c814-kserve-7fd5c84b75-862s9\" (UID: \"7e0e6c17-de15-41b1-b824-3501782d76a6\") " pod="ai-tenant-e2e-shared-b-60c814/auth-test-model-60c814-kserve-7fd5c84b75-862s9" Jul 26 15:48:48.511595 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:48:48.511565 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e0e6c17-de15-41b1-b824-3501782d76a6-home\") pod \"auth-test-model-60c814-kserve-7fd5c84b75-862s9\" (UID: \"7e0e6c17-de15-41b1-b824-3501782d76a6\") " pod="ai-tenant-e2e-shared-b-60c814/auth-test-model-60c814-kserve-7fd5c84b75-862s9" Jul 26 15:48:48.513178 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:48:48.513161 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e0e6c17-de15-41b1-b824-3501782d76a6-dshm\") pod \"auth-test-model-60c814-kserve-7fd5c84b75-862s9\" (UID: \"7e0e6c17-de15-41b1-b824-3501782d76a6\") " pod="ai-tenant-e2e-shared-b-60c814/auth-test-model-60c814-kserve-7fd5c84b75-862s9" Jul 26 15:48:48.513640 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:48:48.513621 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e0e6c17-de15-41b1-b824-3501782d76a6-tls-certs\") pod \"auth-test-model-60c814-kserve-7fd5c84b75-862s9\" (UID: \"7e0e6c17-de15-41b1-b824-3501782d76a6\") " pod="ai-tenant-e2e-shared-b-60c814/auth-test-model-60c814-kserve-7fd5c84b75-862s9" Jul 26 15:48:48.518383 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:48:48.518358 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzrrd\" (UniqueName: \"kubernetes.io/projected/7e0e6c17-de15-41b1-b824-3501782d76a6-kube-api-access-lzrrd\") pod \"auth-test-model-60c814-kserve-7fd5c84b75-862s9\" (UID: \"7e0e6c17-de15-41b1-b824-3501782d76a6\") " pod="ai-tenant-e2e-shared-b-60c814/auth-test-model-60c814-kserve-7fd5c84b75-862s9" Jul 26 15:48:48.553688 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:48:48.553663 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-60c814/auth-test-model-60c814-kserve-7fd5c84b75-862s9" Jul 26 15:48:48.680821 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:48:48.680794 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-60c814/auth-test-model-60c814-kserve-7fd5c84b75-862s9"] Jul 26 15:48:48.682563 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:48:48.682537 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e0e6c17_de15_41b1_b824_3501782d76a6.slice/crio-3007c62c0786963dc2ef011ddd5c3122ce3ed4d3d6eb8f171b545c076bbbb907 WatchSource:0}: Error finding container 3007c62c0786963dc2ef011ddd5c3122ce3ed4d3d6eb8f171b545c076bbbb907: Status 404 returned error can't find the container with id 3007c62c0786963dc2ef011ddd5c3122ce3ed4d3d6eb8f171b545c076bbbb907 Jul 26 15:48:49.461884 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:48:49.461847 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-60c814/auth-test-model-60c814-kserve-7fd5c84b75-862s9" event={"ID":"7e0e6c17-de15-41b1-b824-3501782d76a6","Type":"ContainerStarted","Data":"62467240b43eb6c64356be1fdb54d8c22d105e0a1bfa7f15ba65f230f326ddde"} Jul 26 15:48:49.462381 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:48:49.461892 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-60c814/auth-test-model-60c814-kserve-7fd5c84b75-862s9" event={"ID":"7e0e6c17-de15-41b1-b824-3501782d76a6","Type":"ContainerStarted","Data":"3007c62c0786963dc2ef011ddd5c3122ce3ed4d3d6eb8f171b545c076bbbb907"} Jul 26 15:48:49.479481 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:48:49.479424 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-60c814/auth-test-model-60c814-kserve-7fd5c84b75-862s9" podStartSLOduration=1.479408527 podStartE2EDuration="1.479408527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:48:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:48:49.477811958 +0000 UTC m=+2609.901658827" watchObservedRunningTime="2026-07-26 15:48:49.479408527 +0000 UTC m=+2609.903255398" Jul 26 15:48:58.554584 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:48:58.554552 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-60c814/auth-test-model-60c814-kserve-7fd5c84b75-862s9" Jul 26 15:48:58.555043 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:48:58.554603 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-60c814/auth-test-model-60c814-kserve-7fd5c84b75-862s9" Jul 26 15:48:58.566734 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:48:58.566712 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-60c814/auth-test-model-60c814-kserve-7fd5c84b75-862s9" Jul 26 15:48:59.525597 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:48:59.525558 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-60c814/auth-test-model-60c814-kserve-7fd5c84b75-862s9" Jul 26 15:49:27.301917 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:49:27.301879 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:49:27.354532 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:49:27.354495 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:50:20.192756 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:50:20.192728 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9f857_1fbafda6-6242-400e-b2d4-24584608e5f5/ovn-acl-logging/0.log" Jul 26 15:50:20.199432 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:50:20.199410 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9f857_1fbafda6-6242-400e-b2d4-24584608e5f5/ovn-acl-logging/0.log" Jul 26 15:50:44.841697 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:50:44.841659 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:51:20.263161 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:51:20.263132 2555 scope.go:117] "RemoveContainer" containerID="ecee411fdf3eccba5d10f0306d628518e932f02838ac676a022ec39d6d301494" Jul 26 15:51:20.270864 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:51:20.270845 2555 scope.go:117] "RemoveContainer" containerID="4a762115a6da09577cbc2c18ebd12c736a05a27acef02f80f0d68b561dfb5aa6" Jul 26 15:51:44.467388 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:51:44.467301 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:52:20.291478 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:52:20.291448 2555 scope.go:117] "RemoveContainer" containerID="ad656bd5d26b834f2bad4602711f1578d24528662f3014ef804cfa41dceeb25e" Jul 26 15:52:20.299279 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:52:20.299262 2555 scope.go:117] "RemoveContainer" containerID="ce27ba97198e23a0a2e4c09575b997cd9b8a5fb2b862bf48c3edf7b0b5882cea" Jul 26 15:52:48.336609 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:52:48.336573 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:53:09.296392 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:09.296355 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:53:10.316752 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:10.316715 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-60c814/auth-test-model-60c814-kserve-7fd5c84b75-862s9"] Jul 26 15:53:10.332747 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:10.332720 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:53:20.319865 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:20.319832 2555 scope.go:117] "RemoveContainer" containerID="78fd068c487642b3da3d4bdcbd64c027193cbb8dd69f7caad14295ba32a75faa" Jul 26 15:53:25.336247 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:25.336199 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-60c814/auth-test-model-60c814-kserve-7fd5c84b75-862s9" podUID="7e0e6c17-de15-41b1-b824-3501782d76a6" containerName="main" containerID="cri-o://62467240b43eb6c64356be1fdb54d8c22d105e0a1bfa7f15ba65f230f326ddde" gracePeriod=45 Jul 26 15:53:25.579085 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:25.579059 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-60c814/auth-test-model-60c814-kserve-7fd5c84b75-862s9" Jul 26 15:53:25.698607 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:25.698577 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lzrrd\" (UniqueName: \"kubernetes.io/projected/7e0e6c17-de15-41b1-b824-3501782d76a6-kube-api-access-lzrrd\") pod \"7e0e6c17-de15-41b1-b824-3501782d76a6\" (UID: \"7e0e6c17-de15-41b1-b824-3501782d76a6\") " Jul 26 15:53:25.698765 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:25.698644 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e0e6c17-de15-41b1-b824-3501782d76a6-dshm\") pod \"7e0e6c17-de15-41b1-b824-3501782d76a6\" (UID: \"7e0e6c17-de15-41b1-b824-3501782d76a6\") " Jul 26 15:53:25.698765 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:25.698685 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e0e6c17-de15-41b1-b824-3501782d76a6-tls-certs\") pod \"7e0e6c17-de15-41b1-b824-3501782d76a6\" (UID: \"7e0e6c17-de15-41b1-b824-3501782d76a6\") " Jul 26 15:53:25.698765 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:25.698745 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e0e6c17-de15-41b1-b824-3501782d76a6-home\") pod \"7e0e6c17-de15-41b1-b824-3501782d76a6\" (UID: \"7e0e6c17-de15-41b1-b824-3501782d76a6\") " Jul 26 15:53:25.698962 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:25.698796 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e0e6c17-de15-41b1-b824-3501782d76a6-tmp-dir\") pod \"7e0e6c17-de15-41b1-b824-3501782d76a6\" (UID: \"7e0e6c17-de15-41b1-b824-3501782d76a6\") " Jul 26 15:53:25.698962 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:25.698830 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e0e6c17-de15-41b1-b824-3501782d76a6-model-cache\") pod \"7e0e6c17-de15-41b1-b824-3501782d76a6\" (UID: \"7e0e6c17-de15-41b1-b824-3501782d76a6\") " Jul 26 15:53:25.699079 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:25.699053 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e0e6c17-de15-41b1-b824-3501782d76a6-home" (OuterVolumeSpecName: "home") pod "7e0e6c17-de15-41b1-b824-3501782d76a6" (UID: "7e0e6c17-de15-41b1-b824-3501782d76a6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:53:25.699132 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:25.699066 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e0e6c17-de15-41b1-b824-3501782d76a6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7e0e6c17-de15-41b1-b824-3501782d76a6" (UID: "7e0e6c17-de15-41b1-b824-3501782d76a6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:53:25.699177 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:25.699147 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e0e6c17-de15-41b1-b824-3501782d76a6-model-cache" (OuterVolumeSpecName: "model-cache") pod "7e0e6c17-de15-41b1-b824-3501782d76a6" (UID: "7e0e6c17-de15-41b1-b824-3501782d76a6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:53:25.700918 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:25.700896 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e0e6c17-de15-41b1-b824-3501782d76a6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7e0e6c17-de15-41b1-b824-3501782d76a6" (UID: "7e0e6c17-de15-41b1-b824-3501782d76a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 15:53:25.701121 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:25.701102 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e0e6c17-de15-41b1-b824-3501782d76a6-dshm" (OuterVolumeSpecName: "dshm") pod "7e0e6c17-de15-41b1-b824-3501782d76a6" (UID: "7e0e6c17-de15-41b1-b824-3501782d76a6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:53:25.701121 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:25.701113 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e0e6c17-de15-41b1-b824-3501782d76a6-kube-api-access-lzrrd" (OuterVolumeSpecName: "kube-api-access-lzrrd") pod "7e0e6c17-de15-41b1-b824-3501782d76a6" (UID: "7e0e6c17-de15-41b1-b824-3501782d76a6"). InnerVolumeSpecName "kube-api-access-lzrrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:53:25.800320 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:25.800296 2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e0e6c17-de15-41b1-b824-3501782d76a6-home\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:53:25.800320 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:25.800317 2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e0e6c17-de15-41b1-b824-3501782d76a6-tmp-dir\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:53:25.800473 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:25.800325 2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e0e6c17-de15-41b1-b824-3501782d76a6-model-cache\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:53:25.800473 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:25.800335 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzrrd\" (UniqueName: \"kubernetes.io/projected/7e0e6c17-de15-41b1-b824-3501782d76a6-kube-api-access-lzrrd\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:53:25.800473 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:25.800343 2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e0e6c17-de15-41b1-b824-3501782d76a6-dshm\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:53:25.800473 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:25.800352 2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e0e6c17-de15-41b1-b824-3501782d76a6-tls-certs\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:53:25.802300 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:25.802274 2555 generic.go:356] "Generic (PLEG): container finished" podID="7e0e6c17-de15-41b1-b824-3501782d76a6" containerID="62467240b43eb6c64356be1fdb54d8c22d105e0a1bfa7f15ba65f230f326ddde" exitCode=0 Jul 26 15:53:25.802392 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:25.802312 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-60c814/auth-test-model-60c814-kserve-7fd5c84b75-862s9" event={"ID":"7e0e6c17-de15-41b1-b824-3501782d76a6","Type":"ContainerDied","Data":"62467240b43eb6c64356be1fdb54d8c22d105e0a1bfa7f15ba65f230f326ddde"} Jul 26 15:53:25.802392 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:25.802319 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-60c814/auth-test-model-60c814-kserve-7fd5c84b75-862s9" Jul 26 15:53:25.802392 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:25.802345 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-60c814/auth-test-model-60c814-kserve-7fd5c84b75-862s9" event={"ID":"7e0e6c17-de15-41b1-b824-3501782d76a6","Type":"ContainerDied","Data":"3007c62c0786963dc2ef011ddd5c3122ce3ed4d3d6eb8f171b545c076bbbb907"} Jul 26 15:53:25.802392 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:25.802368 2555 scope.go:117] "RemoveContainer" containerID="62467240b43eb6c64356be1fdb54d8c22d105e0a1bfa7f15ba65f230f326ddde" Jul 26 15:53:25.811194 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:25.811179 2555 scope.go:117] "RemoveContainer" containerID="62467240b43eb6c64356be1fdb54d8c22d105e0a1bfa7f15ba65f230f326ddde" Jul 26 15:53:25.811431 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:53:25.811415 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62467240b43eb6c64356be1fdb54d8c22d105e0a1bfa7f15ba65f230f326ddde\": container with ID starting with 62467240b43eb6c64356be1fdb54d8c22d105e0a1bfa7f15ba65f230f326ddde not found: ID does not exist" containerID="62467240b43eb6c64356be1fdb54d8c22d105e0a1bfa7f15ba65f230f326ddde" Jul 26 15:53:25.811478 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:25.811438 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62467240b43eb6c64356be1fdb54d8c22d105e0a1bfa7f15ba65f230f326ddde"} err="failed to get container status \"62467240b43eb6c64356be1fdb54d8c22d105e0a1bfa7f15ba65f230f326ddde\": rpc error: code = NotFound desc = could not find container \"62467240b43eb6c64356be1fdb54d8c22d105e0a1bfa7f15ba65f230f326ddde\": container with ID starting with 62467240b43eb6c64356be1fdb54d8c22d105e0a1bfa7f15ba65f230f326ddde not found: ID does not exist" Jul 26 15:53:25.821219 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:25.821197 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-60c814/auth-test-model-60c814-kserve-7fd5c84b75-862s9"] Jul 26 15:53:25.825299 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:25.825279 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-60c814/auth-test-model-60c814-kserve-7fd5c84b75-862s9"] Jul 26 15:53:26.063644 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:26.063584 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e0e6c17-de15-41b1-b824-3501782d76a6" path="/var/lib/kubelet/pods/7e0e6c17-de15-41b1-b824-3501782d76a6/volumes" Jul 26 15:53:39.236895 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:39.236863 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-a20d61/sub-test-model-a20d61-kserve-b7788c68c-mqwqb"] Jul 26 15:53:39.237279 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:39.237198 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e0e6c17-de15-41b1-b824-3501782d76a6" containerName="main" Jul 26 15:53:39.237279 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:39.237210 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e0e6c17-de15-41b1-b824-3501782d76a6" containerName="main" Jul 26 15:53:39.237279 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:39.237274 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e0e6c17-de15-41b1-b824-3501782d76a6" containerName="main" Jul 26 15:53:39.240099 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:39.240084 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a20d61/sub-test-model-a20d61-kserve-b7788c68c-mqwqb" Jul 26 15:53:39.242363 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:39.242342 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a20d61\"/\"default-dockercfg-h8f2k\"" Jul 26 15:53:39.242470 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:39.242376 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-a20d61\"/\"kube-root-ca.crt\"" Jul 26 15:53:39.242751 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:39.242732 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a20d61\"/\"sub-test-model-a20d61-kserve-self-signed-certs\"" Jul 26 15:53:39.242832 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:39.242762 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-a20d61\"/\"openshift-service-ca.crt\"" Jul 26 15:53:39.250105 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:39.250083 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a20d61/sub-test-model-a20d61-kserve-b7788c68c-mqwqb"] Jul 26 15:53:39.268098 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:39.268069 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:53:39.295912 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:39.295886 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c-model-cache\") pod \"sub-test-model-a20d61-kserve-b7788c68c-mqwqb\" (UID: \"f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c\") " pod="ai-tenant-e2e-shared-a-a20d61/sub-test-model-a20d61-kserve-b7788c68c-mqwqb" Jul 26 15:53:39.296032 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:39.295948 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c-dshm\") pod \"sub-test-model-a20d61-kserve-b7788c68c-mqwqb\" (UID: \"f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c\") " pod="ai-tenant-e2e-shared-a-a20d61/sub-test-model-a20d61-kserve-b7788c68c-mqwqb" Jul 26 15:53:39.296032 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:39.296016 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c-tls-certs\") pod \"sub-test-model-a20d61-kserve-b7788c68c-mqwqb\" (UID: \"f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c\") " pod="ai-tenant-e2e-shared-a-a20d61/sub-test-model-a20d61-kserve-b7788c68c-mqwqb" Jul 26 15:53:39.296150 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:39.296057 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6pn5\" (UniqueName: \"kubernetes.io/projected/f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c-kube-api-access-r6pn5\") pod \"sub-test-model-a20d61-kserve-b7788c68c-mqwqb\" (UID: \"f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c\") " pod="ai-tenant-e2e-shared-a-a20d61/sub-test-model-a20d61-kserve-b7788c68c-mqwqb" Jul 26 15:53:39.296150 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:39.296105 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c-tmp-dir\") pod \"sub-test-model-a20d61-kserve-b7788c68c-mqwqb\" (UID: \"f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c\") " pod="ai-tenant-e2e-shared-a-a20d61/sub-test-model-a20d61-kserve-b7788c68c-mqwqb" Jul 26 15:53:39.296244 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:39.296168 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c-home\") pod \"sub-test-model-a20d61-kserve-b7788c68c-mqwqb\" (UID: \"f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c\") " pod="ai-tenant-e2e-shared-a-a20d61/sub-test-model-a20d61-kserve-b7788c68c-mqwqb" Jul 26 15:53:39.396537 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:39.396514 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c-tls-certs\") pod \"sub-test-model-a20d61-kserve-b7788c68c-mqwqb\" (UID: \"f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c\") " pod="ai-tenant-e2e-shared-a-a20d61/sub-test-model-a20d61-kserve-b7788c68c-mqwqb" Jul 26 15:53:39.396668 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:39.396546 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6pn5\" (UniqueName: \"kubernetes.io/projected/f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c-kube-api-access-r6pn5\") pod \"sub-test-model-a20d61-kserve-b7788c68c-mqwqb\" (UID: \"f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c\") " pod="ai-tenant-e2e-shared-a-a20d61/sub-test-model-a20d61-kserve-b7788c68c-mqwqb" Jul 26 15:53:39.396668 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:39.396572 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c-tmp-dir\") pod \"sub-test-model-a20d61-kserve-b7788c68c-mqwqb\" (UID: \"f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c\") " pod="ai-tenant-e2e-shared-a-a20d61/sub-test-model-a20d61-kserve-b7788c68c-mqwqb" Jul 26 15:53:39.396668 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:39.396599 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c-home\") pod \"sub-test-model-a20d61-kserve-b7788c68c-mqwqb\" (UID: \"f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c\") " pod="ai-tenant-e2e-shared-a-a20d61/sub-test-model-a20d61-kserve-b7788c68c-mqwqb" Jul 26 15:53:39.396668 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:39.396635 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c-model-cache\") pod \"sub-test-model-a20d61-kserve-b7788c68c-mqwqb\" (UID: \"f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c\") " pod="ai-tenant-e2e-shared-a-a20d61/sub-test-model-a20d61-kserve-b7788c68c-mqwqb" Jul 26 15:53:39.396668 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:39.396656 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c-dshm\") pod \"sub-test-model-a20d61-kserve-b7788c68c-mqwqb\" (UID: \"f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c\") " pod="ai-tenant-e2e-shared-a-a20d61/sub-test-model-a20d61-kserve-b7788c68c-mqwqb" Jul 26 15:53:39.397036 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:39.397016 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c-tmp-dir\") pod \"sub-test-model-a20d61-kserve-b7788c68c-mqwqb\" (UID: \"f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c\") " pod="ai-tenant-e2e-shared-a-a20d61/sub-test-model-a20d61-kserve-b7788c68c-mqwqb" Jul 26 15:53:39.397107 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:39.397060 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c-home\") pod \"sub-test-model-a20d61-kserve-b7788c68c-mqwqb\" (UID: \"f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c\") " pod="ai-tenant-e2e-shared-a-a20d61/sub-test-model-a20d61-kserve-b7788c68c-mqwqb" Jul 26 15:53:39.397107 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:39.397080 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c-model-cache\") pod \"sub-test-model-a20d61-kserve-b7788c68c-mqwqb\" (UID: \"f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c\") " pod="ai-tenant-e2e-shared-a-a20d61/sub-test-model-a20d61-kserve-b7788c68c-mqwqb" Jul 26 15:53:39.398779 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:39.398761 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c-dshm\") pod \"sub-test-model-a20d61-kserve-b7788c68c-mqwqb\" (UID: \"f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c\") " pod="ai-tenant-e2e-shared-a-a20d61/sub-test-model-a20d61-kserve-b7788c68c-mqwqb" Jul 26 15:53:39.398919 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:39.398903 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c-tls-certs\") pod \"sub-test-model-a20d61-kserve-b7788c68c-mqwqb\" (UID: \"f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c\") " pod="ai-tenant-e2e-shared-a-a20d61/sub-test-model-a20d61-kserve-b7788c68c-mqwqb" Jul 26 15:53:39.403384 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:39.403363 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6pn5\" (UniqueName: \"kubernetes.io/projected/f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c-kube-api-access-r6pn5\") pod \"sub-test-model-a20d61-kserve-b7788c68c-mqwqb\" (UID: \"f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c\") " pod="ai-tenant-e2e-shared-a-a20d61/sub-test-model-a20d61-kserve-b7788c68c-mqwqb" Jul 26 15:53:39.550471 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:39.550406 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a20d61/sub-test-model-a20d61-kserve-b7788c68c-mqwqb" Jul 26 15:53:39.677513 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:39.677484 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a20d61/sub-test-model-a20d61-kserve-b7788c68c-mqwqb"] Jul 26 15:53:39.679619 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:53:39.679593 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf31d8e34_a7c6_4ab7_8d7c_e56ce0ca0e0c.slice/crio-b593145390aed26d756c6d77514903ef32ed9f7f40fd68fb55cab94cf98b0ab4 WatchSource:0}: Error finding container b593145390aed26d756c6d77514903ef32ed9f7f40fd68fb55cab94cf98b0ab4: Status 404 returned error can't find the container with id b593145390aed26d756c6d77514903ef32ed9f7f40fd68fb55cab94cf98b0ab4 Jul 26 15:53:39.871513 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:39.871481 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a20d61/sub-test-model-a20d61-kserve-b7788c68c-mqwqb" event={"ID":"f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c","Type":"ContainerStarted","Data":"46f6a61099bdebb4bb50d5335d80e5f1194a391bd9eca30a7bd2d15fcd930c52"} Jul 26 15:53:39.871676 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:39.871526 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a20d61/sub-test-model-a20d61-kserve-b7788c68c-mqwqb" event={"ID":"f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c","Type":"ContainerStarted","Data":"b593145390aed26d756c6d77514903ef32ed9f7f40fd68fb55cab94cf98b0ab4"} Jul 26 15:53:39.888637 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:39.888587 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-a20d61/sub-test-model-a20d61-kserve-b7788c68c-mqwqb" podStartSLOduration=0.888573876 podStartE2EDuration="888.573876ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:53:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:53:39.8877668 +0000 UTC m=+2900.311613671" watchObservedRunningTime="2026-07-26 15:53:39.888573876 +0000 UTC m=+2900.312420745" Jul 26 15:53:49.550676 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:49.550646 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-a20d61/sub-test-model-a20d61-kserve-b7788c68c-mqwqb" Jul 26 15:53:49.550676 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:49.550683 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-a20d61/sub-test-model-a20d61-kserve-b7788c68c-mqwqb" Jul 26 15:53:49.562771 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:49.562744 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-a20d61/sub-test-model-a20d61-kserve-b7788c68c-mqwqb" Jul 26 15:53:49.927323 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:53:49.927294 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-a20d61/sub-test-model-a20d61-kserve-b7788c68c-mqwqb" Jul 26 15:54:11.379728 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:54:11.379693 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:54:11.433217 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:54:11.433185 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:54:20.338951 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:54:20.338902 2555 scope.go:117] "RemoveContainer" containerID="1a72c514acdeac067dfd903849fe1d22f20a95b947b269a440c8463ee80dc23a" Jul 26 15:54:24.126392 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:54:24.126359 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:55:20.224334 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:20.224297 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9f857_1fbafda6-6242-400e-b2d4-24584608e5f5/ovn-acl-logging/0.log" Jul 26 15:55:20.231072 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:20.231050 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9f857_1fbafda6-6242-400e-b2d4-24584608e5f5/ovn-acl-logging/0.log" Jul 26 15:55:29.929442 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:29.929408 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a20d61/sub-test-model-a20d61-kserve-b7788c68c-mqwqb"] Jul 26 15:55:29.949329 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:29.949299 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:55:30.841061 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:30.841026 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:55:44.948232 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:44.948184 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-a20d61/sub-test-model-a20d61-kserve-b7788c68c-mqwqb" podUID="f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c" containerName="main" containerID="cri-o://46f6a61099bdebb4bb50d5335d80e5f1194a391bd9eca30a7bd2d15fcd930c52" gracePeriod=45 Jul 26 15:55:45.201007 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:45.200948 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a20d61/sub-test-model-a20d61-kserve-b7788c68c-mqwqb" Jul 26 15:55:45.354593 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:45.354561 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6pn5\" (UniqueName: \"kubernetes.io/projected/f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c-kube-api-access-r6pn5\") pod \"f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c\" (UID: \"f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c\") " Jul 26 15:55:45.354770 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:45.354644 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c-tmp-dir\") pod \"f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c\" (UID: \"f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c\") " Jul 26 15:55:45.354770 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:45.354682 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c-model-cache\") pod \"f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c\" (UID: \"f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c\") " Jul 26 15:55:45.354770 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:45.354741 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c-dshm\") pod \"f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c\" (UID: \"f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c\") " Jul 26 15:55:45.354961 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:45.354797 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c-tls-certs\") pod \"f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c\" (UID: \"f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c\") " Jul 26 15:55:45.354961 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:45.354821 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c-home\") pod \"f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c\" (UID: \"f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c\") " Jul 26 15:55:45.354961 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:45.354897 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c" (UID: "f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:55:45.355129 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:45.354993 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c-model-cache" (OuterVolumeSpecName: "model-cache") pod "f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c" (UID: "f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:55:45.355208 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:45.355187 2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c-tmp-dir\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:55:45.355266 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:45.355207 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c-home" (OuterVolumeSpecName: "home") pod "f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c" (UID: "f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:55:45.355266 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:45.355213 2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c-model-cache\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:55:45.356858 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:45.356837 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c-dshm" (OuterVolumeSpecName: "dshm") pod "f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c" (UID: "f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:55:45.356858 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:45.356851 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c" (UID: "f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 15:55:45.357062 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:45.356851 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c-kube-api-access-r6pn5" (OuterVolumeSpecName: "kube-api-access-r6pn5") pod "f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c" (UID: "f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c"). InnerVolumeSpecName "kube-api-access-r6pn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:55:45.441719 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:45.441696 2555 generic.go:356] "Generic (PLEG): container finished" podID="f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c" containerID="46f6a61099bdebb4bb50d5335d80e5f1194a391bd9eca30a7bd2d15fcd930c52" exitCode=0 Jul 26 15:55:45.441828 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:45.441735 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a20d61/sub-test-model-a20d61-kserve-b7788c68c-mqwqb" Jul 26 15:55:45.441828 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:45.441758 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a20d61/sub-test-model-a20d61-kserve-b7788c68c-mqwqb" event={"ID":"f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c","Type":"ContainerDied","Data":"46f6a61099bdebb4bb50d5335d80e5f1194a391bd9eca30a7bd2d15fcd930c52"} Jul 26 15:55:45.441828 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:45.441794 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a20d61/sub-test-model-a20d61-kserve-b7788c68c-mqwqb" event={"ID":"f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c","Type":"ContainerDied","Data":"b593145390aed26d756c6d77514903ef32ed9f7f40fd68fb55cab94cf98b0ab4"} Jul 26 15:55:45.441828 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:45.441813 2555 scope.go:117] "RemoveContainer" containerID="46f6a61099bdebb4bb50d5335d80e5f1194a391bd9eca30a7bd2d15fcd930c52" Jul 26 15:55:45.452498 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:45.452479 2555 scope.go:117] "RemoveContainer" containerID="46f6a61099bdebb4bb50d5335d80e5f1194a391bd9eca30a7bd2d15fcd930c52" Jul 26 15:55:45.452752 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:55:45.452731 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46f6a61099bdebb4bb50d5335d80e5f1194a391bd9eca30a7bd2d15fcd930c52\": container with ID starting with 46f6a61099bdebb4bb50d5335d80e5f1194a391bd9eca30a7bd2d15fcd930c52 not found: ID does not exist" containerID="46f6a61099bdebb4bb50d5335d80e5f1194a391bd9eca30a7bd2d15fcd930c52" Jul 26 15:55:45.452815 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:45.452759 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46f6a61099bdebb4bb50d5335d80e5f1194a391bd9eca30a7bd2d15fcd930c52"} err="failed to get container status \"46f6a61099bdebb4bb50d5335d80e5f1194a391bd9eca30a7bd2d15fcd930c52\": rpc error: code = NotFound desc = could not find container \"46f6a61099bdebb4bb50d5335d80e5f1194a391bd9eca30a7bd2d15fcd930c52\": container with ID starting with 46f6a61099bdebb4bb50d5335d80e5f1194a391bd9eca30a7bd2d15fcd930c52 not found: ID does not exist" Jul 26 15:55:45.456128 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:45.456112 2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c-dshm\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:55:45.456187 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:45.456130 2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c-tls-certs\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:55:45.456187 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:45.456139 2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c-home\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:55:45.456187 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:45.456147 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6pn5\" (UniqueName: \"kubernetes.io/projected/f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c-kube-api-access-r6pn5\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:55:45.463200 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:45.463177 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a20d61/sub-test-model-a20d61-kserve-b7788c68c-mqwqb"] Jul 26 15:55:45.466691 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:45.466673 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-a20d61/sub-test-model-a20d61-kserve-b7788c68c-mqwqb"] Jul 26 15:55:46.063710 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:46.063676 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c" path="/var/lib/kubelet/pods/f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c/volumes" Jul 26 15:55:54.618183 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:54.618148 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-a20d61/rate-test-model-a20d61-kserve-66b8b5b5c9-n8h6v"] Jul 26 15:55:54.618738 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:54.618671 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c" containerName="main" Jul 26 15:55:54.618738 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:54.618691 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c" containerName="main" Jul 26 15:55:54.618860 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:54.618794 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f31d8e34-a7c6-4ab7-8d7c-e56ce0ca0e0c" containerName="main" Jul 26 15:55:54.623136 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:54.623116 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a20d61/rate-test-model-a20d61-kserve-66b8b5b5c9-n8h6v" Jul 26 15:55:54.625155 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:54.625134 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-a20d61\"/\"openshift-service-ca.crt\"" Jul 26 15:55:54.625261 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:54.625173 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a20d61\"/\"rate-test-model-a20d61-kserve-self-signed-certs\"" Jul 26 15:55:54.625658 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:54.625641 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-a20d61\"/\"kube-root-ca.crt\"" Jul 26 15:55:54.625748 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:54.625642 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a20d61\"/\"default-dockercfg-h8f2k\"" Jul 26 15:55:54.631482 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:54.631460 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a20d61/rate-test-model-a20d61-kserve-66b8b5b5c9-n8h6v"] Jul 26 15:55:54.637747 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:54.637723 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:55:54.725458 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:54.725436 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58e6b0af-30d9-49c6-9e2b-50d8c0b33850-tls-certs\") pod \"rate-test-model-a20d61-kserve-66b8b5b5c9-n8h6v\" (UID: \"58e6b0af-30d9-49c6-9e2b-50d8c0b33850\") " pod="ai-tenant-e2e-shared-a-a20d61/rate-test-model-a20d61-kserve-66b8b5b5c9-n8h6v" Jul 26 15:55:54.725580 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:54.725464 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58e6b0af-30d9-49c6-9e2b-50d8c0b33850-dshm\") pod \"rate-test-model-a20d61-kserve-66b8b5b5c9-n8h6v\" (UID: \"58e6b0af-30d9-49c6-9e2b-50d8c0b33850\") " pod="ai-tenant-e2e-shared-a-a20d61/rate-test-model-a20d61-kserve-66b8b5b5c9-n8h6v" Jul 26 15:55:54.725580 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:54.725498 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdng5\" (UniqueName: \"kubernetes.io/projected/58e6b0af-30d9-49c6-9e2b-50d8c0b33850-kube-api-access-zdng5\") pod \"rate-test-model-a20d61-kserve-66b8b5b5c9-n8h6v\" (UID: \"58e6b0af-30d9-49c6-9e2b-50d8c0b33850\") " pod="ai-tenant-e2e-shared-a-a20d61/rate-test-model-a20d61-kserve-66b8b5b5c9-n8h6v" Jul 26 15:55:54.725685 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:54.725589 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58e6b0af-30d9-49c6-9e2b-50d8c0b33850-home\") pod \"rate-test-model-a20d61-kserve-66b8b5b5c9-n8h6v\" (UID: \"58e6b0af-30d9-49c6-9e2b-50d8c0b33850\") " pod="ai-tenant-e2e-shared-a-a20d61/rate-test-model-a20d61-kserve-66b8b5b5c9-n8h6v" Jul 26 15:55:54.725685 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:54.725645 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58e6b0af-30d9-49c6-9e2b-50d8c0b33850-tmp-dir\") pod \"rate-test-model-a20d61-kserve-66b8b5b5c9-n8h6v\" (UID: \"58e6b0af-30d9-49c6-9e2b-50d8c0b33850\") " pod="ai-tenant-e2e-shared-a-a20d61/rate-test-model-a20d61-kserve-66b8b5b5c9-n8h6v" Jul 26 15:55:54.725685 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:54.725672 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58e6b0af-30d9-49c6-9e2b-50d8c0b33850-model-cache\") pod \"rate-test-model-a20d61-kserve-66b8b5b5c9-n8h6v\" (UID: \"58e6b0af-30d9-49c6-9e2b-50d8c0b33850\") " pod="ai-tenant-e2e-shared-a-a20d61/rate-test-model-a20d61-kserve-66b8b5b5c9-n8h6v" Jul 26 15:55:54.826758 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:54.826731 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58e6b0af-30d9-49c6-9e2b-50d8c0b33850-tls-certs\") pod \"rate-test-model-a20d61-kserve-66b8b5b5c9-n8h6v\" (UID: \"58e6b0af-30d9-49c6-9e2b-50d8c0b33850\") " pod="ai-tenant-e2e-shared-a-a20d61/rate-test-model-a20d61-kserve-66b8b5b5c9-n8h6v" Jul 26 15:55:54.826863 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:54.826761 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58e6b0af-30d9-49c6-9e2b-50d8c0b33850-dshm\") pod \"rate-test-model-a20d61-kserve-66b8b5b5c9-n8h6v\" (UID: \"58e6b0af-30d9-49c6-9e2b-50d8c0b33850\") " pod="ai-tenant-e2e-shared-a-a20d61/rate-test-model-a20d61-kserve-66b8b5b5c9-n8h6v" Jul 26 15:55:54.826863 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:54.826793 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdng5\" (UniqueName: \"kubernetes.io/projected/58e6b0af-30d9-49c6-9e2b-50d8c0b33850-kube-api-access-zdng5\") pod \"rate-test-model-a20d61-kserve-66b8b5b5c9-n8h6v\" (UID: \"58e6b0af-30d9-49c6-9e2b-50d8c0b33850\") " pod="ai-tenant-e2e-shared-a-a20d61/rate-test-model-a20d61-kserve-66b8b5b5c9-n8h6v" Jul 26 15:55:54.826863 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:54.826821 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58e6b0af-30d9-49c6-9e2b-50d8c0b33850-home\") pod \"rate-test-model-a20d61-kserve-66b8b5b5c9-n8h6v\" (UID: \"58e6b0af-30d9-49c6-9e2b-50d8c0b33850\") " pod="ai-tenant-e2e-shared-a-a20d61/rate-test-model-a20d61-kserve-66b8b5b5c9-n8h6v" Jul 26 15:55:54.826863 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:54.826839 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58e6b0af-30d9-49c6-9e2b-50d8c0b33850-tmp-dir\") pod \"rate-test-model-a20d61-kserve-66b8b5b5c9-n8h6v\" (UID: \"58e6b0af-30d9-49c6-9e2b-50d8c0b33850\") " pod="ai-tenant-e2e-shared-a-a20d61/rate-test-model-a20d61-kserve-66b8b5b5c9-n8h6v" Jul 26 15:55:54.826863 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:54.826855 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58e6b0af-30d9-49c6-9e2b-50d8c0b33850-model-cache\") pod \"rate-test-model-a20d61-kserve-66b8b5b5c9-n8h6v\" (UID: \"58e6b0af-30d9-49c6-9e2b-50d8c0b33850\") " pod="ai-tenant-e2e-shared-a-a20d61/rate-test-model-a20d61-kserve-66b8b5b5c9-n8h6v" Jul 26 15:55:54.827298 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:54.827272 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58e6b0af-30d9-49c6-9e2b-50d8c0b33850-model-cache\") pod \"rate-test-model-a20d61-kserve-66b8b5b5c9-n8h6v\" (UID: \"58e6b0af-30d9-49c6-9e2b-50d8c0b33850\") " pod="ai-tenant-e2e-shared-a-a20d61/rate-test-model-a20d61-kserve-66b8b5b5c9-n8h6v" Jul 26 15:55:54.827361 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:54.827315 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58e6b0af-30d9-49c6-9e2b-50d8c0b33850-home\") pod \"rate-test-model-a20d61-kserve-66b8b5b5c9-n8h6v\" (UID: \"58e6b0af-30d9-49c6-9e2b-50d8c0b33850\") " pod="ai-tenant-e2e-shared-a-a20d61/rate-test-model-a20d61-kserve-66b8b5b5c9-n8h6v" Jul 26 15:55:54.827361 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:54.827331 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58e6b0af-30d9-49c6-9e2b-50d8c0b33850-tmp-dir\") pod \"rate-test-model-a20d61-kserve-66b8b5b5c9-n8h6v\" (UID: \"58e6b0af-30d9-49c6-9e2b-50d8c0b33850\") " pod="ai-tenant-e2e-shared-a-a20d61/rate-test-model-a20d61-kserve-66b8b5b5c9-n8h6v" Jul 26 15:55:54.829054 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:54.829029 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58e6b0af-30d9-49c6-9e2b-50d8c0b33850-dshm\") pod \"rate-test-model-a20d61-kserve-66b8b5b5c9-n8h6v\" (UID: \"58e6b0af-30d9-49c6-9e2b-50d8c0b33850\") " pod="ai-tenant-e2e-shared-a-a20d61/rate-test-model-a20d61-kserve-66b8b5b5c9-n8h6v" Jul 26 15:55:54.829219 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:54.829203 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58e6b0af-30d9-49c6-9e2b-50d8c0b33850-tls-certs\") pod \"rate-test-model-a20d61-kserve-66b8b5b5c9-n8h6v\" (UID: \"58e6b0af-30d9-49c6-9e2b-50d8c0b33850\") " pod="ai-tenant-e2e-shared-a-a20d61/rate-test-model-a20d61-kserve-66b8b5b5c9-n8h6v" Jul 26 15:55:54.833479 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:54.833460 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdng5\" (UniqueName: \"kubernetes.io/projected/58e6b0af-30d9-49c6-9e2b-50d8c0b33850-kube-api-access-zdng5\") pod \"rate-test-model-a20d61-kserve-66b8b5b5c9-n8h6v\" (UID: \"58e6b0af-30d9-49c6-9e2b-50d8c0b33850\") " pod="ai-tenant-e2e-shared-a-a20d61/rate-test-model-a20d61-kserve-66b8b5b5c9-n8h6v" Jul 26 15:55:54.933576 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:54.933524 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a20d61/rate-test-model-a20d61-kserve-66b8b5b5c9-n8h6v" Jul 26 15:55:55.057621 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:55.057594 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a20d61/rate-test-model-a20d61-kserve-66b8b5b5c9-n8h6v"] Jul 26 15:55:55.060245 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:55:55.060178 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod58e6b0af_30d9_49c6_9e2b_50d8c0b33850.slice/crio-054382bb267f51a5c3ab5151b3283652400d39d92f39677b67030a3966dec3a2 WatchSource:0}: Error finding container 054382bb267f51a5c3ab5151b3283652400d39d92f39677b67030a3966dec3a2: Status 404 returned error can't find the container with id 054382bb267f51a5c3ab5151b3283652400d39d92f39677b67030a3966dec3a2 Jul 26 15:55:55.491170 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:55.491138 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a20d61/rate-test-model-a20d61-kserve-66b8b5b5c9-n8h6v" event={"ID":"58e6b0af-30d9-49c6-9e2b-50d8c0b33850","Type":"ContainerStarted","Data":"b4dc56443179870e940c96442ca81355edd28211fb819755f0798b2096adae93"} Jul 26 15:55:55.491330 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:55.491183 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a20d61/rate-test-model-a20d61-kserve-66b8b5b5c9-n8h6v" event={"ID":"58e6b0af-30d9-49c6-9e2b-50d8c0b33850","Type":"ContainerStarted","Data":"054382bb267f51a5c3ab5151b3283652400d39d92f39677b67030a3966dec3a2"} Jul 26 15:55:55.508262 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:55:55.508212 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-a20d61/rate-test-model-a20d61-kserve-66b8b5b5c9-n8h6v" podStartSLOduration=1.508199385 podStartE2EDuration="1.508199385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:55:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:55:55.505986313 +0000 UTC m=+3035.929833194" watchObservedRunningTime="2026-07-26 15:55:55.508199385 +0000 UTC m=+3035.932046255" Jul 26 15:56:04.934519 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:56:04.934470 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-a20d61/rate-test-model-a20d61-kserve-66b8b5b5c9-n8h6v" Jul 26 15:56:04.934999 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:56:04.934538 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-a20d61/rate-test-model-a20d61-kserve-66b8b5b5c9-n8h6v" Jul 26 15:56:04.946700 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:56:04.946676 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-a20d61/rate-test-model-a20d61-kserve-66b8b5b5c9-n8h6v" Jul 26 15:56:05.549501 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:56:05.549470 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-a20d61/rate-test-model-a20d61-kserve-66b8b5b5c9-n8h6v" Jul 26 15:56:16.053263 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:56:16.053177 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:57:31.075009 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:31.074977 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a20d61/rate-test-model-a20d61-kserve-66b8b5b5c9-n8h6v"] Jul 26 15:57:31.100563 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:31.100536 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:57:32.038158 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:32.038126 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:57:32.939123 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:32.939095 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7535-140214310334000"] Jul 26 15:57:32.942237 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:32.942219 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7535-140214310334000" Jul 26 15:57:32.944409 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:32.944390 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-fxpw2\"" Jul 26 15:57:32.948310 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:32.948275 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7535-140214310334000"] Jul 26 15:57:33.003407 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:33.003382 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbgk5\" (UniqueName: \"kubernetes.io/projected/fc166934-7ebe-4bd3-a0bc-353f928a2cb3-kube-api-access-sbgk5\") pod \"test-curl-7535-140214310334000\" (UID: \"fc166934-7ebe-4bd3-a0bc-353f928a2cb3\") " pod="opendatahub/test-curl-7535-140214310334000" Jul 26 15:57:33.104455 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:33.104427 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbgk5\" (UniqueName: \"kubernetes.io/projected/fc166934-7ebe-4bd3-a0bc-353f928a2cb3-kube-api-access-sbgk5\") pod \"test-curl-7535-140214310334000\" (UID: \"fc166934-7ebe-4bd3-a0bc-353f928a2cb3\") " pod="opendatahub/test-curl-7535-140214310334000" Jul 26 15:57:33.112224 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:33.112196 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbgk5\" (UniqueName: \"kubernetes.io/projected/fc166934-7ebe-4bd3-a0bc-353f928a2cb3-kube-api-access-sbgk5\") pod \"test-curl-7535-140214310334000\" (UID: \"fc166934-7ebe-4bd3-a0bc-353f928a2cb3\") " pod="opendatahub/test-curl-7535-140214310334000" Jul 26 15:57:33.252495 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:33.252421 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7535-140214310334000" Jul 26 15:57:33.375013 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:33.374980 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7535-140214310334000"] Jul 26 15:57:33.379011 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:57:33.378985 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfc166934_7ebe_4bd3_a0bc_353f928a2cb3.slice/crio-fd2167d9759ef4f7b4d9c83888a280b085f569e3b4533e97f3fa9c594624abf5 WatchSource:0}: Error finding container fd2167d9759ef4f7b4d9c83888a280b085f569e3b4533e97f3fa9c594624abf5: Status 404 returned error can't find the container with id fd2167d9759ef4f7b4d9c83888a280b085f569e3b4533e97f3fa9c594624abf5 Jul 26 15:57:33.380607 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:33.380587 2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 26 15:57:33.939491 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:33.939454 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7535-140214310334000" event={"ID":"fc166934-7ebe-4bd3-a0bc-353f928a2cb3","Type":"ContainerStarted","Data":"fd2167d9759ef4f7b4d9c83888a280b085f569e3b4533e97f3fa9c594624abf5"} Jul 26 15:57:34.946335 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:34.946306 2555 generic.go:356] "Generic (PLEG): container finished" podID="fc166934-7ebe-4bd3-a0bc-353f928a2cb3" containerID="c44ed4f71ae5eabf7fc5cdb38f4e7c475b51a3cef97e219e71402cc3bfcacee2" exitCode=0 Jul 26 15:57:34.946704 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:34.946356 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7535-140214310334000" event={"ID":"fc166934-7ebe-4bd3-a0bc-353f928a2cb3","Type":"ContainerDied","Data":"c44ed4f71ae5eabf7fc5cdb38f4e7c475b51a3cef97e219e71402cc3bfcacee2"} Jul 26 15:57:36.077640 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:36.077616 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7535-140214310334000" Jul 26 15:57:36.100491 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:36.100460 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7535-140214310334000_fc166934-7ebe-4bd3-a0bc-353f928a2cb3/test-curl-7535-140214310334000/0.log" Jul 26 15:57:36.124997 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:36.124971 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7535-140214310334000"] Jul 26 15:57:36.128386 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:36.128342 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sbgk5\" (UniqueName: \"kubernetes.io/projected/fc166934-7ebe-4bd3-a0bc-353f928a2cb3-kube-api-access-sbgk5\") pod \"fc166934-7ebe-4bd3-a0bc-353f928a2cb3\" (UID: \"fc166934-7ebe-4bd3-a0bc-353f928a2cb3\") " Jul 26 15:57:36.130177 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:36.130156 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7535-140214310334000"] Jul 26 15:57:36.130369 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:36.130346 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc166934-7ebe-4bd3-a0bc-353f928a2cb3-kube-api-access-sbgk5" (OuterVolumeSpecName: "kube-api-access-sbgk5") pod "fc166934-7ebe-4bd3-a0bc-353f928a2cb3" (UID: "fc166934-7ebe-4bd3-a0bc-353f928a2cb3"). InnerVolumeSpecName "kube-api-access-sbgk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:57:36.229171 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:36.229116 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbgk5\" (UniqueName: \"kubernetes.io/projected/fc166934-7ebe-4bd3-a0bc-353f928a2cb3-kube-api-access-sbgk5\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:57:36.244115 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:36.244092 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7535-140214310333616"] Jul 26 15:57:36.244431 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:36.244418 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc166934-7ebe-4bd3-a0bc-353f928a2cb3" containerName="test-curl-7535-140214310334000" Jul 26 15:57:36.244476 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:36.244432 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc166934-7ebe-4bd3-a0bc-353f928a2cb3" containerName="test-curl-7535-140214310334000" Jul 26 15:57:36.244509 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:36.244504 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc166934-7ebe-4bd3-a0bc-353f928a2cb3" containerName="test-curl-7535-140214310334000" Jul 26 15:57:36.247485 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:36.247469 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7535-140214310333616" Jul 26 15:57:36.252633 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:36.252611 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7535-140214310333616"] Jul 26 15:57:36.329779 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:36.329757 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpjv4\" (UniqueName: \"kubernetes.io/projected/17b220e6-636b-4e70-aa7a-0bd0328d948a-kube-api-access-wpjv4\") pod \"test-curl-7535-140214310333616\" (UID: \"17b220e6-636b-4e70-aa7a-0bd0328d948a\") " pod="opendatahub/test-curl-7535-140214310333616" Jul 26 15:57:36.430305 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:36.430280 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpjv4\" (UniqueName: \"kubernetes.io/projected/17b220e6-636b-4e70-aa7a-0bd0328d948a-kube-api-access-wpjv4\") pod \"test-curl-7535-140214310333616\" (UID: \"17b220e6-636b-4e70-aa7a-0bd0328d948a\") " pod="opendatahub/test-curl-7535-140214310333616" Jul 26 15:57:36.451661 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:36.451635 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpjv4\" (UniqueName: \"kubernetes.io/projected/17b220e6-636b-4e70-aa7a-0bd0328d948a-kube-api-access-wpjv4\") pod \"test-curl-7535-140214310333616\" (UID: \"17b220e6-636b-4e70-aa7a-0bd0328d948a\") " pod="opendatahub/test-curl-7535-140214310333616" Jul 26 15:57:36.558024 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:36.557960 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7535-140214310333616" Jul 26 15:57:36.680067 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:36.680043 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7535-140214310333616"] Jul 26 15:57:36.682104 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:57:36.682080 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod17b220e6_636b_4e70_aa7a_0bd0328d948a.slice/crio-196e759786528de6067369c21a506fdc4f9d8b8a363d3c2826a1c7b3b88d0ca3 WatchSource:0}: Error finding container 196e759786528de6067369c21a506fdc4f9d8b8a363d3c2826a1c7b3b88d0ca3: Status 404 returned error can't find the container with id 196e759786528de6067369c21a506fdc4f9d8b8a363d3c2826a1c7b3b88d0ca3 Jul 26 15:57:36.956873 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:36.956845 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7535-140214310333616" event={"ID":"17b220e6-636b-4e70-aa7a-0bd0328d948a","Type":"ContainerStarted","Data":"196e759786528de6067369c21a506fdc4f9d8b8a363d3c2826a1c7b3b88d0ca3"} Jul 26 15:57:36.958087 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:36.958070 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd2167d9759ef4f7b4d9c83888a280b085f569e3b4533e97f3fa9c594624abf5" Jul 26 15:57:36.958183 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:36.958107 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7535-140214310334000" Jul 26 15:57:36.968383 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:57:36.968354 2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7535-140214310334000\" is forbidden: User \"system:node:ip-10-0-130-21.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-21.ec2.internal' and this object" podUID="fc166934-7ebe-4bd3-a0bc-353f928a2cb3" pod="opendatahub/test-curl-7535-140214310334000" Jul 26 15:57:37.964453 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:37.964415 2555 generic.go:356] "Generic (PLEG): container finished" podID="17b220e6-636b-4e70-aa7a-0bd0328d948a" containerID="db4651dcc4577fa0373db9783b924827fd4b7d77f3cc2f0153b895a199d2a8a6" exitCode=0 Jul 26 15:57:37.964823 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:37.964508 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7535-140214310333616" event={"ID":"17b220e6-636b-4e70-aa7a-0bd0328d948a","Type":"ContainerDied","Data":"db4651dcc4577fa0373db9783b924827fd4b7d77f3cc2f0153b895a199d2a8a6"} Jul 26 15:57:37.966260 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:57:37.966237 2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7535-140214310334000\" is forbidden: User \"system:node:ip-10-0-130-21.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-21.ec2.internal' and this object" podUID="fc166934-7ebe-4bd3-a0bc-353f928a2cb3" pod="opendatahub/test-curl-7535-140214310334000" Jul 26 15:57:38.063560 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:38.063529 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc166934-7ebe-4bd3-a0bc-353f928a2cb3" path="/var/lib/kubelet/pods/fc166934-7ebe-4bd3-a0bc-353f928a2cb3/volumes" Jul 26 15:57:39.090797 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:39.090777 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7535-140214310333616" Jul 26 15:57:39.111625 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:39.111605 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7535-140214310333616_17b220e6-636b-4e70-aa7a-0bd0328d948a/test-curl-7535-140214310333616/0.log" Jul 26 15:57:39.132633 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:39.132606 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7535-140214310333616"] Jul 26 15:57:39.136212 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:39.136189 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7535-140214310333616"] Jul 26 15:57:39.149146 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:39.149130 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wpjv4\" (UniqueName: \"kubernetes.io/projected/17b220e6-636b-4e70-aa7a-0bd0328d948a-kube-api-access-wpjv4\") pod \"17b220e6-636b-4e70-aa7a-0bd0328d948a\" (UID: \"17b220e6-636b-4e70-aa7a-0bd0328d948a\") " Jul 26 15:57:39.151008 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:39.150987 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17b220e6-636b-4e70-aa7a-0bd0328d948a-kube-api-access-wpjv4" (OuterVolumeSpecName: "kube-api-access-wpjv4") pod "17b220e6-636b-4e70-aa7a-0bd0328d948a" (UID: "17b220e6-636b-4e70-aa7a-0bd0328d948a"). InnerVolumeSpecName "kube-api-access-wpjv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:57:39.250613 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:39.250557 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpjv4\" (UniqueName: \"kubernetes.io/projected/17b220e6-636b-4e70-aa7a-0bd0328d948a-kube-api-access-wpjv4\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:57:39.974640 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:39.974612 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7535-140214310333616" Jul 26 15:57:39.974794 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:39.974649 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="196e759786528de6067369c21a506fdc4f9d8b8a363d3c2826a1c7b3b88d0ca3" Jul 26 15:57:39.983667 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:57:39.983642 2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7535-140214310333616\" is forbidden: User \"system:node:ip-10-0-130-21.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-21.ec2.internal' and this object" podUID="17b220e6-636b-4e70-aa7a-0bd0328d948a" pod="opendatahub/test-curl-7535-140214310333616" Jul 26 15:57:40.061660 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:57:40.061629 2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7535-140214310333616\" is forbidden: User \"system:node:ip-10-0-130-21.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-21.ec2.internal' and this object" podUID="17b220e6-636b-4e70-aa7a-0bd0328d948a" pod="opendatahub/test-curl-7535-140214310333616" Jul 26 15:57:40.063452 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:40.063430 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17b220e6-636b-4e70-aa7a-0bd0328d948a" path="/var/lib/kubelet/pods/17b220e6-636b-4e70-aa7a-0bd0328d948a/volumes" Jul 26 15:57:45.734440 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:45.734362 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7535-140214310254176"] Jul 26 15:57:45.734786 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:45.734710 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17b220e6-636b-4e70-aa7a-0bd0328d948a" containerName="test-curl-7535-140214310333616" Jul 26 15:57:45.734786 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:45.734721 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="17b220e6-636b-4e70-aa7a-0bd0328d948a" containerName="test-curl-7535-140214310333616" Jul 26 15:57:45.734786 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:45.734786 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="17b220e6-636b-4e70-aa7a-0bd0328d948a" containerName="test-curl-7535-140214310333616" Jul 26 15:57:45.737822 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:45.737805 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7535-140214310254176" Jul 26 15:57:45.739755 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:45.739735 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-fxpw2\"" Jul 26 15:57:45.745353 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:45.745328 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7535-140214310254176"] Jul 26 15:57:45.800341 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:45.800309 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4xtj\" (UniqueName: \"kubernetes.io/projected/4df5cebc-e6c6-4a80-a541-7c1dc8121509-kube-api-access-v4xtj\") pod \"test-curl-7535-140214310254176\" (UID: \"4df5cebc-e6c6-4a80-a541-7c1dc8121509\") " pod="opendatahub/test-curl-7535-140214310254176" Jul 26 15:57:45.901677 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:45.901643 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4xtj\" (UniqueName: \"kubernetes.io/projected/4df5cebc-e6c6-4a80-a541-7c1dc8121509-kube-api-access-v4xtj\") pod \"test-curl-7535-140214310254176\" (UID: \"4df5cebc-e6c6-4a80-a541-7c1dc8121509\") " pod="opendatahub/test-curl-7535-140214310254176" Jul 26 15:57:45.909015 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:45.908991 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4xtj\" (UniqueName: \"kubernetes.io/projected/4df5cebc-e6c6-4a80-a541-7c1dc8121509-kube-api-access-v4xtj\") pod \"test-curl-7535-140214310254176\" (UID: \"4df5cebc-e6c6-4a80-a541-7c1dc8121509\") " pod="opendatahub/test-curl-7535-140214310254176" Jul 26 15:57:46.047880 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:46.047804 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7535-140214310254176" Jul 26 15:57:46.092968 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:46.092901 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-a20d61/rate-test-model-a20d61-kserve-66b8b5b5c9-n8h6v" podUID="58e6b0af-30d9-49c6-9e2b-50d8c0b33850" containerName="main" containerID="cri-o://b4dc56443179870e940c96442ca81355edd28211fb819755f0798b2096adae93" gracePeriod=45 Jul 26 15:57:46.172845 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:46.172821 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7535-140214310254176"] Jul 26 15:57:46.175305 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:57:46.175265 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4df5cebc_e6c6_4a80_a541_7c1dc8121509.slice/crio-e1a665e6da8b45dc32c78a5872e7fc4a21e6cc1cbdcde67d3fbc47528c613ae8 WatchSource:0}: Error finding container e1a665e6da8b45dc32c78a5872e7fc4a21e6cc1cbdcde67d3fbc47528c613ae8: Status 404 returned error can't find the container with id e1a665e6da8b45dc32c78a5872e7fc4a21e6cc1cbdcde67d3fbc47528c613ae8 Jul 26 15:57:46.336716 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:46.336692 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a20d61/rate-test-model-a20d61-kserve-66b8b5b5c9-n8h6v" Jul 26 15:57:46.405710 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:46.405676 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58e6b0af-30d9-49c6-9e2b-50d8c0b33850-dshm\") pod \"58e6b0af-30d9-49c6-9e2b-50d8c0b33850\" (UID: \"58e6b0af-30d9-49c6-9e2b-50d8c0b33850\") " Jul 26 15:57:46.405710 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:46.405714 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zdng5\" (UniqueName: \"kubernetes.io/projected/58e6b0af-30d9-49c6-9e2b-50d8c0b33850-kube-api-access-zdng5\") pod \"58e6b0af-30d9-49c6-9e2b-50d8c0b33850\" (UID: \"58e6b0af-30d9-49c6-9e2b-50d8c0b33850\") " Jul 26 15:57:46.405877 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:46.405730 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58e6b0af-30d9-49c6-9e2b-50d8c0b33850-home\") pod \"58e6b0af-30d9-49c6-9e2b-50d8c0b33850\" (UID: \"58e6b0af-30d9-49c6-9e2b-50d8c0b33850\") " Jul 26 15:57:46.405877 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:46.405771 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58e6b0af-30d9-49c6-9e2b-50d8c0b33850-tls-certs\") pod \"58e6b0af-30d9-49c6-9e2b-50d8c0b33850\" (UID: \"58e6b0af-30d9-49c6-9e2b-50d8c0b33850\") " Jul 26 15:57:46.405877 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:46.405865 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58e6b0af-30d9-49c6-9e2b-50d8c0b33850-tmp-dir\") pod \"58e6b0af-30d9-49c6-9e2b-50d8c0b33850\" (UID: \"58e6b0af-30d9-49c6-9e2b-50d8c0b33850\") " Jul 26 15:57:46.406064 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:46.405879 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58e6b0af-30d9-49c6-9e2b-50d8c0b33850-model-cache\") pod \"58e6b0af-30d9-49c6-9e2b-50d8c0b33850\" (UID: \"58e6b0af-30d9-49c6-9e2b-50d8c0b33850\") " Jul 26 15:57:46.406064 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:46.406042 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58e6b0af-30d9-49c6-9e2b-50d8c0b33850-home" (OuterVolumeSpecName: "home") pod "58e6b0af-30d9-49c6-9e2b-50d8c0b33850" (UID: "58e6b0af-30d9-49c6-9e2b-50d8c0b33850"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:57:46.406170 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:46.406139 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58e6b0af-30d9-49c6-9e2b-50d8c0b33850-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "58e6b0af-30d9-49c6-9e2b-50d8c0b33850" (UID: "58e6b0af-30d9-49c6-9e2b-50d8c0b33850"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:57:46.406227 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:46.406184 2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58e6b0af-30d9-49c6-9e2b-50d8c0b33850-home\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:57:46.406274 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:46.406220 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58e6b0af-30d9-49c6-9e2b-50d8c0b33850-model-cache" (OuterVolumeSpecName: "model-cache") pod "58e6b0af-30d9-49c6-9e2b-50d8c0b33850" (UID: "58e6b0af-30d9-49c6-9e2b-50d8c0b33850"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:57:46.407786 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:46.407767 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58e6b0af-30d9-49c6-9e2b-50d8c0b33850-dshm" (OuterVolumeSpecName: "dshm") pod "58e6b0af-30d9-49c6-9e2b-50d8c0b33850" (UID: "58e6b0af-30d9-49c6-9e2b-50d8c0b33850"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:57:46.407883 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:46.407819 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58e6b0af-30d9-49c6-9e2b-50d8c0b33850-kube-api-access-zdng5" (OuterVolumeSpecName: "kube-api-access-zdng5") pod "58e6b0af-30d9-49c6-9e2b-50d8c0b33850" (UID: "58e6b0af-30d9-49c6-9e2b-50d8c0b33850"). InnerVolumeSpecName "kube-api-access-zdng5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:57:46.407883 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:46.407862 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58e6b0af-30d9-49c6-9e2b-50d8c0b33850-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "58e6b0af-30d9-49c6-9e2b-50d8c0b33850" (UID: "58e6b0af-30d9-49c6-9e2b-50d8c0b33850"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 15:57:46.506732 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:46.506712 2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58e6b0af-30d9-49c6-9e2b-50d8c0b33850-tmp-dir\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:57:46.506806 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:46.506733 2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58e6b0af-30d9-49c6-9e2b-50d8c0b33850-model-cache\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:57:46.506806 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:46.506742 2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58e6b0af-30d9-49c6-9e2b-50d8c0b33850-dshm\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:57:46.506806 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:46.506752 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdng5\" (UniqueName: \"kubernetes.io/projected/58e6b0af-30d9-49c6-9e2b-50d8c0b33850-kube-api-access-zdng5\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:57:46.506806 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:46.506760 2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58e6b0af-30d9-49c6-9e2b-50d8c0b33850-tls-certs\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:57:47.013562 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:47.013531 2555 generic.go:356] "Generic (PLEG): container finished" podID="4df5cebc-e6c6-4a80-a541-7c1dc8121509" containerID="a7a7e7221d4842f1c14b7b37b40277febc66026d881a4d9c8f3f1f10fb3ea129" exitCode=0 Jul 26 15:57:47.013958 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:47.013579 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7535-140214310254176" event={"ID":"4df5cebc-e6c6-4a80-a541-7c1dc8121509","Type":"ContainerDied","Data":"a7a7e7221d4842f1c14b7b37b40277febc66026d881a4d9c8f3f1f10fb3ea129"} Jul 26 15:57:47.013958 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:47.013624 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7535-140214310254176" event={"ID":"4df5cebc-e6c6-4a80-a541-7c1dc8121509","Type":"ContainerStarted","Data":"e1a665e6da8b45dc32c78a5872e7fc4a21e6cc1cbdcde67d3fbc47528c613ae8"} Jul 26 15:57:47.014953 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:47.014918 2555 generic.go:356] "Generic (PLEG): container finished" podID="58e6b0af-30d9-49c6-9e2b-50d8c0b33850" containerID="b4dc56443179870e940c96442ca81355edd28211fb819755f0798b2096adae93" exitCode=0 Jul 26 15:57:47.015017 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:47.014983 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a20d61/rate-test-model-a20d61-kserve-66b8b5b5c9-n8h6v" Jul 26 15:57:47.015071 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:47.014981 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a20d61/rate-test-model-a20d61-kserve-66b8b5b5c9-n8h6v" event={"ID":"58e6b0af-30d9-49c6-9e2b-50d8c0b33850","Type":"ContainerDied","Data":"b4dc56443179870e940c96442ca81355edd28211fb819755f0798b2096adae93"} Jul 26 15:57:47.015117 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:47.015087 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a20d61/rate-test-model-a20d61-kserve-66b8b5b5c9-n8h6v" event={"ID":"58e6b0af-30d9-49c6-9e2b-50d8c0b33850","Type":"ContainerDied","Data":"054382bb267f51a5c3ab5151b3283652400d39d92f39677b67030a3966dec3a2"} Jul 26 15:57:47.015117 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:47.015103 2555 scope.go:117] "RemoveContainer" containerID="b4dc56443179870e940c96442ca81355edd28211fb819755f0798b2096adae93" Jul 26 15:57:47.023658 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:47.023640 2555 scope.go:117] "RemoveContainer" containerID="b4dc56443179870e940c96442ca81355edd28211fb819755f0798b2096adae93" Jul 26 15:57:47.023921 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:57:47.023901 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4dc56443179870e940c96442ca81355edd28211fb819755f0798b2096adae93\": container with ID starting with b4dc56443179870e940c96442ca81355edd28211fb819755f0798b2096adae93 not found: ID does not exist" containerID="b4dc56443179870e940c96442ca81355edd28211fb819755f0798b2096adae93" Jul 26 15:57:47.023999 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:47.023944 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4dc56443179870e940c96442ca81355edd28211fb819755f0798b2096adae93"} err="failed to get container status \"b4dc56443179870e940c96442ca81355edd28211fb819755f0798b2096adae93\": rpc error: code = NotFound desc = could not find container \"b4dc56443179870e940c96442ca81355edd28211fb819755f0798b2096adae93\": container with ID starting with b4dc56443179870e940c96442ca81355edd28211fb819755f0798b2096adae93 not found: ID does not exist" Jul 26 15:57:47.042633 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:47.042610 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a20d61/rate-test-model-a20d61-kserve-66b8b5b5c9-n8h6v"] Jul 26 15:57:47.048399 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:47.048381 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-a20d61/rate-test-model-a20d61-kserve-66b8b5b5c9-n8h6v"] Jul 26 15:57:48.064004 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:48.063973 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58e6b0af-30d9-49c6-9e2b-50d8c0b33850" path="/var/lib/kubelet/pods/58e6b0af-30d9-49c6-9e2b-50d8c0b33850/volumes" Jul 26 15:57:48.154870 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:48.154845 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7535-140214310254176" Jul 26 15:57:48.172149 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:48.172127 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7535-140214310254176_4df5cebc-e6c6-4a80-a541-7c1dc8121509/test-curl-7535-140214310254176/0.log" Jul 26 15:57:48.194359 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:48.194330 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7535-140214310254176"] Jul 26 15:57:48.199610 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:48.199585 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7535-140214310254176"] Jul 26 15:57:48.223106 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:48.223083 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v4xtj\" (UniqueName: \"kubernetes.io/projected/4df5cebc-e6c6-4a80-a541-7c1dc8121509-kube-api-access-v4xtj\") pod \"4df5cebc-e6c6-4a80-a541-7c1dc8121509\" (UID: \"4df5cebc-e6c6-4a80-a541-7c1dc8121509\") " Jul 26 15:57:48.225081 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:48.225061 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4df5cebc-e6c6-4a80-a541-7c1dc8121509-kube-api-access-v4xtj" (OuterVolumeSpecName: "kube-api-access-v4xtj") pod "4df5cebc-e6c6-4a80-a541-7c1dc8121509" (UID: "4df5cebc-e6c6-4a80-a541-7c1dc8121509"). InnerVolumeSpecName "kube-api-access-v4xtj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:57:48.324677 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:48.324607 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4xtj\" (UniqueName: \"kubernetes.io/projected/4df5cebc-e6c6-4a80-a541-7c1dc8121509-kube-api-access-v4xtj\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:57:49.028692 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:49.028662 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7535-140214310254176" Jul 26 15:57:49.028692 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:49.028677 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1a665e6da8b45dc32c78a5872e7fc4a21e6cc1cbdcde67d3fbc47528c613ae8" Jul 26 15:57:49.037964 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:57:49.037924 2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7535-140214310254176\" is forbidden: User \"system:node:ip-10-0-130-21.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-21.ec2.internal' and this object" podUID="4df5cebc-e6c6-4a80-a541-7c1dc8121509" pod="opendatahub/test-curl-7535-140214310254176" Jul 26 15:57:50.061667 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:57:50.061632 2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7535-140214310254176\" is forbidden: User \"system:node:ip-10-0-130-21.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-21.ec2.internal' and this object" podUID="4df5cebc-e6c6-4a80-a541-7c1dc8121509" pod="opendatahub/test-curl-7535-140214310254176" Jul 26 15:57:50.063783 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:50.063757 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4df5cebc-e6c6-4a80-a541-7c1dc8121509" path="/var/lib/kubelet/pods/4df5cebc-e6c6-4a80-a541-7c1dc8121509/volumes" Jul 26 15:57:51.163754 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:51.163718 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:57:52.208818 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:52.208783 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:57:54.411888 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:54.411848 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:57:54.519736 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:54.519705 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:57:54.693913 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:54.693828 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:57:54.779467 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:54.779413 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:57:55.117757 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:55.117727 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:57:55.217541 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:55.217512 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:57:58.912855 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:57:58.912822 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:58:09.813121 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:09.813087 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:58:10.111718 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:10.111630 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:58:10.371883 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:10.371843 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:58:13.570384 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:13.570332 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:58:13.636267 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:13.636233 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:58:13.640487 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:13.640462 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-cb6b99-64cb4746c-29zb6"] Jul 26 15:58:13.640837 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:13.640824 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58e6b0af-30d9-49c6-9e2b-50d8c0b33850" containerName="main" Jul 26 15:58:13.640878 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:13.640838 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="58e6b0af-30d9-49c6-9e2b-50d8c0b33850" containerName="main" Jul 26 15:58:13.640878 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:13.640854 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4df5cebc-e6c6-4a80-a541-7c1dc8121509" containerName="test-curl-7535-140214310254176" Jul 26 15:58:13.640878 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:13.640860 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4df5cebc-e6c6-4a80-a541-7c1dc8121509" containerName="test-curl-7535-140214310254176" Jul 26 15:58:13.641009 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:13.640995 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4df5cebc-e6c6-4a80-a541-7c1dc8121509" containerName="test-curl-7535-140214310254176" Jul 26 15:58:13.641044 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:13.641017 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="58e6b0af-30d9-49c6-9e2b-50d8c0b33850" containerName="main" Jul 26 15:58:13.646000 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:13.645982 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cb6b99-64cb4746c-29zb6" Jul 26 15:58:13.649290 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:13.649270 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-cb6b99-serving-cert\"" Jul 26 15:58:13.650297 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:13.650275 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 26 15:58:13.650413 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:13.650352 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nnwcs\"" Jul 26 15:58:13.651544 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:13.651525 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 26 15:58:13.653737 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:13.653709 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-cb6b99-64cb4746c-29zb6"] Jul 26 15:58:13.719609 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:13.719578 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:58:13.721664 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:13.721641 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97300bf8-0063-4987-b27b-a915287153a2-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-cb6b99-64cb4746c-29zb6\" (UID: \"97300bf8-0063-4987-b27b-a915287153a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cb6b99-64cb4746c-29zb6" Jul 26 15:58:13.721768 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:13.721701 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkxbx\" (UniqueName: \"kubernetes.io/projected/97300bf8-0063-4987-b27b-a915287153a2-kube-api-access-lkxbx\") pod \"maas-api-e2e-ipp-b-cb6b99-64cb4746c-29zb6\" (UID: \"97300bf8-0063-4987-b27b-a915287153a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cb6b99-64cb4746c-29zb6" Jul 26 15:58:13.721809 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:13.721782 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97300bf8-0063-4987-b27b-a915287153a2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-cb6b99-64cb4746c-29zb6\" (UID: \"97300bf8-0063-4987-b27b-a915287153a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cb6b99-64cb4746c-29zb6" Jul 26 15:58:13.823138 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:13.823074 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkxbx\" (UniqueName: \"kubernetes.io/projected/97300bf8-0063-4987-b27b-a915287153a2-kube-api-access-lkxbx\") pod \"maas-api-e2e-ipp-b-cb6b99-64cb4746c-29zb6\" (UID: \"97300bf8-0063-4987-b27b-a915287153a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cb6b99-64cb4746c-29zb6" Jul 26 15:58:13.823138 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:13.823115 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97300bf8-0063-4987-b27b-a915287153a2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-cb6b99-64cb4746c-29zb6\" (UID: \"97300bf8-0063-4987-b27b-a915287153a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cb6b99-64cb4746c-29zb6" Jul 26 15:58:13.823306 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:13.823187 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97300bf8-0063-4987-b27b-a915287153a2-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-cb6b99-64cb4746c-29zb6\" (UID: \"97300bf8-0063-4987-b27b-a915287153a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cb6b99-64cb4746c-29zb6" Jul 26 15:58:13.823306 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:58:13.823253 2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-cb6b99-serving-cert: secret "maas-api-e2e-ipp-b-cb6b99-serving-cert" not found Jul 26 15:58:13.823417 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:58:13.823321 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97300bf8-0063-4987-b27b-a915287153a2-maas-api-tls podName:97300bf8-0063-4987-b27b-a915287153a2 nodeName:}" failed. No retries permitted until 2026-07-26 15:58:14.323306628 +0000 UTC m=+3174.747153480 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/97300bf8-0063-4987-b27b-a915287153a2-maas-api-tls") pod "maas-api-e2e-ipp-b-cb6b99-64cb4746c-29zb6" (UID: "97300bf8-0063-4987-b27b-a915287153a2") : secret "maas-api-e2e-ipp-b-cb6b99-serving-cert" not found Jul 26 15:58:13.823808 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:13.823787 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97300bf8-0063-4987-b27b-a915287153a2-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-cb6b99-64cb4746c-29zb6\" (UID: \"97300bf8-0063-4987-b27b-a915287153a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cb6b99-64cb4746c-29zb6" Jul 26 15:58:13.831288 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:13.831264 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkxbx\" (UniqueName: \"kubernetes.io/projected/97300bf8-0063-4987-b27b-a915287153a2-kube-api-access-lkxbx\") pod \"maas-api-e2e-ipp-b-cb6b99-64cb4746c-29zb6\" (UID: \"97300bf8-0063-4987-b27b-a915287153a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cb6b99-64cb4746c-29zb6" Jul 26 15:58:14.071744 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:14.071715 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:58:14.327377 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:14.327348 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97300bf8-0063-4987-b27b-a915287153a2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-cb6b99-64cb4746c-29zb6\" (UID: \"97300bf8-0063-4987-b27b-a915287153a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cb6b99-64cb4746c-29zb6" Jul 26 15:58:14.329675 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:14.329648 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97300bf8-0063-4987-b27b-a915287153a2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-cb6b99-64cb4746c-29zb6\" (UID: \"97300bf8-0063-4987-b27b-a915287153a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cb6b99-64cb4746c-29zb6" Jul 26 15:58:14.349470 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:14.349440 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:58:14.491569 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:14.491537 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:58:14.558949 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:14.558900 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cb6b99-64cb4746c-29zb6" Jul 26 15:58:14.898529 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:14.898506 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-cb6b99-64cb4746c-29zb6"] Jul 26 15:58:14.900608 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:58:14.900580 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97300bf8_0063_4987_b27b_a915287153a2.slice/crio-a7162495307e7a54c57ce169bee8bb71b0e7f5fa668d390ad8f00ddaf879da5f WatchSource:0}: Error finding container a7162495307e7a54c57ce169bee8bb71b0e7f5fa668d390ad8f00ddaf879da5f: Status 404 returned error can't find the container with id a7162495307e7a54c57ce169bee8bb71b0e7f5fa668d390ad8f00ddaf879da5f Jul 26 15:58:15.148856 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:15.148783 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cb6b99-64cb4746c-29zb6" event={"ID":"97300bf8-0063-4987-b27b-a915287153a2","Type":"ContainerStarted","Data":"a7162495307e7a54c57ce169bee8bb71b0e7f5fa668d390ad8f00ddaf879da5f"} Jul 26 15:58:17.161713 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:17.161680 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cb6b99-64cb4746c-29zb6" event={"ID":"97300bf8-0063-4987-b27b-a915287153a2","Type":"ContainerStarted","Data":"ebc03874dac4c4fcb472a7bf35bd7f16ecb089508720373ec6335025de83bf71"} Jul 26 15:58:17.162128 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:17.161737 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cb6b99-64cb4746c-29zb6" Jul 26 15:58:17.176774 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:17.176711 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cb6b99-64cb4746c-29zb6" podStartSLOduration=2.722638309 podStartE2EDuration="4.176696537s" totalImagesPullingTime="1.454058228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:58:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:58:17.176070419 +0000 UTC m=+3177.599917291" watchObservedRunningTime="2026-07-26 15:58:17.176696537 +0000 UTC m=+3177.600543408" Jul 26 15:58:18.058699 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:18.058663 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:58:23.170910 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:23.170880 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cb6b99-64cb4746c-29zb6" Jul 26 15:58:52.942260 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:52.942221 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-d57e1d/ipp-route-d57e1d-kserve-67444b75f9-j48ms"] Jul 26 15:58:52.946968 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:52.946951 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-d57e1d/ipp-route-d57e1d-kserve-67444b75f9-j48ms" Jul 26 15:58:52.949197 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:52.949174 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-d57e1d\"/\"default-dockercfg-2nmwx\"" Jul 26 15:58:52.949635 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:52.949610 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-d57e1d\"/\"kube-root-ca.crt\"" Jul 26 15:58:52.949737 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:52.949623 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-d57e1d\"/\"ipp-route-d57e1d-kserve-self-signed-certs\"" Jul 26 15:58:52.949737 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:52.949678 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-d57e1d\"/\"openshift-service-ca.crt\"" Jul 26 15:58:52.954276 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:52.954238 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-d57e1d/ipp-route-d57e1d-kserve-67444b75f9-j48ms"] Jul 26 15:58:53.026057 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:53.026029 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1999071c-78d9-494e-9128-3b72e9c593b8-dshm\") pod \"ipp-route-d57e1d-kserve-67444b75f9-j48ms\" (UID: \"1999071c-78d9-494e-9128-3b72e9c593b8\") " pod="ai-tenant-e2e-ipp-a-d57e1d/ipp-route-d57e1d-kserve-67444b75f9-j48ms" Jul 26 15:58:53.026198 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:53.026083 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1999071c-78d9-494e-9128-3b72e9c593b8-home\") pod \"ipp-route-d57e1d-kserve-67444b75f9-j48ms\" (UID: \"1999071c-78d9-494e-9128-3b72e9c593b8\") " pod="ai-tenant-e2e-ipp-a-d57e1d/ipp-route-d57e1d-kserve-67444b75f9-j48ms" Jul 26 15:58:53.026198 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:53.026160 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnr24\" (UniqueName: \"kubernetes.io/projected/1999071c-78d9-494e-9128-3b72e9c593b8-kube-api-access-gnr24\") pod \"ipp-route-d57e1d-kserve-67444b75f9-j48ms\" (UID: \"1999071c-78d9-494e-9128-3b72e9c593b8\") " pod="ai-tenant-e2e-ipp-a-d57e1d/ipp-route-d57e1d-kserve-67444b75f9-j48ms" Jul 26 15:58:53.026295 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:53.026198 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1999071c-78d9-494e-9128-3b72e9c593b8-tls-certs\") pod \"ipp-route-d57e1d-kserve-67444b75f9-j48ms\" (UID: \"1999071c-78d9-494e-9128-3b72e9c593b8\") " pod="ai-tenant-e2e-ipp-a-d57e1d/ipp-route-d57e1d-kserve-67444b75f9-j48ms" Jul 26 15:58:53.026295 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:53.026277 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1999071c-78d9-494e-9128-3b72e9c593b8-model-cache\") pod \"ipp-route-d57e1d-kserve-67444b75f9-j48ms\" (UID: \"1999071c-78d9-494e-9128-3b72e9c593b8\") " pod="ai-tenant-e2e-ipp-a-d57e1d/ipp-route-d57e1d-kserve-67444b75f9-j48ms" Jul 26 15:58:53.026359 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:53.026316 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1999071c-78d9-494e-9128-3b72e9c593b8-tmp-dir\") pod \"ipp-route-d57e1d-kserve-67444b75f9-j48ms\" (UID: \"1999071c-78d9-494e-9128-3b72e9c593b8\") " pod="ai-tenant-e2e-ipp-a-d57e1d/ipp-route-d57e1d-kserve-67444b75f9-j48ms" Jul 26 15:58:53.038557 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:53.038527 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:58:53.127068 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:53.127039 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1999071c-78d9-494e-9128-3b72e9c593b8-dshm\") pod \"ipp-route-d57e1d-kserve-67444b75f9-j48ms\" (UID: \"1999071c-78d9-494e-9128-3b72e9c593b8\") " pod="ai-tenant-e2e-ipp-a-d57e1d/ipp-route-d57e1d-kserve-67444b75f9-j48ms" Jul 26 15:58:53.127174 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:53.127081 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1999071c-78d9-494e-9128-3b72e9c593b8-home\") pod \"ipp-route-d57e1d-kserve-67444b75f9-j48ms\" (UID: \"1999071c-78d9-494e-9128-3b72e9c593b8\") " pod="ai-tenant-e2e-ipp-a-d57e1d/ipp-route-d57e1d-kserve-67444b75f9-j48ms" Jul 26 15:58:53.127174 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:53.127101 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnr24\" (UniqueName: \"kubernetes.io/projected/1999071c-78d9-494e-9128-3b72e9c593b8-kube-api-access-gnr24\") pod \"ipp-route-d57e1d-kserve-67444b75f9-j48ms\" (UID: \"1999071c-78d9-494e-9128-3b72e9c593b8\") " pod="ai-tenant-e2e-ipp-a-d57e1d/ipp-route-d57e1d-kserve-67444b75f9-j48ms" Jul 26 15:58:53.127174 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:53.127120 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1999071c-78d9-494e-9128-3b72e9c593b8-tls-certs\") pod \"ipp-route-d57e1d-kserve-67444b75f9-j48ms\" (UID: \"1999071c-78d9-494e-9128-3b72e9c593b8\") " pod="ai-tenant-e2e-ipp-a-d57e1d/ipp-route-d57e1d-kserve-67444b75f9-j48ms" Jul 26 15:58:53.127174 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:53.127150 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1999071c-78d9-494e-9128-3b72e9c593b8-model-cache\") pod \"ipp-route-d57e1d-kserve-67444b75f9-j48ms\" (UID: \"1999071c-78d9-494e-9128-3b72e9c593b8\") " pod="ai-tenant-e2e-ipp-a-d57e1d/ipp-route-d57e1d-kserve-67444b75f9-j48ms" Jul 26 15:58:53.127367 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:53.127192 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1999071c-78d9-494e-9128-3b72e9c593b8-tmp-dir\") pod \"ipp-route-d57e1d-kserve-67444b75f9-j48ms\" (UID: \"1999071c-78d9-494e-9128-3b72e9c593b8\") " pod="ai-tenant-e2e-ipp-a-d57e1d/ipp-route-d57e1d-kserve-67444b75f9-j48ms" Jul 26 15:58:53.127524 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:53.127499 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1999071c-78d9-494e-9128-3b72e9c593b8-home\") pod \"ipp-route-d57e1d-kserve-67444b75f9-j48ms\" (UID: \"1999071c-78d9-494e-9128-3b72e9c593b8\") " pod="ai-tenant-e2e-ipp-a-d57e1d/ipp-route-d57e1d-kserve-67444b75f9-j48ms" Jul 26 15:58:53.127622 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:53.127581 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1999071c-78d9-494e-9128-3b72e9c593b8-model-cache\") pod \"ipp-route-d57e1d-kserve-67444b75f9-j48ms\" (UID: \"1999071c-78d9-494e-9128-3b72e9c593b8\") " pod="ai-tenant-e2e-ipp-a-d57e1d/ipp-route-d57e1d-kserve-67444b75f9-j48ms" Jul 26 15:58:53.127664 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:53.127638 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1999071c-78d9-494e-9128-3b72e9c593b8-tmp-dir\") pod \"ipp-route-d57e1d-kserve-67444b75f9-j48ms\" (UID: \"1999071c-78d9-494e-9128-3b72e9c593b8\") " pod="ai-tenant-e2e-ipp-a-d57e1d/ipp-route-d57e1d-kserve-67444b75f9-j48ms" Jul 26 15:58:53.129331 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:53.129316 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1999071c-78d9-494e-9128-3b72e9c593b8-dshm\") pod \"ipp-route-d57e1d-kserve-67444b75f9-j48ms\" (UID: \"1999071c-78d9-494e-9128-3b72e9c593b8\") " pod="ai-tenant-e2e-ipp-a-d57e1d/ipp-route-d57e1d-kserve-67444b75f9-j48ms" Jul 26 15:58:53.129609 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:53.129593 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1999071c-78d9-494e-9128-3b72e9c593b8-tls-certs\") pod \"ipp-route-d57e1d-kserve-67444b75f9-j48ms\" (UID: \"1999071c-78d9-494e-9128-3b72e9c593b8\") " pod="ai-tenant-e2e-ipp-a-d57e1d/ipp-route-d57e1d-kserve-67444b75f9-j48ms" Jul 26 15:58:53.136125 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:53.136102 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnr24\" (UniqueName: \"kubernetes.io/projected/1999071c-78d9-494e-9128-3b72e9c593b8-kube-api-access-gnr24\") pod \"ipp-route-d57e1d-kserve-67444b75f9-j48ms\" (UID: \"1999071c-78d9-494e-9128-3b72e9c593b8\") " pod="ai-tenant-e2e-ipp-a-d57e1d/ipp-route-d57e1d-kserve-67444b75f9-j48ms" Jul 26 15:58:53.256522 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:53.256451 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-d57e1d/ipp-route-d57e1d-kserve-67444b75f9-j48ms" Jul 26 15:58:53.381147 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:53.381111 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-d57e1d/ipp-route-d57e1d-kserve-67444b75f9-j48ms"] Jul 26 15:58:53.383529 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:58:53.383503 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1999071c_78d9_494e_9128_3b72e9c593b8.slice/crio-7d4c60afaac7400e46986be893d4a74836a0d01e1a1c50468ce1cc8facf79aae WatchSource:0}: Error finding container 7d4c60afaac7400e46986be893d4a74836a0d01e1a1c50468ce1cc8facf79aae: Status 404 returned error can't find the container with id 7d4c60afaac7400e46986be893d4a74836a0d01e1a1c50468ce1cc8facf79aae Jul 26 15:58:54.331424 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:54.331388 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-d57e1d/ipp-route-d57e1d-kserve-67444b75f9-j48ms" event={"ID":"1999071c-78d9-494e-9128-3b72e9c593b8","Type":"ContainerStarted","Data":"639dc7c8b2e0b2c774b0d2bb94a5a6d9481b0e267afd1d6ab0c12204e13d3257"} Jul 26 15:58:54.331849 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:54.331439 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-d57e1d/ipp-route-d57e1d-kserve-67444b75f9-j48ms" event={"ID":"1999071c-78d9-494e-9128-3b72e9c593b8","Type":"ContainerStarted","Data":"7d4c60afaac7400e46986be893d4a74836a0d01e1a1c50468ce1cc8facf79aae"} Jul 26 15:58:54.347243 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:58:54.347195 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-d57e1d/ipp-route-d57e1d-kserve-67444b75f9-j48ms" podStartSLOduration=2.347181782 podStartE2EDuration="2.347181782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:58:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:58:54.3451086 +0000 UTC m=+3214.768955470" watchObservedRunningTime="2026-07-26 15:58:54.347181782 +0000 UTC m=+3214.771028651" Jul 26 15:59:03.256566 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:59:03.256532 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-d57e1d/ipp-route-d57e1d-kserve-67444b75f9-j48ms" Jul 26 15:59:03.256955 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:59:03.256579 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-d57e1d/ipp-route-d57e1d-kserve-67444b75f9-j48ms" Jul 26 15:59:03.268724 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:59:03.268700 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-d57e1d/ipp-route-d57e1d-kserve-67444b75f9-j48ms" Jul 26 15:59:03.383601 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:59:03.383577 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-d57e1d/ipp-route-d57e1d-kserve-67444b75f9-j48ms" Jul 26 15:59:30.380849 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:59:30.380809 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:59:35.853387 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:59:35.853353 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:59:36.170879 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:59:36.170848 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:59:36.299679 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:59:36.299646 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:59:38.991536 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:59:38.991502 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:59:39.073915 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:59:39.073883 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-35252a-5dfdd66966-46j42"] Jul 26 15:59:39.075867 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:59:39.075836 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:59:39.077855 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:59:39.077836 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-35252a-5dfdd66966-46j42" Jul 26 15:59:39.080096 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:59:39.080076 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-35252a-serving-cert\"" Jul 26 15:59:39.085978 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:59:39.085958 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-35252a-5dfdd66966-46j42"] Jul 26 15:59:39.142956 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:59:39.142915 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:59:39.207634 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:59:39.207606 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b57f313-7122-430d-b4b3-238f46f69f15-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-35252a-5dfdd66966-46j42\" (UID: \"0b57f313-7122-430d-b4b3-238f46f69f15\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-35252a-5dfdd66966-46j42" Jul 26 15:59:39.207734 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:59:39.207644 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b57f313-7122-430d-b4b3-238f46f69f15-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-35252a-5dfdd66966-46j42\" (UID: \"0b57f313-7122-430d-b4b3-238f46f69f15\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-35252a-5dfdd66966-46j42" Jul 26 15:59:39.207779 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:59:39.207762 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2skqt\" (UniqueName: \"kubernetes.io/projected/0b57f313-7122-430d-b4b3-238f46f69f15-kube-api-access-2skqt\") pod \"maas-api-e2e-ipp-cleanup-35252a-5dfdd66966-46j42\" (UID: \"0b57f313-7122-430d-b4b3-238f46f69f15\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-35252a-5dfdd66966-46j42" Jul 26 15:59:39.308172 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:59:39.308108 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2skqt\" (UniqueName: \"kubernetes.io/projected/0b57f313-7122-430d-b4b3-238f46f69f15-kube-api-access-2skqt\") pod \"maas-api-e2e-ipp-cleanup-35252a-5dfdd66966-46j42\" (UID: \"0b57f313-7122-430d-b4b3-238f46f69f15\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-35252a-5dfdd66966-46j42" Jul 26 15:59:39.308172 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:59:39.308157 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b57f313-7122-430d-b4b3-238f46f69f15-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-35252a-5dfdd66966-46j42\" (UID: \"0b57f313-7122-430d-b4b3-238f46f69f15\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-35252a-5dfdd66966-46j42" Jul 26 15:59:39.308352 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:59:39.308232 2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-35252a-serving-cert: secret "maas-api-e2e-ipp-cleanup-35252a-serving-cert" not found Jul 26 15:59:39.308352 ip-10-0-130-21 kubenswrapper[2555]: E0726 15:59:39.308275 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b57f313-7122-430d-b4b3-238f46f69f15-maas-api-tls podName:0b57f313-7122-430d-b4b3-238f46f69f15 nodeName:}" failed. No retries permitted until 2026-07-26 15:59:39.808260437 +0000 UTC m=+3260.232107287 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0b57f313-7122-430d-b4b3-238f46f69f15-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-35252a-5dfdd66966-46j42" (UID: "0b57f313-7122-430d-b4b3-238f46f69f15") : secret "maas-api-e2e-ipp-cleanup-35252a-serving-cert" not found Jul 26 15:59:39.308352 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:59:39.308270 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b57f313-7122-430d-b4b3-238f46f69f15-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-35252a-5dfdd66966-46j42\" (UID: \"0b57f313-7122-430d-b4b3-238f46f69f15\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-35252a-5dfdd66966-46j42" Jul 26 15:59:39.308920 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:59:39.308898 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b57f313-7122-430d-b4b3-238f46f69f15-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-35252a-5dfdd66966-46j42\" (UID: \"0b57f313-7122-430d-b4b3-238f46f69f15\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-35252a-5dfdd66966-46j42" Jul 26 15:59:39.318770 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:59:39.318745 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2skqt\" (UniqueName: \"kubernetes.io/projected/0b57f313-7122-430d-b4b3-238f46f69f15-kube-api-access-2skqt\") pod \"maas-api-e2e-ipp-cleanup-35252a-5dfdd66966-46j42\" (UID: \"0b57f313-7122-430d-b4b3-238f46f69f15\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-35252a-5dfdd66966-46j42" Jul 26 15:59:39.486087 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:59:39.486054 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:59:39.690728 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:59:39.690700 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:59:39.785883 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:59:39.785850 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:59:39.810924 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:59:39.810885 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b57f313-7122-430d-b4b3-238f46f69f15-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-35252a-5dfdd66966-46j42\" (UID: \"0b57f313-7122-430d-b4b3-238f46f69f15\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-35252a-5dfdd66966-46j42" Jul 26 15:59:39.813235 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:59:39.813210 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b57f313-7122-430d-b4b3-238f46f69f15-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-35252a-5dfdd66966-46j42\" (UID: \"0b57f313-7122-430d-b4b3-238f46f69f15\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-35252a-5dfdd66966-46j42" Jul 26 15:59:39.989382 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:59:39.989288 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-35252a-5dfdd66966-46j42" Jul 26 15:59:40.121704 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:59:40.121671 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-35252a-5dfdd66966-46j42"] Jul 26 15:59:40.123241 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:59:40.123220 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b57f313_7122_430d_b4b3_238f46f69f15.slice/crio-ece4213a243cc07181541a4b0a9c8bc6055fd2c067a90aec5e41b1c9a0c89101 WatchSource:0}: Error finding container ece4213a243cc07181541a4b0a9c8bc6055fd2c067a90aec5e41b1c9a0c89101: Status 404 returned error can't find the container with id ece4213a243cc07181541a4b0a9c8bc6055fd2c067a90aec5e41b1c9a0c89101 Jul 26 15:59:40.544577 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:59:40.544547 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-35252a-5dfdd66966-46j42" event={"ID":"0b57f313-7122-430d-b4b3-238f46f69f15","Type":"ContainerStarted","Data":"0ff6682a8c959ee96031b6be552e976ae43c433132c1d30d0db69ae734ba3f04"} Jul 26 15:59:40.544676 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:59:40.544593 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-35252a-5dfdd66966-46j42" event={"ID":"0b57f313-7122-430d-b4b3-238f46f69f15","Type":"ContainerStarted","Data":"ece4213a243cc07181541a4b0a9c8bc6055fd2c067a90aec5e41b1c9a0c89101"} Jul 26 15:59:40.544676 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:59:40.544615 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-35252a-5dfdd66966-46j42" Jul 26 15:59:40.564175 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:59:40.564123 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-35252a-5dfdd66966-46j42" podStartSLOduration=1.21709435 podStartE2EDuration="1.564103256s" totalImagesPullingTime="347.008906ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:59:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:59:40.56146354 +0000 UTC m=+3260.985310415" watchObservedRunningTime="2026-07-26 15:59:40.564103256 +0000 UTC m=+3260.987950128" Jul 26 15:59:43.501048 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:59:43.501010 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 15:59:46.555222 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:59:46.555194 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-35252a-5dfdd66966-46j42" Jul 26 15:59:53.173915 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:59:53.173887 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zmkg4"] Jul 26 15:59:53.177341 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:59:53.177323 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-zmkg4" Jul 26 15:59:53.179383 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:59:53.179361 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-w8jkr\"" Jul 26 15:59:53.185506 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:59:53.185482 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zmkg4"] Jul 26 15:59:53.213730 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:59:53.213705 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md672\" (UniqueName: \"kubernetes.io/projected/fec36f9a-e64e-4bfd-819b-e92ba317e9e8-kube-api-access-md672\") pod \"kuadrant-operator-catalog-zmkg4\" (UID: \"fec36f9a-e64e-4bfd-819b-e92ba317e9e8\") " pod="kuadrant-system/kuadrant-operator-catalog-zmkg4" Jul 26 15:59:53.314547 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:59:53.314523 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md672\" (UniqueName: \"kubernetes.io/projected/fec36f9a-e64e-4bfd-819b-e92ba317e9e8-kube-api-access-md672\") pod \"kuadrant-operator-catalog-zmkg4\" (UID: \"fec36f9a-e64e-4bfd-819b-e92ba317e9e8\") " pod="kuadrant-system/kuadrant-operator-catalog-zmkg4" Jul 26 15:59:53.322860 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:59:53.322831 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-md672\" (UniqueName: \"kubernetes.io/projected/fec36f9a-e64e-4bfd-819b-e92ba317e9e8-kube-api-access-md672\") pod \"kuadrant-operator-catalog-zmkg4\" (UID: \"fec36f9a-e64e-4bfd-819b-e92ba317e9e8\") " pod="kuadrant-system/kuadrant-operator-catalog-zmkg4" Jul 26 15:59:53.486494 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:59:53.486439 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-zmkg4" Jul 26 15:59:53.612287 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:59:53.612265 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zmkg4"] Jul 26 15:59:53.613833 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:59:53.613799 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfec36f9a_e64e_4bfd_819b_e92ba317e9e8.slice/crio-66cb86c348e3fde933d4377803e555a1628a2f7dca36808aca0ff92cb042acb7 WatchSource:0}: Error finding container 66cb86c348e3fde933d4377803e555a1628a2f7dca36808aca0ff92cb042acb7: Status 404 returned error can't find the container with id 66cb86c348e3fde933d4377803e555a1628a2f7dca36808aca0ff92cb042acb7 Jul 26 15:59:54.379561 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:59:54.379508 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-35252a-5xkzv"] Jul 26 15:59:54.383691 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:59:54.383664 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-35252a-5xkzv" Jul 26 15:59:54.389582 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:59:54.389555 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d665x\"" Jul 26 15:59:54.395315 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:59:54.395006 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-35252a-5xkzv"] Jul 26 15:59:54.423330 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:59:54.423282 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f81ca00-4950-46ec-b9cd-e76c329ea576-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-35252a-5xkzv\" (UID: \"0f81ca00-4950-46ec-b9cd-e76c329ea576\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-35252a-5xkzv" Jul 26 15:59:54.524397 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:59:54.524358 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f81ca00-4950-46ec-b9cd-e76c329ea576-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-35252a-5xkzv\" (UID: \"0f81ca00-4950-46ec-b9cd-e76c329ea576\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-35252a-5xkzv" Jul 26 15:59:54.525156 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:59:54.525123 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f81ca00-4950-46ec-b9cd-e76c329ea576-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-35252a-5xkzv\" (UID: \"0f81ca00-4950-46ec-b9cd-e76c329ea576\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-35252a-5xkzv" Jul 26 15:59:54.614117 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:59:54.614060 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-zmkg4" event={"ID":"fec36f9a-e64e-4bfd-819b-e92ba317e9e8","Type":"ContainerStarted","Data":"66cb86c348e3fde933d4377803e555a1628a2f7dca36808aca0ff92cb042acb7"} Jul 26 15:59:54.706516 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:59:54.706428 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-35252a-5xkzv" Jul 26 15:59:55.385227 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:59:55.385202 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-35252a-5xkzv"] Jul 26 15:59:55.387172 ip-10-0-130-21 kubenswrapper[2555]: W0726 15:59:55.387149 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f81ca00_4950_46ec_b9cd_e76c329ea576.slice/crio-d8144cf127dcd0377b1612e1c3f6af2171989c34bf2daa7cea68dba56b3e6219 WatchSource:0}: Error finding container d8144cf127dcd0377b1612e1c3f6af2171989c34bf2daa7cea68dba56b3e6219: Status 404 returned error can't find the container with id d8144cf127dcd0377b1612e1c3f6af2171989c34bf2daa7cea68dba56b3e6219 Jul 26 15:59:55.620387 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:59:55.620354 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-zmkg4" event={"ID":"fec36f9a-e64e-4bfd-819b-e92ba317e9e8","Type":"ContainerStarted","Data":"77180440e058628ac06629b1280a561feec6ea675caaae17f40b630a32a7be5d"} Jul 26 15:59:55.621652 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:59:55.621626 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-35252a-5xkzv" event={"ID":"0f81ca00-4950-46ec-b9cd-e76c329ea576","Type":"ContainerStarted","Data":"d8144cf127dcd0377b1612e1c3f6af2171989c34bf2daa7cea68dba56b3e6219"} Jul 26 15:59:55.634945 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:59:55.634891 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-zmkg4" podStartSLOduration=0.938301128 podStartE2EDuration="2.6348783s" totalImagesPullingTime="1.696577172s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:59:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:59:55.633779991 +0000 UTC m=+3276.057626883" watchObservedRunningTime="2026-07-26 15:59:55.6348783 +0000 UTC m=+3276.058725170" Jul 26 15:59:56.628033 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:59:56.628004 2555 generic.go:356] "Generic (PLEG): container finished" podID="0f81ca00-4950-46ec-b9cd-e76c329ea576" containerID="3d5c92014e2379a26d66a3d1dae672a5e8360b2337029b8abf51be4e256b8a9a" exitCode=0 Jul 26 15:59:56.628433 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:59:56.628100 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-35252a-5xkzv" event={"ID":"0f81ca00-4950-46ec-b9cd-e76c329ea576","Type":"ContainerDied","Data":"3d5c92014e2379a26d66a3d1dae672a5e8360b2337029b8abf51be4e256b8a9a"} Jul 26 15:59:57.768092 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:59:57.768064 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-35252a-5xkzv" Jul 26 15:59:57.852039 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:59:57.852015 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f81ca00-4950-46ec-b9cd-e76c329ea576-maas-api-service-ca\") pod \"0f81ca00-4950-46ec-b9cd-e76c329ea576\" (UID: \"0f81ca00-4950-46ec-b9cd-e76c329ea576\") " Jul 26 15:59:57.852354 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:59:57.852331 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f81ca00-4950-46ec-b9cd-e76c329ea576-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0f81ca00-4950-46ec-b9cd-e76c329ea576" (UID: "0f81ca00-4950-46ec-b9cd-e76c329ea576"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:59:57.952798 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:59:57.952731 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f81ca00-4950-46ec-b9cd-e76c329ea576-maas-api-service-ca\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 15:59:58.638276 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:59:58.638249 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-35252a-5xkzv" Jul 26 15:59:58.638462 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:59:58.638288 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-35252a-5xkzv" event={"ID":"0f81ca00-4950-46ec-b9cd-e76c329ea576","Type":"ContainerDied","Data":"d8144cf127dcd0377b1612e1c3f6af2171989c34bf2daa7cea68dba56b3e6219"} Jul 26 15:59:58.638462 ip-10-0-130-21 kubenswrapper[2555]: I0726 15:59:58.638338 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8144cf127dcd0377b1612e1c3f6af2171989c34bf2daa7cea68dba56b3e6219" Jul 26 16:00:00.135204 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:00.135162 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d57e1d-29751360-f74hp"] Jul 26 16:00:00.135706 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:00.135687 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f81ca00-4950-46ec-b9cd-e76c329ea576" containerName="revoke-keys" Jul 26 16:00:00.135788 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:00.135709 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f81ca00-4950-46ec-b9cd-e76c329ea576" containerName="revoke-keys" Jul 26 16:00:00.135860 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:00.135847 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f81ca00-4950-46ec-b9cd-e76c329ea576" containerName="revoke-keys" Jul 26 16:00:00.139058 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:00.139039 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d57e1d-29751360-f74hp" Jul 26 16:00:00.142653 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:00.142636 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d665x\"" Jul 26 16:00:00.142851 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:00.142836 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a20d61-29751360-fqvrk"] Jul 26 16:00:00.146379 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:00.146360 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29751360-lwkqg"] Jul 26 16:00:00.146574 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:00.146554 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a20d61-29751360-fqvrk" Jul 26 16:00:00.153737 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:00.153714 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d57e1d-29751360-f74hp" Jul 26 16:00:00.157606 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:00.157583 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cb6b99-29751360-v4gl8"] Jul 26 16:00:00.158069 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:00.158049 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29751360-lwkqg" Jul 26 16:00:00.163118 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:00.162875 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-60c814-29751360-sgxhq"] Jul 26 16:00:00.163799 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:00.163767 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cb6b99-29751360-v4gl8" Jul 26 16:00:00.168263 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:00.168228 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d57e1d-29751360-f74hp"] Jul 26 16:00:00.168374 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:00.168279 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a20d61-29751360-fqvrk"] Jul 26 16:00:00.168374 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:00.168316 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29751360-lwkqg"] Jul 26 16:00:00.168374 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:00.168349 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cb6b99-29751360-v4gl8"] Jul 26 16:00:00.168538 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:00.168447 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-60c814-29751360-sgxhq" Jul 26 16:00:00.169041 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:00.168868 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a20d61-29751360-fqvrk" Jul 26 16:00:00.179243 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:00.175439 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-35252a-29751360-fgtxs"] Jul 26 16:00:00.192332 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:00.191170 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-60c814-29751360-sgxhq"] Jul 26 16:00:00.192332 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:00.191214 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-35252a-29751360-fgtxs"] Jul 26 16:00:00.192332 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:00.192008 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-35252a-29751360-fgtxs" Jul 26 16:00:00.220092 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:00.220017 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29751360-lwkqg" Jul 26 16:00:00.227186 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:00.227161 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cb6b99-29751360-v4gl8" Jul 26 16:00:00.241560 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:00.241055 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-60c814-29751360-sgxhq" Jul 26 16:00:00.287847 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:00.287626 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-35252a-29751360-fgtxs" Jul 26 16:00:00.361630 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:00.361294 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d57e1d-29751360-f74hp"] Jul 26 16:00:00.397369 ip-10-0-130-21 kubenswrapper[2555]: W0726 16:00:00.397315 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod114f21a5_0216_40c1_8038_842e96adc6e6.slice/crio-ef419565d9d308ec500a373e063a5dbff03be256de9f838e21453eccba59677c WatchSource:0}: Error finding container ef419565d9d308ec500a373e063a5dbff03be256de9f838e21453eccba59677c: Status 404 returned error can't find the container with id ef419565d9d308ec500a373e063a5dbff03be256de9f838e21453eccba59677c Jul 26 16:00:00.397645 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:00.397621 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a20d61-29751360-fqvrk"] Jul 26 16:00:00.441631 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:00.441602 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29751360-lwkqg"] Jul 26 16:00:00.456247 ip-10-0-130-21 kubenswrapper[2555]: W0726 16:00:00.456213 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1672b2bd_09f2_45ef_83aa_d58c0b3279ee.slice/crio-8fa667722f7f07455274aaf27849547d09b6d04dd26c8744926b2c7d138abfd2 WatchSource:0}: Error finding container 8fa667722f7f07455274aaf27849547d09b6d04dd26c8744926b2c7d138abfd2: Status 404 returned error can't find the container with id 8fa667722f7f07455274aaf27849547d09b6d04dd26c8744926b2c7d138abfd2 Jul 26 16:00:00.514059 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:00.514034 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-35252a-29751360-fgtxs"] Jul 26 16:00:00.561171 ip-10-0-130-21 kubenswrapper[2555]: W0726 16:00:00.561139 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20cb9db2_c276_424a_8b3d_45f275bf739d.slice/crio-bece1915404eb5848bb8af870afc09474d3e0041436b03514ffcc2310981b6b9 WatchSource:0}: Error finding container bece1915404eb5848bb8af870afc09474d3e0041436b03514ffcc2310981b6b9: Status 404 returned error can't find the container with id bece1915404eb5848bb8af870afc09474d3e0041436b03514ffcc2310981b6b9 Jul 26 16:00:00.649329 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:00.649264 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a20d61-29751360-fqvrk" event={"ID":"114f21a5-0216-40c1-8038-842e96adc6e6","Type":"ContainerStarted","Data":"ef419565d9d308ec500a373e063a5dbff03be256de9f838e21453eccba59677c"} Jul 26 16:00:00.650437 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:00.650408 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29751360-lwkqg" event={"ID":"1672b2bd-09f2-45ef-83aa-d58c0b3279ee","Type":"ContainerStarted","Data":"8fa667722f7f07455274aaf27849547d09b6d04dd26c8744926b2c7d138abfd2"} Jul 26 16:00:00.651539 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:00.651511 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-35252a-29751360-fgtxs" event={"ID":"20cb9db2-c276-424a-8b3d-45f275bf739d","Type":"ContainerStarted","Data":"bece1915404eb5848bb8af870afc09474d3e0041436b03514ffcc2310981b6b9"} Jul 26 16:00:00.652566 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:00.652545 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d57e1d-29751360-f74hp" event={"ID":"d54f00bd-fc34-4e8d-a0e5-e889d3e7769d","Type":"ContainerStarted","Data":"4b11d1ccedf9f47f1ea7ee9bc17e3ee55756ec56f91bd2d2e97c7fc85d1211f5"} Jul 26 16:00:00.680701 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:00.680677 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cb6b99-29751360-v4gl8"] Jul 26 16:00:00.683096 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:00.683068 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-60c814-29751360-sgxhq"] Jul 26 16:00:00.683231 ip-10-0-130-21 kubenswrapper[2555]: W0726 16:00:00.683212 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fb61f27_8774_44c9_8883_909475554baf.slice/crio-ac9e03e64888e8fbc20f6d2c2691742686d012307c3206cba6f4d18bd6f447d1 WatchSource:0}: Error finding container ac9e03e64888e8fbc20f6d2c2691742686d012307c3206cba6f4d18bd6f447d1: Status 404 returned error can't find the container with id ac9e03e64888e8fbc20f6d2c2691742686d012307c3206cba6f4d18bd6f447d1 Jul 26 16:00:00.684286 ip-10-0-130-21 kubenswrapper[2555]: W0726 16:00:00.684261 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b6a63a2_264c_48d5_9c10_e5a56bfbecb5.slice/crio-3324a280b5b09adb8afb93a4cc47c909d488128ce3c2ed17c75af8d270759f87 WatchSource:0}: Error finding container 3324a280b5b09adb8afb93a4cc47c909d488128ce3c2ed17c75af8d270759f87: Status 404 returned error can't find the container with id 3324a280b5b09adb8afb93a4cc47c909d488128ce3c2ed17c75af8d270759f87 Jul 26 16:00:01.660026 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:01.659947 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-35252a-29751360-fgtxs" event={"ID":"20cb9db2-c276-424a-8b3d-45f275bf739d","Type":"ContainerStarted","Data":"dfceedd02b33b5f213a9540d08c604728829b8ed0b2ad31eeabdfd708e8fdb8c"} Jul 26 16:00:01.663136 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:01.663076 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-60c814-29751360-sgxhq" event={"ID":"8b6a63a2-264c-48d5-9c10-e5a56bfbecb5","Type":"ContainerStarted","Data":"33720573569eccfac3f3d51eb33a8f6cf5b9429d78408239514a8be29e4c213c"} Jul 26 16:00:01.663341 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:01.663315 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-60c814-29751360-sgxhq" event={"ID":"8b6a63a2-264c-48d5-9c10-e5a56bfbecb5","Type":"ContainerStarted","Data":"3324a280b5b09adb8afb93a4cc47c909d488128ce3c2ed17c75af8d270759f87"} Jul 26 16:00:01.664791 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:01.664763 2555 generic.go:356] "Generic (PLEG): container finished" podID="d54f00bd-fc34-4e8d-a0e5-e889d3e7769d" containerID="ca35e72cffe03dbe44e220f6ba4cf0d8834e48b3782498f4592d5b1de79da76f" exitCode=0 Jul 26 16:00:01.664920 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:01.664870 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d57e1d-29751360-f74hp" event={"ID":"d54f00bd-fc34-4e8d-a0e5-e889d3e7769d","Type":"ContainerDied","Data":"ca35e72cffe03dbe44e220f6ba4cf0d8834e48b3782498f4592d5b1de79da76f"} Jul 26 16:00:01.666726 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:01.666700 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cb6b99-29751360-v4gl8" event={"ID":"0fb61f27-8774-44c9-8883-909475554baf","Type":"ContainerStarted","Data":"7f1078de0e5b371673a8cc05bca912af2bd3f57905270785580540230ce6f73f"} Jul 26 16:00:01.666842 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:01.666750 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cb6b99-29751360-v4gl8" event={"ID":"0fb61f27-8774-44c9-8883-909475554baf","Type":"ContainerStarted","Data":"ac9e03e64888e8fbc20f6d2c2691742686d012307c3206cba6f4d18bd6f447d1"} Jul 26 16:00:01.668353 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:01.668329 2555 generic.go:356] "Generic (PLEG): container finished" podID="114f21a5-0216-40c1-8038-842e96adc6e6" containerID="c1a72ebff851b608066fbf55d2da11e7135a53fb2afb676318cf0b0a217d3a80" exitCode=0 Jul 26 16:00:01.668432 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:01.668374 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a20d61-29751360-fqvrk" event={"ID":"114f21a5-0216-40c1-8038-842e96adc6e6","Type":"ContainerDied","Data":"c1a72ebff851b608066fbf55d2da11e7135a53fb2afb676318cf0b0a217d3a80"} Jul 26 16:00:01.669876 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:01.669857 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29751360-lwkqg" event={"ID":"1672b2bd-09f2-45ef-83aa-d58c0b3279ee","Type":"ContainerStarted","Data":"8c3f0c4e702ed9661382bb071eab98a29e1b8f22d2ed02f9e0c5fb0c66802fc4"} Jul 26 16:00:01.675007 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:01.674970 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-35252a-29751360-fgtxs" podStartSLOduration=1.674956501 podStartE2EDuration="1.674956501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:00:01.673322833 +0000 UTC m=+3282.097169704" watchObservedRunningTime="2026-07-26 16:00:01.674956501 +0000 UTC m=+3282.098803373" Jul 26 16:00:01.687173 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:01.687137 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29751360-lwkqg" podStartSLOduration=1.687125631 podStartE2EDuration="1.687125631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:00:01.685597415 +0000 UTC m=+3282.109444287" watchObservedRunningTime="2026-07-26 16:00:01.687125631 +0000 UTC m=+3282.110972501" Jul 26 16:00:01.697968 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:01.697908 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cb6b99-29751360-v4gl8" podStartSLOduration=1.69789447 podStartE2EDuration="1.69789447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:00:01.697581438 +0000 UTC m=+3282.121428310" watchObservedRunningTime="2026-07-26 16:00:01.69789447 +0000 UTC m=+3282.121741341" Jul 26 16:00:01.739420 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:01.739386 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-60c814-29751360-sgxhq" podStartSLOduration=1.739375318 podStartE2EDuration="1.739375318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:00:01.738987003 +0000 UTC m=+3282.162833873" watchObservedRunningTime="2026-07-26 16:00:01.739375318 +0000 UTC m=+3282.163222189" Jul 26 16:00:02.676500 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:02.676461 2555 generic.go:356] "Generic (PLEG): container finished" podID="0fb61f27-8774-44c9-8883-909475554baf" containerID="7f1078de0e5b371673a8cc05bca912af2bd3f57905270785580540230ce6f73f" exitCode=0 Jul 26 16:00:02.676869 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:02.676550 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cb6b99-29751360-v4gl8" event={"ID":"0fb61f27-8774-44c9-8883-909475554baf","Type":"ContainerDied","Data":"7f1078de0e5b371673a8cc05bca912af2bd3f57905270785580540230ce6f73f"} Jul 26 16:00:02.677995 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:02.677970 2555 generic.go:356] "Generic (PLEG): container finished" podID="1672b2bd-09f2-45ef-83aa-d58c0b3279ee" containerID="8c3f0c4e702ed9661382bb071eab98a29e1b8f22d2ed02f9e0c5fb0c66802fc4" exitCode=0 Jul 26 16:00:02.678111 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:02.678048 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29751360-lwkqg" event={"ID":"1672b2bd-09f2-45ef-83aa-d58c0b3279ee","Type":"ContainerDied","Data":"8c3f0c4e702ed9661382bb071eab98a29e1b8f22d2ed02f9e0c5fb0c66802fc4"} Jul 26 16:00:02.679474 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:02.679455 2555 generic.go:356] "Generic (PLEG): container finished" podID="20cb9db2-c276-424a-8b3d-45f275bf739d" containerID="dfceedd02b33b5f213a9540d08c604728829b8ed0b2ad31eeabdfd708e8fdb8c" exitCode=0 Jul 26 16:00:02.679570 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:02.679522 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-35252a-29751360-fgtxs" event={"ID":"20cb9db2-c276-424a-8b3d-45f275bf739d","Type":"ContainerDied","Data":"dfceedd02b33b5f213a9540d08c604728829b8ed0b2ad31eeabdfd708e8fdb8c"} Jul 26 16:00:02.680848 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:02.680829 2555 generic.go:356] "Generic (PLEG): container finished" podID="8b6a63a2-264c-48d5-9c10-e5a56bfbecb5" containerID="33720573569eccfac3f3d51eb33a8f6cf5b9429d78408239514a8be29e4c213c" exitCode=0 Jul 26 16:00:02.680967 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:02.680880 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-60c814-29751360-sgxhq" event={"ID":"8b6a63a2-264c-48d5-9c10-e5a56bfbecb5","Type":"ContainerDied","Data":"33720573569eccfac3f3d51eb33a8f6cf5b9429d78408239514a8be29e4c213c"} Jul 26 16:00:02.820696 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:02.820673 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a20d61-29751360-fqvrk" Jul 26 16:00:02.846448 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:02.846428 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d57e1d-29751360-f74hp" Jul 26 16:00:03.487457 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:03.487423 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-zmkg4" Jul 26 16:00:03.487651 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:03.487486 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-zmkg4" Jul 26 16:00:03.508819 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:03.508789 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-zmkg4" Jul 26 16:00:03.687661 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:03.687633 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d57e1d-29751360-f74hp" Jul 26 16:00:03.688102 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:03.687674 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d57e1d-29751360-f74hp" event={"ID":"d54f00bd-fc34-4e8d-a0e5-e889d3e7769d","Type":"ContainerDied","Data":"4b11d1ccedf9f47f1ea7ee9bc17e3ee55756ec56f91bd2d2e97c7fc85d1211f5"} Jul 26 16:00:03.688102 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:03.687722 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b11d1ccedf9f47f1ea7ee9bc17e3ee55756ec56f91bd2d2e97c7fc85d1211f5" Jul 26 16:00:03.689012 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:03.688993 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a20d61-29751360-fqvrk" Jul 26 16:00:03.689093 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:03.689038 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a20d61-29751360-fqvrk" event={"ID":"114f21a5-0216-40c1-8038-842e96adc6e6","Type":"ContainerDied","Data":"ef419565d9d308ec500a373e063a5dbff03be256de9f838e21453eccba59677c"} Jul 26 16:00:03.689093 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:03.689074 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef419565d9d308ec500a373e063a5dbff03be256de9f838e21453eccba59677c" Jul 26 16:00:03.717564 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:03.717515 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-zmkg4" Jul 26 16:00:03.755071 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:03.754999 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zmkg4"] Jul 26 16:00:03.850372 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:03.850350 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-60c814-29751360-sgxhq" Jul 26 16:00:03.854284 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:03.854260 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cb6b99-29751360-v4gl8" Jul 26 16:00:03.913711 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:03.913684 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-35252a-29751360-fgtxs" Jul 26 16:00:03.916624 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:03.916606 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29751360-lwkqg" Jul 26 16:00:04.702852 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:04.702822 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29751360-lwkqg" Jul 26 16:00:04.703324 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:04.702828 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29751360-lwkqg" event={"ID":"1672b2bd-09f2-45ef-83aa-d58c0b3279ee","Type":"ContainerDied","Data":"8fa667722f7f07455274aaf27849547d09b6d04dd26c8744926b2c7d138abfd2"} Jul 26 16:00:04.703324 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:04.702988 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8fa667722f7f07455274aaf27849547d09b6d04dd26c8744926b2c7d138abfd2" Jul 26 16:00:04.704410 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:04.704376 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-35252a-29751360-fgtxs" event={"ID":"20cb9db2-c276-424a-8b3d-45f275bf739d","Type":"ContainerDied","Data":"bece1915404eb5848bb8af870afc09474d3e0041436b03514ffcc2310981b6b9"} Jul 26 16:00:04.704533 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:04.704430 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bece1915404eb5848bb8af870afc09474d3e0041436b03514ffcc2310981b6b9" Jul 26 16:00:04.704533 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:04.704387 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-35252a-29751360-fgtxs" Jul 26 16:00:04.705805 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:04.705790 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-60c814-29751360-sgxhq" Jul 26 16:00:04.705900 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:04.705834 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-60c814-29751360-sgxhq" event={"ID":"8b6a63a2-264c-48d5-9c10-e5a56bfbecb5","Type":"ContainerDied","Data":"3324a280b5b09adb8afb93a4cc47c909d488128ce3c2ed17c75af8d270759f87"} Jul 26 16:00:04.705900 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:04.705872 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3324a280b5b09adb8afb93a4cc47c909d488128ce3c2ed17c75af8d270759f87" Jul 26 16:00:04.707985 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:04.707962 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cb6b99-29751360-v4gl8" event={"ID":"0fb61f27-8774-44c9-8883-909475554baf","Type":"ContainerDied","Data":"ac9e03e64888e8fbc20f6d2c2691742686d012307c3206cba6f4d18bd6f447d1"} Jul 26 16:00:04.708083 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:04.708005 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac9e03e64888e8fbc20f6d2c2691742686d012307c3206cba6f4d18bd6f447d1" Jul 26 16:00:04.708083 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:04.707971 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cb6b99-29751360-v4gl8" Jul 26 16:00:04.847137 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:04.847107 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-35252a-5dfdd66966-46j42"] Jul 26 16:00:04.847336 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:04.847314 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-35252a-5dfdd66966-46j42" podUID="0b57f313-7122-430d-b4b3-238f46f69f15" containerName="maas-api" containerID="cri-o://0ff6682a8c959ee96031b6be552e976ae43c433132c1d30d0db69ae734ba3f04" gracePeriod=30 Jul 26 16:00:04.863921 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:04.863898 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-35252a-29751360-fgtxs"] Jul 26 16:00:04.870573 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:04.870549 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-35252a-29751360-fgtxs"] Jul 26 16:00:04.870742 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:04.870724 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 16:00:04.976138 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:04.976105 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 16:00:05.039830 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:05.039798 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 16:00:05.094884 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:05.094862 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-35252a-5dfdd66966-46j42" Jul 26 16:00:05.223101 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:05.223023 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b57f313-7122-430d-b4b3-238f46f69f15-maas-api-tls\") pod \"0b57f313-7122-430d-b4b3-238f46f69f15\" (UID: \"0b57f313-7122-430d-b4b3-238f46f69f15\") " Jul 26 16:00:05.223101 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:05.223071 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2skqt\" (UniqueName: \"kubernetes.io/projected/0b57f313-7122-430d-b4b3-238f46f69f15-kube-api-access-2skqt\") pod \"0b57f313-7122-430d-b4b3-238f46f69f15\" (UID: \"0b57f313-7122-430d-b4b3-238f46f69f15\") " Jul 26 16:00:05.223101 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:05.223092 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b57f313-7122-430d-b4b3-238f46f69f15-openshift-service-ca\") pod \"0b57f313-7122-430d-b4b3-238f46f69f15\" (UID: \"0b57f313-7122-430d-b4b3-238f46f69f15\") " Jul 26 16:00:05.223508 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:05.223488 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b57f313-7122-430d-b4b3-238f46f69f15-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0b57f313-7122-430d-b4b3-238f46f69f15" (UID: "0b57f313-7122-430d-b4b3-238f46f69f15"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:00:05.225204 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:05.225182 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b57f313-7122-430d-b4b3-238f46f69f15-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0b57f313-7122-430d-b4b3-238f46f69f15" (UID: "0b57f313-7122-430d-b4b3-238f46f69f15"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 16:00:05.225357 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:05.225335 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b57f313-7122-430d-b4b3-238f46f69f15-kube-api-access-2skqt" (OuterVolumeSpecName: "kube-api-access-2skqt") pod "0b57f313-7122-430d-b4b3-238f46f69f15" (UID: "0b57f313-7122-430d-b4b3-238f46f69f15"). InnerVolumeSpecName "kube-api-access-2skqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:00:05.324012 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:05.323986 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b57f313-7122-430d-b4b3-238f46f69f15-maas-api-tls\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 16:00:05.324012 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:05.324011 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2skqt\" (UniqueName: \"kubernetes.io/projected/0b57f313-7122-430d-b4b3-238f46f69f15-kube-api-access-2skqt\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 16:00:05.324168 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:05.324020 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b57f313-7122-430d-b4b3-238f46f69f15-openshift-service-ca\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 16:00:05.717495 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:05.717459 2555 generic.go:356] "Generic (PLEG): container finished" podID="0b57f313-7122-430d-b4b3-238f46f69f15" containerID="0ff6682a8c959ee96031b6be552e976ae43c433132c1d30d0db69ae734ba3f04" exitCode=0 Jul 26 16:00:05.717906 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:05.717505 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-35252a-5dfdd66966-46j42" Jul 26 16:00:05.717906 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:05.717541 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-35252a-5dfdd66966-46j42" event={"ID":"0b57f313-7122-430d-b4b3-238f46f69f15","Type":"ContainerDied","Data":"0ff6682a8c959ee96031b6be552e976ae43c433132c1d30d0db69ae734ba3f04"} Jul 26 16:00:05.717906 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:05.717593 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-35252a-5dfdd66966-46j42" event={"ID":"0b57f313-7122-430d-b4b3-238f46f69f15","Type":"ContainerDied","Data":"ece4213a243cc07181541a4b0a9c8bc6055fd2c067a90aec5e41b1c9a0c89101"} Jul 26 16:00:05.717906 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:05.717614 2555 scope.go:117] "RemoveContainer" containerID="0ff6682a8c959ee96031b6be552e976ae43c433132c1d30d0db69ae734ba3f04" Jul 26 16:00:05.717906 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:05.717821 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-zmkg4" podUID="fec36f9a-e64e-4bfd-819b-e92ba317e9e8" containerName="registry-server" containerID="cri-o://77180440e058628ac06629b1280a561feec6ea675caaae17f40b630a32a7be5d" gracePeriod=2 Jul 26 16:00:05.727373 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:05.727354 2555 scope.go:117] "RemoveContainer" containerID="0ff6682a8c959ee96031b6be552e976ae43c433132c1d30d0db69ae734ba3f04" Jul 26 16:00:05.727648 ip-10-0-130-21 kubenswrapper[2555]: E0726 16:00:05.727625 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ff6682a8c959ee96031b6be552e976ae43c433132c1d30d0db69ae734ba3f04\": container with ID starting with 0ff6682a8c959ee96031b6be552e976ae43c433132c1d30d0db69ae734ba3f04 not found: ID does not exist" containerID="0ff6682a8c959ee96031b6be552e976ae43c433132c1d30d0db69ae734ba3f04" Jul 26 16:00:05.727728 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:05.727656 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ff6682a8c959ee96031b6be552e976ae43c433132c1d30d0db69ae734ba3f04"} err="failed to get container status \"0ff6682a8c959ee96031b6be552e976ae43c433132c1d30d0db69ae734ba3f04\": rpc error: code = NotFound desc = could not find container \"0ff6682a8c959ee96031b6be552e976ae43c433132c1d30d0db69ae734ba3f04\": container with ID starting with 0ff6682a8c959ee96031b6be552e976ae43c433132c1d30d0db69ae734ba3f04 not found: ID does not exist" Jul 26 16:00:05.746810 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:05.746788 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-35252a-5dfdd66966-46j42"] Jul 26 16:00:05.752410 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:05.752389 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-35252a-5dfdd66966-46j42"] Jul 26 16:00:05.961476 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:05.961449 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-zmkg4" Jul 26 16:00:06.028700 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:06.028634 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-md672\" (UniqueName: \"kubernetes.io/projected/fec36f9a-e64e-4bfd-819b-e92ba317e9e8-kube-api-access-md672\") pod \"fec36f9a-e64e-4bfd-819b-e92ba317e9e8\" (UID: \"fec36f9a-e64e-4bfd-819b-e92ba317e9e8\") " Jul 26 16:00:06.030571 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:06.030550 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fec36f9a-e64e-4bfd-819b-e92ba317e9e8-kube-api-access-md672" (OuterVolumeSpecName: "kube-api-access-md672") pod "fec36f9a-e64e-4bfd-819b-e92ba317e9e8" (UID: "fec36f9a-e64e-4bfd-819b-e92ba317e9e8"). InnerVolumeSpecName "kube-api-access-md672". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:00:06.065548 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:06.065518 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b57f313-7122-430d-b4b3-238f46f69f15" path="/var/lib/kubelet/pods/0b57f313-7122-430d-b4b3-238f46f69f15/volumes" Jul 26 16:00:06.065791 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:06.065779 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20cb9db2-c276-424a-8b3d-45f275bf739d" path="/var/lib/kubelet/pods/20cb9db2-c276-424a-8b3d-45f275bf739d/volumes" Jul 26 16:00:06.129954 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:06.129915 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-md672\" (UniqueName: \"kubernetes.io/projected/fec36f9a-e64e-4bfd-819b-e92ba317e9e8-kube-api-access-md672\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 16:00:06.724597 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:06.724559 2555 generic.go:356] "Generic (PLEG): container finished" podID="fec36f9a-e64e-4bfd-819b-e92ba317e9e8" containerID="77180440e058628ac06629b1280a561feec6ea675caaae17f40b630a32a7be5d" exitCode=0 Jul 26 16:00:06.725053 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:06.724604 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-zmkg4" Jul 26 16:00:06.725053 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:06.724638 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-zmkg4" event={"ID":"fec36f9a-e64e-4bfd-819b-e92ba317e9e8","Type":"ContainerDied","Data":"77180440e058628ac06629b1280a561feec6ea675caaae17f40b630a32a7be5d"} Jul 26 16:00:06.725053 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:06.724684 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-zmkg4" event={"ID":"fec36f9a-e64e-4bfd-819b-e92ba317e9e8","Type":"ContainerDied","Data":"66cb86c348e3fde933d4377803e555a1628a2f7dca36808aca0ff92cb042acb7"} Jul 26 16:00:06.725053 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:06.724705 2555 scope.go:117] "RemoveContainer" containerID="77180440e058628ac06629b1280a561feec6ea675caaae17f40b630a32a7be5d" Jul 26 16:00:06.734015 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:06.733996 2555 scope.go:117] "RemoveContainer" containerID="77180440e058628ac06629b1280a561feec6ea675caaae17f40b630a32a7be5d" Jul 26 16:00:06.734283 ip-10-0-130-21 kubenswrapper[2555]: E0726 16:00:06.734255 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77180440e058628ac06629b1280a561feec6ea675caaae17f40b630a32a7be5d\": container with ID starting with 77180440e058628ac06629b1280a561feec6ea675caaae17f40b630a32a7be5d not found: ID does not exist" containerID="77180440e058628ac06629b1280a561feec6ea675caaae17f40b630a32a7be5d" Jul 26 16:00:06.734335 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:06.734292 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77180440e058628ac06629b1280a561feec6ea675caaae17f40b630a32a7be5d"} err="failed to get container status \"77180440e058628ac06629b1280a561feec6ea675caaae17f40b630a32a7be5d\": rpc error: code = NotFound desc = could not find container \"77180440e058628ac06629b1280a561feec6ea675caaae17f40b630a32a7be5d\": container with ID starting with 77180440e058628ac06629b1280a561feec6ea675caaae17f40b630a32a7be5d not found: ID does not exist" Jul 26 16:00:06.740500 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:06.740482 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zmkg4"] Jul 26 16:00:06.744278 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:06.744259 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zmkg4"] Jul 26 16:00:08.070449 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:08.070414 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fec36f9a-e64e-4bfd-819b-e92ba317e9e8" path="/var/lib/kubelet/pods/fec36f9a-e64e-4bfd-819b-e92ba317e9e8/volumes" Jul 26 16:00:09.507755 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:09.507717 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 16:00:09.513677 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:09.513649 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-35252a-5xkzv"] Jul 26 16:00:09.515597 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:09.515577 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-35252a-5xkzv"] Jul 26 16:00:09.852396 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:09.852311 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 16:00:10.063745 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:10.063717 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f81ca00-4950-46ec-b9cd-e76c329ea576" path="/var/lib/kubelet/pods/0f81ca00-4950-46ec-b9cd-e76c329ea576/volumes" Jul 26 16:00:15.664367 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:15.664326 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 16:00:15.765102 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:15.765071 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 16:00:15.943890 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:15.943794 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 16:00:16.143709 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:16.143676 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 16:00:19.510524 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:19.510487 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d57e1d-jd9b6"] Jul 26 16:00:19.511023 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:19.511004 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fb61f27-8774-44c9-8883-909475554baf" containerName="cleanup" Jul 26 16:00:19.511023 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:19.511025 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fb61f27-8774-44c9-8883-909475554baf" containerName="cleanup" Jul 26 16:00:19.511182 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:19.511039 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d54f00bd-fc34-4e8d-a0e5-e889d3e7769d" containerName="cleanup" Jul 26 16:00:19.511182 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:19.511048 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d54f00bd-fc34-4e8d-a0e5-e889d3e7769d" containerName="cleanup" Jul 26 16:00:19.511182 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:19.511056 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b6a63a2-264c-48d5-9c10-e5a56bfbecb5" containerName="cleanup" Jul 26 16:00:19.511182 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:19.511064 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b6a63a2-264c-48d5-9c10-e5a56bfbecb5" containerName="cleanup" Jul 26 16:00:19.511182 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:19.511080 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b57f313-7122-430d-b4b3-238f46f69f15" containerName="maas-api" Jul 26 16:00:19.511182 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:19.511089 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b57f313-7122-430d-b4b3-238f46f69f15" containerName="maas-api" Jul 26 16:00:19.511182 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:19.511116 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20cb9db2-c276-424a-8b3d-45f275bf739d" containerName="cleanup" Jul 26 16:00:19.511182 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:19.511124 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="20cb9db2-c276-424a-8b3d-45f275bf739d" containerName="cleanup" Jul 26 16:00:19.511182 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:19.511138 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fec36f9a-e64e-4bfd-819b-e92ba317e9e8" containerName="registry-server" Jul 26 16:00:19.511182 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:19.511146 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="fec36f9a-e64e-4bfd-819b-e92ba317e9e8" containerName="registry-server" Jul 26 16:00:19.511182 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:19.511160 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1672b2bd-09f2-45ef-83aa-d58c0b3279ee" containerName="cleanup" Jul 26 16:00:19.511182 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:19.511168 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="1672b2bd-09f2-45ef-83aa-d58c0b3279ee" containerName="cleanup" Jul 26 16:00:19.511182 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:19.511180 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="114f21a5-0216-40c1-8038-842e96adc6e6" containerName="cleanup" Jul 26 16:00:19.511773 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:19.511188 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="114f21a5-0216-40c1-8038-842e96adc6e6" containerName="cleanup" Jul 26 16:00:19.511773 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:19.511284 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="114f21a5-0216-40c1-8038-842e96adc6e6" containerName="cleanup" Jul 26 16:00:19.511773 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:19.511299 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="1672b2bd-09f2-45ef-83aa-d58c0b3279ee" containerName="cleanup" Jul 26 16:00:19.511773 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:19.511318 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fb61f27-8774-44c9-8883-909475554baf" containerName="cleanup" Jul 26 16:00:19.511773 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:19.511328 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="fec36f9a-e64e-4bfd-819b-e92ba317e9e8" containerName="registry-server" Jul 26 16:00:19.511773 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:19.511336 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b6a63a2-264c-48d5-9c10-e5a56bfbecb5" containerName="cleanup" Jul 26 16:00:19.511773 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:19.511345 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="20cb9db2-c276-424a-8b3d-45f275bf739d" containerName="cleanup" Jul 26 16:00:19.511773 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:19.511354 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b57f313-7122-430d-b4b3-238f46f69f15" containerName="maas-api" Jul 26 16:00:19.511773 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:19.511363 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d54f00bd-fc34-4e8d-a0e5-e889d3e7769d" containerName="cleanup" Jul 26 16:00:19.516341 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:19.516287 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d57e1d-jd9b6" Jul 26 16:00:19.518675 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:19.518653 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d665x\"" Jul 26 16:00:19.519946 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:19.519913 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d57e1d-jd9b6"] Jul 26 16:00:19.642182 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:19.642155 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ed5c964-0bce-406d-9e4a-155f051a6545-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d57e1d-jd9b6\" (UID: \"6ed5c964-0bce-406d-9e4a-155f051a6545\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d57e1d-jd9b6" Jul 26 16:00:19.743544 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:19.743510 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ed5c964-0bce-406d-9e4a-155f051a6545-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d57e1d-jd9b6\" (UID: \"6ed5c964-0bce-406d-9e4a-155f051a6545\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d57e1d-jd9b6" Jul 26 16:00:19.744121 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:19.744101 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ed5c964-0bce-406d-9e4a-155f051a6545-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d57e1d-jd9b6\" (UID: \"6ed5c964-0bce-406d-9e4a-155f051a6545\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d57e1d-jd9b6" Jul 26 16:00:19.828119 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:19.828055 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d57e1d-jd9b6" Jul 26 16:00:20.159538 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:20.159512 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d57e1d-jd9b6"] Jul 26 16:00:20.161033 ip-10-0-130-21 kubenswrapper[2555]: W0726 16:00:20.161008 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ed5c964_0bce_406d_9e4a_155f051a6545.slice/crio-d4786fd05e0007091c145baebc06e2be62cb3b58fdb9514e147ffc70312f9fe9 WatchSource:0}: Error finding container d4786fd05e0007091c145baebc06e2be62cb3b58fdb9514e147ffc70312f9fe9: Status 404 returned error can't find the container with id d4786fd05e0007091c145baebc06e2be62cb3b58fdb9514e147ffc70312f9fe9 Jul 26 16:00:20.255435 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:20.255410 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9f857_1fbafda6-6242-400e-b2d4-24584608e5f5/ovn-acl-logging/0.log" Jul 26 16:00:20.262591 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:20.262572 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9f857_1fbafda6-6242-400e-b2d4-24584608e5f5/ovn-acl-logging/0.log" Jul 26 16:00:20.797667 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:20.797572 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d57e1d-jd9b6" event={"ID":"6ed5c964-0bce-406d-9e4a-155f051a6545","Type":"ContainerStarted","Data":"019a92352c0a685b1538a32bd147a56467947c357f33f3dec4f742cae6f4220f"} Jul 26 16:00:20.797667 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:20.797630 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d57e1d-jd9b6" event={"ID":"6ed5c964-0bce-406d-9e4a-155f051a6545","Type":"ContainerStarted","Data":"d4786fd05e0007091c145baebc06e2be62cb3b58fdb9514e147ffc70312f9fe9"} Jul 26 16:00:20.811053 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:20.811009 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d57e1d-jd9b6" podStartSLOduration=1.810990987 podStartE2EDuration="1.810990987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:00:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:00:20.810183703 +0000 UTC m=+3301.234030565" watchObservedRunningTime="2026-07-26 16:00:20.810990987 +0000 UTC m=+3301.234837859" Jul 26 16:00:21.803880 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:21.803849 2555 generic.go:356] "Generic (PLEG): container finished" podID="6ed5c964-0bce-406d-9e4a-155f051a6545" containerID="019a92352c0a685b1538a32bd147a56467947c357f33f3dec4f742cae6f4220f" exitCode=0 Jul 26 16:00:21.804259 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:21.803908 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d57e1d-jd9b6" event={"ID":"6ed5c964-0bce-406d-9e4a-155f051a6545","Type":"ContainerDied","Data":"019a92352c0a685b1538a32bd147a56467947c357f33f3dec4f742cae6f4220f"} Jul 26 16:00:22.937487 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:22.937460 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d57e1d-jd9b6" Jul 26 16:00:23.073832 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:23.073753 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ed5c964-0bce-406d-9e4a-155f051a6545-maas-api-service-ca\") pod \"6ed5c964-0bce-406d-9e4a-155f051a6545\" (UID: \"6ed5c964-0bce-406d-9e4a-155f051a6545\") " Jul 26 16:00:23.074111 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:23.074089 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ed5c964-0bce-406d-9e4a-155f051a6545-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6ed5c964-0bce-406d-9e4a-155f051a6545" (UID: "6ed5c964-0bce-406d-9e4a-155f051a6545"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:00:23.174631 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:23.174597 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ed5c964-0bce-406d-9e4a-155f051a6545-maas-api-service-ca\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 16:00:23.814616 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:23.814585 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d57e1d-jd9b6" Jul 26 16:00:23.814616 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:23.814602 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d57e1d-jd9b6" event={"ID":"6ed5c964-0bce-406d-9e4a-155f051a6545","Type":"ContainerDied","Data":"d4786fd05e0007091c145baebc06e2be62cb3b58fdb9514e147ffc70312f9fe9"} Jul 26 16:00:23.814872 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:23.814651 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4786fd05e0007091c145baebc06e2be62cb3b58fdb9514e147ffc70312f9fe9" Jul 26 16:00:35.106414 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:35.106374 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 16:00:35.119267 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:35.119242 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d57e1d-29751360-f74hp"] Jul 26 16:00:35.122467 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:35.122448 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d57e1d-29751360-f74hp"] Jul 26 16:00:35.217476 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:35.217434 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 16:00:35.308080 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:35.308042 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 16:00:36.063806 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:36.063768 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d54f00bd-fc34-4e8d-a0e5-e889d3e7769d" path="/var/lib/kubelet/pods/d54f00bd-fc34-4e8d-a0e5-e889d3e7769d/volumes" Jul 26 16:00:39.589727 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:39.589692 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 16:00:39.596039 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:39.596012 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d57e1d-jd9b6"] Jul 26 16:00:39.598289 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:39.598264 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d57e1d-jd9b6"] Jul 26 16:00:39.965728 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:39.965688 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 16:00:40.064341 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:40.064316 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ed5c964-0bce-406d-9e4a-155f051a6545" path="/var/lib/kubelet/pods/6ed5c964-0bce-406d-9e4a-155f051a6545/volumes" Jul 26 16:00:44.945080 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:44.944991 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-d57e1d/ipp-route-d57e1d-kserve-67444b75f9-j48ms"] Jul 26 16:00:44.989366 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:44.989332 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 16:00:57.685628 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:57.685593 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 16:00:59.963362 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:00:59.963295 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-d57e1d/ipp-route-d57e1d-kserve-67444b75f9-j48ms" podUID="1999071c-78d9-494e-9128-3b72e9c593b8" containerName="main" containerID="cri-o://639dc7c8b2e0b2c774b0d2bb94a5a6d9481b0e267afd1d6ab0c12204e13d3257" gracePeriod=45 Jul 26 16:01:00.217290 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:00.217239 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-d57e1d/ipp-route-d57e1d-kserve-67444b75f9-j48ms" Jul 26 16:01:00.365898 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:00.365876 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1999071c-78d9-494e-9128-3b72e9c593b8-dshm\") pod \"1999071c-78d9-494e-9128-3b72e9c593b8\" (UID: \"1999071c-78d9-494e-9128-3b72e9c593b8\") " Jul 26 16:01:00.366070 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:00.365907 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1999071c-78d9-494e-9128-3b72e9c593b8-home\") pod \"1999071c-78d9-494e-9128-3b72e9c593b8\" (UID: \"1999071c-78d9-494e-9128-3b72e9c593b8\") " Jul 26 16:01:00.366070 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:00.365925 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1999071c-78d9-494e-9128-3b72e9c593b8-model-cache\") pod \"1999071c-78d9-494e-9128-3b72e9c593b8\" (UID: \"1999071c-78d9-494e-9128-3b72e9c593b8\") " Jul 26 16:01:00.366070 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:00.366022 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gnr24\" (UniqueName: \"kubernetes.io/projected/1999071c-78d9-494e-9128-3b72e9c593b8-kube-api-access-gnr24\") pod \"1999071c-78d9-494e-9128-3b72e9c593b8\" (UID: \"1999071c-78d9-494e-9128-3b72e9c593b8\") " Jul 26 16:01:00.366243 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:00.366173 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1999071c-78d9-494e-9128-3b72e9c593b8-tls-certs\") pod \"1999071c-78d9-494e-9128-3b72e9c593b8\" (UID: \"1999071c-78d9-494e-9128-3b72e9c593b8\") " Jul 26 16:01:00.366243 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:00.366210 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1999071c-78d9-494e-9128-3b72e9c593b8-home" (OuterVolumeSpecName: "home") pod "1999071c-78d9-494e-9128-3b72e9c593b8" (UID: "1999071c-78d9-494e-9128-3b72e9c593b8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:01:00.366243 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:00.366230 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1999071c-78d9-494e-9128-3b72e9c593b8-tmp-dir\") pod \"1999071c-78d9-494e-9128-3b72e9c593b8\" (UID: \"1999071c-78d9-494e-9128-3b72e9c593b8\") " Jul 26 16:01:00.366384 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:00.366257 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1999071c-78d9-494e-9128-3b72e9c593b8-model-cache" (OuterVolumeSpecName: "model-cache") pod "1999071c-78d9-494e-9128-3b72e9c593b8" (UID: "1999071c-78d9-494e-9128-3b72e9c593b8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:01:00.366525 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:00.366503 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1999071c-78d9-494e-9128-3b72e9c593b8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1999071c-78d9-494e-9128-3b72e9c593b8" (UID: "1999071c-78d9-494e-9128-3b72e9c593b8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:01:00.366620 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:00.366507 2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1999071c-78d9-494e-9128-3b72e9c593b8-home\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 16:01:00.366620 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:00.366552 2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1999071c-78d9-494e-9128-3b72e9c593b8-model-cache\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 16:01:00.368131 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:00.368108 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1999071c-78d9-494e-9128-3b72e9c593b8-kube-api-access-gnr24" (OuterVolumeSpecName: "kube-api-access-gnr24") pod "1999071c-78d9-494e-9128-3b72e9c593b8" (UID: "1999071c-78d9-494e-9128-3b72e9c593b8"). InnerVolumeSpecName "kube-api-access-gnr24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:01:00.368241 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:00.368221 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1999071c-78d9-494e-9128-3b72e9c593b8-dshm" (OuterVolumeSpecName: "dshm") pod "1999071c-78d9-494e-9128-3b72e9c593b8" (UID: "1999071c-78d9-494e-9128-3b72e9c593b8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:01:00.368283 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:00.368242 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1999071c-78d9-494e-9128-3b72e9c593b8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1999071c-78d9-494e-9128-3b72e9c593b8" (UID: "1999071c-78d9-494e-9128-3b72e9c593b8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 16:01:00.467639 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:00.467585 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnr24\" (UniqueName: \"kubernetes.io/projected/1999071c-78d9-494e-9128-3b72e9c593b8-kube-api-access-gnr24\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 16:01:00.467639 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:00.467607 2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1999071c-78d9-494e-9128-3b72e9c593b8-tls-certs\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 16:01:00.467639 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:00.467620 2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1999071c-78d9-494e-9128-3b72e9c593b8-tmp-dir\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 16:01:00.467639 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:00.467627 2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1999071c-78d9-494e-9128-3b72e9c593b8-dshm\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 16:01:01.000952 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:01.000906 2555 generic.go:356] "Generic (PLEG): container finished" podID="1999071c-78d9-494e-9128-3b72e9c593b8" containerID="639dc7c8b2e0b2c774b0d2bb94a5a6d9481b0e267afd1d6ab0c12204e13d3257" exitCode=0 Jul 26 16:01:01.001387 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:01.000969 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-d57e1d/ipp-route-d57e1d-kserve-67444b75f9-j48ms" Jul 26 16:01:01.001387 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:01.000989 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-d57e1d/ipp-route-d57e1d-kserve-67444b75f9-j48ms" event={"ID":"1999071c-78d9-494e-9128-3b72e9c593b8","Type":"ContainerDied","Data":"639dc7c8b2e0b2c774b0d2bb94a5a6d9481b0e267afd1d6ab0c12204e13d3257"} Jul 26 16:01:01.001387 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:01.001027 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-d57e1d/ipp-route-d57e1d-kserve-67444b75f9-j48ms" event={"ID":"1999071c-78d9-494e-9128-3b72e9c593b8","Type":"ContainerDied","Data":"7d4c60afaac7400e46986be893d4a74836a0d01e1a1c50468ce1cc8facf79aae"} Jul 26 16:01:01.001387 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:01.001044 2555 scope.go:117] "RemoveContainer" containerID="639dc7c8b2e0b2c774b0d2bb94a5a6d9481b0e267afd1d6ab0c12204e13d3257" Jul 26 16:01:01.010585 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:01.010577 2555 scope.go:117] "RemoveContainer" containerID="639dc7c8b2e0b2c774b0d2bb94a5a6d9481b0e267afd1d6ab0c12204e13d3257" Jul 26 16:01:01.010791 ip-10-0-130-21 kubenswrapper[2555]: E0726 16:01:01.010775 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"639dc7c8b2e0b2c774b0d2bb94a5a6d9481b0e267afd1d6ab0c12204e13d3257\": container with ID starting with 639dc7c8b2e0b2c774b0d2bb94a5a6d9481b0e267afd1d6ab0c12204e13d3257 not found: ID does not exist" containerID="639dc7c8b2e0b2c774b0d2bb94a5a6d9481b0e267afd1d6ab0c12204e13d3257" Jul 26 16:01:01.010833 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:01.010797 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"639dc7c8b2e0b2c774b0d2bb94a5a6d9481b0e267afd1d6ab0c12204e13d3257"} err="failed to get container status \"639dc7c8b2e0b2c774b0d2bb94a5a6d9481b0e267afd1d6ab0c12204e13d3257\": rpc error: code = NotFound desc = could not find container \"639dc7c8b2e0b2c774b0d2bb94a5a6d9481b0e267afd1d6ab0c12204e13d3257\": container with ID starting with 639dc7c8b2e0b2c774b0d2bb94a5a6d9481b0e267afd1d6ab0c12204e13d3257 not found: ID does not exist" Jul 26 16:01:01.021569 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:01.021507 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-d57e1d/ipp-route-d57e1d-kserve-67444b75f9-j48ms"] Jul 26 16:01:01.023859 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:01.023835 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-d57e1d/ipp-route-d57e1d-kserve-67444b75f9-j48ms"] Jul 26 16:01:02.063520 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:02.063489 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1999071c-78d9-494e-9128-3b72e9c593b8" path="/var/lib/kubelet/pods/1999071c-78d9-494e-9128-3b72e9c593b8/volumes" Jul 26 16:01:21.395073 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:21.395026 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 16:01:22.006479 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:22.006437 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 16:01:25.844916 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:25.844875 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 16:01:27.660135 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:27.660095 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 16:01:33.294752 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:33.294715 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 16:01:33.393288 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:33.393240 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 16:01:33.596210 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:33.596113 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 16:01:33.791202 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:33.791167 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 16:01:33.908903 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:33.908869 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cb6b99-nvjkr"] Jul 26 16:01:33.909255 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:33.909242 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1999071c-78d9-494e-9128-3b72e9c593b8" containerName="main" Jul 26 16:01:33.909298 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:33.909257 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="1999071c-78d9-494e-9128-3b72e9c593b8" containerName="main" Jul 26 16:01:33.909298 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:33.909267 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ed5c964-0bce-406d-9e4a-155f051a6545" containerName="revoke-keys" Jul 26 16:01:33.909298 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:33.909272 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ed5c964-0bce-406d-9e4a-155f051a6545" containerName="revoke-keys" Jul 26 16:01:33.909392 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:33.909330 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ed5c964-0bce-406d-9e4a-155f051a6545" containerName="revoke-keys" Jul 26 16:01:33.909392 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:33.909339 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="1999071c-78d9-494e-9128-3b72e9c593b8" containerName="main" Jul 26 16:01:33.912340 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:33.912323 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cb6b99-nvjkr" Jul 26 16:01:33.914315 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:33.914297 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d665x\"" Jul 26 16:01:33.918274 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:33.918249 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cb6b99-nvjkr"] Jul 26 16:01:33.928144 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:33.928122 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c1d298c-fc77-4eeb-a399-4e37ebe3eb08-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-cb6b99-nvjkr\" (UID: \"0c1d298c-fc77-4eeb-a399-4e37ebe3eb08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cb6b99-nvjkr" Jul 26 16:01:34.028470 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:34.028439 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c1d298c-fc77-4eeb-a399-4e37ebe3eb08-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-cb6b99-nvjkr\" (UID: \"0c1d298c-fc77-4eeb-a399-4e37ebe3eb08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cb6b99-nvjkr" Jul 26 16:01:34.029065 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:34.029048 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c1d298c-fc77-4eeb-a399-4e37ebe3eb08-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-cb6b99-nvjkr\" (UID: \"0c1d298c-fc77-4eeb-a399-4e37ebe3eb08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cb6b99-nvjkr" Jul 26 16:01:34.224126 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:34.224056 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cb6b99-nvjkr" Jul 26 16:01:34.351425 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:34.351399 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cb6b99-nvjkr"] Jul 26 16:01:34.353495 ip-10-0-130-21 kubenswrapper[2555]: W0726 16:01:34.353464 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c1d298c_fc77_4eeb_a399_4e37ebe3eb08.slice/crio-bd5cb6c00c9b0509296031858ec9729f5b1f3040494cc9d68b7d739040e87ddf WatchSource:0}: Error finding container bd5cb6c00c9b0509296031858ec9729f5b1f3040494cc9d68b7d739040e87ddf: Status 404 returned error can't find the container with id bd5cb6c00c9b0509296031858ec9729f5b1f3040494cc9d68b7d739040e87ddf Jul 26 16:01:35.177826 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:35.177790 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cb6b99-nvjkr" event={"ID":"0c1d298c-fc77-4eeb-a399-4e37ebe3eb08","Type":"ContainerStarted","Data":"76e43662f44d8dfb9b203851e038ec73904bc637565a847b69238dc43c593f37"} Jul 26 16:01:35.177991 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:35.177858 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cb6b99-nvjkr" event={"ID":"0c1d298c-fc77-4eeb-a399-4e37ebe3eb08","Type":"ContainerStarted","Data":"bd5cb6c00c9b0509296031858ec9729f5b1f3040494cc9d68b7d739040e87ddf"} Jul 26 16:01:35.191906 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:35.191863 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cb6b99-nvjkr" podStartSLOduration=2.191849693 podStartE2EDuration="2.191849693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:01:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:01:35.19060403 +0000 UTC m=+3375.614450918" watchObservedRunningTime="2026-07-26 16:01:35.191849693 +0000 UTC m=+3375.615696563" Jul 26 16:01:36.187821 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:36.187778 2555 generic.go:356] "Generic (PLEG): container finished" podID="0c1d298c-fc77-4eeb-a399-4e37ebe3eb08" containerID="76e43662f44d8dfb9b203851e038ec73904bc637565a847b69238dc43c593f37" exitCode=0 Jul 26 16:01:36.188290 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:36.187870 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cb6b99-nvjkr" event={"ID":"0c1d298c-fc77-4eeb-a399-4e37ebe3eb08","Type":"ContainerDied","Data":"76e43662f44d8dfb9b203851e038ec73904bc637565a847b69238dc43c593f37"} Jul 26 16:01:37.325360 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:37.325334 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cb6b99-nvjkr" Jul 26 16:01:37.356686 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:37.356662 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c1d298c-fc77-4eeb-a399-4e37ebe3eb08-maas-api-service-ca\") pod \"0c1d298c-fc77-4eeb-a399-4e37ebe3eb08\" (UID: \"0c1d298c-fc77-4eeb-a399-4e37ebe3eb08\") " Jul 26 16:01:37.356996 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:37.356976 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c1d298c-fc77-4eeb-a399-4e37ebe3eb08-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0c1d298c-fc77-4eeb-a399-4e37ebe3eb08" (UID: "0c1d298c-fc77-4eeb-a399-4e37ebe3eb08"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:01:37.457917 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:37.457861 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c1d298c-fc77-4eeb-a399-4e37ebe3eb08-maas-api-service-ca\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 16:01:38.199678 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:38.199652 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cb6b99-nvjkr" Jul 26 16:01:38.199678 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:38.199665 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cb6b99-nvjkr" event={"ID":"0c1d298c-fc77-4eeb-a399-4e37ebe3eb08","Type":"ContainerDied","Data":"bd5cb6c00c9b0509296031858ec9729f5b1f3040494cc9d68b7d739040e87ddf"} Jul 26 16:01:38.199851 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:38.199710 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd5cb6c00c9b0509296031858ec9729f5b1f3040494cc9d68b7d739040e87ddf" Jul 26 16:01:44.335079 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:44.334788 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-cb6b99-64cb4746c-29zb6"] Jul 26 16:01:44.335546 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:44.335137 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cb6b99-64cb4746c-29zb6" podUID="97300bf8-0063-4987-b27b-a915287153a2" containerName="maas-api" containerID="cri-o://ebc03874dac4c4fcb472a7bf35bd7f16ecb089508720373ec6335025de83bf71" gracePeriod=30 Jul 26 16:01:44.359146 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:44.359113 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cb6b99-29751360-v4gl8"] Jul 26 16:01:44.361130 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:44.361107 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cb6b99-29751360-v4gl8"] Jul 26 16:01:44.362546 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:44.362524 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 16:01:44.431543 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:44.431511 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 16:01:44.499921 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:44.499893 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 16:01:44.583672 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:44.583641 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cb6b99-64cb4746c-29zb6" Jul 26 16:01:44.608604 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:44.608547 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97300bf8-0063-4987-b27b-a915287153a2-openshift-service-ca\") pod \"97300bf8-0063-4987-b27b-a915287153a2\" (UID: \"97300bf8-0063-4987-b27b-a915287153a2\") " Jul 26 16:01:44.608715 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:44.608602 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lkxbx\" (UniqueName: \"kubernetes.io/projected/97300bf8-0063-4987-b27b-a915287153a2-kube-api-access-lkxbx\") pod \"97300bf8-0063-4987-b27b-a915287153a2\" (UID: \"97300bf8-0063-4987-b27b-a915287153a2\") " Jul 26 16:01:44.608715 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:44.608674 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97300bf8-0063-4987-b27b-a915287153a2-maas-api-tls\") pod \"97300bf8-0063-4987-b27b-a915287153a2\" (UID: \"97300bf8-0063-4987-b27b-a915287153a2\") " Jul 26 16:01:44.608995 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:44.608961 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97300bf8-0063-4987-b27b-a915287153a2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "97300bf8-0063-4987-b27b-a915287153a2" (UID: "97300bf8-0063-4987-b27b-a915287153a2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:01:44.610533 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:44.610510 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97300bf8-0063-4987-b27b-a915287153a2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "97300bf8-0063-4987-b27b-a915287153a2" (UID: "97300bf8-0063-4987-b27b-a915287153a2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 16:01:44.610609 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:44.610563 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97300bf8-0063-4987-b27b-a915287153a2-kube-api-access-lkxbx" (OuterVolumeSpecName: "kube-api-access-lkxbx") pod "97300bf8-0063-4987-b27b-a915287153a2" (UID: "97300bf8-0063-4987-b27b-a915287153a2"). InnerVolumeSpecName "kube-api-access-lkxbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:01:44.710245 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:44.710216 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97300bf8-0063-4987-b27b-a915287153a2-maas-api-tls\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 16:01:44.710327 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:44.710246 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97300bf8-0063-4987-b27b-a915287153a2-openshift-service-ca\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 16:01:44.710327 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:44.710261 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkxbx\" (UniqueName: \"kubernetes.io/projected/97300bf8-0063-4987-b27b-a915287153a2-kube-api-access-lkxbx\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 16:01:45.236119 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:45.236081 2555 generic.go:356] "Generic (PLEG): container finished" podID="97300bf8-0063-4987-b27b-a915287153a2" containerID="ebc03874dac4c4fcb472a7bf35bd7f16ecb089508720373ec6335025de83bf71" exitCode=0 Jul 26 16:01:45.236291 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:45.236129 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cb6b99-64cb4746c-29zb6" Jul 26 16:01:45.236291 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:45.236151 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cb6b99-64cb4746c-29zb6" event={"ID":"97300bf8-0063-4987-b27b-a915287153a2","Type":"ContainerDied","Data":"ebc03874dac4c4fcb472a7bf35bd7f16ecb089508720373ec6335025de83bf71"} Jul 26 16:01:45.236291 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:45.236201 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cb6b99-64cb4746c-29zb6" event={"ID":"97300bf8-0063-4987-b27b-a915287153a2","Type":"ContainerDied","Data":"a7162495307e7a54c57ce169bee8bb71b0e7f5fa668d390ad8f00ddaf879da5f"} Jul 26 16:01:45.236291 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:45.236230 2555 scope.go:117] "RemoveContainer" containerID="ebc03874dac4c4fcb472a7bf35bd7f16ecb089508720373ec6335025de83bf71" Jul 26 16:01:45.245169 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:45.245152 2555 scope.go:117] "RemoveContainer" containerID="ebc03874dac4c4fcb472a7bf35bd7f16ecb089508720373ec6335025de83bf71" Jul 26 16:01:45.245400 ip-10-0-130-21 kubenswrapper[2555]: E0726 16:01:45.245379 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebc03874dac4c4fcb472a7bf35bd7f16ecb089508720373ec6335025de83bf71\": container with ID starting with ebc03874dac4c4fcb472a7bf35bd7f16ecb089508720373ec6335025de83bf71 not found: ID does not exist" containerID="ebc03874dac4c4fcb472a7bf35bd7f16ecb089508720373ec6335025de83bf71" Jul 26 16:01:45.245456 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:45.245407 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebc03874dac4c4fcb472a7bf35bd7f16ecb089508720373ec6335025de83bf71"} err="failed to get container status \"ebc03874dac4c4fcb472a7bf35bd7f16ecb089508720373ec6335025de83bf71\": rpc error: code = NotFound desc = could not find container \"ebc03874dac4c4fcb472a7bf35bd7f16ecb089508720373ec6335025de83bf71\": container with ID starting with ebc03874dac4c4fcb472a7bf35bd7f16ecb089508720373ec6335025de83bf71 not found: ID does not exist" Jul 26 16:01:45.255480 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:45.255457 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-cb6b99-64cb4746c-29zb6"] Jul 26 16:01:45.258815 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:45.258795 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-cb6b99-64cb4746c-29zb6"] Jul 26 16:01:46.064128 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:46.064095 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fb61f27-8774-44c9-8883-909475554baf" path="/var/lib/kubelet/pods/0fb61f27-8774-44c9-8883-909475554baf/volumes" Jul 26 16:01:46.064503 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:46.064427 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97300bf8-0063-4987-b27b-a915287153a2" path="/var/lib/kubelet/pods/97300bf8-0063-4987-b27b-a915287153a2/volumes" Jul 26 16:01:48.978101 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:48.978065 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cb6b99-nvjkr"] Jul 26 16:01:48.980992 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:48.980960 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 16:01:48.981601 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:48.981582 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cb6b99-nvjkr"] Jul 26 16:01:49.355736 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:49.355657 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 16:01:50.064180 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:50.064144 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c1d298c-fc77-4eeb-a399-4e37ebe3eb08" path="/var/lib/kubelet/pods/0c1d298c-fc77-4eeb-a399-4e37ebe3eb08/volumes" Jul 26 16:01:55.121944 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:55.121905 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 16:01:55.187394 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:55.187366 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 16:01:55.400681 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:55.400597 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 16:01:55.638614 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:55.638583 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 16:01:55.773889 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:55.773813 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a20d61-v7q7c"] Jul 26 16:01:55.774370 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:55.774351 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c1d298c-fc77-4eeb-a399-4e37ebe3eb08" containerName="revoke-keys" Jul 26 16:01:55.774432 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:55.774375 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c1d298c-fc77-4eeb-a399-4e37ebe3eb08" containerName="revoke-keys" Jul 26 16:01:55.774432 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:55.774393 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97300bf8-0063-4987-b27b-a915287153a2" containerName="maas-api" Jul 26 16:01:55.774432 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:55.774402 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="97300bf8-0063-4987-b27b-a915287153a2" containerName="maas-api" Jul 26 16:01:55.774551 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:55.774489 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c1d298c-fc77-4eeb-a399-4e37ebe3eb08" containerName="revoke-keys" Jul 26 16:01:55.774551 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:55.774508 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="97300bf8-0063-4987-b27b-a915287153a2" containerName="maas-api" Jul 26 16:01:55.779436 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:55.779416 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a20d61-v7q7c" Jul 26 16:01:55.782223 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:55.781894 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d665x\"" Jul 26 16:01:55.782223 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:55.781897 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 26 16:01:55.792230 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:55.788580 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a20d61-v7q7c"] Jul 26 16:01:55.899066 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:55.899033 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c775980a-1d13-40cd-998c-6501d2b98b41-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-a20d61-v7q7c\" (UID: \"c775980a-1d13-40cd-998c-6501d2b98b41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a20d61-v7q7c" Jul 26 16:01:55.999716 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:55.999687 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c775980a-1d13-40cd-998c-6501d2b98b41-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-a20d61-v7q7c\" (UID: \"c775980a-1d13-40cd-998c-6501d2b98b41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a20d61-v7q7c" Jul 26 16:01:56.000261 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:56.000244 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c775980a-1d13-40cd-998c-6501d2b98b41-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-a20d61-v7q7c\" (UID: \"c775980a-1d13-40cd-998c-6501d2b98b41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a20d61-v7q7c" Jul 26 16:01:56.097054 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:56.096991 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a20d61-v7q7c" Jul 26 16:01:56.224041 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:56.224018 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a20d61-v7q7c"] Jul 26 16:01:56.226142 ip-10-0-130-21 kubenswrapper[2555]: W0726 16:01:56.226115 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc775980a_1d13_40cd_998c_6501d2b98b41.slice/crio-1b90d288433c24b200c31238bcea7437d67af79b78d8a6b2d4f88aee007bff71 WatchSource:0}: Error finding container 1b90d288433c24b200c31238bcea7437d67af79b78d8a6b2d4f88aee007bff71: Status 404 returned error can't find the container with id 1b90d288433c24b200c31238bcea7437d67af79b78d8a6b2d4f88aee007bff71 Jul 26 16:01:56.294985 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:56.294956 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a20d61-v7q7c" event={"ID":"c775980a-1d13-40cd-998c-6501d2b98b41","Type":"ContainerStarted","Data":"1b90d288433c24b200c31238bcea7437d67af79b78d8a6b2d4f88aee007bff71"} Jul 26 16:01:57.301252 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:57.301173 2555 generic.go:356] "Generic (PLEG): container finished" podID="c775980a-1d13-40cd-998c-6501d2b98b41" containerID="48407b6e6414209e86935c5b8e97204d8baccc6a3873ec81e41ed2f505f83485" exitCode=0 Jul 26 16:01:57.301582 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:57.301259 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a20d61-v7q7c" event={"ID":"c775980a-1d13-40cd-998c-6501d2b98b41","Type":"ContainerDied","Data":"48407b6e6414209e86935c5b8e97204d8baccc6a3873ec81e41ed2f505f83485"} Jul 26 16:01:58.433383 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:58.433360 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a20d61-v7q7c" Jul 26 16:01:58.518914 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:58.518884 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c775980a-1d13-40cd-998c-6501d2b98b41-maas-api-service-ca\") pod \"c775980a-1d13-40cd-998c-6501d2b98b41\" (UID: \"c775980a-1d13-40cd-998c-6501d2b98b41\") " Jul 26 16:01:58.519229 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:58.519208 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c775980a-1d13-40cd-998c-6501d2b98b41-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c775980a-1d13-40cd-998c-6501d2b98b41" (UID: "c775980a-1d13-40cd-998c-6501d2b98b41"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:01:58.620076 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:58.620051 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c775980a-1d13-40cd-998c-6501d2b98b41-maas-api-service-ca\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 16:01:59.311868 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:59.311836 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a20d61-v7q7c" Jul 26 16:01:59.312077 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:59.311831 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a20d61-v7q7c" event={"ID":"c775980a-1d13-40cd-998c-6501d2b98b41","Type":"ContainerDied","Data":"1b90d288433c24b200c31238bcea7437d67af79b78d8a6b2d4f88aee007bff71"} Jul 26 16:01:59.312077 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:01:59.312003 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b90d288433c24b200c31238bcea7437d67af79b78d8a6b2d4f88aee007bff71" Jul 26 16:02:06.280238 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:06.280203 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a20d61-29751360-fqvrk"] Jul 26 16:02:06.302623 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:06.302589 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a20d61-29751360-fqvrk"] Jul 26 16:02:06.314343 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:06.314294 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 16:02:06.432513 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:06.432478 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 16:02:08.064404 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:08.064367 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="114f21a5-0216-40c1-8038-842e96adc6e6" path="/var/lib/kubelet/pods/114f21a5-0216-40c1-8038-842e96adc6e6/volumes" Jul 26 16:02:10.854643 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:10.854609 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a20d61-v7q7c"] Jul 26 16:02:10.857694 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:10.857662 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a20d61-v7q7c"] Jul 26 16:02:10.861046 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:10.861021 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 16:02:11.224838 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:11.224803 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 16:02:12.063392 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:12.063325 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c775980a-1d13-40cd-998c-6501d2b98b41" path="/var/lib/kubelet/pods/c775980a-1d13-40cd-998c-6501d2b98b41/volumes" Jul 26 16:02:17.144049 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:17.144012 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 16:02:17.224772 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:17.224746 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 16:02:17.415797 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:17.415710 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 16:02:17.649634 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:17.649597 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 16:02:17.766120 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:17.766026 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-60c814-6cns7"] Jul 26 16:02:17.766622 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:17.766601 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c775980a-1d13-40cd-998c-6501d2b98b41" containerName="revoke-keys" Jul 26 16:02:17.766668 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:17.766627 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c775980a-1d13-40cd-998c-6501d2b98b41" containerName="revoke-keys" Jul 26 16:02:17.766752 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:17.766741 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c775980a-1d13-40cd-998c-6501d2b98b41" containerName="revoke-keys" Jul 26 16:02:17.770104 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:17.770076 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-60c814-6cns7" Jul 26 16:02:17.772573 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:17.772545 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d665x\"" Jul 26 16:02:17.772834 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:17.772788 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 26 16:02:17.774989 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:17.774953 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-60c814-6cns7"] Jul 26 16:02:17.878797 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:17.878761 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fc592d8-5b68-42cc-a137-1a848de13442-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-60c814-6cns7\" (UID: \"9fc592d8-5b68-42cc-a137-1a848de13442\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-60c814-6cns7" Jul 26 16:02:17.980222 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:17.980197 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fc592d8-5b68-42cc-a137-1a848de13442-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-60c814-6cns7\" (UID: \"9fc592d8-5b68-42cc-a137-1a848de13442\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-60c814-6cns7" Jul 26 16:02:17.980794 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:17.980774 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fc592d8-5b68-42cc-a137-1a848de13442-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-60c814-6cns7\" (UID: \"9fc592d8-5b68-42cc-a137-1a848de13442\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-60c814-6cns7" Jul 26 16:02:18.083966 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:18.083877 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-60c814-6cns7" Jul 26 16:02:18.411042 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:18.411018 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-60c814-6cns7"] Jul 26 16:02:18.413817 ip-10-0-130-21 kubenswrapper[2555]: W0726 16:02:18.413788 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fc592d8_5b68_42cc_a137_1a848de13442.slice/crio-1e847b16a7742091bc948759a6b5afcb8986f8bbc3550a9ceed8190f4aa56fbd WatchSource:0}: Error finding container 1e847b16a7742091bc948759a6b5afcb8986f8bbc3550a9ceed8190f4aa56fbd: Status 404 returned error can't find the container with id 1e847b16a7742091bc948759a6b5afcb8986f8bbc3550a9ceed8190f4aa56fbd Jul 26 16:02:19.423855 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:19.423823 2555 generic.go:356] "Generic (PLEG): container finished" podID="9fc592d8-5b68-42cc-a137-1a848de13442" containerID="d33a5def4112f069479231ddef8522a5f27f3dc4d3a4369c5aac9d0f1da78beb" exitCode=0 Jul 26 16:02:19.424267 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:19.423881 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-60c814-6cns7" event={"ID":"9fc592d8-5b68-42cc-a137-1a848de13442","Type":"ContainerDied","Data":"d33a5def4112f069479231ddef8522a5f27f3dc4d3a4369c5aac9d0f1da78beb"} Jul 26 16:02:19.424267 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:19.423917 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-60c814-6cns7" event={"ID":"9fc592d8-5b68-42cc-a137-1a848de13442","Type":"ContainerStarted","Data":"1e847b16a7742091bc948759a6b5afcb8986f8bbc3550a9ceed8190f4aa56fbd"} Jul 26 16:02:20.561265 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:20.561241 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-60c814-6cns7" Jul 26 16:02:20.700240 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:20.700172 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fc592d8-5b68-42cc-a137-1a848de13442-maas-api-service-ca\") pod \"9fc592d8-5b68-42cc-a137-1a848de13442\" (UID: \"9fc592d8-5b68-42cc-a137-1a848de13442\") " Jul 26 16:02:20.700502 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:20.700478 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fc592d8-5b68-42cc-a137-1a848de13442-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9fc592d8-5b68-42cc-a137-1a848de13442" (UID: "9fc592d8-5b68-42cc-a137-1a848de13442"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:02:20.800865 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:20.800834 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fc592d8-5b68-42cc-a137-1a848de13442-maas-api-service-ca\") on node \"ip-10-0-130-21.ec2.internal\" DevicePath \"\"" Jul 26 16:02:21.434913 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:21.434887 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-60c814-6cns7" Jul 26 16:02:21.434913 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:21.434887 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-60c814-6cns7" event={"ID":"9fc592d8-5b68-42cc-a137-1a848de13442","Type":"ContainerDied","Data":"1e847b16a7742091bc948759a6b5afcb8986f8bbc3550a9ceed8190f4aa56fbd"} Jul 26 16:02:21.435204 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:21.434957 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e847b16a7742091bc948759a6b5afcb8986f8bbc3550a9ceed8190f4aa56fbd" Jul 26 16:02:28.346832 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:28.346788 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 16:02:28.351939 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:28.351904 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-60c814-29751360-sgxhq"] Jul 26 16:02:28.356064 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:28.356044 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-60c814-29751360-sgxhq"] Jul 26 16:02:28.440770 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:28.440740 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 16:02:28.540780 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:28.540750 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 16:02:30.063612 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:30.063578 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b6a63a2-264c-48d5-9c10-e5a56bfbecb5" path="/var/lib/kubelet/pods/8b6a63a2-264c-48d5-9c10-e5a56bfbecb5/volumes" Jul 26 16:02:32.873126 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:32.873083 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 16:02:32.875489 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:32.873967 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-60c814-6cns7"] Jul 26 16:02:32.876243 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:32.876223 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-60c814-6cns7"] Jul 26 16:02:33.256355 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:33.256274 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 16:02:34.063915 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:34.063882 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fc592d8-5b68-42cc-a137-1a848de13442" path="/var/lib/kubelet/pods/9fc592d8-5b68-42cc-a137-1a848de13442/volumes" Jul 26 16:02:39.101558 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:39.101520 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 16:02:39.201711 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:39.201677 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 16:02:39.382847 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:39.382815 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 16:02:39.591037 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:39.591004 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 16:02:40.032565 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:40.032529 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c6bc86c6d-74nmr_4096ea8f-3158-4364-9858-346b97b711a7/authorino/0.log" Jul 26 16:02:45.580627 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:45.580593 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-766dd44bbb-2dzqh_c5dceda4-6cc2-47f0-ade7-574acf5bd536/manager/0.log" Jul 26 16:02:46.173877 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:46.173848 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c55b8954b-vbslq_08867a8f-2cc2-493d-b5b0-2c06659d50db/copy-manifests/0.log" Jul 26 16:02:46.180600 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:46.180573 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c55b8954b-vbslq_08867a8f-2cc2-493d-b5b0-2c06659d50db/manager/0.log" Jul 26 16:02:46.745639 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:46.745604 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29751345-2t29j_1fe785cd-5fb9-4e0b-a888-ece9ab03d6eb/cleanup/0.log" Jul 26 16:02:46.853335 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:46.853307 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29751360-lwkqg_1672b2bd-09f2-45ef-83aa-d58c0b3279ee/cleanup/0.log" Jul 26 16:02:48.227317 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:48.227284 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c6bc86c6d-74nmr_4096ea8f-3158-4364-9858-346b97b711a7/authorino/0.log" Jul 26 16:02:48.558080 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:48.558008 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-w7kd4_9c09ff1b-831c-4351-a943-71edd17a8845/kuadrant-console-plugin/0.log" Jul 26 16:02:49.027240 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:49.027210 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-r9hq7_51d0e3e9-db56-494a-85ce-12179457089d/manager/0.log" Jul 26 16:02:49.706639 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:49.706605 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-69f589b649-xvn52_781bfb24-f7d7-45a3-bf9d-b8adc44aa6c2/kube-auth-proxy/0.log" Jul 26 16:02:50.730368 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:50.730334 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-f6bg2_ee63dd78-4e6c-41dc-a11e-8f1b6ad335ab/main/0.log" Jul 26 16:02:50.844287 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:50.844250 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrck9_56c46022-da23-42c1-aa80-b3e0229ce1a2/main/0.log" Jul 26 16:02:51.064411 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:51.064346 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-6dp5m_8073ae4b-3724-4254-b3c6-0a4507caae4b/main/0.log" Jul 26 16:02:56.697390 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:56.697356 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 16:02:58.713229 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:58.713193 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" limit=1024 Jul 26 16:02:59.293884 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:59.293844 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-b8spm_706538eb-583d-4fca-8f24-285cd3ab940c/global-pull-secret-syncer/0.log" Jul 26 16:02:59.416809 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:59.416777 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-bndqd_64483b31-45cb-4fa9-93a4-9726dc1bbd99/konnectivity-agent/0.log" Jul 26 16:02:59.505355 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:02:59.505328 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-21.ec2.internal_b544e0067e3546297215dd780b690606/haproxy/0.log" Jul 26 16:03:04.681773 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:04.681732 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c6bc86c6d-74nmr_4096ea8f-3158-4364-9858-346b97b711a7/authorino/0.log" Jul 26 16:03:04.790691 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:04.790668 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-w7kd4_9c09ff1b-831c-4351-a943-71edd17a8845/kuadrant-console-plugin/0.log" Jul 26 16:03:05.180449 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:05.180410 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-r9hq7_51d0e3e9-db56-494a-85ce-12179457089d/manager/0.log" Jul 26 16:03:07.078318 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:07.078238 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ljbtm_c6ef366b-7f71-4f17-8383-7c1c87a3a1f1/node-exporter/0.log" Jul 26 16:03:07.096955 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:07.096913 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ljbtm_c6ef366b-7f71-4f17-8383-7c1c87a3a1f1/kube-rbac-proxy/0.log" Jul 26 16:03:07.122643 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:07.122621 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ljbtm_c6ef366b-7f71-4f17-8383-7c1c87a3a1f1/init-textfile/0.log" Jul 26 16:03:07.240316 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:07.240286 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-54pc5_43ab9a7f-db10-4445-b2b6-b6793c88f75d/kube-rbac-proxy-main/0.log" Jul 26 16:03:07.260533 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:07.260512 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-54pc5_43ab9a7f-db10-4445-b2b6-b6793c88f75d/kube-rbac-proxy-self/0.log" Jul 26 16:03:07.281058 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:07.281041 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-54pc5_43ab9a7f-db10-4445-b2b6-b6793c88f75d/openshift-state-metrics/0.log" Jul 26 16:03:07.342462 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:07.342398 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_082e4a33-5a14-4366-b955-fe623586d569/prometheus/0.log" Jul 26 16:03:07.396404 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:07.396380 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_082e4a33-5a14-4366-b955-fe623586d569/config-reloader/0.log" Jul 26 16:03:07.437834 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:07.437812 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_082e4a33-5a14-4366-b955-fe623586d569/thanos-sidecar/0.log" Jul 26 16:03:07.511336 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:07.511318 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_082e4a33-5a14-4366-b955-fe623586d569/kube-rbac-proxy-web/0.log" Jul 26 16:03:07.554221 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:07.554202 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_082e4a33-5a14-4366-b955-fe623586d569/kube-rbac-proxy/0.log" Jul 26 16:03:07.577057 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:07.577033 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_082e4a33-5a14-4366-b955-fe623586d569/kube-rbac-proxy-thanos/0.log" Jul 26 16:03:07.597688 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:07.597634 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_082e4a33-5a14-4366-b955-fe623586d569/init-config-reloader/0.log" Jul 26 16:03:07.664799 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:07.664780 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-6brxk_4461086d-e84a-46f8-bbf7-a8c9cba0aabb/prometheus-operator-admission-webhook/0.log" Jul 26 16:03:09.713195 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:09.713159 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4ldhb/perf-node-gather-daemonset-dlnhh"] Jul 26 16:03:09.713657 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:09.713636 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fc592d8-5b68-42cc-a137-1a848de13442" containerName="revoke-keys" Jul 26 16:03:09.713657 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:09.713655 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fc592d8-5b68-42cc-a137-1a848de13442" containerName="revoke-keys" Jul 26 16:03:09.713791 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:09.713775 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fc592d8-5b68-42cc-a137-1a848de13442" containerName="revoke-keys" Jul 26 16:03:09.716772 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:09.716751 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4ldhb/perf-node-gather-daemonset-dlnhh" Jul 26 16:03:09.718622 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:09.718596 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4ldhb\"/\"kube-root-ca.crt\"" Jul 26 16:03:09.718735 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:09.718681 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4ldhb\"/\"openshift-service-ca.crt\"" Jul 26 16:03:09.719435 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:09.719418 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4ldhb\"/\"default-dockercfg-5tfm4\"" Jul 26 16:03:09.724220 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:09.724158 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4ldhb/perf-node-gather-daemonset-dlnhh"] Jul 26 16:03:09.758269 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:09.758247 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bad3ee5e-6c98-4ad7-8c7c-a40b3b8cd266-proc\") pod \"perf-node-gather-daemonset-dlnhh\" (UID: \"bad3ee5e-6c98-4ad7-8c7c-a40b3b8cd266\") " pod="openshift-must-gather-4ldhb/perf-node-gather-daemonset-dlnhh" Jul 26 16:03:09.758371 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:09.758279 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4xjh\" (UniqueName: \"kubernetes.io/projected/bad3ee5e-6c98-4ad7-8c7c-a40b3b8cd266-kube-api-access-p4xjh\") pod \"perf-node-gather-daemonset-dlnhh\" (UID: \"bad3ee5e-6c98-4ad7-8c7c-a40b3b8cd266\") " pod="openshift-must-gather-4ldhb/perf-node-gather-daemonset-dlnhh" Jul 26 16:03:09.758371 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:09.758302 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bad3ee5e-6c98-4ad7-8c7c-a40b3b8cd266-sys\") pod \"perf-node-gather-daemonset-dlnhh\" (UID: \"bad3ee5e-6c98-4ad7-8c7c-a40b3b8cd266\") " pod="openshift-must-gather-4ldhb/perf-node-gather-daemonset-dlnhh" Jul 26 16:03:09.758371 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:09.758328 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bad3ee5e-6c98-4ad7-8c7c-a40b3b8cd266-podres\") pod \"perf-node-gather-daemonset-dlnhh\" (UID: \"bad3ee5e-6c98-4ad7-8c7c-a40b3b8cd266\") " pod="openshift-must-gather-4ldhb/perf-node-gather-daemonset-dlnhh" Jul 26 16:03:09.758371 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:09.758370 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bad3ee5e-6c98-4ad7-8c7c-a40b3b8cd266-lib-modules\") pod \"perf-node-gather-daemonset-dlnhh\" (UID: \"bad3ee5e-6c98-4ad7-8c7c-a40b3b8cd266\") " pod="openshift-must-gather-4ldhb/perf-node-gather-daemonset-dlnhh" Jul 26 16:03:09.858745 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:09.858720 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bad3ee5e-6c98-4ad7-8c7c-a40b3b8cd266-lib-modules\") pod \"perf-node-gather-daemonset-dlnhh\" (UID: \"bad3ee5e-6c98-4ad7-8c7c-a40b3b8cd266\") " pod="openshift-must-gather-4ldhb/perf-node-gather-daemonset-dlnhh" Jul 26 16:03:09.858845 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:09.858791 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bad3ee5e-6c98-4ad7-8c7c-a40b3b8cd266-proc\") pod \"perf-node-gather-daemonset-dlnhh\" (UID: \"bad3ee5e-6c98-4ad7-8c7c-a40b3b8cd266\") " pod="openshift-must-gather-4ldhb/perf-node-gather-daemonset-dlnhh" Jul 26 16:03:09.858845 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:09.858820 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4xjh\" (UniqueName: \"kubernetes.io/projected/bad3ee5e-6c98-4ad7-8c7c-a40b3b8cd266-kube-api-access-p4xjh\") pod \"perf-node-gather-daemonset-dlnhh\" (UID: \"bad3ee5e-6c98-4ad7-8c7c-a40b3b8cd266\") " pod="openshift-must-gather-4ldhb/perf-node-gather-daemonset-dlnhh" Jul 26 16:03:09.858951 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:09.858850 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bad3ee5e-6c98-4ad7-8c7c-a40b3b8cd266-sys\") pod \"perf-node-gather-daemonset-dlnhh\" (UID: \"bad3ee5e-6c98-4ad7-8c7c-a40b3b8cd266\") " pod="openshift-must-gather-4ldhb/perf-node-gather-daemonset-dlnhh" Jul 26 16:03:09.858951 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:09.858854 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bad3ee5e-6c98-4ad7-8c7c-a40b3b8cd266-lib-modules\") pod \"perf-node-gather-daemonset-dlnhh\" (UID: \"bad3ee5e-6c98-4ad7-8c7c-a40b3b8cd266\") " pod="openshift-must-gather-4ldhb/perf-node-gather-daemonset-dlnhh" Jul 26 16:03:09.858951 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:09.858878 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bad3ee5e-6c98-4ad7-8c7c-a40b3b8cd266-podres\") pod \"perf-node-gather-daemonset-dlnhh\" (UID: \"bad3ee5e-6c98-4ad7-8c7c-a40b3b8cd266\") " pod="openshift-must-gather-4ldhb/perf-node-gather-daemonset-dlnhh" Jul 26 16:03:09.858951 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:09.858907 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bad3ee5e-6c98-4ad7-8c7c-a40b3b8cd266-proc\") pod \"perf-node-gather-daemonset-dlnhh\" (UID: \"bad3ee5e-6c98-4ad7-8c7c-a40b3b8cd266\") " pod="openshift-must-gather-4ldhb/perf-node-gather-daemonset-dlnhh" Jul 26 16:03:09.858951 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:09.858913 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bad3ee5e-6c98-4ad7-8c7c-a40b3b8cd266-sys\") pod \"perf-node-gather-daemonset-dlnhh\" (UID: \"bad3ee5e-6c98-4ad7-8c7c-a40b3b8cd266\") " pod="openshift-must-gather-4ldhb/perf-node-gather-daemonset-dlnhh" Jul 26 16:03:09.859141 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:09.858999 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bad3ee5e-6c98-4ad7-8c7c-a40b3b8cd266-podres\") pod \"perf-node-gather-daemonset-dlnhh\" (UID: \"bad3ee5e-6c98-4ad7-8c7c-a40b3b8cd266\") " pod="openshift-must-gather-4ldhb/perf-node-gather-daemonset-dlnhh" Jul 26 16:03:09.866044 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:09.866026 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4xjh\" (UniqueName: \"kubernetes.io/projected/bad3ee5e-6c98-4ad7-8c7c-a40b3b8cd266-kube-api-access-p4xjh\") pod \"perf-node-gather-daemonset-dlnhh\" (UID: \"bad3ee5e-6c98-4ad7-8c7c-a40b3b8cd266\") " pod="openshift-must-gather-4ldhb/perf-node-gather-daemonset-dlnhh" Jul 26 16:03:10.027050 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:10.026991 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4ldhb/perf-node-gather-daemonset-dlnhh" Jul 26 16:03:10.150650 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:10.150627 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4ldhb/perf-node-gather-daemonset-dlnhh"] Jul 26 16:03:10.152501 ip-10-0-130-21 kubenswrapper[2555]: W0726 16:03:10.152463 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbad3ee5e_6c98_4ad7_8c7c_a40b3b8cd266.slice/crio-0caea95a113a15f7e07dd99de0ca2f7765cb0fbad66889af668ce52d7776af91 WatchSource:0}: Error finding container 0caea95a113a15f7e07dd99de0ca2f7765cb0fbad66889af668ce52d7776af91: Status 404 returned error can't find the container with id 0caea95a113a15f7e07dd99de0ca2f7765cb0fbad66889af668ce52d7776af91 Jul 26 16:03:10.322527 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:10.322460 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-754ff4fd64-zkr5f_35dbf057-b87d-4846-91af-4445273f3272/console/0.log" Jul 26 16:03:10.686471 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:10.686431 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4ldhb/perf-node-gather-daemonset-dlnhh" event={"ID":"bad3ee5e-6c98-4ad7-8c7c-a40b3b8cd266","Type":"ContainerStarted","Data":"049a35f7de59680e8ae2b3a47ba347b2812e5f03bcf207e6a38fe7a4eaa06270"} Jul 26 16:03:10.686653 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:10.686527 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4ldhb/perf-node-gather-daemonset-dlnhh" event={"ID":"bad3ee5e-6c98-4ad7-8c7c-a40b3b8cd266","Type":"ContainerStarted","Data":"0caea95a113a15f7e07dd99de0ca2f7765cb0fbad66889af668ce52d7776af91"} Jul 26 16:03:10.686653 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:10.686556 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-4ldhb/perf-node-gather-daemonset-dlnhh" Jul 26 16:03:10.700884 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:10.700837 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4ldhb/perf-node-gather-daemonset-dlnhh" podStartSLOduration=1.700822707 podStartE2EDuration="1.700822707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 16:03:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:03:10.699998735 +0000 UTC m=+3471.123845617" watchObservedRunningTime="2026-07-26 16:03:10.700822707 +0000 UTC m=+3471.124669579" Jul 26 16:03:10.937718 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:10.937636 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-qkmnq_aafc3a3d-6f48-4f65-86b8-760837db363d/volume-data-source-validator/0.log" Jul 26 16:03:11.957790 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:11.957756 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-x57rg_184d66ee-e80d-4766-9642-c9de748e1d7b/dns/0.log" Jul 26 16:03:11.976509 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:11.976487 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-x57rg_184d66ee-e80d-4766-9642-c9de748e1d7b/kube-rbac-proxy/0.log" Jul 26 16:03:12.074818 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:12.074797 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rfxhs_57b3ab2c-f95e-45d9-9caa-6c1eac37919c/dns-node-resolver/0.log" Jul 26 16:03:12.625864 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:12.625837 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-j6hp5_53c30547-0ebf-4139-80c9-e452ed2b9578/node-ca/0.log" Jul 26 16:03:13.725433 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:13.725404 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-69f589b649-xvn52_781bfb24-f7d7-45a3-bf9d-b8adc44aa6c2/kube-auth-proxy/0.log" Jul 26 16:03:14.700568 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:14.700530 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vmblz_da36cb43-fdd5-41dc-a3d9-e799d58a9ae4/serve-healthcheck-canary/0.log" Jul 26 16:03:15.140312 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:15.140285 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-rxf4b_84fece67-b584-4229-800c-e74f02667c8d/insights-operator/0.log" Jul 26 16:03:15.143402 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:15.143380 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-rxf4b_84fece67-b584-4229-800c-e74f02667c8d/insights-operator/1.log" Jul 26 16:03:15.221750 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:15.221719 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hbjsw_a65a53c9-99a1-4097-9686-73738e044085/kube-rbac-proxy/0.log" Jul 26 16:03:15.239054 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:15.239036 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hbjsw_a65a53c9-99a1-4097-9686-73738e044085/exporter/0.log" Jul 26 16:03:15.257568 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:15.257547 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hbjsw_a65a53c9-99a1-4097-9686-73738e044085/extractor/0.log" Jul 26 16:03:16.701176 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:16.701149 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-4ldhb/perf-node-gather-daemonset-dlnhh" Jul 26 16:03:17.964964 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:17.964919 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-766dd44bbb-2dzqh_c5dceda4-6cc2-47f0-ade7-574acf5bd536/manager/0.log" Jul 26 16:03:18.266687 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:18.266612 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c55b8954b-vbslq_08867a8f-2cc2-493d-b5b0-2c06659d50db/manager/0.log" Jul 26 16:03:18.288215 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:18.288189 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c55b8954b-vbslq_08867a8f-2cc2-493d-b5b0-2c06659d50db/copy-manifests/0.log" Jul 26 16:03:19.705547 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:19.705518 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-694b8c44c9-bq52h_daf5b316-e8be-40ca-a73c-6580b46f079b/manager/0.log" Jul 26 16:03:20.774874 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:20.774845 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-69f589b649-xvn52_781bfb24-f7d7-45a3-bf9d-b8adc44aa6c2/kube-auth-proxy/0.log" Jul 26 16:03:22.507691 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:22.507658 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-69f589b649-xvn52_781bfb24-f7d7-45a3-bf9d-b8adc44aa6c2/kube-auth-proxy/0.log" Jul 26 16:03:24.754085 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:24.754027 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-6v4n5_cc080b47-4ca2-4dfb-aaeb-239c0918e09d/migrator/0.log" Jul 26 16:03:24.771330 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:24.771292 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-6v4n5_cc080b47-4ca2-4dfb-aaeb-239c0918e09d/graceful-termination/0.log" Jul 26 16:03:25.118754 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:25.118724 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-qklsw_e5f83acc-8163-48d2-94de-bf52dd9c0e42/kube-storage-version-migrator-operator/1.log" Jul 26 16:03:25.120266 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:25.120244 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-qklsw_e5f83acc-8163-48d2-94de-bf52dd9c0e42/kube-storage-version-migrator-operator/0.log" Jul 26 16:03:26.033803 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:26.033775 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4kk2l_96d47908-2b56-4fbe-a6e4-6472cd3a5cd4/kube-multus-additional-cni-plugins/0.log" Jul 26 16:03:26.051713 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:26.051686 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4kk2l_96d47908-2b56-4fbe-a6e4-6472cd3a5cd4/egress-router-binary-copy/0.log" Jul 26 16:03:26.068363 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:26.068335 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4kk2l_96d47908-2b56-4fbe-a6e4-6472cd3a5cd4/cni-plugins/0.log" Jul 26 16:03:26.085420 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:26.085394 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4kk2l_96d47908-2b56-4fbe-a6e4-6472cd3a5cd4/bond-cni-plugin/0.log" Jul 26 16:03:26.102408 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:26.102388 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4kk2l_96d47908-2b56-4fbe-a6e4-6472cd3a5cd4/routeoverride-cni/0.log" Jul 26 16:03:26.119757 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:26.119734 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4kk2l_96d47908-2b56-4fbe-a6e4-6472cd3a5cd4/whereabouts-cni-bincopy/0.log" Jul 26 16:03:26.139107 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:26.139085 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4kk2l_96d47908-2b56-4fbe-a6e4-6472cd3a5cd4/whereabouts-cni/0.log" Jul 26 16:03:26.471646 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:26.471618 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-nzw98_db621adf-49db-46b5-9224-4c526e9ae434/kube-multus/0.log" Jul 26 16:03:26.603618 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:26.603566 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-qlw9m_1b6101bd-09a1-444f-ae02-50384ba66923/network-metrics-daemon/0.log" Jul 26 16:03:26.620485 ip-10-0-130-21 kubenswrapper[2555]: I0726 16:03:26.620465 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-qlw9m_1b6101bd-09a1-444f-ae02-50384ba66923/kube-rbac-proxy/0.log"