Jul 06 22:18:06.435912 ip-10-0-137-147 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 06 22:18:06.435924 ip-10-0-137-147 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 06 22:18:06.435931 ip-10-0-137-147 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 06 22:18:06.436150 ip-10-0-137-147 systemd[1]: Failed to start Kubernetes Kubelet. Jul 06 22:18:16.485863 ip-10-0-137-147 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 06 22:18:16.485883 ip-10-0-137-147 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot fec19d8c9e7641888f77ea132918fd24 -- Jul 06 22:19:52.970511 ip-10-0-137-147 systemd[1]: Starting Kubernetes Kubelet... Jul 06 22:19:53.392870 ip-10-0-137-147 kubenswrapper[2571]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 22:19:53.392870 ip-10-0-137-147 kubenswrapper[2571]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 06 22:19:53.392870 ip-10-0-137-147 kubenswrapper[2571]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 22:19:53.392870 ip-10-0-137-147 kubenswrapper[2571]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 06 22:19:53.392870 ip-10-0-137-147 kubenswrapper[2571]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 22:19:53.395431 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.395270 2571 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 06 22:19:53.400412 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400393 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 22:19:53.400412 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400412 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 22:19:53.400484 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400417 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 22:19:53.400484 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400421 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 22:19:53.400484 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400424 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 22:19:53.400484 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400427 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 22:19:53.400484 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400430 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 22:19:53.400484 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400432 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 22:19:53.400484 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400435 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 22:19:53.400484 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400438 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 22:19:53.400484 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400440 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 22:19:53.400484 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400443 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 22:19:53.400484 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400446 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 22:19:53.400484 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400448 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 22:19:53.400484 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400451 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 22:19:53.400484 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400462 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 22:19:53.400484 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400465 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 22:19:53.400484 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400468 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 22:19:53.400484 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400470 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 22:19:53.400484 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400473 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 22:19:53.400484 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400475 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 22:19:53.400484 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400478 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 22:19:53.400952 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400482 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 22:19:53.400952 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400486 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 22:19:53.400952 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400489 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 22:19:53.400952 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400492 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 22:19:53.400952 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400495 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 22:19:53.400952 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400498 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 22:19:53.400952 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400500 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 22:19:53.400952 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400503 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 22:19:53.400952 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400505 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 22:19:53.400952 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400508 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 22:19:53.400952 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400510 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 22:19:53.400952 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400513 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 22:19:53.400952 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400515 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 22:19:53.400952 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400518 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 22:19:53.400952 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400520 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 22:19:53.400952 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400523 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 22:19:53.400952 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400526 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 22:19:53.400952 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400528 2571 feature_gate.go:328] unrecognized feature gate: Example Jul 06 22:19:53.400952 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400531 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 22:19:53.400952 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400534 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 22:19:53.401477 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400536 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 22:19:53.401477 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400538 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 22:19:53.401477 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400541 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 22:19:53.401477 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400543 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 22:19:53.401477 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400546 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 22:19:53.401477 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400549 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 22:19:53.401477 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400551 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 22:19:53.401477 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400553 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 22:19:53.401477 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400556 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 22:19:53.401477 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400558 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 22:19:53.401477 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400561 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 22:19:53.401477 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400563 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 22:19:53.401477 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400565 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 22:19:53.401477 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400569 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 22:19:53.401477 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400571 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 22:19:53.401477 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400575 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 22:19:53.401477 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400578 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 22:19:53.401477 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400581 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 22:19:53.401477 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400583 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 22:19:53.401477 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400585 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 22:19:53.401981 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400589 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 22:19:53.401981 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400593 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 22:19:53.401981 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400596 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 22:19:53.401981 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400599 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 22:19:53.401981 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400601 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 22:19:53.401981 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400604 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 22:19:53.401981 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400606 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 22:19:53.401981 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400609 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 22:19:53.401981 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400611 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 22:19:53.401981 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400614 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 22:19:53.401981 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400617 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 22:19:53.401981 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400619 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 22:19:53.401981 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400622 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 22:19:53.401981 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400625 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 22:19:53.401981 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400627 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 22:19:53.401981 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400630 2571 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 22:19:53.401981 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400632 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 22:19:53.401981 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400634 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 22:19:53.401981 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400637 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 22:19:53.401981 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400639 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 22:19:53.402500 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400642 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 22:19:53.402500 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400646 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 22:19:53.402500 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400649 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 22:19:53.402500 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400651 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 22:19:53.402500 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400654 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 22:19:53.402500 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400658 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 22:19:53.402500 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400661 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 22:19:53.402500 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400664 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 22:19:53.402500 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400666 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 22:19:53.402500 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400668 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 22:19:53.402500 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400671 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 22:19:53.402500 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400674 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 22:19:53.402500 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400677 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 22:19:53.402500 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400681 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 22:19:53.402500 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400684 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 22:19:53.402500 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400687 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 22:19:53.402500 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.400690 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 22:19:53.402500 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401425 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 22:19:53.402500 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401433 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 22:19:53.402967 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401436 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 22:19:53.402967 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401439 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 22:19:53.402967 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401442 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 22:19:53.402967 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401452 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 22:19:53.402967 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401455 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 22:19:53.402967 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401458 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 22:19:53.402967 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401461 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 22:19:53.402967 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401464 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 22:19:53.402967 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401467 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 22:19:53.402967 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401469 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 22:19:53.402967 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401472 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 22:19:53.402967 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401474 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 22:19:53.402967 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401478 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 22:19:53.402967 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401480 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 22:19:53.402967 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401483 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 22:19:53.402967 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401486 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 22:19:53.402967 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401489 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 22:19:53.402967 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401491 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 22:19:53.402967 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401495 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 22:19:53.402967 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401498 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 22:19:53.403524 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401502 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 22:19:53.403524 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401506 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 22:19:53.403524 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401509 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 22:19:53.403524 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401511 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 22:19:53.403524 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401514 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 22:19:53.403524 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401518 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 22:19:53.403524 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401521 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 22:19:53.403524 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401524 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 22:19:53.403524 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401527 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 22:19:53.403524 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401529 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 22:19:53.403524 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401532 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 22:19:53.403524 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401534 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 22:19:53.403524 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401536 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 22:19:53.403524 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401539 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 22:19:53.403524 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401542 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 22:19:53.403524 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401544 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 22:19:53.403524 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401548 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 22:19:53.403524 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401550 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 22:19:53.403524 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401553 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 22:19:53.404012 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401556 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 22:19:53.404012 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401559 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 22:19:53.404012 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401562 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 22:19:53.404012 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401564 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 22:19:53.404012 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401567 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 22:19:53.404012 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401569 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 22:19:53.404012 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401572 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 22:19:53.404012 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401574 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 22:19:53.404012 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401576 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 22:19:53.404012 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401579 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 22:19:53.404012 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401582 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 22:19:53.404012 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401585 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 22:19:53.404012 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401588 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 22:19:53.404012 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401591 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 22:19:53.404012 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401593 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 22:19:53.404012 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401597 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 22:19:53.404012 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401600 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 22:19:53.404012 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401602 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 22:19:53.404012 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401605 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 22:19:53.404012 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401607 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 22:19:53.404503 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401610 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 22:19:53.404503 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401613 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 22:19:53.404503 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401616 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 22:19:53.404503 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401618 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 22:19:53.404503 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401620 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 22:19:53.404503 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401623 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 22:19:53.404503 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401625 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 22:19:53.404503 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401628 2571 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 22:19:53.404503 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401631 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 22:19:53.404503 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401633 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 22:19:53.404503 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401636 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 22:19:53.404503 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401639 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 22:19:53.404503 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401642 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 22:19:53.404503 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401644 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 22:19:53.404503 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401647 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 22:19:53.404503 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401650 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 22:19:53.404503 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401652 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 22:19:53.404503 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401655 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 22:19:53.404503 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401658 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 22:19:53.404503 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401661 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 22:19:53.404985 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401664 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 22:19:53.404985 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401666 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 22:19:53.404985 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401670 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 22:19:53.404985 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401673 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 22:19:53.404985 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401676 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 22:19:53.404985 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401678 2571 feature_gate.go:328] unrecognized feature gate: Example Jul 06 22:19:53.404985 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401680 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 22:19:53.404985 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401683 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 22:19:53.404985 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401685 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 22:19:53.404985 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401688 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 22:19:53.404985 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401690 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 22:19:53.404985 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401694 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 22:19:53.404985 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401697 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 22:19:53.404985 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401700 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 22:19:53.404985 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401703 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 22:19:53.404985 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401705 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 22:19:53.404985 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401708 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 22:19:53.404985 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.401710 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 22:19:53.404985 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402084 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 22:19:53.405462 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402088 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 22:19:53.405462 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402091 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 22:19:53.405462 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402094 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 22:19:53.405462 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402097 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 22:19:53.405462 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402100 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 22:19:53.405462 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402102 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 22:19:53.405462 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402105 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 22:19:53.405462 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402108 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 22:19:53.405462 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402113 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 22:19:53.405462 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402117 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 22:19:53.405462 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402120 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 22:19:53.405462 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402123 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 22:19:53.405462 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402127 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 22:19:53.405462 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402130 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 22:19:53.405462 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402132 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 22:19:53.405462 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402136 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 22:19:53.405462 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402138 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 22:19:53.405462 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402141 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 22:19:53.405462 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402143 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 22:19:53.405942 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402146 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 22:19:53.405942 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402149 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 22:19:53.405942 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402151 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 22:19:53.405942 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402154 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 22:19:53.405942 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402157 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 22:19:53.405942 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402159 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 22:19:53.405942 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402162 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 22:19:53.405942 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402164 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 22:19:53.405942 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402167 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 22:19:53.405942 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402169 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 22:19:53.405942 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402171 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 22:19:53.405942 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402174 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 22:19:53.405942 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402177 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 22:19:53.405942 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402179 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 22:19:53.405942 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402182 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 22:19:53.405942 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402184 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 22:19:53.405942 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402187 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 22:19:53.405942 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402189 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 22:19:53.405942 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402192 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 22:19:53.405942 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402196 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 22:19:53.406438 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402199 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 22:19:53.406438 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402201 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 22:19:53.406438 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402221 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 22:19:53.406438 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402224 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 22:19:53.406438 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402226 2571 feature_gate.go:328] unrecognized feature gate: Example Jul 06 22:19:53.406438 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402232 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 22:19:53.406438 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402235 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 22:19:53.406438 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402238 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 22:19:53.406438 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402240 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 22:19:53.406438 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402243 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 22:19:53.406438 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402245 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 22:19:53.406438 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402248 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 22:19:53.406438 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402250 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 22:19:53.406438 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402253 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 22:19:53.406438 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402255 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 22:19:53.406438 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402258 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 22:19:53.406438 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402260 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 22:19:53.406438 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402263 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 22:19:53.406438 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402266 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 22:19:53.406438 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402268 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 22:19:53.406924 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402271 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 22:19:53.406924 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402273 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 22:19:53.406924 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402276 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 22:19:53.406924 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402278 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 22:19:53.406924 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402283 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 22:19:53.406924 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402287 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 22:19:53.406924 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402290 2571 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 22:19:53.406924 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402293 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 22:19:53.406924 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402295 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 22:19:53.406924 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402298 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 22:19:53.406924 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402301 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 22:19:53.406924 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402303 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 22:19:53.406924 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402306 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 22:19:53.406924 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402308 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 22:19:53.406924 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402311 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 22:19:53.406924 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402313 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 22:19:53.406924 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402316 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 22:19:53.406924 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402320 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 22:19:53.406924 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402323 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 22:19:53.407405 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402325 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 22:19:53.407405 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402328 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 22:19:53.407405 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402330 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 22:19:53.407405 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402333 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 22:19:53.407405 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402335 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 22:19:53.407405 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402338 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 22:19:53.407405 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402340 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 22:19:53.407405 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402343 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 22:19:53.407405 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402345 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 22:19:53.407405 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402348 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 22:19:53.407405 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402350 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 22:19:53.407405 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402353 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 22:19:53.407405 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402355 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 22:19:53.407405 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402358 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 22:19:53.407405 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402360 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 22:19:53.407405 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402364 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 22:19:53.407405 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402367 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 22:19:53.407405 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402371 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 22:19:53.407405 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402374 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 22:19:53.407405 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.402376 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 22:19:53.407895 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402449 2571 flags.go:64] FLAG: --address="0.0.0.0" Jul 06 22:19:53.407895 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402462 2571 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 06 22:19:53.407895 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402469 2571 flags.go:64] FLAG: --anonymous-auth="true" Jul 06 22:19:53.407895 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402475 2571 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 06 22:19:53.407895 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402482 2571 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 06 22:19:53.407895 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402487 2571 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 06 22:19:53.407895 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402493 2571 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 06 22:19:53.407895 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402499 2571 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 06 22:19:53.407895 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402504 2571 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 06 22:19:53.407895 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402507 2571 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 06 22:19:53.407895 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402512 2571 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 06 22:19:53.407895 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402516 2571 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 06 22:19:53.407895 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402519 2571 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 06 22:19:53.407895 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402522 2571 flags.go:64] FLAG: --cgroup-root="" Jul 06 22:19:53.407895 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402525 2571 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 06 22:19:53.407895 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402528 2571 flags.go:64] FLAG: --client-ca-file="" Jul 06 22:19:53.407895 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402531 2571 flags.go:64] FLAG: --cloud-provider="external" Jul 06 22:19:53.407895 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402534 2571 flags.go:64] FLAG: --cluster-dns="[]" Jul 06 22:19:53.407895 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402538 2571 flags.go:64] FLAG: --cluster-domain="" Jul 06 22:19:53.407895 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402541 2571 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 06 22:19:53.407895 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402544 2571 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 06 22:19:53.407895 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402547 2571 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 06 22:19:53.407895 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402551 2571 flags.go:64] FLAG: --container-log-max-files="5" Jul 06 22:19:53.407895 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402555 2571 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 06 22:19:53.408501 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402558 2571 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 06 22:19:53.408501 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402561 2571 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 06 22:19:53.408501 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402564 2571 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 06 22:19:53.408501 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402567 2571 flags.go:64] FLAG: --contention-profiling="false" Jul 06 22:19:53.408501 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402570 2571 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 06 22:19:53.408501 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402574 2571 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 06 22:19:53.408501 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402577 2571 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 06 22:19:53.408501 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402580 2571 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 06 22:19:53.408501 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402584 2571 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 06 22:19:53.408501 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402587 2571 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 06 22:19:53.408501 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402590 2571 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 06 22:19:53.408501 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402593 2571 flags.go:64] FLAG: --enable-load-reader="false" Jul 06 22:19:53.408501 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402595 2571 flags.go:64] FLAG: --enable-server="true" Jul 06 22:19:53.408501 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402599 2571 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 06 22:19:53.408501 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402604 2571 flags.go:64] FLAG: --event-burst="100" Jul 06 22:19:53.408501 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402607 2571 flags.go:64] FLAG: --event-qps="50" Jul 06 22:19:53.408501 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402610 2571 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 06 22:19:53.408501 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402613 2571 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 06 22:19:53.408501 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402617 2571 flags.go:64] FLAG: --eviction-hard="" Jul 06 22:19:53.408501 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402621 2571 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 06 22:19:53.408501 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402623 2571 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 06 22:19:53.408501 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402626 2571 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 06 22:19:53.408501 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402629 2571 flags.go:64] FLAG: --eviction-soft="" Jul 06 22:19:53.408501 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402632 2571 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 06 22:19:53.408501 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402635 2571 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 06 22:19:53.409122 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402638 2571 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 06 22:19:53.409122 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402641 2571 flags.go:64] FLAG: --experimental-mounter-path="" Jul 06 22:19:53.409122 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402643 2571 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 06 22:19:53.409122 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402646 2571 flags.go:64] FLAG: --fail-swap-on="true" Jul 06 22:19:53.409122 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402649 2571 flags.go:64] FLAG: --feature-gates="" Jul 06 22:19:53.409122 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402653 2571 flags.go:64] FLAG: --file-check-frequency="20s" Jul 06 22:19:53.409122 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402655 2571 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 06 22:19:53.409122 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402658 2571 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 06 22:19:53.409122 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402661 2571 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 06 22:19:53.409122 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402664 2571 flags.go:64] FLAG: --healthz-port="10248" Jul 06 22:19:53.409122 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402667 2571 flags.go:64] FLAG: --help="false" Jul 06 22:19:53.409122 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402671 2571 flags.go:64] FLAG: --hostname-override="ip-10-0-137-147.ec2.internal" Jul 06 22:19:53.409122 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402674 2571 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 06 22:19:53.409122 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402677 2571 flags.go:64] FLAG: --http-check-frequency="20s" Jul 06 22:19:53.409122 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402680 2571 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 06 22:19:53.409122 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402684 2571 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 06 22:19:53.409122 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402687 2571 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 06 22:19:53.409122 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402690 2571 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 06 22:19:53.409122 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402693 2571 flags.go:64] FLAG: --image-service-endpoint="" Jul 06 22:19:53.409122 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402696 2571 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 06 22:19:53.409122 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402699 2571 flags.go:64] FLAG: --kube-api-burst="100" Jul 06 22:19:53.409122 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402702 2571 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 06 22:19:53.409122 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402705 2571 flags.go:64] FLAG: --kube-api-qps="50" Jul 06 22:19:53.409122 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402708 2571 flags.go:64] FLAG: --kube-reserved="" Jul 06 22:19:53.409727 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402711 2571 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 06 22:19:53.409727 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402715 2571 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 06 22:19:53.409727 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402718 2571 flags.go:64] FLAG: --kubelet-cgroups="" Jul 06 22:19:53.409727 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402721 2571 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 06 22:19:53.409727 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402723 2571 flags.go:64] FLAG: --lock-file="" Jul 06 22:19:53.409727 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402726 2571 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 06 22:19:53.409727 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402729 2571 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 06 22:19:53.409727 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402732 2571 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 06 22:19:53.409727 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402738 2571 flags.go:64] FLAG: --log-json-split-stream="false" Jul 06 22:19:53.409727 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402740 2571 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 06 22:19:53.409727 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402743 2571 flags.go:64] FLAG: --log-text-split-stream="false" Jul 06 22:19:53.409727 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402746 2571 flags.go:64] FLAG: --logging-format="text" Jul 06 22:19:53.409727 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402749 2571 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 06 22:19:53.409727 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402753 2571 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 06 22:19:53.409727 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402755 2571 flags.go:64] FLAG: --manifest-url="" Jul 06 22:19:53.409727 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402758 2571 flags.go:64] FLAG: --manifest-url-header="" Jul 06 22:19:53.409727 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402762 2571 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 06 22:19:53.409727 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402765 2571 flags.go:64] FLAG: --max-open-files="1000000" Jul 06 22:19:53.409727 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402769 2571 flags.go:64] FLAG: --max-pods="110" Jul 06 22:19:53.409727 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402772 2571 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 06 22:19:53.409727 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402775 2571 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 06 22:19:53.409727 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402778 2571 flags.go:64] FLAG: --memory-manager-policy="None" Jul 06 22:19:53.409727 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402781 2571 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 06 22:19:53.409727 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402784 2571 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 06 22:19:53.409727 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402787 2571 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 06 22:19:53.410520 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402790 2571 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 06 22:19:53.410520 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402798 2571 flags.go:64] FLAG: --node-status-max-images="50" Jul 06 22:19:53.410520 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402800 2571 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 06 22:19:53.410520 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402803 2571 flags.go:64] FLAG: --oom-score-adj="-999" Jul 06 22:19:53.410520 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402807 2571 flags.go:64] FLAG: --pod-cidr="" Jul 06 22:19:53.410520 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402810 2571 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 06 22:19:53.410520 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402816 2571 flags.go:64] FLAG: --pod-manifest-path="" Jul 06 22:19:53.410520 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402819 2571 flags.go:64] FLAG: --pod-max-pids="-1" Jul 06 22:19:53.410520 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402823 2571 flags.go:64] FLAG: --pods-per-core="0" Jul 06 22:19:53.410520 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402827 2571 flags.go:64] FLAG: --port="10250" Jul 06 22:19:53.410520 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402830 2571 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 06 22:19:53.410520 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402832 2571 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f0d45676e5ddafb1" Jul 06 22:19:53.410520 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402835 2571 flags.go:64] FLAG: --qos-reserved="" Jul 06 22:19:53.410520 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402838 2571 flags.go:64] FLAG: --read-only-port="10255" Jul 06 22:19:53.410520 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402841 2571 flags.go:64] FLAG: --register-node="true" Jul 06 22:19:53.410520 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402858 2571 flags.go:64] FLAG: --register-with-taints="" Jul 06 22:19:53.410520 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402863 2571 flags.go:64] FLAG: --registry-burst="10" Jul 06 22:19:53.410520 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402866 2571 flags.go:64] FLAG: --registry-qps="5" Jul 06 22:19:53.410520 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402870 2571 flags.go:64] FLAG: --reserved-cpus="" Jul 06 22:19:53.410520 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402873 2571 flags.go:64] FLAG: --reserved-memory="" Jul 06 22:19:53.410520 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402876 2571 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 06 22:19:53.410520 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402879 2571 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 06 22:19:53.410520 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402882 2571 flags.go:64] FLAG: --rotate-certificates="false" Jul 06 22:19:53.410520 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402885 2571 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 06 22:19:53.410520 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402888 2571 flags.go:64] FLAG: --runonce="false" Jul 06 22:19:53.411133 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402891 2571 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 06 22:19:53.411133 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402894 2571 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 06 22:19:53.411133 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402898 2571 flags.go:64] FLAG: --seccomp-default="false" Jul 06 22:19:53.411133 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402900 2571 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 06 22:19:53.411133 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402903 2571 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 06 22:19:53.411133 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402906 2571 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 06 22:19:53.411133 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402909 2571 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 06 22:19:53.411133 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402912 2571 flags.go:64] FLAG: --storage-driver-password="root" Jul 06 22:19:53.411133 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402916 2571 flags.go:64] FLAG: --storage-driver-secure="false" Jul 06 22:19:53.411133 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402918 2571 flags.go:64] FLAG: --storage-driver-table="stats" Jul 06 22:19:53.411133 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402921 2571 flags.go:64] FLAG: --storage-driver-user="root" Jul 06 22:19:53.411133 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402924 2571 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 06 22:19:53.411133 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402927 2571 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 06 22:19:53.411133 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402931 2571 flags.go:64] FLAG: --system-cgroups="" Jul 06 22:19:53.411133 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402934 2571 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 06 22:19:53.411133 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402939 2571 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 06 22:19:53.411133 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402943 2571 flags.go:64] FLAG: --tls-cert-file="" Jul 06 22:19:53.411133 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402946 2571 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 06 22:19:53.411133 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402950 2571 flags.go:64] FLAG: --tls-min-version="" Jul 06 22:19:53.411133 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402952 2571 flags.go:64] FLAG: --tls-private-key-file="" Jul 06 22:19:53.411133 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402955 2571 flags.go:64] FLAG: --topology-manager-policy="none" Jul 06 22:19:53.411133 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402958 2571 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 06 22:19:53.411133 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402961 2571 flags.go:64] FLAG: --topology-manager-scope="container" Jul 06 22:19:53.411133 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402964 2571 flags.go:64] FLAG: --v="2" Jul 06 22:19:53.411133 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402968 2571 flags.go:64] FLAG: --version="false" Jul 06 22:19:53.411760 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402972 2571 flags.go:64] FLAG: --vmodule="" Jul 06 22:19:53.411760 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402976 2571 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 06 22:19:53.411760 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.402979 2571 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 06 22:19:53.411760 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403109 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 22:19:53.411760 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403113 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 22:19:53.411760 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403116 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 22:19:53.411760 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403119 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 22:19:53.411760 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403123 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 22:19:53.411760 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403126 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 22:19:53.411760 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403130 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 22:19:53.411760 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403133 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 22:19:53.411760 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403135 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 22:19:53.411760 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403139 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 22:19:53.411760 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403143 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 22:19:53.411760 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403146 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 22:19:53.411760 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403148 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 22:19:53.411760 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403151 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 22:19:53.411760 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403153 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 22:19:53.411760 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403156 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 22:19:53.411760 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403159 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 22:19:53.412271 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403161 2571 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 22:19:53.412271 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403164 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 22:19:53.412271 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403166 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 22:19:53.412271 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403169 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 22:19:53.412271 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403172 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 22:19:53.412271 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403174 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 22:19:53.412271 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403177 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 22:19:53.412271 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403179 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 22:19:53.412271 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403182 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 22:19:53.412271 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403184 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 22:19:53.412271 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403187 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 22:19:53.412271 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403189 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 22:19:53.412271 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403192 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 22:19:53.412271 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403194 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 22:19:53.412271 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403196 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 22:19:53.412271 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403199 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 22:19:53.412271 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403201 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 22:19:53.412271 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403216 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 22:19:53.412271 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403219 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 22:19:53.412736 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403223 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 22:19:53.412736 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403227 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 22:19:53.412736 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403237 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 22:19:53.412736 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403241 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 22:19:53.412736 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403244 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 22:19:53.412736 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403247 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 22:19:53.412736 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403250 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 22:19:53.412736 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403253 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 22:19:53.412736 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403255 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 22:19:53.412736 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403258 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 22:19:53.412736 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403260 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 22:19:53.412736 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403263 2571 feature_gate.go:328] unrecognized feature gate: Example Jul 06 22:19:53.412736 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403265 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 22:19:53.412736 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403268 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 22:19:53.412736 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403271 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 22:19:53.412736 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403274 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 22:19:53.412736 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403277 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 22:19:53.412736 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403280 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 22:19:53.412736 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403283 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 22:19:53.413277 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403285 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 22:19:53.413277 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403287 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 22:19:53.413277 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403290 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 22:19:53.413277 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403293 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 22:19:53.413277 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403295 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 22:19:53.413277 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403298 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 22:19:53.413277 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403300 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 22:19:53.413277 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403303 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 22:19:53.413277 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403305 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 22:19:53.413277 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403308 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 22:19:53.413277 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403310 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 22:19:53.413277 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403313 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 22:19:53.413277 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403315 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 22:19:53.413277 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403318 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 22:19:53.413277 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403320 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 22:19:53.413277 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403324 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 22:19:53.413277 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403327 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 22:19:53.413277 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403329 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 22:19:53.413277 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403332 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 22:19:53.413277 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403334 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 22:19:53.413758 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403336 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 22:19:53.413758 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403339 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 22:19:53.413758 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403341 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 22:19:53.413758 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403344 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 22:19:53.413758 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403346 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 22:19:53.413758 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403348 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 22:19:53.413758 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403351 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 22:19:53.413758 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403354 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 22:19:53.413758 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403356 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 22:19:53.413758 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403359 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 22:19:53.413758 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403362 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 22:19:53.413758 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403365 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 22:19:53.413758 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403368 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 22:19:53.413758 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403370 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 22:19:53.413758 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403373 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 22:19:53.413758 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403375 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 22:19:53.413758 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403378 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 22:19:53.413758 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403380 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 22:19:53.413758 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403384 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 22:19:53.414222 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403387 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 22:19:53.414222 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403390 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 22:19:53.414222 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403392 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 22:19:53.414222 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403395 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 22:19:53.414222 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.403397 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 22:19:53.414222 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.404304 2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 22:19:53.414222 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.411528 2571 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 06 22:19:53.414222 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.411545 2571 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 06 22:19:53.414222 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.411567 2571 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 06 22:19:53.414222 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.411573 2571 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 06 22:19:53.414222 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411646 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 22:19:53.414222 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411651 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 22:19:53.414222 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411655 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 22:19:53.414222 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411657 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 22:19:53.414222 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411660 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 22:19:53.414612 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411663 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 22:19:53.414612 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411666 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 22:19:53.414612 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411669 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 22:19:53.414612 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411672 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 22:19:53.414612 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411675 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 22:19:53.414612 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411677 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 22:19:53.414612 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411680 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 22:19:53.414612 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411682 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 22:19:53.414612 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411685 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 22:19:53.414612 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411687 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 22:19:53.414612 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411690 2571 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 22:19:53.414612 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411692 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 22:19:53.414612 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411695 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 22:19:53.414612 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411698 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 22:19:53.414612 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411700 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 22:19:53.414612 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411703 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 22:19:53.414612 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411705 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 22:19:53.414612 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411708 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 22:19:53.414612 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411710 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 22:19:53.414612 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411713 2571 feature_gate.go:328] unrecognized feature gate: Example Jul 06 22:19:53.415107 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411715 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 22:19:53.415107 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411718 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 22:19:53.415107 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411721 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 22:19:53.415107 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411724 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 22:19:53.415107 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411728 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 22:19:53.415107 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411731 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 22:19:53.415107 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411733 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 22:19:53.415107 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411736 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 22:19:53.415107 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411738 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 22:19:53.415107 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411741 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 22:19:53.415107 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411743 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 22:19:53.415107 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411746 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 22:19:53.415107 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411748 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 22:19:53.415107 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411751 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 22:19:53.415107 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411753 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 22:19:53.415107 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411756 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 22:19:53.415107 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411758 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 22:19:53.415107 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411762 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 22:19:53.415107 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411767 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 22:19:53.415107 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411770 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 22:19:53.415636 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411773 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 22:19:53.415636 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411776 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 22:19:53.415636 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411779 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 22:19:53.415636 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411781 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 22:19:53.415636 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411784 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 22:19:53.415636 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411787 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 22:19:53.415636 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411789 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 22:19:53.415636 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411792 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 22:19:53.415636 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411794 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 22:19:53.415636 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411797 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 22:19:53.415636 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411799 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 22:19:53.415636 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411802 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 22:19:53.415636 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411804 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 22:19:53.415636 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411807 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 22:19:53.415636 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411810 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 22:19:53.415636 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411812 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 22:19:53.415636 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411815 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 22:19:53.415636 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411818 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 22:19:53.415636 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411821 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 22:19:53.415636 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411823 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 22:19:53.416134 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411826 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 22:19:53.416134 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411829 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 22:19:53.416134 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411831 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 22:19:53.416134 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411833 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 22:19:53.416134 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411836 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 22:19:53.416134 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411838 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 22:19:53.416134 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411841 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 22:19:53.416134 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411843 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 22:19:53.416134 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411846 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 22:19:53.416134 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411848 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 22:19:53.416134 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411851 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 22:19:53.416134 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411853 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 22:19:53.416134 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411855 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 22:19:53.416134 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411858 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 22:19:53.416134 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411860 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 22:19:53.416134 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411863 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 22:19:53.416134 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411865 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 22:19:53.416134 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411868 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 22:19:53.416134 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411870 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 22:19:53.416134 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411872 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 22:19:53.416626 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411875 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 22:19:53.416626 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411877 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 22:19:53.416626 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411880 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 22:19:53.416626 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411882 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 22:19:53.416626 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411886 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 22:19:53.416626 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411890 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 22:19:53.416626 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411892 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 22:19:53.416626 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411895 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 22:19:53.416626 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411898 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 22:19:53.416626 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411901 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 22:19:53.416626 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411904 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 22:19:53.416626 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411907 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 22:19:53.416626 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411909 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 22:19:53.416626 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.411913 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 22:19:53.416976 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.411919 2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 22:19:53.416976 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412037 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 22:19:53.416976 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412042 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 22:19:53.416976 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412045 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 22:19:53.416976 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412048 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 22:19:53.416976 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412051 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 22:19:53.416976 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412054 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 22:19:53.416976 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412058 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 22:19:53.416976 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412061 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 22:19:53.416976 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412064 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 22:19:53.416976 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412066 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 22:19:53.416976 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412069 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 22:19:53.416976 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412071 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 22:19:53.416976 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412074 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 22:19:53.416976 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412076 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 22:19:53.417366 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412079 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 22:19:53.417366 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412081 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 22:19:53.417366 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412083 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 22:19:53.417366 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412086 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 22:19:53.417366 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412089 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 22:19:53.417366 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412091 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 22:19:53.417366 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412094 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 22:19:53.417366 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412096 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 22:19:53.417366 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412100 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 22:19:53.417366 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412104 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 22:19:53.417366 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412107 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 22:19:53.417366 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412109 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 22:19:53.417366 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412112 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 22:19:53.417366 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412115 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 22:19:53.417366 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412118 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 22:19:53.417366 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412120 2571 feature_gate.go:328] unrecognized feature gate: Example Jul 06 22:19:53.417366 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412123 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 22:19:53.417366 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412126 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 22:19:53.417366 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412128 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 22:19:53.417366 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412131 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 22:19:53.417862 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412133 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 22:19:53.417862 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412135 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 22:19:53.417862 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412138 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 22:19:53.417862 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412140 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 22:19:53.417862 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412143 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 22:19:53.417862 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412145 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 22:19:53.417862 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412147 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 22:19:53.417862 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412151 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 22:19:53.417862 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412154 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 22:19:53.417862 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412157 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 22:19:53.417862 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412160 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 22:19:53.417862 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412162 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 22:19:53.417862 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412165 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 22:19:53.417862 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412168 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 22:19:53.417862 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412170 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 22:19:53.417862 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412173 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 22:19:53.417862 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412175 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 22:19:53.417862 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412178 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 22:19:53.417862 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412180 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 22:19:53.418353 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412182 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 22:19:53.418353 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412185 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 22:19:53.418353 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412188 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 22:19:53.418353 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412190 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 22:19:53.418353 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412193 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 22:19:53.418353 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412195 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 22:19:53.418353 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412198 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 22:19:53.418353 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412200 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 22:19:53.418353 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412218 2571 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 22:19:53.418353 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412224 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 22:19:53.418353 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412226 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 22:19:53.418353 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412229 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 22:19:53.418353 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412231 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 22:19:53.418353 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412234 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 22:19:53.418353 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412237 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 22:19:53.418353 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412239 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 22:19:53.418353 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412242 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 22:19:53.418353 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412244 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 22:19:53.418353 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412247 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 22:19:53.418353 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412250 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 22:19:53.418833 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412252 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 22:19:53.418833 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412255 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 22:19:53.418833 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412257 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 22:19:53.418833 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412259 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 22:19:53.418833 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412262 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 22:19:53.418833 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412264 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 22:19:53.418833 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412267 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 22:19:53.418833 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412269 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 22:19:53.418833 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412273 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 22:19:53.418833 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412276 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 22:19:53.418833 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412280 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 22:19:53.418833 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412282 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 22:19:53.418833 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412286 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 22:19:53.418833 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412289 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 22:19:53.418833 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412292 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 22:19:53.418833 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412294 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 22:19:53.418833 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412297 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 22:19:53.418833 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412299 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 22:19:53.418833 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412302 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 22:19:53.419427 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412304 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 22:19:53.419427 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412307 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 22:19:53.419427 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412310 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 22:19:53.419427 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412313 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 22:19:53.419427 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412315 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 22:19:53.419427 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412318 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 22:19:53.419427 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:53.412320 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 22:19:53.419427 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.412325 2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 22:19:53.419427 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.412446 2571 server.go:962] "Client rotation is on, will bootstrap in background" Jul 06 22:19:53.419427 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.416786 2571 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 06 22:19:53.419427 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.418924 2571 server.go:1019] "Starting client certificate rotation" Jul 06 22:19:53.419427 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.419025 2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 22:19:53.419427 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.419054 2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 22:19:53.421765 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.421748 2571 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 22:19:53.428162 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.428143 2571 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 22:19:53.432195 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.432174 2571 log.go:25] "Validated CRI v1 runtime API" Jul 06 22:19:53.435339 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.435323 2571 log.go:25] "Validated CRI v1 image API" Jul 06 22:19:53.436101 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.436085 2571 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 06 22:19:53.438757 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.438736 2571 fs.go:135] Filesystem UUIDs: map[1d51b819-9226-4cc3-b0c2-c91a852b6a95:/dev/nvme0n1p3 5e4db5df-f1c6-4e12-8842-a6b07a10cd88:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Jul 06 22:19:53.438842 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.438755 2571 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 06 22:19:53.444854 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.444742 2571 manager.go:216] Machine: {Timestamp:2026-07-06 22:19:53.443381373 +0000 UTC m=+0.361624221 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3119292 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28bbbfa3a1e1cbe35695a20e1ae3ea SystemUUID:ec28bbbf-a3a1-e1cb-e356-95a20e1ae3ea BootID:fec19d8c-9e76-4188-8f77-ea132918fd24 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:93:b7:28:c8:1b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:93:b7:28:c8:1b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b6:d1:38:9c:b4:02 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 06 22:19:53.444854 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.444846 2571 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 06 22:19:53.445002 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.444930 2571 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 06 22:19:53.448442 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.448414 2571 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 06 22:19:53.448613 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.448445 2571 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-147.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 06 22:19:53.449331 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.449318 2571 topology_manager.go:138] "Creating topology manager with none policy" Jul 06 22:19:53.449331 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.449332 2571 container_manager_linux.go:309] "Creating device plugin manager" Jul 06 22:19:53.449406 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.449340 2571 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 22:19:53.450927 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.450915 2571 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 22:19:53.452884 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.452872 2571 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 06 22:19:53.457313 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.457291 2571 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 22:19:53.457433 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.457420 2571 state_mem.go:36] "Initialized new in-memory state store" Jul 06 22:19:53.457565 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.457555 2571 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 06 22:19:53.457602 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.457593 2571 kubelet.go:486] "Attempting to sync node with API server" Jul 06 22:19:53.457630 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.457619 2571 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 06 22:19:53.457658 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.457631 2571 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 06 22:19:53.457658 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.457642 2571 kubelet.go:398] "Adding apiserver pod source" Jul 06 22:19:53.457727 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.457717 2571 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 06 22:19:53.460011 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.459987 2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 22:19:53.460156 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.460144 2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 22:19:53.463198 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.463178 2571 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 06 22:19:53.465653 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.465638 2571 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 06 22:19:53.465719 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.465690 2571 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 06 22:19:53.467198 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.467185 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 06 22:19:53.467282 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.467267 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 06 22:19:53.467317 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.467290 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 06 22:19:53.467317 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.467297 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 06 22:19:53.467317 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.467309 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 06 22:19:53.467317 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.467314 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 06 22:19:53.467427 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.467323 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 06 22:19:53.467427 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.467328 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 06 22:19:53.467427 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.467335 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 06 22:19:53.467427 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.467341 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 06 22:19:53.467427 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.467349 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 06 22:19:53.467427 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.467355 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 06 22:19:53.468267 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.468256 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 06 22:19:53.468267 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.468266 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 06 22:19:53.469824 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:53.469778 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-147.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 06 22:19:53.469824 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:53.469808 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 06 22:19:53.469922 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.469897 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-147.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 06 22:19:53.471668 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.471656 2571 server.go:1268] "Started kubelet" Jul 06 22:19:53.471776 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.471733 2571 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 06 22:19:53.471822 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.471792 2571 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 06 22:19:53.471875 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.471856 2571 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 06 22:19:53.472483 ip-10-0-137-147 systemd[1]: Started Kubernetes Kubelet. Jul 06 22:19:53.474010 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.473990 2571 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 06 22:19:53.474504 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.474489 2571 server.go:310] "Adding debug handlers to kubelet server" Jul 06 22:19:53.478139 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:53.476956 2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-147.ec2.internal.18bfd36191b19be8 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-147.ec2.internal,UID:ip-10-0-137-147.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-147.ec2.internal,},FirstTimestamp:2026-07-06 22:19:53.471630312 +0000 UTC m=+0.389873159,LastTimestamp:2026-07-06 22:19:53.471630312 +0000 UTC m=+0.389873159,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-147.ec2.internal,}" Jul 06 22:19:53.478265 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:53.478191 2571 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 06 22:19:53.482300 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.482284 2571 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 06 22:19:53.482300 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.482297 2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 06 22:19:53.482434 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:53.482413 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-147.ec2.internal\" not found" Jul 06 22:19:53.483176 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.483156 2571 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 06 22:19:53.483288 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.483183 2571 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 06 22:19:53.483288 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.483163 2571 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 06 22:19:53.483382 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.483327 2571 reconstruct.go:97] "Volume reconstruction finished" Jul 06 22:19:53.483382 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.483336 2571 reconciler.go:29] "Reconciler: start to sync state" Jul 06 22:19:53.483455 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.483430 2571 factory.go:153] Registering CRI-O factory Jul 06 22:19:53.483455 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.483446 2571 factory.go:223] Registration of the crio container factory successfully Jul 06 22:19:53.483515 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.483496 2571 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 06 22:19:53.483515 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.483504 2571 factory.go:55] Registering systemd factory Jul 06 22:19:53.483515 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.483514 2571 factory.go:223] Registration of the systemd container factory successfully Jul 06 22:19:53.483612 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.483541 2571 factory.go:103] Registering Raw factory Jul 06 22:19:53.483612 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.483560 2571 manager.go:1196] Started watching for new ooms in manager Jul 06 22:19:53.484372 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.484357 2571 manager.go:349] Starting recovery of all containers Jul 06 22:19:53.494026 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:53.493845 2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-147.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 06 22:19:53.494174 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:53.494149 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 06 22:19:53.495336 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.495316 2571 manager.go:354] Recovery completed Jul 06 22:19:53.500397 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.500383 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 22:19:53.502519 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.502483 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-147.ec2.internal" event="NodeHasSufficientMemory" Jul 06 22:19:53.502595 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.502537 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-147.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 22:19:53.502595 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.502552 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-147.ec2.internal" event="NodeHasSufficientPID" Jul 06 22:19:53.503116 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.503099 2571 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 06 22:19:53.503116 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.503114 2571 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 06 22:19:53.503252 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.503133 2571 state_mem.go:36] "Initialized new in-memory state store" Jul 06 22:19:53.503252 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.503152 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 22:19:53.503946 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.503932 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-147.ec2.internal" event="NodeHasSufficientMemory" Jul 06 22:19:53.503983 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.503960 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-147.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 22:19:53.503983 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.503971 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-147.ec2.internal" event="NodeHasSufficientPID" Jul 06 22:19:53.504846 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:53.504776 2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-147.ec2.internal.18bfd3619388eb71 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-147.ec2.internal,UID:ip-10-0-137-147.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-147.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-147.ec2.internal,},FirstTimestamp:2026-07-06 22:19:53.502518129 +0000 UTC m=+0.420760976,LastTimestamp:2026-07-06 22:19:53.502518129 +0000 UTC m=+0.420760976,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-147.ec2.internal,}" Jul 06 22:19:53.505386 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.505374 2571 policy_none.go:49] "None policy: Start" Jul 06 22:19:53.505443 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.505392 2571 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 06 22:19:53.505443 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.505406 2571 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 06 22:19:53.507422 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.507408 2571 policy_none.go:47] "Start" Jul 06 22:19:53.513241 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.513202 2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hkbb4" Jul 06 22:19:53.513818 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:53.513749 2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-147.ec2.internal.18bfd36193895003 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-147.ec2.internal,UID:ip-10-0-137-147.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-137-147.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-137-147.ec2.internal,},FirstTimestamp:2026-07-06 22:19:53.502543875 +0000 UTC m=+0.420786722,LastTimestamp:2026-07-06 22:19:53.502543875 +0000 UTC m=+0.420786722,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-147.ec2.internal,}" Jul 06 22:19:53.522139 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.522115 2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hkbb4" Jul 06 22:19:53.525475 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:53.525380 2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-147.ec2.internal.18bfd361938988d6 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-147.ec2.internal,UID:ip-10-0-137-147.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-137-147.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-137-147.ec2.internal,},FirstTimestamp:2026-07-06 22:19:53.502558422 +0000 UTC m=+0.420801270,LastTimestamp:2026-07-06 22:19:53.502558422 +0000 UTC m=+0.420801270,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-147.ec2.internal,}" Jul 06 22:19:53.549870 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.549852 2571 manager.go:337] "Starting Device Plugin manager" Jul 06 22:19:53.568644 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:53.549904 2571 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 06 22:19:53.568644 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.549920 2571 server.go:87] "Starting device plugin registration server" Jul 06 22:19:53.568644 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.550267 2571 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 06 22:19:53.568644 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.550284 2571 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 06 22:19:53.568644 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.550369 2571 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 06 22:19:53.568644 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.550465 2571 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 06 22:19:53.568644 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.550473 2571 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 06 22:19:53.568644 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:53.550940 2571 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 06 22:19:53.568644 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:53.550994 2571 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-147.ec2.internal\" not found" Jul 06 22:19:53.626549 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.626456 2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 06 22:19:53.627921 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.627905 2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 06 22:19:53.627983 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.627942 2571 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 06 22:19:53.627983 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.627975 2571 kubelet.go:2448] "Starting kubelet main sync loop" Jul 06 22:19:53.628059 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:53.628022 2571 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 06 22:19:53.630517 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.630494 2571 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 22:19:53.650674 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.650630 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 22:19:53.651614 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.651595 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-147.ec2.internal" event="NodeHasSufficientMemory" Jul 06 22:19:53.651704 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.651631 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-147.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 22:19:53.651704 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.651649 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-147.ec2.internal" event="NodeHasSufficientPID" Jul 06 22:19:53.651704 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.651674 2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-147.ec2.internal" Jul 06 22:19:53.659862 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.659844 2571 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-147.ec2.internal" Jul 06 22:19:53.659914 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:53.659871 2571 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-147.ec2.internal\": node \"ip-10-0-137-147.ec2.internal\" not found" Jul 06 22:19:53.682118 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:53.682095 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-147.ec2.internal\" not found" Jul 06 22:19:53.728220 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.728153 2571 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-147.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-147.ec2.internal"] Jul 06 22:19:53.728298 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.728271 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 22:19:53.729842 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.729825 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-147.ec2.internal" event="NodeHasSufficientMemory" Jul 06 22:19:53.729925 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.729856 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-147.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 22:19:53.729925 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.729870 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-147.ec2.internal" event="NodeHasSufficientPID" Jul 06 22:19:53.732385 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.732372 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 22:19:53.732530 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.732510 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-147.ec2.internal" Jul 06 22:19:53.732570 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.732551 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 22:19:53.733296 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.733274 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-147.ec2.internal" event="NodeHasSufficientMemory" Jul 06 22:19:53.733405 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.733309 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-147.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 22:19:53.733405 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.733323 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-147.ec2.internal" event="NodeHasSufficientPID" Jul 06 22:19:53.733724 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.733707 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-147.ec2.internal" event="NodeHasSufficientMemory" Jul 06 22:19:53.733808 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.733745 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-147.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 22:19:53.733808 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.733761 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-147.ec2.internal" event="NodeHasSufficientPID" Jul 06 22:19:53.736193 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.736174 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-147.ec2.internal" Jul 06 22:19:53.736311 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.736209 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 22:19:53.736960 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.736942 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-147.ec2.internal" event="NodeHasSufficientMemory" Jul 06 22:19:53.737058 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.736975 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-147.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 22:19:53.737058 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.736991 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-147.ec2.internal" event="NodeHasSufficientPID" Jul 06 22:19:53.765299 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:53.765272 2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-147.ec2.internal\" not found" node="ip-10-0-137-147.ec2.internal" Jul 06 22:19:53.770254 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:53.770233 2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-147.ec2.internal\" not found" node="ip-10-0-137-147.ec2.internal" Jul 06 22:19:53.782887 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:53.782863 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-147.ec2.internal\" not found" Jul 06 22:19:53.785311 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.785293 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/023ce87fee0cc66352510ecf45bcb1ac-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-147.ec2.internal\" (UID: \"023ce87fee0cc66352510ecf45bcb1ac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-147.ec2.internal" Jul 06 22:19:53.785392 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.785327 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/023ce87fee0cc66352510ecf45bcb1ac-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-147.ec2.internal\" (UID: \"023ce87fee0cc66352510ecf45bcb1ac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-147.ec2.internal" Jul 06 22:19:53.785392 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.785358 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/13c2e0a524b9d1d1320d115d1eeca7b4-config\") pod \"kube-apiserver-proxy-ip-10-0-137-147.ec2.internal\" (UID: \"13c2e0a524b9d1d1320d115d1eeca7b4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-147.ec2.internal" Jul 06 22:19:53.883817 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:53.883720 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-147.ec2.internal\" not found" Jul 06 22:19:53.886133 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.886110 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/023ce87fee0cc66352510ecf45bcb1ac-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-147.ec2.internal\" (UID: \"023ce87fee0cc66352510ecf45bcb1ac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-147.ec2.internal" Jul 06 22:19:53.886244 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.886148 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/023ce87fee0cc66352510ecf45bcb1ac-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-147.ec2.internal\" (UID: \"023ce87fee0cc66352510ecf45bcb1ac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-147.ec2.internal" Jul 06 22:19:53.886244 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.886175 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/13c2e0a524b9d1d1320d115d1eeca7b4-config\") pod \"kube-apiserver-proxy-ip-10-0-137-147.ec2.internal\" (UID: \"13c2e0a524b9d1d1320d115d1eeca7b4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-147.ec2.internal" Jul 06 22:19:53.886244 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.886217 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/13c2e0a524b9d1d1320d115d1eeca7b4-config\") pod \"kube-apiserver-proxy-ip-10-0-137-147.ec2.internal\" (UID: \"13c2e0a524b9d1d1320d115d1eeca7b4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-147.ec2.internal" Jul 06 22:19:53.886390 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.886247 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/023ce87fee0cc66352510ecf45bcb1ac-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-147.ec2.internal\" (UID: \"023ce87fee0cc66352510ecf45bcb1ac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-147.ec2.internal" Jul 06 22:19:53.886390 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:53.886243 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/023ce87fee0cc66352510ecf45bcb1ac-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-147.ec2.internal\" (UID: \"023ce87fee0cc66352510ecf45bcb1ac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-147.ec2.internal" Jul 06 22:19:53.983909 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:53.983864 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-147.ec2.internal\" not found" Jul 06 22:19:54.074129 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:54.074104 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-147.ec2.internal" Jul 06 22:19:54.074235 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:54.074136 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-147.ec2.internal" Jul 06 22:19:54.084447 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:54.084428 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-147.ec2.internal\" not found" Jul 06 22:19:54.186938 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:54.186818 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-147.ec2.internal\" not found" Jul 06 22:19:54.287710 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:54.287670 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-147.ec2.internal\" not found" Jul 06 22:19:54.389783 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:54.389736 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-147.ec2.internal\" not found" Jul 06 22:19:54.419159 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:54.419117 2571 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 06 22:19:54.419700 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:54.419355 2571 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 22:19:54.490512 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:54.490448 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-147.ec2.internal\" not found" Jul 06 22:19:54.525470 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:54.525429 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-05 22:14:53 +0000 UTC" deadline="2028-03-13 02:02:12.702320454 +0000 UTC" Jul 06 22:19:54.525470 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:54.525463 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14763h42m18.176861114s" Jul 06 22:19:54.577063 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:54.577032 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 22:19:54.578259 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:54.578239 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-147.ec2.internal" event="NodeHasSufficientMemory" Jul 06 22:19:54.578361 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:54.578273 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-147.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 22:19:54.578361 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:54.578285 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-147.ec2.internal" event="NodeHasSufficientPID" Jul 06 22:19:54.590865 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:54.590840 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-147.ec2.internal\" not found" Jul 06 22:19:54.690948 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:54.690904 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-147.ec2.internal\" not found" Jul 06 22:19:54.722487 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:54.722441 2571 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 22:19:54.791370 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:54.791324 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-147.ec2.internal\" not found" Jul 06 22:19:54.845173 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:54.845139 2571 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 22:19:54.883455 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:54.883422 2571 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-147.ec2.internal" Jul 06 22:19:54.894871 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:54.894838 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod023ce87fee0cc66352510ecf45bcb1ac.slice/crio-b5c2685dbd0719cc494d3e8b18c3275ec5c2651ca8746643720dd7f1b2ebd6f3 WatchSource:0}: Error finding container b5c2685dbd0719cc494d3e8b18c3275ec5c2651ca8746643720dd7f1b2ebd6f3: Status 404 returned error can't find the container with id b5c2685dbd0719cc494d3e8b18c3275ec5c2651ca8746643720dd7f1b2ebd6f3 Jul 06 22:19:54.899641 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:54.899621 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 22:19:54.899938 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:54.899909 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13c2e0a524b9d1d1320d115d1eeca7b4.slice/crio-1ebff8e3fc351277dad7b038c27486bca600d57d253087d11511f7aac69238e7 WatchSource:0}: Error finding container 1ebff8e3fc351277dad7b038c27486bca600d57d253087d11511f7aac69238e7: Status 404 returned error can't find the container with id 1ebff8e3fc351277dad7b038c27486bca600d57d253087d11511f7aac69238e7 Jul 06 22:19:54.900227 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:54.900204 2571 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 22:19:54.901117 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:54.901096 2571 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-147.ec2.internal" Jul 06 22:19:54.908674 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:54.908654 2571 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 22:19:54.914428 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:54.914410 2571 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 22:19:55.462136 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.462103 2571 apiserver.go:52] "Watching apiserver" Jul 06 22:19:55.472338 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.472301 2571 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 06 22:19:55.473957 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.473931 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-pwdsc","openshift-network-operator/iptables-alerter-fk7hp","kube-system/global-pull-secret-syncer-ccgsb","openshift-dns/node-resolver-l7h9q","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-147.ec2.internal","openshift-multus/multus-additional-cni-plugins-qhhbl","openshift-network-diagnostics/network-check-target-fg2lc","openshift-ovn-kubernetes/ovnkube-node-qbbrs","kube-system/konnectivity-agent-65zfn","kube-system/kube-apiserver-proxy-ip-10-0-137-147.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47dsw","openshift-cluster-node-tuning-operator/tuned-2rsjs","openshift-image-registry/node-ca-tj7dx","openshift-multus/multus-xxc6f"] Jul 06 22:19:55.478002 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.477982 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qhhbl" Jul 06 22:19:55.480207 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.480178 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xxc6f" Jul 06 22:19:55.480966 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.480943 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 06 22:19:55.481073 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.480945 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 06 22:19:55.481073 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.480949 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-t7m6v\"" Jul 06 22:19:55.481073 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.481048 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 06 22:19:55.481233 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.481107 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 06 22:19:55.481314 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.481251 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 06 22:19:55.482576 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.482554 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47dsw" Jul 06 22:19:55.482723 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.482705 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 06 22:19:55.483185 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.483169 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-fhq8g\"" Jul 06 22:19:55.483262 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.483207 2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 06 22:19:55.485084 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.484943 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 06 22:19:55.485084 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.484975 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 06 22:19:55.485211 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.485153 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rgv2c\"" Jul 06 22:19:55.485385 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.485367 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 06 22:19:55.485468 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.485407 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pwdsc" Jul 06 22:19:55.485524 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:55.485478 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pwdsc" podUID="cfc24edf-03da-480b-b0d2-528c9098f001" Jul 06 22:19:55.488745 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.488725 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fg2lc" Jul 06 22:19:55.488829 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:55.488805 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fg2lc" podUID="5f44d847-874a-4f0a-b3ab-6dcae0173f2d" Jul 06 22:19:55.491188 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.491169 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-l7h9q" Jul 06 22:19:55.493678 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.493648 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:19:55.493762 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.493746 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xk5qq\"" Jul 06 22:19:55.493821 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.493769 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 06 22:19:55.493821 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.493804 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 06 22:19:55.494113 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.494091 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cb6edb1a-9b3d-4809-89e1-5668948aa33f-host-run-multus-certs\") pod \"multus-xxc6f\" (UID: \"cb6edb1a-9b3d-4809-89e1-5668948aa33f\") " pod="openshift-multus/multus-xxc6f" Jul 06 22:19:55.494192 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.494132 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2dea44d6-fcc2-44f2-bace-20919aa7d774-sys-fs\") pod \"aws-ebs-csi-driver-node-47dsw\" (UID: \"2dea44d6-fcc2-44f2-bace-20919aa7d774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47dsw" Jul 06 22:19:55.494192 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.494161 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cb6edb1a-9b3d-4809-89e1-5668948aa33f-os-release\") pod \"multus-xxc6f\" (UID: \"cb6edb1a-9b3d-4809-89e1-5668948aa33f\") " pod="openshift-multus/multus-xxc6f" Jul 06 22:19:55.494192 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.494178 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cb6edb1a-9b3d-4809-89e1-5668948aa33f-multus-conf-dir\") pod \"multus-xxc6f\" (UID: \"cb6edb1a-9b3d-4809-89e1-5668948aa33f\") " pod="openshift-multus/multus-xxc6f" Jul 06 22:19:55.494364 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.494193 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvh72\" (UniqueName: \"kubernetes.io/projected/cb6edb1a-9b3d-4809-89e1-5668948aa33f-kube-api-access-lvh72\") pod \"multus-xxc6f\" (UID: \"cb6edb1a-9b3d-4809-89e1-5668948aa33f\") " pod="openshift-multus/multus-xxc6f" Jul 06 22:19:55.494364 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.494210 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2dea44d6-fcc2-44f2-bace-20919aa7d774-device-dir\") pod \"aws-ebs-csi-driver-node-47dsw\" (UID: \"2dea44d6-fcc2-44f2-bace-20919aa7d774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47dsw" Jul 06 22:19:55.494364 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.494225 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb6edb1a-9b3d-4809-89e1-5668948aa33f-system-cni-dir\") pod \"multus-xxc6f\" (UID: \"cb6edb1a-9b3d-4809-89e1-5668948aa33f\") " pod="openshift-multus/multus-xxc6f" Jul 06 22:19:55.494364 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.494241 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cb6edb1a-9b3d-4809-89e1-5668948aa33f-multus-daemon-config\") pod \"multus-xxc6f\" (UID: \"cb6edb1a-9b3d-4809-89e1-5668948aa33f\") " pod="openshift-multus/multus-xxc6f" Jul 06 22:19:55.494364 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.494257 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2gbc\" (UniqueName: \"kubernetes.io/projected/cfc24edf-03da-480b-b0d2-528c9098f001-kube-api-access-p2gbc\") pod \"network-metrics-daemon-pwdsc\" (UID: \"cfc24edf-03da-480b-b0d2-528c9098f001\") " pod="openshift-multus/network-metrics-daemon-pwdsc" Jul 06 22:19:55.494364 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.494312 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5x22\" (UniqueName: \"kubernetes.io/projected/528a881a-8c62-4448-ae10-af9c1a6243ed-kube-api-access-v5x22\") pod \"multus-additional-cni-plugins-qhhbl\" (UID: \"528a881a-8c62-4448-ae10-af9c1a6243ed\") " pod="openshift-multus/multus-additional-cni-plugins-qhhbl" Jul 06 22:19:55.494364 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.494338 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cb6edb1a-9b3d-4809-89e1-5668948aa33f-hostroot\") pod \"multus-xxc6f\" (UID: \"cb6edb1a-9b3d-4809-89e1-5668948aa33f\") " pod="openshift-multus/multus-xxc6f" Jul 06 22:19:55.494364 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.494361 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/528a881a-8c62-4448-ae10-af9c1a6243ed-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qhhbl\" (UID: \"528a881a-8c62-4448-ae10-af9c1a6243ed\") " pod="openshift-multus/multus-additional-cni-plugins-qhhbl" Jul 06 22:19:55.494668 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.494388 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cb6edb1a-9b3d-4809-89e1-5668948aa33f-cni-binary-copy\") pod \"multus-xxc6f\" (UID: \"cb6edb1a-9b3d-4809-89e1-5668948aa33f\") " pod="openshift-multus/multus-xxc6f" Jul 06 22:19:55.494668 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.494412 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb6edb1a-9b3d-4809-89e1-5668948aa33f-etc-kubernetes\") pod \"multus-xxc6f\" (UID: \"cb6edb1a-9b3d-4809-89e1-5668948aa33f\") " pod="openshift-multus/multus-xxc6f" Jul 06 22:19:55.494668 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.494429 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2dea44d6-fcc2-44f2-bace-20919aa7d774-kubelet-dir\") pod \"aws-ebs-csi-driver-node-47dsw\" (UID: \"2dea44d6-fcc2-44f2-bace-20919aa7d774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47dsw" Jul 06 22:19:55.494668 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.494444 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2dea44d6-fcc2-44f2-bace-20919aa7d774-socket-dir\") pod \"aws-ebs-csi-driver-node-47dsw\" (UID: \"2dea44d6-fcc2-44f2-bace-20919aa7d774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47dsw" Jul 06 22:19:55.494668 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.494460 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2dea44d6-fcc2-44f2-bace-20919aa7d774-registration-dir\") pod \"aws-ebs-csi-driver-node-47dsw\" (UID: \"2dea44d6-fcc2-44f2-bace-20919aa7d774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47dsw" Jul 06 22:19:55.494668 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.494490 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnmqr\" (UniqueName: \"kubernetes.io/projected/2dea44d6-fcc2-44f2-bace-20919aa7d774-kube-api-access-gnmqr\") pod \"aws-ebs-csi-driver-node-47dsw\" (UID: \"2dea44d6-fcc2-44f2-bace-20919aa7d774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47dsw" Jul 06 22:19:55.494668 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.494518 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/528a881a-8c62-4448-ae10-af9c1a6243ed-system-cni-dir\") pod \"multus-additional-cni-plugins-qhhbl\" (UID: \"528a881a-8c62-4448-ae10-af9c1a6243ed\") " pod="openshift-multus/multus-additional-cni-plugins-qhhbl" Jul 06 22:19:55.494668 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.494540 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/528a881a-8c62-4448-ae10-af9c1a6243ed-cnibin\") pod \"multus-additional-cni-plugins-qhhbl\" (UID: \"528a881a-8c62-4448-ae10-af9c1a6243ed\") " pod="openshift-multus/multus-additional-cni-plugins-qhhbl" Jul 06 22:19:55.494668 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.494564 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/528a881a-8c62-4448-ae10-af9c1a6243ed-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qhhbl\" (UID: \"528a881a-8c62-4448-ae10-af9c1a6243ed\") " pod="openshift-multus/multus-additional-cni-plugins-qhhbl" Jul 06 22:19:55.494668 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.494580 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2dea44d6-fcc2-44f2-bace-20919aa7d774-etc-selinux\") pod \"aws-ebs-csi-driver-node-47dsw\" (UID: \"2dea44d6-fcc2-44f2-bace-20919aa7d774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47dsw" Jul 06 22:19:55.494668 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.494596 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cb6edb1a-9b3d-4809-89e1-5668948aa33f-cnibin\") pod \"multus-xxc6f\" (UID: \"cb6edb1a-9b3d-4809-89e1-5668948aa33f\") " pod="openshift-multus/multus-xxc6f" Jul 06 22:19:55.494668 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.494615 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/528a881a-8c62-4448-ae10-af9c1a6243ed-os-release\") pod \"multus-additional-cni-plugins-qhhbl\" (UID: \"528a881a-8c62-4448-ae10-af9c1a6243ed\") " pod="openshift-multus/multus-additional-cni-plugins-qhhbl" Jul 06 22:19:55.494668 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.494639 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/528a881a-8c62-4448-ae10-af9c1a6243ed-cni-binary-copy\") pod \"multus-additional-cni-plugins-qhhbl\" (UID: \"528a881a-8c62-4448-ae10-af9c1a6243ed\") " pod="openshift-multus/multus-additional-cni-plugins-qhhbl" Jul 06 22:19:55.494668 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.494664 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cb6edb1a-9b3d-4809-89e1-5668948aa33f-host-run-k8s-cni-cncf-io\") pod \"multus-xxc6f\" (UID: \"cb6edb1a-9b3d-4809-89e1-5668948aa33f\") " pod="openshift-multus/multus-xxc6f" Jul 06 22:19:55.495139 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.494682 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cb6edb1a-9b3d-4809-89e1-5668948aa33f-host-run-netns\") pod \"multus-xxc6f\" (UID: \"cb6edb1a-9b3d-4809-89e1-5668948aa33f\") " pod="openshift-multus/multus-xxc6f" Jul 06 22:19:55.495139 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.494712 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cb6edb1a-9b3d-4809-89e1-5668948aa33f-host-var-lib-cni-bin\") pod \"multus-xxc6f\" (UID: \"cb6edb1a-9b3d-4809-89e1-5668948aa33f\") " pod="openshift-multus/multus-xxc6f" Jul 06 22:19:55.495139 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.494732 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfc24edf-03da-480b-b0d2-528c9098f001-metrics-certs\") pod \"network-metrics-daemon-pwdsc\" (UID: \"cfc24edf-03da-480b-b0d2-528c9098f001\") " pod="openshift-multus/network-metrics-daemon-pwdsc" Jul 06 22:19:55.495139 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.494750 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/528a881a-8c62-4448-ae10-af9c1a6243ed-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qhhbl\" (UID: \"528a881a-8c62-4448-ae10-af9c1a6243ed\") " pod="openshift-multus/multus-additional-cni-plugins-qhhbl" Jul 06 22:19:55.495139 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.494767 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb6edb1a-9b3d-4809-89e1-5668948aa33f-multus-cni-dir\") pod \"multus-xxc6f\" (UID: \"cb6edb1a-9b3d-4809-89e1-5668948aa33f\") " pod="openshift-multus/multus-xxc6f" Jul 06 22:19:55.495139 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.494788 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cb6edb1a-9b3d-4809-89e1-5668948aa33f-multus-socket-dir-parent\") pod \"multus-xxc6f\" (UID: \"cb6edb1a-9b3d-4809-89e1-5668948aa33f\") " pod="openshift-multus/multus-xxc6f" Jul 06 22:19:55.495139 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.494804 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cb6edb1a-9b3d-4809-89e1-5668948aa33f-host-var-lib-cni-multus\") pod \"multus-xxc6f\" (UID: \"cb6edb1a-9b3d-4809-89e1-5668948aa33f\") " pod="openshift-multus/multus-xxc6f" Jul 06 22:19:55.495139 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.494821 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb6edb1a-9b3d-4809-89e1-5668948aa33f-host-var-lib-kubelet\") pod \"multus-xxc6f\" (UID: \"cb6edb1a-9b3d-4809-89e1-5668948aa33f\") " pod="openshift-multus/multus-xxc6f" Jul 06 22:19:55.496113 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.496093 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-65zfn" Jul 06 22:19:55.496501 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.496479 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 06 22:19:55.496501 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.496496 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 06 22:19:55.496651 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.496496 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 06 22:19:55.497505 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.497487 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 06 22:19:55.497505 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.497495 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 06 22:19:55.497650 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.497522 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 06 22:19:55.497650 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.497495 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-jhwq7\"" Jul 06 22:19:55.498588 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.498558 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 06 22:19:55.498588 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.498578 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 06 22:19:55.498746 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.498657 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ccgsb" Jul 06 22:19:55.498746 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:55.498709 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ccgsb" podUID="d115b038-4f94-41df-82e3-6d42109603e0" Jul 06 22:19:55.499119 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.499098 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-dkxxp\"" Jul 06 22:19:55.500298 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.500271 2571 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 22:19:55.501172 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.501155 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2rsjs" Jul 06 22:19:55.503737 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.503719 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tj7dx" Jul 06 22:19:55.503831 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.503738 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 06 22:19:55.503921 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.503720 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-mpxbz\"" Jul 06 22:19:55.503986 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.503974 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 06 22:19:55.506187 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.506170 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-56btr\"" Jul 06 22:19:55.506298 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.506252 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 06 22:19:55.506407 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.506384 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 06 22:19:55.506510 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.506409 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 06 22:19:55.506633 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.506617 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fk7hp" Jul 06 22:19:55.509188 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.509168 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 06 22:19:55.509515 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.509499 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qtp99\"" Jul 06 22:19:55.509610 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.509598 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 06 22:19:55.509673 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.509660 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 06 22:19:55.535262 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.534338 2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-frmkp" Jul 06 22:19:55.544595 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.544567 2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-frmkp" Jul 06 22:19:55.584475 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.584436 2571 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 06 22:19:55.595287 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.595255 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5x22\" (UniqueName: \"kubernetes.io/projected/528a881a-8c62-4448-ae10-af9c1a6243ed-kube-api-access-v5x22\") pod \"multus-additional-cni-plugins-qhhbl\" (UID: \"528a881a-8c62-4448-ae10-af9c1a6243ed\") " pod="openshift-multus/multus-additional-cni-plugins-qhhbl" Jul 06 22:19:55.595445 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.595294 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cb6edb1a-9b3d-4809-89e1-5668948aa33f-hostroot\") pod \"multus-xxc6f\" (UID: \"cb6edb1a-9b3d-4809-89e1-5668948aa33f\") " pod="openshift-multus/multus-xxc6f" Jul 06 22:19:55.595445 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.595321 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/48762cd8-7ac2-449f-8f6d-49e08ec0eb54-etc-kubernetes\") pod \"tuned-2rsjs\" (UID: \"48762cd8-7ac2-449f-8f6d-49e08ec0eb54\") " pod="openshift-cluster-node-tuning-operator/tuned-2rsjs" Jul 06 22:19:55.595445 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.595349 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bf23ef00-a2d0-499c-8c87-5d6b98510ea0-run-ovn\") pod \"ovnkube-node-qbbrs\" (UID: \"bf23ef00-a2d0-499c-8c87-5d6b98510ea0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:19:55.595445 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.595373 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bf23ef00-a2d0-499c-8c87-5d6b98510ea0-node-log\") pod \"ovnkube-node-qbbrs\" (UID: \"bf23ef00-a2d0-499c-8c87-5d6b98510ea0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:19:55.595445 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.595400 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cb6edb1a-9b3d-4809-89e1-5668948aa33f-cni-binary-copy\") pod \"multus-xxc6f\" (UID: \"cb6edb1a-9b3d-4809-89e1-5668948aa33f\") " pod="openshift-multus/multus-xxc6f" Jul 06 22:19:55.595445 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.595404 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cb6edb1a-9b3d-4809-89e1-5668948aa33f-hostroot\") pod \"multus-xxc6f\" (UID: \"cb6edb1a-9b3d-4809-89e1-5668948aa33f\") " pod="openshift-multus/multus-xxc6f" Jul 06 22:19:55.595445 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.595424 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb6edb1a-9b3d-4809-89e1-5668948aa33f-etc-kubernetes\") pod \"multus-xxc6f\" (UID: \"cb6edb1a-9b3d-4809-89e1-5668948aa33f\") " pod="openshift-multus/multus-xxc6f" Jul 06 22:19:55.595832 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.595452 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bf23ef00-a2d0-499c-8c87-5d6b98510ea0-ovn-node-metrics-cert\") pod \"ovnkube-node-qbbrs\" (UID: \"bf23ef00-a2d0-499c-8c87-5d6b98510ea0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:19:55.595832 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.595476 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4ebc390e-a912-4351-acdc-23f8ea5aa971-host-slash\") pod \"iptables-alerter-fk7hp\" (UID: \"4ebc390e-a912-4351-acdc-23f8ea5aa971\") " pod="openshift-network-operator/iptables-alerter-fk7hp" Jul 06 22:19:55.595832 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.595505 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/48762cd8-7ac2-449f-8f6d-49e08ec0eb54-etc-sysctl-conf\") pod \"tuned-2rsjs\" (UID: \"48762cd8-7ac2-449f-8f6d-49e08ec0eb54\") " pod="openshift-cluster-node-tuning-operator/tuned-2rsjs" Jul 06 22:19:55.595832 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.595530 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2dea44d6-fcc2-44f2-bace-20919aa7d774-etc-selinux\") pod \"aws-ebs-csi-driver-node-47dsw\" (UID: \"2dea44d6-fcc2-44f2-bace-20919aa7d774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47dsw" Jul 06 22:19:55.595832 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.595556 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf23ef00-a2d0-499c-8c87-5d6b98510ea0-host-kubelet\") pod \"ovnkube-node-qbbrs\" (UID: \"bf23ef00-a2d0-499c-8c87-5d6b98510ea0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:19:55.595832 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.595581 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bf23ef00-a2d0-499c-8c87-5d6b98510ea0-ovnkube-config\") pod \"ovnkube-node-qbbrs\" (UID: \"bf23ef00-a2d0-499c-8c87-5d6b98510ea0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:19:55.595832 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.595614 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/48762cd8-7ac2-449f-8f6d-49e08ec0eb54-etc-systemd\") pod \"tuned-2rsjs\" (UID: \"48762cd8-7ac2-449f-8f6d-49e08ec0eb54\") " pod="openshift-cluster-node-tuning-operator/tuned-2rsjs" Jul 06 22:19:55.595832 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.595648 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cb6edb1a-9b3d-4809-89e1-5668948aa33f-cnibin\") pod \"multus-xxc6f\" (UID: \"cb6edb1a-9b3d-4809-89e1-5668948aa33f\") " pod="openshift-multus/multus-xxc6f" Jul 06 22:19:55.595832 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.595731 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cb6edb1a-9b3d-4809-89e1-5668948aa33f-cnibin\") pod \"multus-xxc6f\" (UID: \"cb6edb1a-9b3d-4809-89e1-5668948aa33f\") " pod="openshift-multus/multus-xxc6f" Jul 06 22:19:55.595832 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.595770 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e472a07e-8c98-4b7c-9da5-13a2240aaa4c-hosts-file\") pod \"node-resolver-l7h9q\" (UID: \"e472a07e-8c98-4b7c-9da5-13a2240aaa4c\") " pod="openshift-dns/node-resolver-l7h9q" Jul 06 22:19:55.595832 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.595792 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bf23ef00-a2d0-499c-8c87-5d6b98510ea0-log-socket\") pod \"ovnkube-node-qbbrs\" (UID: \"bf23ef00-a2d0-499c-8c87-5d6b98510ea0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:19:55.595832 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.595818 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfzsg\" (UniqueName: \"kubernetes.io/projected/bf23ef00-a2d0-499c-8c87-5d6b98510ea0-kube-api-access-cfzsg\") pod \"ovnkube-node-qbbrs\" (UID: \"bf23ef00-a2d0-499c-8c87-5d6b98510ea0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:19:55.596552 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.595863 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/48762cd8-7ac2-449f-8f6d-49e08ec0eb54-etc-tuned\") pod \"tuned-2rsjs\" (UID: \"48762cd8-7ac2-449f-8f6d-49e08ec0eb54\") " pod="openshift-cluster-node-tuning-operator/tuned-2rsjs" Jul 06 22:19:55.596552 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.595867 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb6edb1a-9b3d-4809-89e1-5668948aa33f-etc-kubernetes\") pod \"multus-xxc6f\" (UID: \"cb6edb1a-9b3d-4809-89e1-5668948aa33f\") " pod="openshift-multus/multus-xxc6f" Jul 06 22:19:55.596552 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.595888 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bf23ef00-a2d0-499c-8c87-5d6b98510ea0-host-cni-bin\") pod \"ovnkube-node-qbbrs\" (UID: \"bf23ef00-a2d0-499c-8c87-5d6b98510ea0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:19:55.596552 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.595917 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/528a881a-8c62-4448-ae10-af9c1a6243ed-cni-binary-copy\") pod \"multus-additional-cni-plugins-qhhbl\" (UID: \"528a881a-8c62-4448-ae10-af9c1a6243ed\") " pod="openshift-multus/multus-additional-cni-plugins-qhhbl" Jul 06 22:19:55.596552 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.595835 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2dea44d6-fcc2-44f2-bace-20919aa7d774-etc-selinux\") pod \"aws-ebs-csi-driver-node-47dsw\" (UID: \"2dea44d6-fcc2-44f2-bace-20919aa7d774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47dsw" Jul 06 22:19:55.596552 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.595944 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cb6edb1a-9b3d-4809-89e1-5668948aa33f-host-run-k8s-cni-cncf-io\") pod \"multus-xxc6f\" (UID: \"cb6edb1a-9b3d-4809-89e1-5668948aa33f\") " pod="openshift-multus/multus-xxc6f" Jul 06 22:19:55.596552 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.595993 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45bks\" (UniqueName: \"kubernetes.io/projected/e472a07e-8c98-4b7c-9da5-13a2240aaa4c-kube-api-access-45bks\") pod \"node-resolver-l7h9q\" (UID: \"e472a07e-8c98-4b7c-9da5-13a2240aaa4c\") " pod="openshift-dns/node-resolver-l7h9q" Jul 06 22:19:55.596552 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.596027 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf23ef00-a2d0-499c-8c87-5d6b98510ea0-etc-openvswitch\") pod \"ovnkube-node-qbbrs\" (UID: \"bf23ef00-a2d0-499c-8c87-5d6b98510ea0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:19:55.596552 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.596054 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4ebc390e-a912-4351-acdc-23f8ea5aa971-iptables-alerter-script\") pod \"iptables-alerter-fk7hp\" (UID: \"4ebc390e-a912-4351-acdc-23f8ea5aa971\") " pod="openshift-network-operator/iptables-alerter-fk7hp" Jul 06 22:19:55.596552 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.596082 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/528a881a-8c62-4448-ae10-af9c1a6243ed-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qhhbl\" (UID: \"528a881a-8c62-4448-ae10-af9c1a6243ed\") " pod="openshift-multus/multus-additional-cni-plugins-qhhbl" Jul 06 22:19:55.596552 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.596107 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cb6edb1a-9b3d-4809-89e1-5668948aa33f-cni-binary-copy\") pod \"multus-xxc6f\" (UID: \"cb6edb1a-9b3d-4809-89e1-5668948aa33f\") " pod="openshift-multus/multus-xxc6f" Jul 06 22:19:55.596552 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.596122 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb6edb1a-9b3d-4809-89e1-5668948aa33f-multus-cni-dir\") pod \"multus-xxc6f\" (UID: \"cb6edb1a-9b3d-4809-89e1-5668948aa33f\") " pod="openshift-multus/multus-xxc6f" Jul 06 22:19:55.596552 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.596148 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cb6edb1a-9b3d-4809-89e1-5668948aa33f-host-var-lib-cni-multus\") pod \"multus-xxc6f\" (UID: \"cb6edb1a-9b3d-4809-89e1-5668948aa33f\") " pod="openshift-multus/multus-xxc6f" Jul 06 22:19:55.596552 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.596174 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb6edb1a-9b3d-4809-89e1-5668948aa33f-host-var-lib-kubelet\") pod \"multus-xxc6f\" (UID: \"cb6edb1a-9b3d-4809-89e1-5668948aa33f\") " pod="openshift-multus/multus-xxc6f" Jul 06 22:19:55.596552 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.596181 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cb6edb1a-9b3d-4809-89e1-5668948aa33f-host-run-k8s-cni-cncf-io\") pod \"multus-xxc6f\" (UID: \"cb6edb1a-9b3d-4809-89e1-5668948aa33f\") " pod="openshift-multus/multus-xxc6f" Jul 06 22:19:55.596552 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.596202 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e5f17bc2-eb9a-4dbc-a599-2f6902a646f9-konnectivity-ca\") pod \"konnectivity-agent-65zfn\" (UID: \"e5f17bc2-eb9a-4dbc-a599-2f6902a646f9\") " pod="kube-system/konnectivity-agent-65zfn" Jul 06 22:19:55.596552 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.596233 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cb6edb1a-9b3d-4809-89e1-5668948aa33f-os-release\") pod \"multus-xxc6f\" (UID: \"cb6edb1a-9b3d-4809-89e1-5668948aa33f\") " pod="openshift-multus/multus-xxc6f" Jul 06 22:19:55.597443 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.596278 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cb6edb1a-9b3d-4809-89e1-5668948aa33f-multus-conf-dir\") pod \"multus-xxc6f\" (UID: \"cb6edb1a-9b3d-4809-89e1-5668948aa33f\") " pod="openshift-multus/multus-xxc6f" Jul 06 22:19:55.597443 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.596349 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2dea44d6-fcc2-44f2-bace-20919aa7d774-device-dir\") pod \"aws-ebs-csi-driver-node-47dsw\" (UID: \"2dea44d6-fcc2-44f2-bace-20919aa7d774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47dsw" Jul 06 22:19:55.597443 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.596376 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/48762cd8-7ac2-449f-8f6d-49e08ec0eb54-etc-modprobe-d\") pod \"tuned-2rsjs\" (UID: \"48762cd8-7ac2-449f-8f6d-49e08ec0eb54\") " pod="openshift-cluster-node-tuning-operator/tuned-2rsjs" Jul 06 22:19:55.597443 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.596400 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/48762cd8-7ac2-449f-8f6d-49e08ec0eb54-host\") pod \"tuned-2rsjs\" (UID: \"48762cd8-7ac2-449f-8f6d-49e08ec0eb54\") " pod="openshift-cluster-node-tuning-operator/tuned-2rsjs" Jul 06 22:19:55.597443 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.596426 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb6edb1a-9b3d-4809-89e1-5668948aa33f-system-cni-dir\") pod \"multus-xxc6f\" (UID: \"cb6edb1a-9b3d-4809-89e1-5668948aa33f\") " pod="openshift-multus/multus-xxc6f" Jul 06 22:19:55.597443 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.596436 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cb6edb1a-9b3d-4809-89e1-5668948aa33f-host-var-lib-cni-multus\") pod \"multus-xxc6f\" (UID: \"cb6edb1a-9b3d-4809-89e1-5668948aa33f\") " pod="openshift-multus/multus-xxc6f" Jul 06 22:19:55.597443 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.596488 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cb6edb1a-9b3d-4809-89e1-5668948aa33f-multus-daemon-config\") pod \"multus-xxc6f\" (UID: \"cb6edb1a-9b3d-4809-89e1-5668948aa33f\") " pod="openshift-multus/multus-xxc6f" Jul 06 22:19:55.597443 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.596504 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb6edb1a-9b3d-4809-89e1-5668948aa33f-system-cni-dir\") pod \"multus-xxc6f\" (UID: \"cb6edb1a-9b3d-4809-89e1-5668948aa33f\") " pod="openshift-multus/multus-xxc6f" Jul 06 22:19:55.597443 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.596525 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bf23ef00-a2d0-499c-8c87-5d6b98510ea0-host-slash\") pod \"ovnkube-node-qbbrs\" (UID: \"bf23ef00-a2d0-499c-8c87-5d6b98510ea0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:19:55.597443 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.596544 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb6edb1a-9b3d-4809-89e1-5668948aa33f-host-var-lib-kubelet\") pod \"multus-xxc6f\" (UID: \"cb6edb1a-9b3d-4809-89e1-5668948aa33f\") " pod="openshift-multus/multus-xxc6f" Jul 06 22:19:55.597443 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.596561 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf23ef00-a2d0-499c-8c87-5d6b98510ea0-run-openvswitch\") pod \"ovnkube-node-qbbrs\" (UID: \"bf23ef00-a2d0-499c-8c87-5d6b98510ea0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:19:55.597443 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.596593 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bf23ef00-a2d0-499c-8c87-5d6b98510ea0-host-cni-netd\") pod \"ovnkube-node-qbbrs\" (UID: \"bf23ef00-a2d0-499c-8c87-5d6b98510ea0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:19:55.597443 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.596595 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2dea44d6-fcc2-44f2-bace-20919aa7d774-device-dir\") pod \"aws-ebs-csi-driver-node-47dsw\" (UID: \"2dea44d6-fcc2-44f2-bace-20919aa7d774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47dsw" Jul 06 22:19:55.597443 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.596652 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cb6edb1a-9b3d-4809-89e1-5668948aa33f-multus-conf-dir\") pod \"multus-xxc6f\" (UID: \"cb6edb1a-9b3d-4809-89e1-5668948aa33f\") " pod="openshift-multus/multus-xxc6f" Jul 06 22:19:55.597443 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.596655 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/528a881a-8c62-4448-ae10-af9c1a6243ed-cni-binary-copy\") pod \"multus-additional-cni-plugins-qhhbl\" (UID: \"528a881a-8c62-4448-ae10-af9c1a6243ed\") " pod="openshift-multus/multus-additional-cni-plugins-qhhbl" Jul 06 22:19:55.597443 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.596710 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/48762cd8-7ac2-449f-8f6d-49e08ec0eb54-run\") pod \"tuned-2rsjs\" (UID: \"48762cd8-7ac2-449f-8f6d-49e08ec0eb54\") " pod="openshift-cluster-node-tuning-operator/tuned-2rsjs" Jul 06 22:19:55.597443 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.596748 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/528a881a-8c62-4448-ae10-af9c1a6243ed-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qhhbl\" (UID: \"528a881a-8c62-4448-ae10-af9c1a6243ed\") " pod="openshift-multus/multus-additional-cni-plugins-qhhbl" Jul 06 22:19:55.598303 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.596776 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2dea44d6-fcc2-44f2-bace-20919aa7d774-kubelet-dir\") pod \"aws-ebs-csi-driver-node-47dsw\" (UID: \"2dea44d6-fcc2-44f2-bace-20919aa7d774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47dsw" Jul 06 22:19:55.598303 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.596807 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb6edb1a-9b3d-4809-89e1-5668948aa33f-multus-cni-dir\") pod \"multus-xxc6f\" (UID: \"cb6edb1a-9b3d-4809-89e1-5668948aa33f\") " pod="openshift-multus/multus-xxc6f" Jul 06 22:19:55.598303 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.596827 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2dea44d6-fcc2-44f2-bace-20919aa7d774-socket-dir\") pod \"aws-ebs-csi-driver-node-47dsw\" (UID: \"2dea44d6-fcc2-44f2-bace-20919aa7d774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47dsw" Jul 06 22:19:55.598303 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.596856 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2dea44d6-fcc2-44f2-bace-20919aa7d774-registration-dir\") pod \"aws-ebs-csi-driver-node-47dsw\" (UID: \"2dea44d6-fcc2-44f2-bace-20919aa7d774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47dsw" Jul 06 22:19:55.598303 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.596865 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2dea44d6-fcc2-44f2-bace-20919aa7d774-kubelet-dir\") pod \"aws-ebs-csi-driver-node-47dsw\" (UID: \"2dea44d6-fcc2-44f2-bace-20919aa7d774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47dsw" Jul 06 22:19:55.598303 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.596918 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2dea44d6-fcc2-44f2-bace-20919aa7d774-registration-dir\") pod \"aws-ebs-csi-driver-node-47dsw\" (UID: \"2dea44d6-fcc2-44f2-bace-20919aa7d774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47dsw" Jul 06 22:19:55.598303 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.596960 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnmqr\" (UniqueName: \"kubernetes.io/projected/2dea44d6-fcc2-44f2-bace-20919aa7d774-kube-api-access-gnmqr\") pod \"aws-ebs-csi-driver-node-47dsw\" (UID: \"2dea44d6-fcc2-44f2-bace-20919aa7d774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47dsw" Jul 06 22:19:55.598303 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.596995 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf23ef00-a2d0-499c-8c87-5d6b98510ea0-var-lib-openvswitch\") pod \"ovnkube-node-qbbrs\" (UID: \"bf23ef00-a2d0-499c-8c87-5d6b98510ea0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:19:55.598303 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.597005 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/528a881a-8c62-4448-ae10-af9c1a6243ed-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qhhbl\" (UID: \"528a881a-8c62-4448-ae10-af9c1a6243ed\") " pod="openshift-multus/multus-additional-cni-plugins-qhhbl" Jul 06 22:19:55.598303 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.597073 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cb6edb1a-9b3d-4809-89e1-5668948aa33f-multus-daemon-config\") pod \"multus-xxc6f\" (UID: \"cb6edb1a-9b3d-4809-89e1-5668948aa33f\") " pod="openshift-multus/multus-xxc6f" Jul 06 22:19:55.598303 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.597115 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d115b038-4f94-41df-82e3-6d42109603e0-original-pull-secret\") pod \"global-pull-secret-syncer-ccgsb\" (UID: \"d115b038-4f94-41df-82e3-6d42109603e0\") " pod="kube-system/global-pull-secret-syncer-ccgsb" Jul 06 22:19:55.598303 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.597132 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cb6edb1a-9b3d-4809-89e1-5668948aa33f-os-release\") pod \"multus-xxc6f\" (UID: \"cb6edb1a-9b3d-4809-89e1-5668948aa33f\") " pod="openshift-multus/multus-xxc6f" Jul 06 22:19:55.598303 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.597151 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cddcda7d-c04f-4fca-a269-9c034ad1e2a1-host\") pod \"node-ca-tj7dx\" (UID: \"cddcda7d-c04f-4fca-a269-9c034ad1e2a1\") " pod="openshift-image-registry/node-ca-tj7dx" Jul 06 22:19:55.598303 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.597145 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2dea44d6-fcc2-44f2-bace-20919aa7d774-socket-dir\") pod \"aws-ebs-csi-driver-node-47dsw\" (UID: \"2dea44d6-fcc2-44f2-bace-20919aa7d774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47dsw" Jul 06 22:19:55.598303 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.597207 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/528a881a-8c62-4448-ae10-af9c1a6243ed-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qhhbl\" (UID: \"528a881a-8c62-4448-ae10-af9c1a6243ed\") " pod="openshift-multus/multus-additional-cni-plugins-qhhbl" Jul 06 22:19:55.598303 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.597227 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/528a881a-8c62-4448-ae10-af9c1a6243ed-system-cni-dir\") pod \"multus-additional-cni-plugins-qhhbl\" (UID: \"528a881a-8c62-4448-ae10-af9c1a6243ed\") " pod="openshift-multus/multus-additional-cni-plugins-qhhbl" Jul 06 22:19:55.598303 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.597277 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/528a881a-8c62-4448-ae10-af9c1a6243ed-cnibin\") pod \"multus-additional-cni-plugins-qhhbl\" (UID: \"528a881a-8c62-4448-ae10-af9c1a6243ed\") " pod="openshift-multus/multus-additional-cni-plugins-qhhbl" Jul 06 22:19:55.599182 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.597299 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/528a881a-8c62-4448-ae10-af9c1a6243ed-system-cni-dir\") pod \"multus-additional-cni-plugins-qhhbl\" (UID: \"528a881a-8c62-4448-ae10-af9c1a6243ed\") " pod="openshift-multus/multus-additional-cni-plugins-qhhbl" Jul 06 22:19:55.599182 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.597363 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/528a881a-8c62-4448-ae10-af9c1a6243ed-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qhhbl\" (UID: \"528a881a-8c62-4448-ae10-af9c1a6243ed\") " pod="openshift-multus/multus-additional-cni-plugins-qhhbl" Jul 06 22:19:55.599182 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.597398 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e472a07e-8c98-4b7c-9da5-13a2240aaa4c-tmp-dir\") pod \"node-resolver-l7h9q\" (UID: \"e472a07e-8c98-4b7c-9da5-13a2240aaa4c\") " pod="openshift-dns/node-resolver-l7h9q" Jul 06 22:19:55.599182 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.597401 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/528a881a-8c62-4448-ae10-af9c1a6243ed-cnibin\") pod \"multus-additional-cni-plugins-qhhbl\" (UID: \"528a881a-8c62-4448-ae10-af9c1a6243ed\") " pod="openshift-multus/multus-additional-cni-plugins-qhhbl" Jul 06 22:19:55.599182 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.597466 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bf23ef00-a2d0-499c-8c87-5d6b98510ea0-host-run-netns\") pod \"ovnkube-node-qbbrs\" (UID: \"bf23ef00-a2d0-499c-8c87-5d6b98510ea0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:19:55.599182 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.597588 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf23ef00-a2d0-499c-8c87-5d6b98510ea0-host-run-ovn-kubernetes\") pod \"ovnkube-node-qbbrs\" (UID: \"bf23ef00-a2d0-499c-8c87-5d6b98510ea0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:19:55.599182 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.597621 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brm7n\" (UniqueName: \"kubernetes.io/projected/4ebc390e-a912-4351-acdc-23f8ea5aa971-kube-api-access-brm7n\") pod \"iptables-alerter-fk7hp\" (UID: \"4ebc390e-a912-4351-acdc-23f8ea5aa971\") " pod="openshift-network-operator/iptables-alerter-fk7hp" Jul 06 22:19:55.599182 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.597697 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bf23ef00-a2d0-499c-8c87-5d6b98510ea0-env-overrides\") pod \"ovnkube-node-qbbrs\" (UID: \"bf23ef00-a2d0-499c-8c87-5d6b98510ea0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:19:55.599182 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.597775 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bf23ef00-a2d0-499c-8c87-5d6b98510ea0-ovnkube-script-lib\") pod \"ovnkube-node-qbbrs\" (UID: \"bf23ef00-a2d0-499c-8c87-5d6b98510ea0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:19:55.599182 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.597819 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/48762cd8-7ac2-449f-8f6d-49e08ec0eb54-etc-sysconfig\") pod \"tuned-2rsjs\" (UID: \"48762cd8-7ac2-449f-8f6d-49e08ec0eb54\") " pod="openshift-cluster-node-tuning-operator/tuned-2rsjs" Jul 06 22:19:55.599182 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.597854 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/48762cd8-7ac2-449f-8f6d-49e08ec0eb54-var-lib-kubelet\") pod \"tuned-2rsjs\" (UID: \"48762cd8-7ac2-449f-8f6d-49e08ec0eb54\") " pod="openshift-cluster-node-tuning-operator/tuned-2rsjs" Jul 06 22:19:55.599182 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.597904 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cddcda7d-c04f-4fca-a269-9c034ad1e2a1-serviceca\") pod \"node-ca-tj7dx\" (UID: \"cddcda7d-c04f-4fca-a269-9c034ad1e2a1\") " pod="openshift-image-registry/node-ca-tj7dx" Jul 06 22:19:55.599182 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.597939 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/528a881a-8c62-4448-ae10-af9c1a6243ed-os-release\") pod \"multus-additional-cni-plugins-qhhbl\" (UID: \"528a881a-8c62-4448-ae10-af9c1a6243ed\") " pod="openshift-multus/multus-additional-cni-plugins-qhhbl" Jul 06 22:19:55.599182 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.597968 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/528a881a-8c62-4448-ae10-af9c1a6243ed-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qhhbl\" (UID: \"528a881a-8c62-4448-ae10-af9c1a6243ed\") " pod="openshift-multus/multus-additional-cni-plugins-qhhbl" Jul 06 22:19:55.599182 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.597982 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cb6edb1a-9b3d-4809-89e1-5668948aa33f-host-run-netns\") pod \"multus-xxc6f\" (UID: \"cb6edb1a-9b3d-4809-89e1-5668948aa33f\") " pod="openshift-multus/multus-xxc6f" Jul 06 22:19:55.599182 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.598009 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cb6edb1a-9b3d-4809-89e1-5668948aa33f-host-var-lib-cni-bin\") pod \"multus-xxc6f\" (UID: \"cb6edb1a-9b3d-4809-89e1-5668948aa33f\") " pod="openshift-multus/multus-xxc6f" Jul 06 22:19:55.599182 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.598037 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfc24edf-03da-480b-b0d2-528c9098f001-metrics-certs\") pod \"network-metrics-daemon-pwdsc\" (UID: \"cfc24edf-03da-480b-b0d2-528c9098f001\") " pod="openshift-multus/network-metrics-daemon-pwdsc" Jul 06 22:19:55.600065 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.598047 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/528a881a-8c62-4448-ae10-af9c1a6243ed-os-release\") pod \"multus-additional-cni-plugins-qhhbl\" (UID: \"528a881a-8c62-4448-ae10-af9c1a6243ed\") " pod="openshift-multus/multus-additional-cni-plugins-qhhbl" Jul 06 22:19:55.600065 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.598068 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf23ef00-a2d0-499c-8c87-5d6b98510ea0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qbbrs\" (UID: \"bf23ef00-a2d0-499c-8c87-5d6b98510ea0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:19:55.600065 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.598092 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cb6edb1a-9b3d-4809-89e1-5668948aa33f-host-run-netns\") pod \"multus-xxc6f\" (UID: \"cb6edb1a-9b3d-4809-89e1-5668948aa33f\") " pod="openshift-multus/multus-xxc6f" Jul 06 22:19:55.600065 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.598110 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cb6edb1a-9b3d-4809-89e1-5668948aa33f-host-var-lib-cni-bin\") pod \"multus-xxc6f\" (UID: \"cb6edb1a-9b3d-4809-89e1-5668948aa33f\") " pod="openshift-multus/multus-xxc6f" Jul 06 22:19:55.600065 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:55.598181 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:19:55.600065 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.598194 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d115b038-4f94-41df-82e3-6d42109603e0-dbus\") pod \"global-pull-secret-syncer-ccgsb\" (UID: \"d115b038-4f94-41df-82e3-6d42109603e0\") " pod="kube-system/global-pull-secret-syncer-ccgsb" Jul 06 22:19:55.600065 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.598224 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwd96\" (UniqueName: \"kubernetes.io/projected/cddcda7d-c04f-4fca-a269-9c034ad1e2a1-kube-api-access-gwd96\") pod \"node-ca-tj7dx\" (UID: \"cddcda7d-c04f-4fca-a269-9c034ad1e2a1\") " pod="openshift-image-registry/node-ca-tj7dx" Jul 06 22:19:55.600065 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:55.598246 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfc24edf-03da-480b-b0d2-528c9098f001-metrics-certs podName:cfc24edf-03da-480b-b0d2-528c9098f001 nodeName:}" failed. No retries permitted until 2026-07-06 22:19:56.098224872 +0000 UTC m=+3.016467711 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cfc24edf-03da-480b-b0d2-528c9098f001-metrics-certs") pod "network-metrics-daemon-pwdsc" (UID: "cfc24edf-03da-480b-b0d2-528c9098f001") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:19:55.600065 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.598297 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cb6edb1a-9b3d-4809-89e1-5668948aa33f-multus-socket-dir-parent\") pod \"multus-xxc6f\" (UID: \"cb6edb1a-9b3d-4809-89e1-5668948aa33f\") " pod="openshift-multus/multus-xxc6f" Jul 06 22:19:55.600065 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.598358 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cb6edb1a-9b3d-4809-89e1-5668948aa33f-host-run-multus-certs\") pod \"multus-xxc6f\" (UID: \"cb6edb1a-9b3d-4809-89e1-5668948aa33f\") " pod="openshift-multus/multus-xxc6f" Jul 06 22:19:55.600065 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.598384 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2dea44d6-fcc2-44f2-bace-20919aa7d774-sys-fs\") pod \"aws-ebs-csi-driver-node-47dsw\" (UID: \"2dea44d6-fcc2-44f2-bace-20919aa7d774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47dsw" Jul 06 22:19:55.600065 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.598533 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bf23ef00-a2d0-499c-8c87-5d6b98510ea0-systemd-units\") pod \"ovnkube-node-qbbrs\" (UID: \"bf23ef00-a2d0-499c-8c87-5d6b98510ea0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:19:55.600065 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.598568 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbw84\" (UniqueName: \"kubernetes.io/projected/5f44d847-874a-4f0a-b3ab-6dcae0173f2d-kube-api-access-vbw84\") pod \"network-check-target-fg2lc\" (UID: \"5f44d847-874a-4f0a-b3ab-6dcae0173f2d\") " pod="openshift-network-diagnostics/network-check-target-fg2lc" Jul 06 22:19:55.600065 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.598597 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d115b038-4f94-41df-82e3-6d42109603e0-kubelet-config\") pod \"global-pull-secret-syncer-ccgsb\" (UID: \"d115b038-4f94-41df-82e3-6d42109603e0\") " pod="kube-system/global-pull-secret-syncer-ccgsb" Jul 06 22:19:55.600065 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.598621 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48762cd8-7ac2-449f-8f6d-49e08ec0eb54-sys\") pod \"tuned-2rsjs\" (UID: \"48762cd8-7ac2-449f-8f6d-49e08ec0eb54\") " pod="openshift-cluster-node-tuning-operator/tuned-2rsjs" Jul 06 22:19:55.600065 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.598648 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvh72\" (UniqueName: \"kubernetes.io/projected/cb6edb1a-9b3d-4809-89e1-5668948aa33f-kube-api-access-lvh72\") pod \"multus-xxc6f\" (UID: \"cb6edb1a-9b3d-4809-89e1-5668948aa33f\") " pod="openshift-multus/multus-xxc6f" Jul 06 22:19:55.600826 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.598672 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/48762cd8-7ac2-449f-8f6d-49e08ec0eb54-etc-sysctl-d\") pod \"tuned-2rsjs\" (UID: \"48762cd8-7ac2-449f-8f6d-49e08ec0eb54\") " pod="openshift-cluster-node-tuning-operator/tuned-2rsjs" Jul 06 22:19:55.600826 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.598695 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/48762cd8-7ac2-449f-8f6d-49e08ec0eb54-lib-modules\") pod \"tuned-2rsjs\" (UID: \"48762cd8-7ac2-449f-8f6d-49e08ec0eb54\") " pod="openshift-cluster-node-tuning-operator/tuned-2rsjs" Jul 06 22:19:55.600826 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.598724 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48762cd8-7ac2-449f-8f6d-49e08ec0eb54-tmp\") pod \"tuned-2rsjs\" (UID: \"48762cd8-7ac2-449f-8f6d-49e08ec0eb54\") " pod="openshift-cluster-node-tuning-operator/tuned-2rsjs" Jul 06 22:19:55.600826 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.598750 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2gbc\" (UniqueName: \"kubernetes.io/projected/cfc24edf-03da-480b-b0d2-528c9098f001-kube-api-access-p2gbc\") pod \"network-metrics-daemon-pwdsc\" (UID: \"cfc24edf-03da-480b-b0d2-528c9098f001\") " pod="openshift-multus/network-metrics-daemon-pwdsc" Jul 06 22:19:55.600826 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.598779 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bf23ef00-a2d0-499c-8c87-5d6b98510ea0-run-systemd\") pod \"ovnkube-node-qbbrs\" (UID: \"bf23ef00-a2d0-499c-8c87-5d6b98510ea0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:19:55.600826 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.598805 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e5f17bc2-eb9a-4dbc-a599-2f6902a646f9-agent-certs\") pod \"konnectivity-agent-65zfn\" (UID: \"e5f17bc2-eb9a-4dbc-a599-2f6902a646f9\") " pod="kube-system/konnectivity-agent-65zfn" Jul 06 22:19:55.600826 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.598829 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4bnb\" (UniqueName: \"kubernetes.io/projected/48762cd8-7ac2-449f-8f6d-49e08ec0eb54-kube-api-access-s4bnb\") pod \"tuned-2rsjs\" (UID: \"48762cd8-7ac2-449f-8f6d-49e08ec0eb54\") " pod="openshift-cluster-node-tuning-operator/tuned-2rsjs" Jul 06 22:19:55.600826 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.599118 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cb6edb1a-9b3d-4809-89e1-5668948aa33f-multus-socket-dir-parent\") pod \"multus-xxc6f\" (UID: \"cb6edb1a-9b3d-4809-89e1-5668948aa33f\") " pod="openshift-multus/multus-xxc6f" Jul 06 22:19:55.600826 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.599167 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cb6edb1a-9b3d-4809-89e1-5668948aa33f-host-run-multus-certs\") pod \"multus-xxc6f\" (UID: \"cb6edb1a-9b3d-4809-89e1-5668948aa33f\") " pod="openshift-multus/multus-xxc6f" Jul 06 22:19:55.600826 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.599229 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2dea44d6-fcc2-44f2-bace-20919aa7d774-sys-fs\") pod \"aws-ebs-csi-driver-node-47dsw\" (UID: \"2dea44d6-fcc2-44f2-bace-20919aa7d774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47dsw" Jul 06 22:19:55.604107 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.604080 2571 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 06 22:19:55.608539 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.608482 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5x22\" (UniqueName: \"kubernetes.io/projected/528a881a-8c62-4448-ae10-af9c1a6243ed-kube-api-access-v5x22\") pod \"multus-additional-cni-plugins-qhhbl\" (UID: \"528a881a-8c62-4448-ae10-af9c1a6243ed\") " pod="openshift-multus/multus-additional-cni-plugins-qhhbl" Jul 06 22:19:55.610589 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.610559 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnmqr\" (UniqueName: \"kubernetes.io/projected/2dea44d6-fcc2-44f2-bace-20919aa7d774-kube-api-access-gnmqr\") pod \"aws-ebs-csi-driver-node-47dsw\" (UID: \"2dea44d6-fcc2-44f2-bace-20919aa7d774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47dsw" Jul 06 22:19:55.614299 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.614222 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvh72\" (UniqueName: \"kubernetes.io/projected/cb6edb1a-9b3d-4809-89e1-5668948aa33f-kube-api-access-lvh72\") pod \"multus-xxc6f\" (UID: \"cb6edb1a-9b3d-4809-89e1-5668948aa33f\") " pod="openshift-multus/multus-xxc6f" Jul 06 22:19:55.614626 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.614605 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2gbc\" (UniqueName: \"kubernetes.io/projected/cfc24edf-03da-480b-b0d2-528c9098f001-kube-api-access-p2gbc\") pod \"network-metrics-daemon-pwdsc\" (UID: \"cfc24edf-03da-480b-b0d2-528c9098f001\") " pod="openshift-multus/network-metrics-daemon-pwdsc" Jul 06 22:19:55.636668 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.636598 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-147.ec2.internal" event={"ID":"13c2e0a524b9d1d1320d115d1eeca7b4","Type":"ContainerStarted","Data":"1ebff8e3fc351277dad7b038c27486bca600d57d253087d11511f7aac69238e7"} Jul 06 22:19:55.638618 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.638586 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-147.ec2.internal" event={"ID":"023ce87fee0cc66352510ecf45bcb1ac","Type":"ContainerStarted","Data":"b5c2685dbd0719cc494d3e8b18c3275ec5c2651ca8746643720dd7f1b2ebd6f3"} Jul 06 22:19:55.699573 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.699535 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bf23ef00-a2d0-499c-8c87-5d6b98510ea0-systemd-units\") pod \"ovnkube-node-qbbrs\" (UID: \"bf23ef00-a2d0-499c-8c87-5d6b98510ea0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:19:55.699779 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.699581 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbw84\" (UniqueName: \"kubernetes.io/projected/5f44d847-874a-4f0a-b3ab-6dcae0173f2d-kube-api-access-vbw84\") pod \"network-check-target-fg2lc\" (UID: \"5f44d847-874a-4f0a-b3ab-6dcae0173f2d\") " pod="openshift-network-diagnostics/network-check-target-fg2lc" Jul 06 22:19:55.699779 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.699607 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d115b038-4f94-41df-82e3-6d42109603e0-kubelet-config\") pod \"global-pull-secret-syncer-ccgsb\" (UID: \"d115b038-4f94-41df-82e3-6d42109603e0\") " pod="kube-system/global-pull-secret-syncer-ccgsb" Jul 06 22:19:55.699779 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.699677 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bf23ef00-a2d0-499c-8c87-5d6b98510ea0-systemd-units\") pod \"ovnkube-node-qbbrs\" (UID: \"bf23ef00-a2d0-499c-8c87-5d6b98510ea0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:19:55.699779 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.699693 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d115b038-4f94-41df-82e3-6d42109603e0-kubelet-config\") pod \"global-pull-secret-syncer-ccgsb\" (UID: \"d115b038-4f94-41df-82e3-6d42109603e0\") " pod="kube-system/global-pull-secret-syncer-ccgsb" Jul 06 22:19:55.699779 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.699731 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48762cd8-7ac2-449f-8f6d-49e08ec0eb54-sys\") pod \"tuned-2rsjs\" (UID: \"48762cd8-7ac2-449f-8f6d-49e08ec0eb54\") " pod="openshift-cluster-node-tuning-operator/tuned-2rsjs" Jul 06 22:19:55.699779 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.699771 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/48762cd8-7ac2-449f-8f6d-49e08ec0eb54-etc-sysctl-d\") pod \"tuned-2rsjs\" (UID: \"48762cd8-7ac2-449f-8f6d-49e08ec0eb54\") " pod="openshift-cluster-node-tuning-operator/tuned-2rsjs" Jul 06 22:19:55.700098 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.699800 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/48762cd8-7ac2-449f-8f6d-49e08ec0eb54-lib-modules\") pod \"tuned-2rsjs\" (UID: \"48762cd8-7ac2-449f-8f6d-49e08ec0eb54\") " pod="openshift-cluster-node-tuning-operator/tuned-2rsjs" Jul 06 22:19:55.700098 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.699828 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48762cd8-7ac2-449f-8f6d-49e08ec0eb54-tmp\") pod \"tuned-2rsjs\" (UID: \"48762cd8-7ac2-449f-8f6d-49e08ec0eb54\") " pod="openshift-cluster-node-tuning-operator/tuned-2rsjs" Jul 06 22:19:55.700098 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.699837 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48762cd8-7ac2-449f-8f6d-49e08ec0eb54-sys\") pod \"tuned-2rsjs\" (UID: \"48762cd8-7ac2-449f-8f6d-49e08ec0eb54\") " pod="openshift-cluster-node-tuning-operator/tuned-2rsjs" Jul 06 22:19:55.700098 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.699872 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bf23ef00-a2d0-499c-8c87-5d6b98510ea0-run-systemd\") pod \"ovnkube-node-qbbrs\" (UID: \"bf23ef00-a2d0-499c-8c87-5d6b98510ea0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:19:55.700098 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.699905 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e5f17bc2-eb9a-4dbc-a599-2f6902a646f9-agent-certs\") pod \"konnectivity-agent-65zfn\" (UID: \"e5f17bc2-eb9a-4dbc-a599-2f6902a646f9\") " pod="kube-system/konnectivity-agent-65zfn" Jul 06 22:19:55.700098 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.699930 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4bnb\" (UniqueName: \"kubernetes.io/projected/48762cd8-7ac2-449f-8f6d-49e08ec0eb54-kube-api-access-s4bnb\") pod \"tuned-2rsjs\" (UID: \"48762cd8-7ac2-449f-8f6d-49e08ec0eb54\") " pod="openshift-cluster-node-tuning-operator/tuned-2rsjs" Jul 06 22:19:55.700098 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.699956 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/48762cd8-7ac2-449f-8f6d-49e08ec0eb54-etc-kubernetes\") pod \"tuned-2rsjs\" (UID: \"48762cd8-7ac2-449f-8f6d-49e08ec0eb54\") " pod="openshift-cluster-node-tuning-operator/tuned-2rsjs" Jul 06 22:19:55.700098 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.699969 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bf23ef00-a2d0-499c-8c87-5d6b98510ea0-run-systemd\") pod \"ovnkube-node-qbbrs\" (UID: \"bf23ef00-a2d0-499c-8c87-5d6b98510ea0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:19:55.700098 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.699981 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bf23ef00-a2d0-499c-8c87-5d6b98510ea0-run-ovn\") pod \"ovnkube-node-qbbrs\" (UID: \"bf23ef00-a2d0-499c-8c87-5d6b98510ea0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:19:55.700098 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.699997 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/48762cd8-7ac2-449f-8f6d-49e08ec0eb54-etc-sysctl-d\") pod \"tuned-2rsjs\" (UID: \"48762cd8-7ac2-449f-8f6d-49e08ec0eb54\") " pod="openshift-cluster-node-tuning-operator/tuned-2rsjs" Jul 06 22:19:55.700098 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.700004 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/48762cd8-7ac2-449f-8f6d-49e08ec0eb54-lib-modules\") pod \"tuned-2rsjs\" (UID: \"48762cd8-7ac2-449f-8f6d-49e08ec0eb54\") " pod="openshift-cluster-node-tuning-operator/tuned-2rsjs" Jul 06 22:19:55.700098 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.700009 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bf23ef00-a2d0-499c-8c87-5d6b98510ea0-node-log\") pod \"ovnkube-node-qbbrs\" (UID: \"bf23ef00-a2d0-499c-8c87-5d6b98510ea0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:19:55.700098 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.700050 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bf23ef00-a2d0-499c-8c87-5d6b98510ea0-ovn-node-metrics-cert\") pod \"ovnkube-node-qbbrs\" (UID: \"bf23ef00-a2d0-499c-8c87-5d6b98510ea0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:19:55.700098 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.700064 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/48762cd8-7ac2-449f-8f6d-49e08ec0eb54-etc-kubernetes\") pod \"tuned-2rsjs\" (UID: \"48762cd8-7ac2-449f-8f6d-49e08ec0eb54\") " pod="openshift-cluster-node-tuning-operator/tuned-2rsjs" Jul 06 22:19:55.700098 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.700082 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4ebc390e-a912-4351-acdc-23f8ea5aa971-host-slash\") pod \"iptables-alerter-fk7hp\" (UID: \"4ebc390e-a912-4351-acdc-23f8ea5aa971\") " pod="openshift-network-operator/iptables-alerter-fk7hp" Jul 06 22:19:55.700935 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.700111 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/48762cd8-7ac2-449f-8f6d-49e08ec0eb54-etc-sysctl-conf\") pod \"tuned-2rsjs\" (UID: \"48762cd8-7ac2-449f-8f6d-49e08ec0eb54\") " pod="openshift-cluster-node-tuning-operator/tuned-2rsjs" Jul 06 22:19:55.700935 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.700138 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf23ef00-a2d0-499c-8c87-5d6b98510ea0-host-kubelet\") pod \"ovnkube-node-qbbrs\" (UID: \"bf23ef00-a2d0-499c-8c87-5d6b98510ea0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:19:55.700935 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.700149 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bf23ef00-a2d0-499c-8c87-5d6b98510ea0-node-log\") pod \"ovnkube-node-qbbrs\" (UID: \"bf23ef00-a2d0-499c-8c87-5d6b98510ea0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:19:55.700935 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.700170 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bf23ef00-a2d0-499c-8c87-5d6b98510ea0-ovnkube-config\") pod \"ovnkube-node-qbbrs\" (UID: \"bf23ef00-a2d0-499c-8c87-5d6b98510ea0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:19:55.700935 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.700194 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/48762cd8-7ac2-449f-8f6d-49e08ec0eb54-etc-systemd\") pod \"tuned-2rsjs\" (UID: \"48762cd8-7ac2-449f-8f6d-49e08ec0eb54\") " pod="openshift-cluster-node-tuning-operator/tuned-2rsjs" Jul 06 22:19:55.700935 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.700220 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e472a07e-8c98-4b7c-9da5-13a2240aaa4c-hosts-file\") pod \"node-resolver-l7h9q\" (UID: \"e472a07e-8c98-4b7c-9da5-13a2240aaa4c\") " pod="openshift-dns/node-resolver-l7h9q" Jul 06 22:19:55.700935 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.700261 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bf23ef00-a2d0-499c-8c87-5d6b98510ea0-log-socket\") pod \"ovnkube-node-qbbrs\" (UID: \"bf23ef00-a2d0-499c-8c87-5d6b98510ea0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:19:55.700935 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.700286 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfzsg\" (UniqueName: \"kubernetes.io/projected/bf23ef00-a2d0-499c-8c87-5d6b98510ea0-kube-api-access-cfzsg\") pod \"ovnkube-node-qbbrs\" (UID: \"bf23ef00-a2d0-499c-8c87-5d6b98510ea0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:19:55.700935 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.700308 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/48762cd8-7ac2-449f-8f6d-49e08ec0eb54-etc-sysctl-conf\") pod \"tuned-2rsjs\" (UID: \"48762cd8-7ac2-449f-8f6d-49e08ec0eb54\") " pod="openshift-cluster-node-tuning-operator/tuned-2rsjs" Jul 06 22:19:55.700935 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.700314 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bf23ef00-a2d0-499c-8c87-5d6b98510ea0-run-ovn\") pod \"ovnkube-node-qbbrs\" (UID: \"bf23ef00-a2d0-499c-8c87-5d6b98510ea0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:19:55.700935 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.700358 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4ebc390e-a912-4351-acdc-23f8ea5aa971-host-slash\") pod \"iptables-alerter-fk7hp\" (UID: \"4ebc390e-a912-4351-acdc-23f8ea5aa971\") " pod="openshift-network-operator/iptables-alerter-fk7hp" Jul 06 22:19:55.700935 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.700373 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/48762cd8-7ac2-449f-8f6d-49e08ec0eb54-etc-systemd\") pod \"tuned-2rsjs\" (UID: \"48762cd8-7ac2-449f-8f6d-49e08ec0eb54\") " pod="openshift-cluster-node-tuning-operator/tuned-2rsjs" Jul 06 22:19:55.700935 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.700315 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/48762cd8-7ac2-449f-8f6d-49e08ec0eb54-etc-tuned\") pod \"tuned-2rsjs\" (UID: \"48762cd8-7ac2-449f-8f6d-49e08ec0eb54\") " pod="openshift-cluster-node-tuning-operator/tuned-2rsjs" Jul 06 22:19:55.700935 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.700360 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf23ef00-a2d0-499c-8c87-5d6b98510ea0-host-kubelet\") pod \"ovnkube-node-qbbrs\" (UID: \"bf23ef00-a2d0-499c-8c87-5d6b98510ea0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:19:55.700935 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.700416 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bf23ef00-a2d0-499c-8c87-5d6b98510ea0-host-cni-bin\") pod \"ovnkube-node-qbbrs\" (UID: \"bf23ef00-a2d0-499c-8c87-5d6b98510ea0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:19:55.700935 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.700440 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bf23ef00-a2d0-499c-8c87-5d6b98510ea0-log-socket\") pod \"ovnkube-node-qbbrs\" (UID: \"bf23ef00-a2d0-499c-8c87-5d6b98510ea0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:19:55.700935 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.700450 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45bks\" (UniqueName: \"kubernetes.io/projected/e472a07e-8c98-4b7c-9da5-13a2240aaa4c-kube-api-access-45bks\") pod \"node-resolver-l7h9q\" (UID: \"e472a07e-8c98-4b7c-9da5-13a2240aaa4c\") " pod="openshift-dns/node-resolver-l7h9q" Jul 06 22:19:55.700935 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.700482 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf23ef00-a2d0-499c-8c87-5d6b98510ea0-etc-openvswitch\") pod \"ovnkube-node-qbbrs\" (UID: \"bf23ef00-a2d0-499c-8c87-5d6b98510ea0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:19:55.701888 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.700530 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4ebc390e-a912-4351-acdc-23f8ea5aa971-iptables-alerter-script\") pod \"iptables-alerter-fk7hp\" (UID: \"4ebc390e-a912-4351-acdc-23f8ea5aa971\") " pod="openshift-network-operator/iptables-alerter-fk7hp" Jul 06 22:19:55.701888 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.700536 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e472a07e-8c98-4b7c-9da5-13a2240aaa4c-hosts-file\") pod \"node-resolver-l7h9q\" (UID: \"e472a07e-8c98-4b7c-9da5-13a2240aaa4c\") " pod="openshift-dns/node-resolver-l7h9q" Jul 06 22:19:55.701888 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.700560 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e5f17bc2-eb9a-4dbc-a599-2f6902a646f9-konnectivity-ca\") pod \"konnectivity-agent-65zfn\" (UID: \"e5f17bc2-eb9a-4dbc-a599-2f6902a646f9\") " pod="kube-system/konnectivity-agent-65zfn" Jul 06 22:19:55.701888 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.700589 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/48762cd8-7ac2-449f-8f6d-49e08ec0eb54-etc-modprobe-d\") pod \"tuned-2rsjs\" (UID: \"48762cd8-7ac2-449f-8f6d-49e08ec0eb54\") " pod="openshift-cluster-node-tuning-operator/tuned-2rsjs" Jul 06 22:19:55.701888 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.700615 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/48762cd8-7ac2-449f-8f6d-49e08ec0eb54-host\") pod \"tuned-2rsjs\" (UID: \"48762cd8-7ac2-449f-8f6d-49e08ec0eb54\") " pod="openshift-cluster-node-tuning-operator/tuned-2rsjs" Jul 06 22:19:55.701888 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.700642 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bf23ef00-a2d0-499c-8c87-5d6b98510ea0-host-slash\") pod \"ovnkube-node-qbbrs\" (UID: \"bf23ef00-a2d0-499c-8c87-5d6b98510ea0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:19:55.701888 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.700707 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf23ef00-a2d0-499c-8c87-5d6b98510ea0-run-openvswitch\") pod \"ovnkube-node-qbbrs\" (UID: \"bf23ef00-a2d0-499c-8c87-5d6b98510ea0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:19:55.701888 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.700745 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bf23ef00-a2d0-499c-8c87-5d6b98510ea0-host-cni-netd\") pod \"ovnkube-node-qbbrs\" (UID: \"bf23ef00-a2d0-499c-8c87-5d6b98510ea0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:19:55.701888 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.700771 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/48762cd8-7ac2-449f-8f6d-49e08ec0eb54-run\") pod \"tuned-2rsjs\" (UID: \"48762cd8-7ac2-449f-8f6d-49e08ec0eb54\") " pod="openshift-cluster-node-tuning-operator/tuned-2rsjs" Jul 06 22:19:55.701888 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.700804 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf23ef00-a2d0-499c-8c87-5d6b98510ea0-var-lib-openvswitch\") pod \"ovnkube-node-qbbrs\" (UID: \"bf23ef00-a2d0-499c-8c87-5d6b98510ea0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:19:55.701888 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.700811 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bf23ef00-a2d0-499c-8c87-5d6b98510ea0-host-cni-bin\") pod \"ovnkube-node-qbbrs\" (UID: \"bf23ef00-a2d0-499c-8c87-5d6b98510ea0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:19:55.701888 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.700830 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d115b038-4f94-41df-82e3-6d42109603e0-original-pull-secret\") pod \"global-pull-secret-syncer-ccgsb\" (UID: \"d115b038-4f94-41df-82e3-6d42109603e0\") " pod="kube-system/global-pull-secret-syncer-ccgsb" Jul 06 22:19:55.701888 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.700874 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cddcda7d-c04f-4fca-a269-9c034ad1e2a1-host\") pod \"node-ca-tj7dx\" (UID: \"cddcda7d-c04f-4fca-a269-9c034ad1e2a1\") " pod="openshift-image-registry/node-ca-tj7dx" Jul 06 22:19:55.701888 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.700906 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e472a07e-8c98-4b7c-9da5-13a2240aaa4c-tmp-dir\") pod \"node-resolver-l7h9q\" (UID: \"e472a07e-8c98-4b7c-9da5-13a2240aaa4c\") " pod="openshift-dns/node-resolver-l7h9q" Jul 06 22:19:55.701888 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.700932 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bf23ef00-a2d0-499c-8c87-5d6b98510ea0-host-run-netns\") pod \"ovnkube-node-qbbrs\" (UID: \"bf23ef00-a2d0-499c-8c87-5d6b98510ea0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:19:55.701888 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.700950 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bf23ef00-a2d0-499c-8c87-5d6b98510ea0-ovnkube-config\") pod \"ovnkube-node-qbbrs\" (UID: \"bf23ef00-a2d0-499c-8c87-5d6b98510ea0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:19:55.701888 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.700957 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf23ef00-a2d0-499c-8c87-5d6b98510ea0-host-run-ovn-kubernetes\") pod \"ovnkube-node-qbbrs\" (UID: \"bf23ef00-a2d0-499c-8c87-5d6b98510ea0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:19:55.702775 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.700985 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brm7n\" (UniqueName: \"kubernetes.io/projected/4ebc390e-a912-4351-acdc-23f8ea5aa971-kube-api-access-brm7n\") pod \"iptables-alerter-fk7hp\" (UID: \"4ebc390e-a912-4351-acdc-23f8ea5aa971\") " pod="openshift-network-operator/iptables-alerter-fk7hp" Jul 06 22:19:55.702775 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.701012 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bf23ef00-a2d0-499c-8c87-5d6b98510ea0-env-overrides\") pod \"ovnkube-node-qbbrs\" (UID: \"bf23ef00-a2d0-499c-8c87-5d6b98510ea0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:19:55.702775 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.701022 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bf23ef00-a2d0-499c-8c87-5d6b98510ea0-host-cni-netd\") pod \"ovnkube-node-qbbrs\" (UID: \"bf23ef00-a2d0-499c-8c87-5d6b98510ea0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:19:55.702775 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.701044 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bf23ef00-a2d0-499c-8c87-5d6b98510ea0-ovnkube-script-lib\") pod \"ovnkube-node-qbbrs\" (UID: \"bf23ef00-a2d0-499c-8c87-5d6b98510ea0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:19:55.702775 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.701077 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/48762cd8-7ac2-449f-8f6d-49e08ec0eb54-etc-sysconfig\") pod \"tuned-2rsjs\" (UID: \"48762cd8-7ac2-449f-8f6d-49e08ec0eb54\") " pod="openshift-cluster-node-tuning-operator/tuned-2rsjs" Jul 06 22:19:55.702775 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.701079 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/48762cd8-7ac2-449f-8f6d-49e08ec0eb54-host\") pod \"tuned-2rsjs\" (UID: \"48762cd8-7ac2-449f-8f6d-49e08ec0eb54\") " pod="openshift-cluster-node-tuning-operator/tuned-2rsjs" Jul 06 22:19:55.702775 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.701103 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/48762cd8-7ac2-449f-8f6d-49e08ec0eb54-var-lib-kubelet\") pod \"tuned-2rsjs\" (UID: \"48762cd8-7ac2-449f-8f6d-49e08ec0eb54\") " pod="openshift-cluster-node-tuning-operator/tuned-2rsjs" Jul 06 22:19:55.702775 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.701107 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/48762cd8-7ac2-449f-8f6d-49e08ec0eb54-etc-modprobe-d\") pod \"tuned-2rsjs\" (UID: \"48762cd8-7ac2-449f-8f6d-49e08ec0eb54\") " pod="openshift-cluster-node-tuning-operator/tuned-2rsjs" Jul 06 22:19:55.702775 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.701119 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bf23ef00-a2d0-499c-8c87-5d6b98510ea0-host-slash\") pod \"ovnkube-node-qbbrs\" (UID: \"bf23ef00-a2d0-499c-8c87-5d6b98510ea0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:19:55.702775 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.701129 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cddcda7d-c04f-4fca-a269-9c034ad1e2a1-serviceca\") pod \"node-ca-tj7dx\" (UID: \"cddcda7d-c04f-4fca-a269-9c034ad1e2a1\") " pod="openshift-image-registry/node-ca-tj7dx" Jul 06 22:19:55.702775 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.701173 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf23ef00-a2d0-499c-8c87-5d6b98510ea0-etc-openvswitch\") pod \"ovnkube-node-qbbrs\" (UID: \"bf23ef00-a2d0-499c-8c87-5d6b98510ea0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:19:55.702775 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.701217 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf23ef00-a2d0-499c-8c87-5d6b98510ea0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qbbrs\" (UID: \"bf23ef00-a2d0-499c-8c87-5d6b98510ea0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:19:55.702775 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.701273 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d115b038-4f94-41df-82e3-6d42109603e0-dbus\") pod \"global-pull-secret-syncer-ccgsb\" (UID: \"d115b038-4f94-41df-82e3-6d42109603e0\") " pod="kube-system/global-pull-secret-syncer-ccgsb" Jul 06 22:19:55.702775 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.701325 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwd96\" (UniqueName: \"kubernetes.io/projected/cddcda7d-c04f-4fca-a269-9c034ad1e2a1-kube-api-access-gwd96\") pod \"node-ca-tj7dx\" (UID: \"cddcda7d-c04f-4fca-a269-9c034ad1e2a1\") " pod="openshift-image-registry/node-ca-tj7dx" Jul 06 22:19:55.702775 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.701613 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf23ef00-a2d0-499c-8c87-5d6b98510ea0-run-openvswitch\") pod \"ovnkube-node-qbbrs\" (UID: \"bf23ef00-a2d0-499c-8c87-5d6b98510ea0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:19:55.702775 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.701661 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cddcda7d-c04f-4fca-a269-9c034ad1e2a1-serviceca\") pod \"node-ca-tj7dx\" (UID: \"cddcda7d-c04f-4fca-a269-9c034ad1e2a1\") " pod="openshift-image-registry/node-ca-tj7dx" Jul 06 22:19:55.702775 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.701704 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e5f17bc2-eb9a-4dbc-a599-2f6902a646f9-konnectivity-ca\") pod \"konnectivity-agent-65zfn\" (UID: \"e5f17bc2-eb9a-4dbc-a599-2f6902a646f9\") " pod="kube-system/konnectivity-agent-65zfn" Jul 06 22:19:55.702775 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.701743 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/48762cd8-7ac2-449f-8f6d-49e08ec0eb54-run\") pod \"tuned-2rsjs\" (UID: \"48762cd8-7ac2-449f-8f6d-49e08ec0eb54\") " pod="openshift-cluster-node-tuning-operator/tuned-2rsjs" Jul 06 22:19:55.703914 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.701785 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf23ef00-a2d0-499c-8c87-5d6b98510ea0-host-run-ovn-kubernetes\") pod \"ovnkube-node-qbbrs\" (UID: \"bf23ef00-a2d0-499c-8c87-5d6b98510ea0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:19:55.703914 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.701835 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf23ef00-a2d0-499c-8c87-5d6b98510ea0-var-lib-openvswitch\") pod \"ovnkube-node-qbbrs\" (UID: \"bf23ef00-a2d0-499c-8c87-5d6b98510ea0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:19:55.703914 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:55.701916 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 22:19:55.703914 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:55.701990 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d115b038-4f94-41df-82e3-6d42109603e0-original-pull-secret podName:d115b038-4f94-41df-82e3-6d42109603e0 nodeName:}" failed. No retries permitted until 2026-07-06 22:19:56.2019673 +0000 UTC m=+3.120210157 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d115b038-4f94-41df-82e3-6d42109603e0-original-pull-secret") pod "global-pull-secret-syncer-ccgsb" (UID: "d115b038-4f94-41df-82e3-6d42109603e0") : object "kube-system"/"original-pull-secret" not registered Jul 06 22:19:55.703914 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.702174 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4ebc390e-a912-4351-acdc-23f8ea5aa971-iptables-alerter-script\") pod \"iptables-alerter-fk7hp\" (UID: \"4ebc390e-a912-4351-acdc-23f8ea5aa971\") " pod="openshift-network-operator/iptables-alerter-fk7hp" Jul 06 22:19:55.703914 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.702184 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cddcda7d-c04f-4fca-a269-9c034ad1e2a1-host\") pod \"node-ca-tj7dx\" (UID: \"cddcda7d-c04f-4fca-a269-9c034ad1e2a1\") " pod="openshift-image-registry/node-ca-tj7dx" Jul 06 22:19:55.703914 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.702238 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bf23ef00-a2d0-499c-8c87-5d6b98510ea0-host-run-netns\") pod \"ovnkube-node-qbbrs\" (UID: \"bf23ef00-a2d0-499c-8c87-5d6b98510ea0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:19:55.703914 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.702338 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/48762cd8-7ac2-449f-8f6d-49e08ec0eb54-etc-sysconfig\") pod \"tuned-2rsjs\" (UID: \"48762cd8-7ac2-449f-8f6d-49e08ec0eb54\") " pod="openshift-cluster-node-tuning-operator/tuned-2rsjs" Jul 06 22:19:55.703914 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.702370 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bf23ef00-a2d0-499c-8c87-5d6b98510ea0-env-overrides\") pod \"ovnkube-node-qbbrs\" (UID: \"bf23ef00-a2d0-499c-8c87-5d6b98510ea0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:19:55.703914 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.702464 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/48762cd8-7ac2-449f-8f6d-49e08ec0eb54-var-lib-kubelet\") pod \"tuned-2rsjs\" (UID: \"48762cd8-7ac2-449f-8f6d-49e08ec0eb54\") " pod="openshift-cluster-node-tuning-operator/tuned-2rsjs" Jul 06 22:19:55.703914 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.702511 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf23ef00-a2d0-499c-8c87-5d6b98510ea0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qbbrs\" (UID: \"bf23ef00-a2d0-499c-8c87-5d6b98510ea0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:19:55.703914 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.702510 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e472a07e-8c98-4b7c-9da5-13a2240aaa4c-tmp-dir\") pod \"node-resolver-l7h9q\" (UID: \"e472a07e-8c98-4b7c-9da5-13a2240aaa4c\") " pod="openshift-dns/node-resolver-l7h9q" Jul 06 22:19:55.703914 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.702624 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d115b038-4f94-41df-82e3-6d42109603e0-dbus\") pod \"global-pull-secret-syncer-ccgsb\" (UID: \"d115b038-4f94-41df-82e3-6d42109603e0\") " pod="kube-system/global-pull-secret-syncer-ccgsb" Jul 06 22:19:55.703914 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.702871 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bf23ef00-a2d0-499c-8c87-5d6b98510ea0-ovnkube-script-lib\") pod \"ovnkube-node-qbbrs\" (UID: \"bf23ef00-a2d0-499c-8c87-5d6b98510ea0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:19:55.703914 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.702940 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48762cd8-7ac2-449f-8f6d-49e08ec0eb54-tmp\") pod \"tuned-2rsjs\" (UID: \"48762cd8-7ac2-449f-8f6d-49e08ec0eb54\") " pod="openshift-cluster-node-tuning-operator/tuned-2rsjs" Jul 06 22:19:55.705307 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.705278 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bf23ef00-a2d0-499c-8c87-5d6b98510ea0-ovn-node-metrics-cert\") pod \"ovnkube-node-qbbrs\" (UID: \"bf23ef00-a2d0-499c-8c87-5d6b98510ea0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:19:55.705654 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.705632 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/48762cd8-7ac2-449f-8f6d-49e08ec0eb54-etc-tuned\") pod \"tuned-2rsjs\" (UID: \"48762cd8-7ac2-449f-8f6d-49e08ec0eb54\") " pod="openshift-cluster-node-tuning-operator/tuned-2rsjs" Jul 06 22:19:55.706184 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.705860 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e5f17bc2-eb9a-4dbc-a599-2f6902a646f9-agent-certs\") pod \"konnectivity-agent-65zfn\" (UID: \"e5f17bc2-eb9a-4dbc-a599-2f6902a646f9\") " pod="kube-system/konnectivity-agent-65zfn" Jul 06 22:19:55.708430 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:55.707885 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 22:19:55.708430 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:55.707918 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 22:19:55.708430 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:55.707932 2571 projected.go:196] Error preparing data for projected volume kube-api-access-vbw84 for pod openshift-network-diagnostics/network-check-target-fg2lc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:19:55.708430 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:55.708000 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f44d847-874a-4f0a-b3ab-6dcae0173f2d-kube-api-access-vbw84 podName:5f44d847-874a-4f0a-b3ab-6dcae0173f2d nodeName:}" failed. No retries permitted until 2026-07-06 22:19:56.207981681 +0000 UTC m=+3.126224540 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-vbw84" (UniqueName: "kubernetes.io/projected/5f44d847-874a-4f0a-b3ab-6dcae0173f2d-kube-api-access-vbw84") pod "network-check-target-fg2lc" (UID: "5f44d847-874a-4f0a-b3ab-6dcae0173f2d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:19:55.711388 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.711360 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4bnb\" (UniqueName: \"kubernetes.io/projected/48762cd8-7ac2-449f-8f6d-49e08ec0eb54-kube-api-access-s4bnb\") pod \"tuned-2rsjs\" (UID: \"48762cd8-7ac2-449f-8f6d-49e08ec0eb54\") " pod="openshift-cluster-node-tuning-operator/tuned-2rsjs" Jul 06 22:19:55.713896 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.713829 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwd96\" (UniqueName: \"kubernetes.io/projected/cddcda7d-c04f-4fca-a269-9c034ad1e2a1-kube-api-access-gwd96\") pod \"node-ca-tj7dx\" (UID: \"cddcda7d-c04f-4fca-a269-9c034ad1e2a1\") " pod="openshift-image-registry/node-ca-tj7dx" Jul 06 22:19:55.714610 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.714541 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfzsg\" (UniqueName: \"kubernetes.io/projected/bf23ef00-a2d0-499c-8c87-5d6b98510ea0-kube-api-access-cfzsg\") pod \"ovnkube-node-qbbrs\" (UID: \"bf23ef00-a2d0-499c-8c87-5d6b98510ea0\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:19:55.714710 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.714658 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brm7n\" (UniqueName: \"kubernetes.io/projected/4ebc390e-a912-4351-acdc-23f8ea5aa971-kube-api-access-brm7n\") pod \"iptables-alerter-fk7hp\" (UID: \"4ebc390e-a912-4351-acdc-23f8ea5aa971\") " pod="openshift-network-operator/iptables-alerter-fk7hp" Jul 06 22:19:55.714805 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.714782 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45bks\" (UniqueName: \"kubernetes.io/projected/e472a07e-8c98-4b7c-9da5-13a2240aaa4c-kube-api-access-45bks\") pod \"node-resolver-l7h9q\" (UID: \"e472a07e-8c98-4b7c-9da5-13a2240aaa4c\") " pod="openshift-dns/node-resolver-l7h9q" Jul 06 22:19:55.751843 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.751793 2571 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 22:19:55.790015 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.789971 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qhhbl" Jul 06 22:19:55.796527 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.796331 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xxc6f" Jul 06 22:19:55.803298 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:55.803265 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod528a881a_8c62_4448_ae10_af9c1a6243ed.slice/crio-b2fd0d084cacb02ef6d3536a03ac120bc35ac04068f96d6e502d5f38b867663d WatchSource:0}: Error finding container b2fd0d084cacb02ef6d3536a03ac120bc35ac04068f96d6e502d5f38b867663d: Status 404 returned error can't find the container with id b2fd0d084cacb02ef6d3536a03ac120bc35ac04068f96d6e502d5f38b867663d Jul 06 22:19:55.803856 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.803661 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47dsw" Jul 06 22:19:55.808994 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.808973 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-l7h9q" Jul 06 22:19:55.816027 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.815816 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:19:55.822948 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.822770 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-65zfn" Jul 06 22:19:55.828068 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:55.828038 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode472a07e_8c98_4b7c_9da5_13a2240aaa4c.slice/crio-748a3b7a71f3f0dc8db6a53178b4f68d3a07777b0d1cb88d3b2d51c3649102e9 WatchSource:0}: Error finding container 748a3b7a71f3f0dc8db6a53178b4f68d3a07777b0d1cb88d3b2d51c3649102e9: Status 404 returned error can't find the container with id 748a3b7a71f3f0dc8db6a53178b4f68d3a07777b0d1cb88d3b2d51c3649102e9 Jul 06 22:19:55.829319 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.829278 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2rsjs" Jul 06 22:19:55.838063 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.837891 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tj7dx" Jul 06 22:19:55.841886 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:55.841860 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5f17bc2_eb9a_4dbc_a599_2f6902a646f9.slice/crio-aba30cdbf91cf268e40398029f50ffa67fde85727d31f6249f19c511a8049ffd WatchSource:0}: Error finding container aba30cdbf91cf268e40398029f50ffa67fde85727d31f6249f19c511a8049ffd: Status 404 returned error can't find the container with id aba30cdbf91cf268e40398029f50ffa67fde85727d31f6249f19c511a8049ffd Jul 06 22:19:55.844325 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:55.844300 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fk7hp" Jul 06 22:19:55.844658 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:55.844545 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48762cd8_7ac2_449f_8f6d_49e08ec0eb54.slice/crio-7eb987c89527c162d0b8e3779c6802a2f841fa904bbae77e73087e59d51fc5c9 WatchSource:0}: Error finding container 7eb987c89527c162d0b8e3779c6802a2f841fa904bbae77e73087e59d51fc5c9: Status 404 returned error can't find the container with id 7eb987c89527c162d0b8e3779c6802a2f841fa904bbae77e73087e59d51fc5c9 Jul 06 22:19:55.856742 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:55.856697 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcddcda7d_c04f_4fca_a269_9c034ad1e2a1.slice/crio-47e03da6c43aa5d55ecb22a30736b4da8b6fd9d9186e229dabd77e72292bd596 WatchSource:0}: Error finding container 47e03da6c43aa5d55ecb22a30736b4da8b6fd9d9186e229dabd77e72292bd596: Status 404 returned error can't find the container with id 47e03da6c43aa5d55ecb22a30736b4da8b6fd9d9186e229dabd77e72292bd596 Jul 06 22:19:55.859022 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:19:55.858993 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ebc390e_a912_4351_acdc_23f8ea5aa971.slice/crio-5ddb113db1e5b012d6c6f492f8818d1c9fd09b80a4585c2ec426dde08a22ec54 WatchSource:0}: Error finding container 5ddb113db1e5b012d6c6f492f8818d1c9fd09b80a4585c2ec426dde08a22ec54: Status 404 returned error can't find the container with id 5ddb113db1e5b012d6c6f492f8818d1c9fd09b80a4585c2ec426dde08a22ec54 Jul 06 22:19:56.105931 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:56.105843 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfc24edf-03da-480b-b0d2-528c9098f001-metrics-certs\") pod \"network-metrics-daemon-pwdsc\" (UID: \"cfc24edf-03da-480b-b0d2-528c9098f001\") " pod="openshift-multus/network-metrics-daemon-pwdsc" Jul 06 22:19:56.106108 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:56.105975 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:19:56.106108 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:56.106049 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfc24edf-03da-480b-b0d2-528c9098f001-metrics-certs podName:cfc24edf-03da-480b-b0d2-528c9098f001 nodeName:}" failed. No retries permitted until 2026-07-06 22:19:57.10602925 +0000 UTC m=+4.024272104 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cfc24edf-03da-480b-b0d2-528c9098f001-metrics-certs") pod "network-metrics-daemon-pwdsc" (UID: "cfc24edf-03da-480b-b0d2-528c9098f001") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:19:56.206983 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:56.206937 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d115b038-4f94-41df-82e3-6d42109603e0-original-pull-secret\") pod \"global-pull-secret-syncer-ccgsb\" (UID: \"d115b038-4f94-41df-82e3-6d42109603e0\") " pod="kube-system/global-pull-secret-syncer-ccgsb" Jul 06 22:19:56.207303 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:56.207129 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 22:19:56.207303 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:56.207252 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d115b038-4f94-41df-82e3-6d42109603e0-original-pull-secret podName:d115b038-4f94-41df-82e3-6d42109603e0 nodeName:}" failed. No retries permitted until 2026-07-06 22:19:57.207213913 +0000 UTC m=+4.125456749 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d115b038-4f94-41df-82e3-6d42109603e0-original-pull-secret") pod "global-pull-secret-syncer-ccgsb" (UID: "d115b038-4f94-41df-82e3-6d42109603e0") : object "kube-system"/"original-pull-secret" not registered Jul 06 22:19:56.307593 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:56.307551 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbw84\" (UniqueName: \"kubernetes.io/projected/5f44d847-874a-4f0a-b3ab-6dcae0173f2d-kube-api-access-vbw84\") pod \"network-check-target-fg2lc\" (UID: \"5f44d847-874a-4f0a-b3ab-6dcae0173f2d\") " pod="openshift-network-diagnostics/network-check-target-fg2lc" Jul 06 22:19:56.307789 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:56.307759 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 22:19:56.307789 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:56.307786 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 22:19:56.307916 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:56.307801 2571 projected.go:196] Error preparing data for projected volume kube-api-access-vbw84 for pod openshift-network-diagnostics/network-check-target-fg2lc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:19:56.307916 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:56.307871 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f44d847-874a-4f0a-b3ab-6dcae0173f2d-kube-api-access-vbw84 podName:5f44d847-874a-4f0a-b3ab-6dcae0173f2d nodeName:}" failed. No retries permitted until 2026-07-06 22:19:57.307851031 +0000 UTC m=+4.226093891 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-vbw84" (UniqueName: "kubernetes.io/projected/5f44d847-874a-4f0a-b3ab-6dcae0173f2d-kube-api-access-vbw84") pod "network-check-target-fg2lc" (UID: "5f44d847-874a-4f0a-b3ab-6dcae0173f2d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:19:56.548772 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:56.548666 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 22:14:55 +0000 UTC" deadline="2028-02-02 03:57:44.889695195 +0000 UTC" Jul 06 22:19:56.548772 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:56.548714 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13805h37m48.340986915s" Jul 06 22:19:56.629072 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:56.629039 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ccgsb" Jul 06 22:19:56.629280 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:56.629056 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fg2lc" Jul 06 22:19:56.629280 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:56.629186 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ccgsb" podUID="d115b038-4f94-41df-82e3-6d42109603e0" Jul 06 22:19:56.629280 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:56.629267 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fg2lc" podUID="5f44d847-874a-4f0a-b3ab-6dcae0173f2d" Jul 06 22:19:56.642548 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:56.642509 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xxc6f" event={"ID":"cb6edb1a-9b3d-4809-89e1-5668948aa33f","Type":"ContainerStarted","Data":"03383510ea4717d343df18e6bf4aa17a7a3afb0861fde99bbb0e850ee416350e"} Jul 06 22:19:56.644023 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:56.643990 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-l7h9q" event={"ID":"e472a07e-8c98-4b7c-9da5-13a2240aaa4c","Type":"ContainerStarted","Data":"748a3b7a71f3f0dc8db6a53178b4f68d3a07777b0d1cb88d3b2d51c3649102e9"} Jul 06 22:19:56.645395 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:56.645359 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47dsw" event={"ID":"2dea44d6-fcc2-44f2-bace-20919aa7d774","Type":"ContainerStarted","Data":"dbb73b72cfd3503ff78c22eb9c75322c4a18f351f2ac65d763fa21441984ede7"} Jul 06 22:19:56.646653 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:56.646623 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qhhbl" event={"ID":"528a881a-8c62-4448-ae10-af9c1a6243ed","Type":"ContainerStarted","Data":"b2fd0d084cacb02ef6d3536a03ac120bc35ac04068f96d6e502d5f38b867663d"} Jul 06 22:19:56.647934 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:56.647890 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fk7hp" event={"ID":"4ebc390e-a912-4351-acdc-23f8ea5aa971","Type":"ContainerStarted","Data":"5ddb113db1e5b012d6c6f492f8818d1c9fd09b80a4585c2ec426dde08a22ec54"} Jul 06 22:19:56.649563 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:56.649518 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tj7dx" event={"ID":"cddcda7d-c04f-4fca-a269-9c034ad1e2a1","Type":"ContainerStarted","Data":"47e03da6c43aa5d55ecb22a30736b4da8b6fd9d9186e229dabd77e72292bd596"} Jul 06 22:19:56.651054 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:56.650999 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2rsjs" event={"ID":"48762cd8-7ac2-449f-8f6d-49e08ec0eb54","Type":"ContainerStarted","Data":"7eb987c89527c162d0b8e3779c6802a2f841fa904bbae77e73087e59d51fc5c9"} Jul 06 22:19:56.652300 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:56.652255 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-65zfn" event={"ID":"e5f17bc2-eb9a-4dbc-a599-2f6902a646f9","Type":"ContainerStarted","Data":"aba30cdbf91cf268e40398029f50ffa67fde85727d31f6249f19c511a8049ffd"} Jul 06 22:19:56.654017 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:56.653990 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" event={"ID":"bf23ef00-a2d0-499c-8c87-5d6b98510ea0","Type":"ContainerStarted","Data":"ac818f8101b7c6625d017707c15f994366252e04f57623a85d4eb734e2133039"} Jul 06 22:19:57.116000 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:57.115366 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfc24edf-03da-480b-b0d2-528c9098f001-metrics-certs\") pod \"network-metrics-daemon-pwdsc\" (UID: \"cfc24edf-03da-480b-b0d2-528c9098f001\") " pod="openshift-multus/network-metrics-daemon-pwdsc" Jul 06 22:19:57.116000 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:57.115567 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:19:57.116000 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:57.115635 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfc24edf-03da-480b-b0d2-528c9098f001-metrics-certs podName:cfc24edf-03da-480b-b0d2-528c9098f001 nodeName:}" failed. No retries permitted until 2026-07-06 22:19:59.115616253 +0000 UTC m=+6.033859099 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cfc24edf-03da-480b-b0d2-528c9098f001-metrics-certs") pod "network-metrics-daemon-pwdsc" (UID: "cfc24edf-03da-480b-b0d2-528c9098f001") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:19:57.216669 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:57.216042 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d115b038-4f94-41df-82e3-6d42109603e0-original-pull-secret\") pod \"global-pull-secret-syncer-ccgsb\" (UID: \"d115b038-4f94-41df-82e3-6d42109603e0\") " pod="kube-system/global-pull-secret-syncer-ccgsb" Jul 06 22:19:57.216669 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:57.216241 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 22:19:57.216669 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:57.216307 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d115b038-4f94-41df-82e3-6d42109603e0-original-pull-secret podName:d115b038-4f94-41df-82e3-6d42109603e0 nodeName:}" failed. No retries permitted until 2026-07-06 22:19:59.216287384 +0000 UTC m=+6.134530233 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d115b038-4f94-41df-82e3-6d42109603e0-original-pull-secret") pod "global-pull-secret-syncer-ccgsb" (UID: "d115b038-4f94-41df-82e3-6d42109603e0") : object "kube-system"/"original-pull-secret" not registered Jul 06 22:19:57.317641 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:57.316927 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbw84\" (UniqueName: \"kubernetes.io/projected/5f44d847-874a-4f0a-b3ab-6dcae0173f2d-kube-api-access-vbw84\") pod \"network-check-target-fg2lc\" (UID: \"5f44d847-874a-4f0a-b3ab-6dcae0173f2d\") " pod="openshift-network-diagnostics/network-check-target-fg2lc" Jul 06 22:19:57.317641 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:57.317146 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 22:19:57.317641 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:57.317165 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 22:19:57.317641 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:57.317177 2571 projected.go:196] Error preparing data for projected volume kube-api-access-vbw84 for pod openshift-network-diagnostics/network-check-target-fg2lc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:19:57.317641 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:57.317267 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f44d847-874a-4f0a-b3ab-6dcae0173f2d-kube-api-access-vbw84 podName:5f44d847-874a-4f0a-b3ab-6dcae0173f2d nodeName:}" failed. No retries permitted until 2026-07-06 22:19:59.317236474 +0000 UTC m=+6.235479316 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-vbw84" (UniqueName: "kubernetes.io/projected/5f44d847-874a-4f0a-b3ab-6dcae0173f2d-kube-api-access-vbw84") pod "network-check-target-fg2lc" (UID: "5f44d847-874a-4f0a-b3ab-6dcae0173f2d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:19:57.550048 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:57.549876 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 22:14:55 +0000 UTC" deadline="2027-12-04 14:46:36.038910169 +0000 UTC" Jul 06 22:19:57.550048 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:57.549942 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12376h26m38.488972849s" Jul 06 22:19:57.629309 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:57.629278 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pwdsc" Jul 06 22:19:57.629491 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:57.629428 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pwdsc" podUID="cfc24edf-03da-480b-b0d2-528c9098f001" Jul 06 22:19:58.629158 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:58.628483 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ccgsb" Jul 06 22:19:58.629158 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:58.628617 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ccgsb" podUID="d115b038-4f94-41df-82e3-6d42109603e0" Jul 06 22:19:58.629158 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:58.629012 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fg2lc" Jul 06 22:19:58.629158 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:58.629106 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fg2lc" podUID="5f44d847-874a-4f0a-b3ab-6dcae0173f2d" Jul 06 22:19:59.133038 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:59.132458 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfc24edf-03da-480b-b0d2-528c9098f001-metrics-certs\") pod \"network-metrics-daemon-pwdsc\" (UID: \"cfc24edf-03da-480b-b0d2-528c9098f001\") " pod="openshift-multus/network-metrics-daemon-pwdsc" Jul 06 22:19:59.133038 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:59.132599 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:19:59.133038 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:59.132665 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfc24edf-03da-480b-b0d2-528c9098f001-metrics-certs podName:cfc24edf-03da-480b-b0d2-528c9098f001 nodeName:}" failed. No retries permitted until 2026-07-06 22:20:03.132645058 +0000 UTC m=+10.050887907 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cfc24edf-03da-480b-b0d2-528c9098f001-metrics-certs") pod "network-metrics-daemon-pwdsc" (UID: "cfc24edf-03da-480b-b0d2-528c9098f001") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:19:59.233867 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:59.233810 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d115b038-4f94-41df-82e3-6d42109603e0-original-pull-secret\") pod \"global-pull-secret-syncer-ccgsb\" (UID: \"d115b038-4f94-41df-82e3-6d42109603e0\") " pod="kube-system/global-pull-secret-syncer-ccgsb" Jul 06 22:19:59.234053 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:59.233941 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 22:19:59.234053 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:59.234022 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d115b038-4f94-41df-82e3-6d42109603e0-original-pull-secret podName:d115b038-4f94-41df-82e3-6d42109603e0 nodeName:}" failed. No retries permitted until 2026-07-06 22:20:03.233999089 +0000 UTC m=+10.152241933 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d115b038-4f94-41df-82e3-6d42109603e0-original-pull-secret") pod "global-pull-secret-syncer-ccgsb" (UID: "d115b038-4f94-41df-82e3-6d42109603e0") : object "kube-system"/"original-pull-secret" not registered Jul 06 22:19:59.335567 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:59.334952 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbw84\" (UniqueName: \"kubernetes.io/projected/5f44d847-874a-4f0a-b3ab-6dcae0173f2d-kube-api-access-vbw84\") pod \"network-check-target-fg2lc\" (UID: \"5f44d847-874a-4f0a-b3ab-6dcae0173f2d\") " pod="openshift-network-diagnostics/network-check-target-fg2lc" Jul 06 22:19:59.335567 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:59.335113 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 22:19:59.335567 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:59.335133 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 22:19:59.335567 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:59.335147 2571 projected.go:196] Error preparing data for projected volume kube-api-access-vbw84 for pod openshift-network-diagnostics/network-check-target-fg2lc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:19:59.335567 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:59.335224 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f44d847-874a-4f0a-b3ab-6dcae0173f2d-kube-api-access-vbw84 podName:5f44d847-874a-4f0a-b3ab-6dcae0173f2d nodeName:}" failed. No retries permitted until 2026-07-06 22:20:03.335188155 +0000 UTC m=+10.253430992 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-vbw84" (UniqueName: "kubernetes.io/projected/5f44d847-874a-4f0a-b3ab-6dcae0173f2d-kube-api-access-vbw84") pod "network-check-target-fg2lc" (UID: "5f44d847-874a-4f0a-b3ab-6dcae0173f2d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:19:59.628922 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:19:59.628773 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pwdsc" Jul 06 22:19:59.629116 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:19:59.628927 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pwdsc" podUID="cfc24edf-03da-480b-b0d2-528c9098f001" Jul 06 22:20:00.629113 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:00.629020 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fg2lc" Jul 06 22:20:00.629555 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:00.629125 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fg2lc" podUID="5f44d847-874a-4f0a-b3ab-6dcae0173f2d" Jul 06 22:20:00.629555 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:00.629020 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ccgsb" Jul 06 22:20:00.629555 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:00.629490 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ccgsb" podUID="d115b038-4f94-41df-82e3-6d42109603e0" Jul 06 22:20:01.629992 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:01.629185 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pwdsc" Jul 06 22:20:01.629992 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:01.629699 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pwdsc" podUID="cfc24edf-03da-480b-b0d2-528c9098f001" Jul 06 22:20:02.629491 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:02.629112 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ccgsb" Jul 06 22:20:02.629491 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:02.629261 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ccgsb" podUID="d115b038-4f94-41df-82e3-6d42109603e0" Jul 06 22:20:02.629491 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:02.629350 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fg2lc" Jul 06 22:20:02.629491 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:02.629433 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fg2lc" podUID="5f44d847-874a-4f0a-b3ab-6dcae0173f2d" Jul 06 22:20:03.169183 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:03.169136 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfc24edf-03da-480b-b0d2-528c9098f001-metrics-certs\") pod \"network-metrics-daemon-pwdsc\" (UID: \"cfc24edf-03da-480b-b0d2-528c9098f001\") " pod="openshift-multus/network-metrics-daemon-pwdsc" Jul 06 22:20:03.169648 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:03.169297 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:20:03.169648 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:03.169391 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfc24edf-03da-480b-b0d2-528c9098f001-metrics-certs podName:cfc24edf-03da-480b-b0d2-528c9098f001 nodeName:}" failed. No retries permitted until 2026-07-06 22:20:11.169374916 +0000 UTC m=+18.087617753 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cfc24edf-03da-480b-b0d2-528c9098f001-metrics-certs") pod "network-metrics-daemon-pwdsc" (UID: "cfc24edf-03da-480b-b0d2-528c9098f001") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:20:03.269995 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:03.269952 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d115b038-4f94-41df-82e3-6d42109603e0-original-pull-secret\") pod \"global-pull-secret-syncer-ccgsb\" (UID: \"d115b038-4f94-41df-82e3-6d42109603e0\") " pod="kube-system/global-pull-secret-syncer-ccgsb" Jul 06 22:20:03.270176 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:03.270157 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 22:20:03.270275 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:03.270260 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d115b038-4f94-41df-82e3-6d42109603e0-original-pull-secret podName:d115b038-4f94-41df-82e3-6d42109603e0 nodeName:}" failed. No retries permitted until 2026-07-06 22:20:11.270237022 +0000 UTC m=+18.188479861 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d115b038-4f94-41df-82e3-6d42109603e0-original-pull-secret") pod "global-pull-secret-syncer-ccgsb" (UID: "d115b038-4f94-41df-82e3-6d42109603e0") : object "kube-system"/"original-pull-secret" not registered Jul 06 22:20:03.370576 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:03.370526 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbw84\" (UniqueName: \"kubernetes.io/projected/5f44d847-874a-4f0a-b3ab-6dcae0173f2d-kube-api-access-vbw84\") pod \"network-check-target-fg2lc\" (UID: \"5f44d847-874a-4f0a-b3ab-6dcae0173f2d\") " pod="openshift-network-diagnostics/network-check-target-fg2lc" Jul 06 22:20:03.370742 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:03.370711 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 22:20:03.370742 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:03.370730 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 22:20:03.370838 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:03.370744 2571 projected.go:196] Error preparing data for projected volume kube-api-access-vbw84 for pod openshift-network-diagnostics/network-check-target-fg2lc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:20:03.370838 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:03.370810 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f44d847-874a-4f0a-b3ab-6dcae0173f2d-kube-api-access-vbw84 podName:5f44d847-874a-4f0a-b3ab-6dcae0173f2d nodeName:}" failed. No retries permitted until 2026-07-06 22:20:11.370794031 +0000 UTC m=+18.289036884 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-vbw84" (UniqueName: "kubernetes.io/projected/5f44d847-874a-4f0a-b3ab-6dcae0173f2d-kube-api-access-vbw84") pod "network-check-target-fg2lc" (UID: "5f44d847-874a-4f0a-b3ab-6dcae0173f2d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:20:03.629863 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:03.629768 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pwdsc" Jul 06 22:20:03.630014 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:03.629897 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pwdsc" podUID="cfc24edf-03da-480b-b0d2-528c9098f001" Jul 06 22:20:04.629130 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:04.629083 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ccgsb" Jul 06 22:20:04.629602 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:04.629242 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ccgsb" podUID="d115b038-4f94-41df-82e3-6d42109603e0" Jul 06 22:20:04.629602 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:04.629290 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fg2lc" Jul 06 22:20:04.629602 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:04.629367 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fg2lc" podUID="5f44d847-874a-4f0a-b3ab-6dcae0173f2d" Jul 06 22:20:05.628665 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:05.628614 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pwdsc" Jul 06 22:20:05.628852 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:05.628767 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pwdsc" podUID="cfc24edf-03da-480b-b0d2-528c9098f001" Jul 06 22:20:06.629750 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:06.628802 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ccgsb" Jul 06 22:20:06.629750 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:06.628802 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fg2lc" Jul 06 22:20:06.629750 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:06.628940 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ccgsb" podUID="d115b038-4f94-41df-82e3-6d42109603e0" Jul 06 22:20:06.629750 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:06.629033 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fg2lc" podUID="5f44d847-874a-4f0a-b3ab-6dcae0173f2d" Jul 06 22:20:06.683658 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:06.683622 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-147.ec2.internal" event={"ID":"13c2e0a524b9d1d1320d115d1eeca7b4","Type":"ContainerStarted","Data":"ce9ab3f37192dac6534b5a9aedc0a59f29ea214c13941dfabbca611265d64d6a"} Jul 06 22:20:06.702149 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:06.702095 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-147.ec2.internal" podStartSLOduration=11.702075073 podStartE2EDuration="11.702075073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:19:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:20:06.701892761 +0000 UTC m=+13.620135617" watchObservedRunningTime="2026-07-06 22:20:06.702075073 +0000 UTC m=+13.620317930" Jul 06 22:20:07.628991 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:07.628910 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pwdsc" Jul 06 22:20:07.629147 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:07.629067 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pwdsc" podUID="cfc24edf-03da-480b-b0d2-528c9098f001" Jul 06 22:20:07.688847 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:07.688725 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2rsjs" event={"ID":"48762cd8-7ac2-449f-8f6d-49e08ec0eb54","Type":"ContainerStarted","Data":"ab6c044dd6abaea8d55f512073988c459e69ee449213cc5d6df49db96d1f7760"} Jul 06 22:20:08.629267 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:08.629012 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ccgsb" Jul 06 22:20:08.629424 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:08.629058 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fg2lc" Jul 06 22:20:08.629424 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:08.629319 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ccgsb" podUID="d115b038-4f94-41df-82e3-6d42109603e0" Jul 06 22:20:08.629424 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:08.629402 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fg2lc" podUID="5f44d847-874a-4f0a-b3ab-6dcae0173f2d" Jul 06 22:20:08.692451 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:08.692418 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tj7dx" event={"ID":"cddcda7d-c04f-4fca-a269-9c034ad1e2a1","Type":"ContainerStarted","Data":"71b46d75c92738d7b453416ceb3b9393342471ebd47d372f84f03a5b60cf53e0"} Jul 06 22:20:08.694100 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:08.694047 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-65zfn" event={"ID":"e5f17bc2-eb9a-4dbc-a599-2f6902a646f9","Type":"ContainerStarted","Data":"538595eab2ce11719ca046d56a0e5986ec01024163d5be79042b550363bc024f"} Jul 06 22:20:08.695648 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:08.695625 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-l7h9q" event={"ID":"e472a07e-8c98-4b7c-9da5-13a2240aaa4c","Type":"ContainerStarted","Data":"f1b39b9d5a7ed60331f7b4cd4ab3fbebe42bb7fade354c5c504c17ec34b37d24"} Jul 06 22:20:08.697299 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:08.697274 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47dsw" event={"ID":"2dea44d6-fcc2-44f2-bace-20919aa7d774","Type":"ContainerStarted","Data":"ae3332f42749f0a7a104e9636a76e98de1f2fcf308ed6fe73a43f7130300eb36"} Jul 06 22:20:08.698822 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:08.698789 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qhhbl" event={"ID":"528a881a-8c62-4448-ae10-af9c1a6243ed","Type":"ContainerStarted","Data":"2ca8fb89b264645460d44c218dff947b4c14732c8e892c80bc3a4e227b08bc8d"} Jul 06 22:20:08.700310 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:08.700280 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-147.ec2.internal" event={"ID":"023ce87fee0cc66352510ecf45bcb1ac","Type":"ContainerStarted","Data":"cfb8f90db1a11a44397fd5d6539f609623281bc5c459a61ff67e4833f81c59bc"} Jul 06 22:20:08.701846 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:08.701821 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fk7hp" event={"ID":"4ebc390e-a912-4351-acdc-23f8ea5aa971","Type":"ContainerStarted","Data":"996a52f7ea445b75eda23f0eeeb1ef55bf96bd3df72985bf55d43650f1ba23d6"} Jul 06 22:20:08.710126 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:08.710084 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-2rsjs" podStartSLOduration=3.959390398 podStartE2EDuration="14.710069692s" totalImagesPullingTime="10.750679294s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:19:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:20:07.712704741 +0000 UTC m=+14.630947610" watchObservedRunningTime="2026-07-06 22:20:08.710069692 +0000 UTC m=+15.628312548" Jul 06 22:20:08.710283 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:08.710224 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-tj7dx" podStartSLOduration=4.185111114 podStartE2EDuration="14.710199954s" totalImagesPullingTime="10.52508884s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:19:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:20:08.709881517 +0000 UTC m=+15.628124373" watchObservedRunningTime="2026-07-06 22:20:08.710199954 +0000 UTC m=+15.628442811" Jul 06 22:20:08.726738 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:08.726685 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-fk7hp" podStartSLOduration=4.202016092 podStartE2EDuration="14.726671683s" totalImagesPullingTime="10.524655591s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:19:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:20:08.726016883 +0000 UTC m=+15.644259740" watchObservedRunningTime="2026-07-06 22:20:08.726671683 +0000 UTC m=+15.644914541" Jul 06 22:20:08.733186 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:08.733149 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-65zfn" Jul 06 22:20:08.733796 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:08.733774 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-65zfn" Jul 06 22:20:08.863159 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:08.863090 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-l7h9q" podStartSLOduration=4.298806844 podStartE2EDuration="14.863072216s" totalImagesPullingTime="10.564265372s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:19:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:20:08.8620001 +0000 UTC m=+15.780242957" watchObservedRunningTime="2026-07-06 22:20:08.863072216 +0000 UTC m=+15.781315072" Jul 06 22:20:08.919568 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:08.919518 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-65zfn" podStartSLOduration=4.3967396 podStartE2EDuration="14.919506173s" totalImagesPullingTime="10.522766573s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:19:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:20:08.884962223 +0000 UTC m=+15.803205083" watchObservedRunningTime="2026-07-06 22:20:08.919506173 +0000 UTC m=+15.837749029" Jul 06 22:20:09.628269 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:09.628241 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pwdsc" Jul 06 22:20:09.628458 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:09.628362 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pwdsc" podUID="cfc24edf-03da-480b-b0d2-528c9098f001" Jul 06 22:20:09.704820 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:09.704790 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-65zfn" Jul 06 22:20:09.705424 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:09.705345 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-65zfn" Jul 06 22:20:10.628878 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:10.628825 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fg2lc" Jul 06 22:20:10.629117 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:10.628825 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ccgsb" Jul 06 22:20:10.629117 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:10.628971 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fg2lc" podUID="5f44d847-874a-4f0a-b3ab-6dcae0173f2d" Jul 06 22:20:10.629117 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:10.629009 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ccgsb" podUID="d115b038-4f94-41df-82e3-6d42109603e0" Jul 06 22:20:11.232512 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:11.232469 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfc24edf-03da-480b-b0d2-528c9098f001-metrics-certs\") pod \"network-metrics-daemon-pwdsc\" (UID: \"cfc24edf-03da-480b-b0d2-528c9098f001\") " pod="openshift-multus/network-metrics-daemon-pwdsc" Jul 06 22:20:11.233192 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:11.232634 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:20:11.233192 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:11.232713 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfc24edf-03da-480b-b0d2-528c9098f001-metrics-certs podName:cfc24edf-03da-480b-b0d2-528c9098f001 nodeName:}" failed. No retries permitted until 2026-07-06 22:20:27.232691681 +0000 UTC m=+34.150934532 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cfc24edf-03da-480b-b0d2-528c9098f001-metrics-certs") pod "network-metrics-daemon-pwdsc" (UID: "cfc24edf-03da-480b-b0d2-528c9098f001") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:20:11.333806 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:11.333756 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d115b038-4f94-41df-82e3-6d42109603e0-original-pull-secret\") pod \"global-pull-secret-syncer-ccgsb\" (UID: \"d115b038-4f94-41df-82e3-6d42109603e0\") " pod="kube-system/global-pull-secret-syncer-ccgsb" Jul 06 22:20:11.333979 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:11.333924 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 22:20:11.334064 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:11.334005 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d115b038-4f94-41df-82e3-6d42109603e0-original-pull-secret podName:d115b038-4f94-41df-82e3-6d42109603e0 nodeName:}" failed. No retries permitted until 2026-07-06 22:20:27.333984499 +0000 UTC m=+34.252227346 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d115b038-4f94-41df-82e3-6d42109603e0-original-pull-secret") pod "global-pull-secret-syncer-ccgsb" (UID: "d115b038-4f94-41df-82e3-6d42109603e0") : object "kube-system"/"original-pull-secret" not registered Jul 06 22:20:11.434710 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:11.434657 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbw84\" (UniqueName: \"kubernetes.io/projected/5f44d847-874a-4f0a-b3ab-6dcae0173f2d-kube-api-access-vbw84\") pod \"network-check-target-fg2lc\" (UID: \"5f44d847-874a-4f0a-b3ab-6dcae0173f2d\") " pod="openshift-network-diagnostics/network-check-target-fg2lc" Jul 06 22:20:11.434881 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:11.434856 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 22:20:11.434881 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:11.434881 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 22:20:11.435003 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:11.434895 2571 projected.go:196] Error preparing data for projected volume kube-api-access-vbw84 for pod openshift-network-diagnostics/network-check-target-fg2lc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:20:11.435003 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:11.434965 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f44d847-874a-4f0a-b3ab-6dcae0173f2d-kube-api-access-vbw84 podName:5f44d847-874a-4f0a-b3ab-6dcae0173f2d nodeName:}" failed. No retries permitted until 2026-07-06 22:20:27.434945468 +0000 UTC m=+34.353188320 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-vbw84" (UniqueName: "kubernetes.io/projected/5f44d847-874a-4f0a-b3ab-6dcae0173f2d-kube-api-access-vbw84") pod "network-check-target-fg2lc" (UID: "5f44d847-874a-4f0a-b3ab-6dcae0173f2d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:20:11.628646 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:11.628555 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pwdsc" Jul 06 22:20:11.628782 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:11.628718 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pwdsc" podUID="cfc24edf-03da-480b-b0d2-528c9098f001" Jul 06 22:20:11.710281 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:11.710246 2571 generic.go:356] "Generic (PLEG): container finished" podID="023ce87fee0cc66352510ecf45bcb1ac" containerID="cfb8f90db1a11a44397fd5d6539f609623281bc5c459a61ff67e4833f81c59bc" exitCode=0 Jul 06 22:20:11.710426 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:11.710321 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-147.ec2.internal" event={"ID":"023ce87fee0cc66352510ecf45bcb1ac","Type":"ContainerDied","Data":"cfb8f90db1a11a44397fd5d6539f609623281bc5c459a61ff67e4833f81c59bc"} Jul 06 22:20:11.712137 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:11.712105 2571 generic.go:356] "Generic (PLEG): container finished" podID="528a881a-8c62-4448-ae10-af9c1a6243ed" containerID="2ca8fb89b264645460d44c218dff947b4c14732c8e892c80bc3a4e227b08bc8d" exitCode=0 Jul 06 22:20:11.712269 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:11.712190 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qhhbl" event={"ID":"528a881a-8c62-4448-ae10-af9c1a6243ed","Type":"ContainerDied","Data":"2ca8fb89b264645460d44c218dff947b4c14732c8e892c80bc3a4e227b08bc8d"} Jul 06 22:20:12.628496 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:12.628457 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ccgsb" Jul 06 22:20:12.628953 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:12.628465 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fg2lc" Jul 06 22:20:12.628953 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:12.628612 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ccgsb" podUID="d115b038-4f94-41df-82e3-6d42109603e0" Jul 06 22:20:12.628953 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:12.628698 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fg2lc" podUID="5f44d847-874a-4f0a-b3ab-6dcae0173f2d" Jul 06 22:20:13.629598 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:13.629383 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pwdsc" Jul 06 22:20:13.629995 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:13.629695 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pwdsc" podUID="cfc24edf-03da-480b-b0d2-528c9098f001" Jul 06 22:20:14.628517 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:14.628479 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ccgsb" Jul 06 22:20:14.628517 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:14.628517 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fg2lc" Jul 06 22:20:14.628761 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:14.628620 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ccgsb" podUID="d115b038-4f94-41df-82e3-6d42109603e0" Jul 06 22:20:14.628818 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:14.628766 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fg2lc" podUID="5f44d847-874a-4f0a-b3ab-6dcae0173f2d" Jul 06 22:20:15.628572 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:15.628534 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pwdsc" Jul 06 22:20:15.628950 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:15.628678 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pwdsc" podUID="cfc24edf-03da-480b-b0d2-528c9098f001" Jul 06 22:20:15.722056 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:15.721958 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xxc6f" event={"ID":"cb6edb1a-9b3d-4809-89e1-5668948aa33f","Type":"ContainerStarted","Data":"12e5ccbef92a5ff5ab5688697026c16023f6f3c4b8c21353a1c0a14f1579190e"} Jul 06 22:20:15.747564 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:15.747494 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-xxc6f" podStartSLOduration=2.959216296 podStartE2EDuration="22.747474766s" totalImagesPullingTime="19.78825847s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:19:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:20:15.746893224 +0000 UTC m=+22.665136082" watchObservedRunningTime="2026-07-06 22:20:15.747474766 +0000 UTC m=+22.665717623" Jul 06 22:20:16.026803 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:16.026775 2571 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 06 22:20:16.563146 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:16.562847 2571 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-06T22:20:16.026796988Z","UUID":"cc03dfd9-7923-489e-9fd0-5c839c1c13ca","Handler":null,"Name":"","Endpoint":""} Jul 06 22:20:16.564707 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:16.564682 2571 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 06 22:20:16.564834 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:16.564722 2571 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 06 22:20:16.628373 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:16.628335 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ccgsb" Jul 06 22:20:16.628564 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:16.628439 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ccgsb" podUID="d115b038-4f94-41df-82e3-6d42109603e0" Jul 06 22:20:16.628564 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:16.628497 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fg2lc" Jul 06 22:20:16.629025 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:16.628612 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fg2lc" podUID="5f44d847-874a-4f0a-b3ab-6dcae0173f2d" Jul 06 22:20:16.727199 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:16.727171 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qbbrs_bf23ef00-a2d0-499c-8c87-5d6b98510ea0/ovn-acl-logging/0.log" Jul 06 22:20:16.727505 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:16.727484 2571 generic.go:356] "Generic (PLEG): container finished" podID="bf23ef00-a2d0-499c-8c87-5d6b98510ea0" containerID="648907d3b86e357594f4e98c0d7d7e2c4be0f899ec663cd8f4d05770e2cae890" exitCode=1 Jul 06 22:20:16.727590 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:16.727567 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" event={"ID":"bf23ef00-a2d0-499c-8c87-5d6b98510ea0","Type":"ContainerStarted","Data":"1c05c550f6ba5b5838f69c4ebd6ea056b92f90528e79cd6b755567deba6d5139"} Jul 06 22:20:16.727674 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:16.727660 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" event={"ID":"bf23ef00-a2d0-499c-8c87-5d6b98510ea0","Type":"ContainerStarted","Data":"29f021d80e246435224d559175716cb25a5a2a4257a558587f0fb6b4b388479e"} Jul 06 22:20:16.727730 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:16.727717 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" event={"ID":"bf23ef00-a2d0-499c-8c87-5d6b98510ea0","Type":"ContainerStarted","Data":"680534d2fdd0cb48ce6c98a6e7595495cb95f5756f1affe1d38f463c6e48dc40"} Jul 06 22:20:16.727785 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:16.727767 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" event={"ID":"bf23ef00-a2d0-499c-8c87-5d6b98510ea0","Type":"ContainerStarted","Data":"af9599a373d5825549238311b5816d995133ba0640bb669e9b866c863b128e0a"} Jul 06 22:20:16.727837 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:16.727818 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" event={"ID":"bf23ef00-a2d0-499c-8c87-5d6b98510ea0","Type":"ContainerDied","Data":"648907d3b86e357594f4e98c0d7d7e2c4be0f899ec663cd8f4d05770e2cae890"} Jul 06 22:20:16.727888 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:16.727871 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" event={"ID":"bf23ef00-a2d0-499c-8c87-5d6b98510ea0","Type":"ContainerStarted","Data":"df14a70762a0c95b573a95b7f41d3334aeb47a3e225625caada91005fc03afc4"} Jul 06 22:20:16.729431 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:16.729402 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47dsw" event={"ID":"2dea44d6-fcc2-44f2-bace-20919aa7d774","Type":"ContainerStarted","Data":"fc339d14ca10296d0d75fb5f203a70a3d24e58d7134913ea8458ea9c4cf191d0"} Jul 06 22:20:16.731680 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:16.731656 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-147.ec2.internal" event={"ID":"023ce87fee0cc66352510ecf45bcb1ac","Type":"ContainerStarted","Data":"e7631709f2cabe7038730910960711a52f326d2e5de983de5a5352a8287d0c9d"} Jul 06 22:20:17.628994 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:17.628958 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pwdsc" Jul 06 22:20:17.629644 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:17.629092 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pwdsc" podUID="cfc24edf-03da-480b-b0d2-528c9098f001" Jul 06 22:20:17.736918 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:17.736821 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47dsw" event={"ID":"2dea44d6-fcc2-44f2-bace-20919aa7d774","Type":"ContainerStarted","Data":"b6bab0747203a0434b314ddde3350d1627cd0d7079cc544f07aaeb967fdf6fe3"} Jul 06 22:20:17.763272 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:17.763199 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47dsw" podStartSLOduration=7.472506108 podStartE2EDuration="24.763179518s" totalImagesPullingTime="17.29067341s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:19:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:20:17.762830795 +0000 UTC m=+24.681073652" watchObservedRunningTime="2026-07-06 22:20:17.763179518 +0000 UTC m=+24.681422378" Jul 06 22:20:17.763716 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:17.763681 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-147.ec2.internal" podStartSLOduration=22.76366677 podStartE2EDuration="22.76366677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:19:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:20:16.750817597 +0000 UTC m=+23.669060453" watchObservedRunningTime="2026-07-06 22:20:17.76366677 +0000 UTC m=+24.681909628" Jul 06 22:20:18.628748 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:18.628723 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ccgsb" Jul 06 22:20:18.628748 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:18.628741 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fg2lc" Jul 06 22:20:18.628996 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:18.628823 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ccgsb" podUID="d115b038-4f94-41df-82e3-6d42109603e0" Jul 06 22:20:18.628996 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:18.628968 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fg2lc" podUID="5f44d847-874a-4f0a-b3ab-6dcae0173f2d" Jul 06 22:20:18.742538 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:18.742512 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qbbrs_bf23ef00-a2d0-499c-8c87-5d6b98510ea0/ovn-acl-logging/0.log" Jul 06 22:20:18.742953 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:18.742928 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" event={"ID":"bf23ef00-a2d0-499c-8c87-5d6b98510ea0","Type":"ContainerStarted","Data":"676c1c61f085db396c8492bad4052bf1988bb9999157e51e871e6985a2f5873e"} Jul 06 22:20:19.628580 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:19.628541 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pwdsc" Jul 06 22:20:19.628762 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:19.628672 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pwdsc" podUID="cfc24edf-03da-480b-b0d2-528c9098f001" Jul 06 22:20:20.629257 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:20.629181 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ccgsb" Jul 06 22:20:20.629257 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:20.629229 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fg2lc" Jul 06 22:20:20.629770 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:20.629338 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ccgsb" podUID="d115b038-4f94-41df-82e3-6d42109603e0" Jul 06 22:20:20.629837 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:20.629770 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fg2lc" podUID="5f44d847-874a-4f0a-b3ab-6dcae0173f2d" Jul 06 22:20:21.629273 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:21.629100 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pwdsc" Jul 06 22:20:21.629650 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:21.629358 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pwdsc" podUID="cfc24edf-03da-480b-b0d2-528c9098f001" Jul 06 22:20:21.754319 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:21.754228 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qbbrs_bf23ef00-a2d0-499c-8c87-5d6b98510ea0/ovn-acl-logging/0.log" Jul 06 22:20:21.755038 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:21.754717 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" event={"ID":"bf23ef00-a2d0-499c-8c87-5d6b98510ea0","Type":"ContainerStarted","Data":"8e14a1b1197b7aef484f58d3d66d28add1e92fa1d7173cdc6c9d2ae63176106d"} Jul 06 22:20:21.755158 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:21.755084 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:20:21.755158 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:21.755155 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:20:21.755272 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:21.755225 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:20:21.755446 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:21.755426 2571 scope.go:117] "RemoveContainer" containerID="648907d3b86e357594f4e98c0d7d7e2c4be0f899ec663cd8f4d05770e2cae890" Jul 06 22:20:21.776543 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:21.776516 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:20:21.776687 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:21.776668 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:20:22.628328 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:22.628292 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fg2lc" Jul 06 22:20:22.628527 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:22.628292 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ccgsb" Jul 06 22:20:22.628527 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:22.628396 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fg2lc" podUID="5f44d847-874a-4f0a-b3ab-6dcae0173f2d" Jul 06 22:20:22.628527 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:22.628496 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ccgsb" podUID="d115b038-4f94-41df-82e3-6d42109603e0" Jul 06 22:20:22.761756 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:22.761637 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qbbrs_bf23ef00-a2d0-499c-8c87-5d6b98510ea0/ovn-acl-logging/0.log" Jul 06 22:20:22.763685 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:22.762553 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" event={"ID":"bf23ef00-a2d0-499c-8c87-5d6b98510ea0","Type":"ContainerStarted","Data":"852f12a516e0c338a7e96a252fa5c2c7a92cef8202cadd3fc91aa9030a5129e2"} Jul 06 22:20:22.766144 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:22.765552 2571 generic.go:356] "Generic (PLEG): container finished" podID="528a881a-8c62-4448-ae10-af9c1a6243ed" containerID="f6079b76505e8d097cdb8e59ace4d1c4f2feb636aa00232d98b4934e0015dcd6" exitCode=0 Jul 06 22:20:22.766144 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:22.765606 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qhhbl" event={"ID":"528a881a-8c62-4448-ae10-af9c1a6243ed","Type":"ContainerDied","Data":"f6079b76505e8d097cdb8e59ace4d1c4f2feb636aa00232d98b4934e0015dcd6"} Jul 06 22:20:22.796922 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:22.796859 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" podStartSLOduration=8.764078768 podStartE2EDuration="28.796838317s" totalImagesPullingTime="20.032759549s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:19:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:20:22.795480704 +0000 UTC m=+29.713723560" watchObservedRunningTime="2026-07-06 22:20:22.796838317 +0000 UTC m=+29.715081177" Jul 06 22:20:23.629069 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:23.629035 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pwdsc" Jul 06 22:20:23.629327 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:23.629136 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pwdsc" podUID="cfc24edf-03da-480b-b0d2-528c9098f001" Jul 06 22:20:23.677881 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:23.677844 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ccgsb"] Jul 06 22:20:23.678065 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:23.677978 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ccgsb" Jul 06 22:20:23.678134 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:23.678079 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ccgsb" podUID="d115b038-4f94-41df-82e3-6d42109603e0" Jul 06 22:20:23.680579 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:23.680552 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pwdsc"] Jul 06 22:20:23.681030 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:23.681008 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fg2lc"] Jul 06 22:20:23.681153 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:23.681102 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fg2lc" Jul 06 22:20:23.681198 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:23.681182 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fg2lc" podUID="5f44d847-874a-4f0a-b3ab-6dcae0173f2d" Jul 06 22:20:23.767360 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:23.767333 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pwdsc" Jul 06 22:20:23.767778 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:23.767439 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pwdsc" podUID="cfc24edf-03da-480b-b0d2-528c9098f001" Jul 06 22:20:24.771340 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:24.771146 2571 generic.go:356] "Generic (PLEG): container finished" podID="528a881a-8c62-4448-ae10-af9c1a6243ed" containerID="795f03aa9b5256b061bc8b710c7564aeb100d73c8105655c9a190726c17362fe" exitCode=0 Jul 06 22:20:24.771748 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:24.771236 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qhhbl" event={"ID":"528a881a-8c62-4448-ae10-af9c1a6243ed","Type":"ContainerDied","Data":"795f03aa9b5256b061bc8b710c7564aeb100d73c8105655c9a190726c17362fe"} Jul 06 22:20:25.629125 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:25.629094 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ccgsb" Jul 06 22:20:25.629289 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:25.629100 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pwdsc" Jul 06 22:20:25.629289 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:25.629236 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fg2lc" Jul 06 22:20:25.629289 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:25.629235 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ccgsb" podUID="d115b038-4f94-41df-82e3-6d42109603e0" Jul 06 22:20:25.629408 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:25.629342 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pwdsc" podUID="cfc24edf-03da-480b-b0d2-528c9098f001" Jul 06 22:20:25.629445 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:25.629417 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fg2lc" podUID="5f44d847-874a-4f0a-b3ab-6dcae0173f2d" Jul 06 22:20:25.777228 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:25.777177 2571 generic.go:356] "Generic (PLEG): container finished" podID="528a881a-8c62-4448-ae10-af9c1a6243ed" containerID="60dd0528e63d350b73b6dcb09fad237777f555249dbedc02231bb192d34ca9c9" exitCode=0 Jul 06 22:20:25.777739 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:25.777237 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qhhbl" event={"ID":"528a881a-8c62-4448-ae10-af9c1a6243ed","Type":"ContainerDied","Data":"60dd0528e63d350b73b6dcb09fad237777f555249dbedc02231bb192d34ca9c9"} Jul 06 22:20:27.248684 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:27.248641 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfc24edf-03da-480b-b0d2-528c9098f001-metrics-certs\") pod \"network-metrics-daemon-pwdsc\" (UID: \"cfc24edf-03da-480b-b0d2-528c9098f001\") " pod="openshift-multus/network-metrics-daemon-pwdsc" Jul 06 22:20:27.249154 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:27.248815 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:20:27.249154 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:27.248887 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfc24edf-03da-480b-b0d2-528c9098f001-metrics-certs podName:cfc24edf-03da-480b-b0d2-528c9098f001 nodeName:}" failed. No retries permitted until 2026-07-06 22:20:59.248871141 +0000 UTC m=+66.167113976 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cfc24edf-03da-480b-b0d2-528c9098f001-metrics-certs") pod "network-metrics-daemon-pwdsc" (UID: "cfc24edf-03da-480b-b0d2-528c9098f001") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:20:27.349877 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:27.349841 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d115b038-4f94-41df-82e3-6d42109603e0-original-pull-secret\") pod \"global-pull-secret-syncer-ccgsb\" (UID: \"d115b038-4f94-41df-82e3-6d42109603e0\") " pod="kube-system/global-pull-secret-syncer-ccgsb" Jul 06 22:20:27.350048 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:27.349985 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 22:20:27.350099 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:27.350064 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d115b038-4f94-41df-82e3-6d42109603e0-original-pull-secret podName:d115b038-4f94-41df-82e3-6d42109603e0 nodeName:}" failed. No retries permitted until 2026-07-06 22:20:59.350047743 +0000 UTC m=+66.268290602 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d115b038-4f94-41df-82e3-6d42109603e0-original-pull-secret") pod "global-pull-secret-syncer-ccgsb" (UID: "d115b038-4f94-41df-82e3-6d42109603e0") : object "kube-system"/"original-pull-secret" not registered Jul 06 22:20:27.451033 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:27.450997 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbw84\" (UniqueName: \"kubernetes.io/projected/5f44d847-874a-4f0a-b3ab-6dcae0173f2d-kube-api-access-vbw84\") pod \"network-check-target-fg2lc\" (UID: \"5f44d847-874a-4f0a-b3ab-6dcae0173f2d\") " pod="openshift-network-diagnostics/network-check-target-fg2lc" Jul 06 22:20:27.451245 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:27.451229 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 22:20:27.451313 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:27.451274 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 22:20:27.451313 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:27.451289 2571 projected.go:196] Error preparing data for projected volume kube-api-access-vbw84 for pod openshift-network-diagnostics/network-check-target-fg2lc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:20:27.451413 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:27.451364 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f44d847-874a-4f0a-b3ab-6dcae0173f2d-kube-api-access-vbw84 podName:5f44d847-874a-4f0a-b3ab-6dcae0173f2d nodeName:}" failed. No retries permitted until 2026-07-06 22:20:59.451344769 +0000 UTC m=+66.369587606 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-vbw84" (UniqueName: "kubernetes.io/projected/5f44d847-874a-4f0a-b3ab-6dcae0173f2d-kube-api-access-vbw84") pod "network-check-target-fg2lc" (UID: "5f44d847-874a-4f0a-b3ab-6dcae0173f2d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:20:27.629121 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:27.629028 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pwdsc" Jul 06 22:20:27.629121 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:27.629061 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fg2lc" Jul 06 22:20:27.629332 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:27.629028 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ccgsb" Jul 06 22:20:27.629332 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:27.629158 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pwdsc" podUID="cfc24edf-03da-480b-b0d2-528c9098f001" Jul 06 22:20:27.629332 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:27.629243 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fg2lc" podUID="5f44d847-874a-4f0a-b3ab-6dcae0173f2d" Jul 06 22:20:27.629437 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:27.629336 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ccgsb" podUID="d115b038-4f94-41df-82e3-6d42109603e0" Jul 06 22:20:28.321066 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:28.320992 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-147.ec2.internal" event="NodeReady" Jul 06 22:20:28.321430 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:28.321119 2571 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 06 22:20:28.358266 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:28.358230 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7675579c6b-dg9ls"] Jul 06 22:20:28.362047 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:28.362028 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7675579c6b-dg9ls" Jul 06 22:20:28.364999 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:28.364967 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-mnwqj\"" Jul 06 22:20:28.365160 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:28.365137 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 06 22:20:28.365295 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:28.365163 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 06 22:20:28.365496 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:28.365475 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 06 22:20:28.373164 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:28.372988 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 06 22:20:28.375731 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:28.375708 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7675579c6b-dg9ls"] Jul 06 22:20:28.378573 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:28.378516 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xfxdc"] Jul 06 22:20:28.382817 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:28.382797 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-7hlrt"] Jul 06 22:20:28.382946 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:28.382931 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xfxdc" Jul 06 22:20:28.385834 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:28.385793 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 06 22:20:28.385995 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:28.385972 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2q2fk\"" Jul 06 22:20:28.386368 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:28.386352 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 06 22:20:28.386857 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:28.386839 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7hlrt" Jul 06 22:20:28.389280 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:28.389257 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 06 22:20:28.389420 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:28.389387 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 06 22:20:28.389713 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:28.389692 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-gmtbp\"" Jul 06 22:20:28.389794 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:28.389717 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 06 22:20:28.394325 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:28.394303 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7hlrt"] Jul 06 22:20:28.396182 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:28.396163 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xfxdc"] Jul 06 22:20:28.460263 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:28.460202 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/13cb55e3-2959-4b92-acff-2d8750c2dc62-config-volume\") pod \"dns-default-xfxdc\" (UID: \"13cb55e3-2959-4b92-acff-2d8750c2dc62\") " pod="openshift-dns/dns-default-xfxdc" Jul 06 22:20:28.460483 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:28.460307 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/13cb55e3-2959-4b92-acff-2d8750c2dc62-metrics-tls\") pod \"dns-default-xfxdc\" (UID: \"13cb55e3-2959-4b92-acff-2d8750c2dc62\") " pod="openshift-dns/dns-default-xfxdc" Jul 06 22:20:28.460483 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:28.460341 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8acf7e0c-2b66-4c40-a249-58f6fccba39e-registry-certificates\") pod \"image-registry-7675579c6b-dg9ls\" (UID: \"8acf7e0c-2b66-4c40-a249-58f6fccba39e\") " pod="openshift-image-registry/image-registry-7675579c6b-dg9ls" Jul 06 22:20:28.460483 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:28.460385 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8acf7e0c-2b66-4c40-a249-58f6fccba39e-registry-tls\") pod \"image-registry-7675579c6b-dg9ls\" (UID: \"8acf7e0c-2b66-4c40-a249-58f6fccba39e\") " pod="openshift-image-registry/image-registry-7675579c6b-dg9ls" Jul 06 22:20:28.460483 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:28.460438 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8acf7e0c-2b66-4c40-a249-58f6fccba39e-trusted-ca\") pod \"image-registry-7675579c6b-dg9ls\" (UID: \"8acf7e0c-2b66-4c40-a249-58f6fccba39e\") " pod="openshift-image-registry/image-registry-7675579c6b-dg9ls" Jul 06 22:20:28.460483 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:28.460471 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kh4q\" (UniqueName: \"kubernetes.io/projected/8acf7e0c-2b66-4c40-a249-58f6fccba39e-kube-api-access-5kh4q\") pod \"image-registry-7675579c6b-dg9ls\" (UID: \"8acf7e0c-2b66-4c40-a249-58f6fccba39e\") " pod="openshift-image-registry/image-registry-7675579c6b-dg9ls" Jul 06 22:20:28.460816 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:28.460518 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7t8z\" (UniqueName: \"kubernetes.io/projected/13cb55e3-2959-4b92-acff-2d8750c2dc62-kube-api-access-t7t8z\") pod \"dns-default-xfxdc\" (UID: \"13cb55e3-2959-4b92-acff-2d8750c2dc62\") " pod="openshift-dns/dns-default-xfxdc" Jul 06 22:20:28.460816 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:28.460536 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8acf7e0c-2b66-4c40-a249-58f6fccba39e-ca-trust-extracted\") pod \"image-registry-7675579c6b-dg9ls\" (UID: \"8acf7e0c-2b66-4c40-a249-58f6fccba39e\") " pod="openshift-image-registry/image-registry-7675579c6b-dg9ls" Jul 06 22:20:28.460816 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:28.460556 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8acf7e0c-2b66-4c40-a249-58f6fccba39e-installation-pull-secrets\") pod \"image-registry-7675579c6b-dg9ls\" (UID: \"8acf7e0c-2b66-4c40-a249-58f6fccba39e\") " pod="openshift-image-registry/image-registry-7675579c6b-dg9ls" Jul 06 22:20:28.460816 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:28.460605 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13cb55e3-2959-4b92-acff-2d8750c2dc62-tmp-dir\") pod \"dns-default-xfxdc\" (UID: \"13cb55e3-2959-4b92-acff-2d8750c2dc62\") " pod="openshift-dns/dns-default-xfxdc" Jul 06 22:20:28.460816 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:28.460650 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8acf7e0c-2b66-4c40-a249-58f6fccba39e-image-registry-private-configuration\") pod \"image-registry-7675579c6b-dg9ls\" (UID: \"8acf7e0c-2b66-4c40-a249-58f6fccba39e\") " pod="openshift-image-registry/image-registry-7675579c6b-dg9ls" Jul 06 22:20:28.460816 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:28.460704 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8acf7e0c-2b66-4c40-a249-58f6fccba39e-bound-sa-token\") pod \"image-registry-7675579c6b-dg9ls\" (UID: \"8acf7e0c-2b66-4c40-a249-58f6fccba39e\") " pod="openshift-image-registry/image-registry-7675579c6b-dg9ls" Jul 06 22:20:28.543117 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:28.543075 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:20:28.561875 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:28.561839 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kh4q\" (UniqueName: \"kubernetes.io/projected/8acf7e0c-2b66-4c40-a249-58f6fccba39e-kube-api-access-5kh4q\") pod \"image-registry-7675579c6b-dg9ls\" (UID: \"8acf7e0c-2b66-4c40-a249-58f6fccba39e\") " pod="openshift-image-registry/image-registry-7675579c6b-dg9ls" Jul 06 22:20:28.561875 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:28.561875 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/45c9cd76-e64c-4050-a5d2-88d69415cb36-cert\") pod \"ingress-canary-7hlrt\" (UID: \"45c9cd76-e64c-4050-a5d2-88d69415cb36\") " pod="openshift-ingress-canary/ingress-canary-7hlrt" Jul 06 22:20:28.562105 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:28.561893 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7t8z\" (UniqueName: \"kubernetes.io/projected/13cb55e3-2959-4b92-acff-2d8750c2dc62-kube-api-access-t7t8z\") pod \"dns-default-xfxdc\" (UID: \"13cb55e3-2959-4b92-acff-2d8750c2dc62\") " pod="openshift-dns/dns-default-xfxdc" Jul 06 22:20:28.562105 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:28.561909 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8acf7e0c-2b66-4c40-a249-58f6fccba39e-ca-trust-extracted\") pod \"image-registry-7675579c6b-dg9ls\" (UID: \"8acf7e0c-2b66-4c40-a249-58f6fccba39e\") " pod="openshift-image-registry/image-registry-7675579c6b-dg9ls" Jul 06 22:20:28.562105 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:28.561955 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8acf7e0c-2b66-4c40-a249-58f6fccba39e-installation-pull-secrets\") pod \"image-registry-7675579c6b-dg9ls\" (UID: \"8acf7e0c-2b66-4c40-a249-58f6fccba39e\") " pod="openshift-image-registry/image-registry-7675579c6b-dg9ls" Jul 06 22:20:28.562105 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:28.562085 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmgl8\" (UniqueName: \"kubernetes.io/projected/45c9cd76-e64c-4050-a5d2-88d69415cb36-kube-api-access-bmgl8\") pod \"ingress-canary-7hlrt\" (UID: \"45c9cd76-e64c-4050-a5d2-88d69415cb36\") " pod="openshift-ingress-canary/ingress-canary-7hlrt" Jul 06 22:20:28.562342 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:28.562115 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13cb55e3-2959-4b92-acff-2d8750c2dc62-tmp-dir\") pod \"dns-default-xfxdc\" (UID: \"13cb55e3-2959-4b92-acff-2d8750c2dc62\") " pod="openshift-dns/dns-default-xfxdc" Jul 06 22:20:28.562342 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:28.562141 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8acf7e0c-2b66-4c40-a249-58f6fccba39e-image-registry-private-configuration\") pod \"image-registry-7675579c6b-dg9ls\" (UID: \"8acf7e0c-2b66-4c40-a249-58f6fccba39e\") " pod="openshift-image-registry/image-registry-7675579c6b-dg9ls" Jul 06 22:20:28.562342 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:28.562187 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8acf7e0c-2b66-4c40-a249-58f6fccba39e-bound-sa-token\") pod \"image-registry-7675579c6b-dg9ls\" (UID: \"8acf7e0c-2b66-4c40-a249-58f6fccba39e\") " pod="openshift-image-registry/image-registry-7675579c6b-dg9ls" Jul 06 22:20:28.562342 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:28.562238 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/13cb55e3-2959-4b92-acff-2d8750c2dc62-config-volume\") pod \"dns-default-xfxdc\" (UID: \"13cb55e3-2959-4b92-acff-2d8750c2dc62\") " pod="openshift-dns/dns-default-xfxdc" Jul 06 22:20:28.562532 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:28.562502 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/13cb55e3-2959-4b92-acff-2d8750c2dc62-metrics-tls\") pod \"dns-default-xfxdc\" (UID: \"13cb55e3-2959-4b92-acff-2d8750c2dc62\") " pod="openshift-dns/dns-default-xfxdc" Jul 06 22:20:28.562532 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:28.562511 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13cb55e3-2959-4b92-acff-2d8750c2dc62-tmp-dir\") pod \"dns-default-xfxdc\" (UID: \"13cb55e3-2959-4b92-acff-2d8750c2dc62\") " pod="openshift-dns/dns-default-xfxdc" Jul 06 22:20:28.562629 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:28.562527 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8acf7e0c-2b66-4c40-a249-58f6fccba39e-ca-trust-extracted\") pod \"image-registry-7675579c6b-dg9ls\" (UID: \"8acf7e0c-2b66-4c40-a249-58f6fccba39e\") " pod="openshift-image-registry/image-registry-7675579c6b-dg9ls" Jul 06 22:20:28.562629 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:28.562535 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8acf7e0c-2b66-4c40-a249-58f6fccba39e-registry-certificates\") pod \"image-registry-7675579c6b-dg9ls\" (UID: \"8acf7e0c-2b66-4c40-a249-58f6fccba39e\") " pod="openshift-image-registry/image-registry-7675579c6b-dg9ls" Jul 06 22:20:28.562629 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:28.562596 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8acf7e0c-2b66-4c40-a249-58f6fccba39e-registry-tls\") pod \"image-registry-7675579c6b-dg9ls\" (UID: \"8acf7e0c-2b66-4c40-a249-58f6fccba39e\") " pod="openshift-image-registry/image-registry-7675579c6b-dg9ls" Jul 06 22:20:28.562629 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:28.562603 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 22:20:28.562954 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:28.562638 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8acf7e0c-2b66-4c40-a249-58f6fccba39e-trusted-ca\") pod \"image-registry-7675579c6b-dg9ls\" (UID: \"8acf7e0c-2b66-4c40-a249-58f6fccba39e\") " pod="openshift-image-registry/image-registry-7675579c6b-dg9ls" Jul 06 22:20:28.562954 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:28.562662 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13cb55e3-2959-4b92-acff-2d8750c2dc62-metrics-tls podName:13cb55e3-2959-4b92-acff-2d8750c2dc62 nodeName:}" failed. No retries permitted until 2026-07-06 22:20:29.062641357 +0000 UTC m=+35.980884198 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/13cb55e3-2959-4b92-acff-2d8750c2dc62-metrics-tls") pod "dns-default-xfxdc" (UID: "13cb55e3-2959-4b92-acff-2d8750c2dc62") : secret "dns-default-metrics-tls" not found Jul 06 22:20:28.562954 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:28.562737 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 22:20:28.562954 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:28.562752 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7675579c6b-dg9ls: secret "image-registry-tls" not found Jul 06 22:20:28.562954 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:28.562829 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8acf7e0c-2b66-4c40-a249-58f6fccba39e-registry-tls podName:8acf7e0c-2b66-4c40-a249-58f6fccba39e nodeName:}" failed. No retries permitted until 2026-07-06 22:20:29.062812313 +0000 UTC m=+35.981055148 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8acf7e0c-2b66-4c40-a249-58f6fccba39e-registry-tls") pod "image-registry-7675579c6b-dg9ls" (UID: "8acf7e0c-2b66-4c40-a249-58f6fccba39e") : secret "image-registry-tls" not found Jul 06 22:20:28.563247 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:28.563034 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8acf7e0c-2b66-4c40-a249-58f6fccba39e-registry-certificates\") pod \"image-registry-7675579c6b-dg9ls\" (UID: \"8acf7e0c-2b66-4c40-a249-58f6fccba39e\") " pod="openshift-image-registry/image-registry-7675579c6b-dg9ls" Jul 06 22:20:28.563352 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:28.563327 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/13cb55e3-2959-4b92-acff-2d8750c2dc62-config-volume\") pod \"dns-default-xfxdc\" (UID: \"13cb55e3-2959-4b92-acff-2d8750c2dc62\") " pod="openshift-dns/dns-default-xfxdc" Jul 06 22:20:28.563554 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:28.563531 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8acf7e0c-2b66-4c40-a249-58f6fccba39e-trusted-ca\") pod \"image-registry-7675579c6b-dg9ls\" (UID: \"8acf7e0c-2b66-4c40-a249-58f6fccba39e\") " pod="openshift-image-registry/image-registry-7675579c6b-dg9ls" Jul 06 22:20:28.566237 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:28.566191 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8acf7e0c-2b66-4c40-a249-58f6fccba39e-installation-pull-secrets\") pod \"image-registry-7675579c6b-dg9ls\" (UID: \"8acf7e0c-2b66-4c40-a249-58f6fccba39e\") " pod="openshift-image-registry/image-registry-7675579c6b-dg9ls" Jul 06 22:20:28.566371 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:28.566191 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8acf7e0c-2b66-4c40-a249-58f6fccba39e-image-registry-private-configuration\") pod \"image-registry-7675579c6b-dg9ls\" (UID: \"8acf7e0c-2b66-4c40-a249-58f6fccba39e\") " pod="openshift-image-registry/image-registry-7675579c6b-dg9ls" Jul 06 22:20:28.572002 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:28.571654 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7t8z\" (UniqueName: \"kubernetes.io/projected/13cb55e3-2959-4b92-acff-2d8750c2dc62-kube-api-access-t7t8z\") pod \"dns-default-xfxdc\" (UID: \"13cb55e3-2959-4b92-acff-2d8750c2dc62\") " pod="openshift-dns/dns-default-xfxdc" Jul 06 22:20:28.572002 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:28.571903 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8acf7e0c-2b66-4c40-a249-58f6fccba39e-bound-sa-token\") pod \"image-registry-7675579c6b-dg9ls\" (UID: \"8acf7e0c-2b66-4c40-a249-58f6fccba39e\") " pod="openshift-image-registry/image-registry-7675579c6b-dg9ls" Jul 06 22:20:28.572314 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:28.572070 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kh4q\" (UniqueName: \"kubernetes.io/projected/8acf7e0c-2b66-4c40-a249-58f6fccba39e-kube-api-access-5kh4q\") pod \"image-registry-7675579c6b-dg9ls\" (UID: \"8acf7e0c-2b66-4c40-a249-58f6fccba39e\") " pod="openshift-image-registry/image-registry-7675579c6b-dg9ls" Jul 06 22:20:28.630821 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:28.630561 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:20:28.663376 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:28.663340 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/45c9cd76-e64c-4050-a5d2-88d69415cb36-cert\") pod \"ingress-canary-7hlrt\" (UID: \"45c9cd76-e64c-4050-a5d2-88d69415cb36\") " pod="openshift-ingress-canary/ingress-canary-7hlrt" Jul 06 22:20:28.663376 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:28.663384 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmgl8\" (UniqueName: \"kubernetes.io/projected/45c9cd76-e64c-4050-a5d2-88d69415cb36-kube-api-access-bmgl8\") pod \"ingress-canary-7hlrt\" (UID: \"45c9cd76-e64c-4050-a5d2-88d69415cb36\") " pod="openshift-ingress-canary/ingress-canary-7hlrt" Jul 06 22:20:28.666704 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:28.666676 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/45c9cd76-e64c-4050-a5d2-88d69415cb36-cert\") pod \"ingress-canary-7hlrt\" (UID: \"45c9cd76-e64c-4050-a5d2-88d69415cb36\") " pod="openshift-ingress-canary/ingress-canary-7hlrt" Jul 06 22:20:28.673348 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:28.673322 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmgl8\" (UniqueName: \"kubernetes.io/projected/45c9cd76-e64c-4050-a5d2-88d69415cb36-kube-api-access-bmgl8\") pod \"ingress-canary-7hlrt\" (UID: \"45c9cd76-e64c-4050-a5d2-88d69415cb36\") " pod="openshift-ingress-canary/ingress-canary-7hlrt" Jul 06 22:20:28.701570 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:28.701535 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7hlrt" Jul 06 22:20:28.849962 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:28.849884 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7hlrt"] Jul 06 22:20:28.853819 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:20:28.853786 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45c9cd76_e64c_4050_a5d2_88d69415cb36.slice/crio-d0915ef8405f15b918cd672de610175e838d4b82b198abf829e555f97b6b65c1 WatchSource:0}: Error finding container d0915ef8405f15b918cd672de610175e838d4b82b198abf829e555f97b6b65c1: Status 404 returned error can't find the container with id d0915ef8405f15b918cd672de610175e838d4b82b198abf829e555f97b6b65c1 Jul 06 22:20:29.067252 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:29.067193 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/13cb55e3-2959-4b92-acff-2d8750c2dc62-metrics-tls\") pod \"dns-default-xfxdc\" (UID: \"13cb55e3-2959-4b92-acff-2d8750c2dc62\") " pod="openshift-dns/dns-default-xfxdc" Jul 06 22:20:29.067479 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:29.067265 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8acf7e0c-2b66-4c40-a249-58f6fccba39e-registry-tls\") pod \"image-registry-7675579c6b-dg9ls\" (UID: \"8acf7e0c-2b66-4c40-a249-58f6fccba39e\") " pod="openshift-image-registry/image-registry-7675579c6b-dg9ls" Jul 06 22:20:29.067479 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:29.067371 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 22:20:29.067479 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:29.067404 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 22:20:29.067479 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:29.067418 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7675579c6b-dg9ls: secret "image-registry-tls" not found Jul 06 22:20:29.067479 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:29.067447 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13cb55e3-2959-4b92-acff-2d8750c2dc62-metrics-tls podName:13cb55e3-2959-4b92-acff-2d8750c2dc62 nodeName:}" failed. No retries permitted until 2026-07-06 22:20:30.067426646 +0000 UTC m=+36.985669500 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/13cb55e3-2959-4b92-acff-2d8750c2dc62-metrics-tls") pod "dns-default-xfxdc" (UID: "13cb55e3-2959-4b92-acff-2d8750c2dc62") : secret "dns-default-metrics-tls" not found Jul 06 22:20:29.067791 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:29.067503 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8acf7e0c-2b66-4c40-a249-58f6fccba39e-registry-tls podName:8acf7e0c-2b66-4c40-a249-58f6fccba39e nodeName:}" failed. No retries permitted until 2026-07-06 22:20:30.067490957 +0000 UTC m=+36.985733797 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8acf7e0c-2b66-4c40-a249-58f6fccba39e-registry-tls") pod "image-registry-7675579c6b-dg9ls" (UID: "8acf7e0c-2b66-4c40-a249-58f6fccba39e") : secret "image-registry-tls" not found Jul 06 22:20:29.628784 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:29.628695 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ccgsb" Jul 06 22:20:29.629361 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:29.628695 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fg2lc" Jul 06 22:20:29.629361 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:29.628840 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pwdsc" Jul 06 22:20:29.631954 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:29.631819 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 22:20:29.631954 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:29.631832 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 22:20:29.631954 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:29.631858 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 22:20:29.632853 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:29.632836 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zplcn\"" Jul 06 22:20:29.632966 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:29.632861 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-t8ckr\"" Jul 06 22:20:29.633144 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:29.633090 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 22:20:29.789536 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:29.789477 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7hlrt" event={"ID":"45c9cd76-e64c-4050-a5d2-88d69415cb36","Type":"ContainerStarted","Data":"d0915ef8405f15b918cd672de610175e838d4b82b198abf829e555f97b6b65c1"} Jul 06 22:20:30.076106 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:30.076063 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/13cb55e3-2959-4b92-acff-2d8750c2dc62-metrics-tls\") pod \"dns-default-xfxdc\" (UID: \"13cb55e3-2959-4b92-acff-2d8750c2dc62\") " pod="openshift-dns/dns-default-xfxdc" Jul 06 22:20:30.076311 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:30.076132 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8acf7e0c-2b66-4c40-a249-58f6fccba39e-registry-tls\") pod \"image-registry-7675579c6b-dg9ls\" (UID: \"8acf7e0c-2b66-4c40-a249-58f6fccba39e\") " pod="openshift-image-registry/image-registry-7675579c6b-dg9ls" Jul 06 22:20:30.076311 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:30.076261 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 22:20:30.076438 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:30.076263 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 22:20:30.076438 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:30.076364 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7675579c6b-dg9ls: secret "image-registry-tls" not found Jul 06 22:20:30.076438 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:30.076370 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13cb55e3-2959-4b92-acff-2d8750c2dc62-metrics-tls podName:13cb55e3-2959-4b92-acff-2d8750c2dc62 nodeName:}" failed. No retries permitted until 2026-07-06 22:20:32.076350187 +0000 UTC m=+38.994593044 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/13cb55e3-2959-4b92-acff-2d8750c2dc62-metrics-tls") pod "dns-default-xfxdc" (UID: "13cb55e3-2959-4b92-acff-2d8750c2dc62") : secret "dns-default-metrics-tls" not found Jul 06 22:20:30.076438 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:30.076411 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8acf7e0c-2b66-4c40-a249-58f6fccba39e-registry-tls podName:8acf7e0c-2b66-4c40-a249-58f6fccba39e nodeName:}" failed. No retries permitted until 2026-07-06 22:20:32.076400312 +0000 UTC m=+38.994643151 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8acf7e0c-2b66-4c40-a249-58f6fccba39e-registry-tls") pod "image-registry-7675579c6b-dg9ls" (UID: "8acf7e0c-2b66-4c40-a249-58f6fccba39e") : secret "image-registry-tls" not found Jul 06 22:20:32.092587 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:32.092542 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/13cb55e3-2959-4b92-acff-2d8750c2dc62-metrics-tls\") pod \"dns-default-xfxdc\" (UID: \"13cb55e3-2959-4b92-acff-2d8750c2dc62\") " pod="openshift-dns/dns-default-xfxdc" Jul 06 22:20:32.093096 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:32.092609 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8acf7e0c-2b66-4c40-a249-58f6fccba39e-registry-tls\") pod \"image-registry-7675579c6b-dg9ls\" (UID: \"8acf7e0c-2b66-4c40-a249-58f6fccba39e\") " pod="openshift-image-registry/image-registry-7675579c6b-dg9ls" Jul 06 22:20:32.093096 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:32.092720 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 22:20:32.093096 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:32.092722 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 22:20:32.093096 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:32.092732 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7675579c6b-dg9ls: secret "image-registry-tls" not found Jul 06 22:20:32.093096 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:32.092799 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13cb55e3-2959-4b92-acff-2d8750c2dc62-metrics-tls podName:13cb55e3-2959-4b92-acff-2d8750c2dc62 nodeName:}" failed. No retries permitted until 2026-07-06 22:20:36.092778268 +0000 UTC m=+43.011021113 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/13cb55e3-2959-4b92-acff-2d8750c2dc62-metrics-tls") pod "dns-default-xfxdc" (UID: "13cb55e3-2959-4b92-acff-2d8750c2dc62") : secret "dns-default-metrics-tls" not found Jul 06 22:20:32.093096 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:32.092855 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8acf7e0c-2b66-4c40-a249-58f6fccba39e-registry-tls podName:8acf7e0c-2b66-4c40-a249-58f6fccba39e nodeName:}" failed. No retries permitted until 2026-07-06 22:20:36.092840207 +0000 UTC m=+43.011083045 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8acf7e0c-2b66-4c40-a249-58f6fccba39e-registry-tls") pod "image-registry-7675579c6b-dg9ls" (UID: "8acf7e0c-2b66-4c40-a249-58f6fccba39e") : secret "image-registry-tls" not found Jul 06 22:20:32.593086 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:32.593049 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:20:32.706345 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:32.706306 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:20:32.805238 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:32.805177 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:20:32.879063 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:32.878981 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:20:33.628168 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:33.627919 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:20:33.692135 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:33.692080 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:20:33.751927 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:33.751890 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:20:33.801480 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:33.801448 2571 generic.go:356] "Generic (PLEG): container finished" podID="528a881a-8c62-4448-ae10-af9c1a6243ed" containerID="ca8ea441d43790e89ac5a090962362cbcb498d9cd2d10b6c8f4c5e8d087236ca" exitCode=0 Jul 06 22:20:33.801635 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:33.801523 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qhhbl" event={"ID":"528a881a-8c62-4448-ae10-af9c1a6243ed","Type":"ContainerDied","Data":"ca8ea441d43790e89ac5a090962362cbcb498d9cd2d10b6c8f4c5e8d087236ca"} Jul 06 22:20:33.803022 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:33.802997 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7hlrt" event={"ID":"45c9cd76-e64c-4050-a5d2-88d69415cb36","Type":"ContainerStarted","Data":"86a4bb4dbf0a3d9f081025963e1fb296f9b42322d89bc56632748faa5b07258b"} Jul 06 22:20:33.849460 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:33.849411 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-7hlrt" podStartSLOduration=1.876805832 podStartE2EDuration="5.849387944s" totalImagesPullingTime="3.972582112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:20:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:20:33.848839164 +0000 UTC m=+40.767082023" watchObservedRunningTime="2026-07-06 22:20:33.849387944 +0000 UTC m=+40.767630804" Jul 06 22:20:34.810178 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:34.810134 2571 generic.go:356] "Generic (PLEG): container finished" podID="528a881a-8c62-4448-ae10-af9c1a6243ed" containerID="88cd4c947d1274ee3649e40aa02e3feb2cd6cd22bfa132b2439ea83e587bf3bb" exitCode=0 Jul 06 22:20:34.810618 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:34.810201 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qhhbl" event={"ID":"528a881a-8c62-4448-ae10-af9c1a6243ed","Type":"ContainerDied","Data":"88cd4c947d1274ee3649e40aa02e3feb2cd6cd22bfa132b2439ea83e587bf3bb"} Jul 06 22:20:35.817796 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:35.817759 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qhhbl" event={"ID":"528a881a-8c62-4448-ae10-af9c1a6243ed","Type":"ContainerStarted","Data":"8e870d31be9693f527100176005ec96d390bdeb23433450e933bd3baa252a13a"} Jul 06 22:20:35.974924 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:35.974893 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:20:36.075009 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:36.074921 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:20:36.122984 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:36.122949 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/13cb55e3-2959-4b92-acff-2d8750c2dc62-metrics-tls\") pod \"dns-default-xfxdc\" (UID: \"13cb55e3-2959-4b92-acff-2d8750c2dc62\") " pod="openshift-dns/dns-default-xfxdc" Jul 06 22:20:36.123437 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:36.123397 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 22:20:36.123437 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:36.122999 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8acf7e0c-2b66-4c40-a249-58f6fccba39e-registry-tls\") pod \"image-registry-7675579c6b-dg9ls\" (UID: \"8acf7e0c-2b66-4c40-a249-58f6fccba39e\") " pod="openshift-image-registry/image-registry-7675579c6b-dg9ls" Jul 06 22:20:36.123642 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:36.123509 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13cb55e3-2959-4b92-acff-2d8750c2dc62-metrics-tls podName:13cb55e3-2959-4b92-acff-2d8750c2dc62 nodeName:}" failed. No retries permitted until 2026-07-06 22:20:44.123473017 +0000 UTC m=+51.041715864 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/13cb55e3-2959-4b92-acff-2d8750c2dc62-metrics-tls") pod "dns-default-xfxdc" (UID: "13cb55e3-2959-4b92-acff-2d8750c2dc62") : secret "dns-default-metrics-tls" not found Jul 06 22:20:36.123642 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:36.123557 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 22:20:36.123642 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:36.123580 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7675579c6b-dg9ls: secret "image-registry-tls" not found Jul 06 22:20:36.123642 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:36.123645 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8acf7e0c-2b66-4c40-a249-58f6fccba39e-registry-tls podName:8acf7e0c-2b66-4c40-a249-58f6fccba39e nodeName:}" failed. No retries permitted until 2026-07-06 22:20:44.123627806 +0000 UTC m=+51.041870641 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8acf7e0c-2b66-4c40-a249-58f6fccba39e-registry-tls") pod "image-registry-7675579c6b-dg9ls" (UID: "8acf7e0c-2b66-4c40-a249-58f6fccba39e") : secret "image-registry-tls" not found Jul 06 22:20:39.536252 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:39.536187 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-qhhbl" podStartSLOduration=16.600826075 podStartE2EDuration="46.536168598s" totalImagesPullingTime="29.935342523s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:19:53 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:20:35.846746189 +0000 UTC m=+42.764989045" watchObservedRunningTime="2026-07-06 22:20:39.536168598 +0000 UTC m=+46.454411497" Jul 06 22:20:39.536836 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:39.536350 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-829dv"] Jul 06 22:20:39.571570 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:39.571538 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-829dv"] Jul 06 22:20:39.571728 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:39.571654 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-829dv" Jul 06 22:20:39.574369 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:39.574343 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 06 22:20:39.574509 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:39.574379 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-v96t6\"" Jul 06 22:20:39.575199 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:39.575176 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 06 22:20:39.575199 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:39.575181 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 06 22:20:39.638768 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:39.638734 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-4x6pf"] Jul 06 22:20:39.661915 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:39.661854 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-4x6pf"] Jul 06 22:20:39.662082 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:39.661970 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-4x6pf" Jul 06 22:20:39.664852 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:39.664827 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 06 22:20:39.664985 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:39.664830 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 06 22:20:39.664985 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:39.664893 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 06 22:20:39.665086 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:39.664997 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 06 22:20:39.665132 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:39.665117 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-xbd2r\"" Jul 06 22:20:39.673667 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:39.673648 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 06 22:20:39.755179 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:39.755134 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:20:39.756111 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:39.756085 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/63d474a2-5a9a-4c1d-8711-bbac91ebfee2-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-829dv\" (UID: \"63d474a2-5a9a-4c1d-8711-bbac91ebfee2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-829dv" Jul 06 22:20:39.756245 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:39.756115 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgpkt\" (UniqueName: \"kubernetes.io/projected/63d474a2-5a9a-4c1d-8711-bbac91ebfee2-kube-api-access-bgpkt\") pod \"cluster-samples-operator-56fcf86bf-829dv\" (UID: \"63d474a2-5a9a-4c1d-8711-bbac91ebfee2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-829dv" Jul 06 22:20:39.852862 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:39.852775 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:20:39.856742 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:39.856716 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5796a8ea-adcb-4819-bb18-0b85e1ef4681-trusted-ca\") pod \"console-operator-68b757fb67-4x6pf\" (UID: \"5796a8ea-adcb-4819-bb18-0b85e1ef4681\") " pod="openshift-console-operator/console-operator-68b757fb67-4x6pf" Jul 06 22:20:39.856904 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:39.856772 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5796a8ea-adcb-4819-bb18-0b85e1ef4681-serving-cert\") pod \"console-operator-68b757fb67-4x6pf\" (UID: \"5796a8ea-adcb-4819-bb18-0b85e1ef4681\") " pod="openshift-console-operator/console-operator-68b757fb67-4x6pf" Jul 06 22:20:39.856972 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:39.856894 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/63d474a2-5a9a-4c1d-8711-bbac91ebfee2-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-829dv\" (UID: \"63d474a2-5a9a-4c1d-8711-bbac91ebfee2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-829dv" Jul 06 22:20:39.856972 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:39.856927 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgpkt\" (UniqueName: \"kubernetes.io/projected/63d474a2-5a9a-4c1d-8711-bbac91ebfee2-kube-api-access-bgpkt\") pod \"cluster-samples-operator-56fcf86bf-829dv\" (UID: \"63d474a2-5a9a-4c1d-8711-bbac91ebfee2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-829dv" Jul 06 22:20:39.856972 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:39.856965 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5796a8ea-adcb-4819-bb18-0b85e1ef4681-config\") pod \"console-operator-68b757fb67-4x6pf\" (UID: \"5796a8ea-adcb-4819-bb18-0b85e1ef4681\") " pod="openshift-console-operator/console-operator-68b757fb67-4x6pf" Jul 06 22:20:39.857118 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:39.857012 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkd6r\" (UniqueName: \"kubernetes.io/projected/5796a8ea-adcb-4819-bb18-0b85e1ef4681-kube-api-access-nkd6r\") pod \"console-operator-68b757fb67-4x6pf\" (UID: \"5796a8ea-adcb-4819-bb18-0b85e1ef4681\") " pod="openshift-console-operator/console-operator-68b757fb67-4x6pf" Jul 06 22:20:39.857118 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:39.857024 2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 22:20:39.857118 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:39.857074 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63d474a2-5a9a-4c1d-8711-bbac91ebfee2-samples-operator-tls podName:63d474a2-5a9a-4c1d-8711-bbac91ebfee2 nodeName:}" failed. No retries permitted until 2026-07-06 22:20:40.357059521 +0000 UTC m=+47.275302357 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/63d474a2-5a9a-4c1d-8711-bbac91ebfee2-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-829dv" (UID: "63d474a2-5a9a-4c1d-8711-bbac91ebfee2") : secret "samples-operator-tls" not found Jul 06 22:20:39.872002 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:39.871975 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgpkt\" (UniqueName: \"kubernetes.io/projected/63d474a2-5a9a-4c1d-8711-bbac91ebfee2-kube-api-access-bgpkt\") pod \"cluster-samples-operator-56fcf86bf-829dv\" (UID: \"63d474a2-5a9a-4c1d-8711-bbac91ebfee2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-829dv" Jul 06 22:20:39.957871 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:39.957822 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5796a8ea-adcb-4819-bb18-0b85e1ef4681-trusted-ca\") pod \"console-operator-68b757fb67-4x6pf\" (UID: \"5796a8ea-adcb-4819-bb18-0b85e1ef4681\") " pod="openshift-console-operator/console-operator-68b757fb67-4x6pf" Jul 06 22:20:39.958060 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:39.957911 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5796a8ea-adcb-4819-bb18-0b85e1ef4681-serving-cert\") pod \"console-operator-68b757fb67-4x6pf\" (UID: \"5796a8ea-adcb-4819-bb18-0b85e1ef4681\") " pod="openshift-console-operator/console-operator-68b757fb67-4x6pf" Jul 06 22:20:39.958060 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:39.957995 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5796a8ea-adcb-4819-bb18-0b85e1ef4681-config\") pod \"console-operator-68b757fb67-4x6pf\" (UID: \"5796a8ea-adcb-4819-bb18-0b85e1ef4681\") " pod="openshift-console-operator/console-operator-68b757fb67-4x6pf" Jul 06 22:20:39.958060 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:39.958037 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkd6r\" (UniqueName: \"kubernetes.io/projected/5796a8ea-adcb-4819-bb18-0b85e1ef4681-kube-api-access-nkd6r\") pod \"console-operator-68b757fb67-4x6pf\" (UID: \"5796a8ea-adcb-4819-bb18-0b85e1ef4681\") " pod="openshift-console-operator/console-operator-68b757fb67-4x6pf" Jul 06 22:20:39.958711 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:39.958681 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5796a8ea-adcb-4819-bb18-0b85e1ef4681-config\") pod \"console-operator-68b757fb67-4x6pf\" (UID: \"5796a8ea-adcb-4819-bb18-0b85e1ef4681\") " pod="openshift-console-operator/console-operator-68b757fb67-4x6pf" Jul 06 22:20:39.958975 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:39.958953 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5796a8ea-adcb-4819-bb18-0b85e1ef4681-trusted-ca\") pod \"console-operator-68b757fb67-4x6pf\" (UID: \"5796a8ea-adcb-4819-bb18-0b85e1ef4681\") " pod="openshift-console-operator/console-operator-68b757fb67-4x6pf" Jul 06 22:20:39.960337 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:39.960314 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5796a8ea-adcb-4819-bb18-0b85e1ef4681-serving-cert\") pod \"console-operator-68b757fb67-4x6pf\" (UID: \"5796a8ea-adcb-4819-bb18-0b85e1ef4681\") " pod="openshift-console-operator/console-operator-68b757fb67-4x6pf" Jul 06 22:20:39.968181 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:39.968150 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkd6r\" (UniqueName: \"kubernetes.io/projected/5796a8ea-adcb-4819-bb18-0b85e1ef4681-kube-api-access-nkd6r\") pod \"console-operator-68b757fb67-4x6pf\" (UID: \"5796a8ea-adcb-4819-bb18-0b85e1ef4681\") " pod="openshift-console-operator/console-operator-68b757fb67-4x6pf" Jul 06 22:20:39.970822 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:39.970796 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-4x6pf" Jul 06 22:20:39.994387 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:39.994182 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:20:40.094728 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:40.094693 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:20:40.115810 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:40.115724 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-4x6pf"] Jul 06 22:20:40.119660 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:20:40.119637 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5796a8ea_adcb_4819_bb18_0b85e1ef4681.slice/crio-4d5916067beae1c5676137d8a010b218f7e3913fe73fa4d071d9fd5cbb08b398 WatchSource:0}: Error finding container 4d5916067beae1c5676137d8a010b218f7e3913fe73fa4d071d9fd5cbb08b398: Status 404 returned error can't find the container with id 4d5916067beae1c5676137d8a010b218f7e3913fe73fa4d071d9fd5cbb08b398 Jul 06 22:20:40.361336 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:40.361303 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/63d474a2-5a9a-4c1d-8711-bbac91ebfee2-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-829dv\" (UID: \"63d474a2-5a9a-4c1d-8711-bbac91ebfee2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-829dv" Jul 06 22:20:40.361492 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:40.361460 2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 22:20:40.361530 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:40.361524 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63d474a2-5a9a-4c1d-8711-bbac91ebfee2-samples-operator-tls podName:63d474a2-5a9a-4c1d-8711-bbac91ebfee2 nodeName:}" failed. No retries permitted until 2026-07-06 22:20:41.361506125 +0000 UTC m=+48.279748959 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/63d474a2-5a9a-4c1d-8711-bbac91ebfee2-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-829dv" (UID: "63d474a2-5a9a-4c1d-8711-bbac91ebfee2") : secret "samples-operator-tls" not found Jul 06 22:20:40.492037 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:40.492001 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-fn24h"] Jul 06 22:20:40.523440 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:40.523403 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-fn24h"] Jul 06 22:20:40.523597 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:40.523502 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-fn24h" Jul 06 22:20:40.526956 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:40.526921 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 06 22:20:40.526956 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:40.526953 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 06 22:20:40.527179 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:40.526964 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 06 22:20:40.527179 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:40.526964 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-fsznr\"" Jul 06 22:20:40.527179 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:40.526959 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 06 22:20:40.663019 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:40.662985 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9c4f4c11-7942-49f9-9473-c14ea497795d-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-fn24h\" (UID: \"9c4f4c11-7942-49f9-9473-c14ea497795d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-fn24h" Jul 06 22:20:40.663516 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:40.663040 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5scr\" (UniqueName: \"kubernetes.io/projected/9c4f4c11-7942-49f9-9473-c14ea497795d-kube-api-access-n5scr\") pod \"kube-storage-version-migrator-operator-846bf668f7-fn24h\" (UID: \"9c4f4c11-7942-49f9-9473-c14ea497795d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-fn24h" Jul 06 22:20:40.663516 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:40.663163 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9c4f4c11-7942-49f9-9473-c14ea497795d-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-fn24h\" (UID: \"9c4f4c11-7942-49f9-9473-c14ea497795d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-fn24h" Jul 06 22:20:40.764055 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:40.763959 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5scr\" (UniqueName: \"kubernetes.io/projected/9c4f4c11-7942-49f9-9473-c14ea497795d-kube-api-access-n5scr\") pod \"kube-storage-version-migrator-operator-846bf668f7-fn24h\" (UID: \"9c4f4c11-7942-49f9-9473-c14ea497795d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-fn24h" Jul 06 22:20:40.764055 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:40.764007 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9c4f4c11-7942-49f9-9473-c14ea497795d-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-fn24h\" (UID: \"9c4f4c11-7942-49f9-9473-c14ea497795d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-fn24h" Jul 06 22:20:40.764275 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:40.764089 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9c4f4c11-7942-49f9-9473-c14ea497795d-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-fn24h\" (UID: \"9c4f4c11-7942-49f9-9473-c14ea497795d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-fn24h" Jul 06 22:20:40.764594 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:40.764575 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9c4f4c11-7942-49f9-9473-c14ea497795d-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-fn24h\" (UID: \"9c4f4c11-7942-49f9-9473-c14ea497795d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-fn24h" Jul 06 22:20:40.767512 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:40.767484 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9c4f4c11-7942-49f9-9473-c14ea497795d-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-fn24h\" (UID: \"9c4f4c11-7942-49f9-9473-c14ea497795d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-fn24h" Jul 06 22:20:40.774431 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:40.774388 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5scr\" (UniqueName: \"kubernetes.io/projected/9c4f4c11-7942-49f9-9473-c14ea497795d-kube-api-access-n5scr\") pod \"kube-storage-version-migrator-operator-846bf668f7-fn24h\" (UID: \"9c4f4c11-7942-49f9-9473-c14ea497795d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-fn24h" Jul 06 22:20:40.831087 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:40.831040 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-4x6pf" event={"ID":"5796a8ea-adcb-4819-bb18-0b85e1ef4681","Type":"ContainerStarted","Data":"4d5916067beae1c5676137d8a010b218f7e3913fe73fa4d071d9fd5cbb08b398"} Jul 06 22:20:40.833320 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:40.833294 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-fn24h" Jul 06 22:20:40.989183 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:40.989150 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-fn24h"] Jul 06 22:20:40.992937 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:20:40.992901 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c4f4c11_7942_49f9_9473_c14ea497795d.slice/crio-2d8ccedeb06575f485b0645990d0658950c2a81de8feff87b568fc8a34b04edd WatchSource:0}: Error finding container 2d8ccedeb06575f485b0645990d0658950c2a81de8feff87b568fc8a34b04edd: Status 404 returned error can't find the container with id 2d8ccedeb06575f485b0645990d0658950c2a81de8feff87b568fc8a34b04edd Jul 06 22:20:41.368692 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:41.368655 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/63d474a2-5a9a-4c1d-8711-bbac91ebfee2-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-829dv\" (UID: \"63d474a2-5a9a-4c1d-8711-bbac91ebfee2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-829dv" Jul 06 22:20:41.368876 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:41.368803 2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 22:20:41.368938 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:41.368878 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63d474a2-5a9a-4c1d-8711-bbac91ebfee2-samples-operator-tls podName:63d474a2-5a9a-4c1d-8711-bbac91ebfee2 nodeName:}" failed. No retries permitted until 2026-07-06 22:20:43.368854221 +0000 UTC m=+50.287097078 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/63d474a2-5a9a-4c1d-8711-bbac91ebfee2-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-829dv" (UID: "63d474a2-5a9a-4c1d-8711-bbac91ebfee2") : secret "samples-operator-tls" not found Jul 06 22:20:41.835131 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:41.835089 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-fn24h" event={"ID":"9c4f4c11-7942-49f9-9473-c14ea497795d","Type":"ContainerStarted","Data":"2d8ccedeb06575f485b0645990d0658950c2a81de8feff87b568fc8a34b04edd"} Jul 06 22:20:42.202134 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:42.201886 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:20:42.302480 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:42.302444 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:20:42.497090 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:42.497013 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-s8bk9"] Jul 06 22:20:42.517994 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:42.517966 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-s8bk9"] Jul 06 22:20:42.518165 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:42.518063 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-s8bk9" Jul 06 22:20:42.520491 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:42.520468 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-jb85f\"" Jul 06 22:20:42.580066 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:42.580014 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txkg7\" (UniqueName: \"kubernetes.io/projected/3734aee6-8f88-4a2d-b6bb-0687f4c4dba2-kube-api-access-txkg7\") pod \"network-check-source-989f8b8dc-s8bk9\" (UID: \"3734aee6-8f88-4a2d-b6bb-0687f4c4dba2\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-s8bk9" Jul 06 22:20:42.680582 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:42.680558 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txkg7\" (UniqueName: \"kubernetes.io/projected/3734aee6-8f88-4a2d-b6bb-0687f4c4dba2-kube-api-access-txkg7\") pod \"network-check-source-989f8b8dc-s8bk9\" (UID: \"3734aee6-8f88-4a2d-b6bb-0687f4c4dba2\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-s8bk9" Jul 06 22:20:42.690814 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:42.690789 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txkg7\" (UniqueName: \"kubernetes.io/projected/3734aee6-8f88-4a2d-b6bb-0687f4c4dba2-kube-api-access-txkg7\") pod \"network-check-source-989f8b8dc-s8bk9\" (UID: \"3734aee6-8f88-4a2d-b6bb-0687f4c4dba2\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-s8bk9" Jul 06 22:20:42.826825 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:42.826795 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-s8bk9" Jul 06 22:20:42.839819 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:42.839781 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-4x6pf" event={"ID":"5796a8ea-adcb-4819-bb18-0b85e1ef4681","Type":"ContainerStarted","Data":"48c0ddc4762f63ec17b06ac36bec407b3941d000cd5a6f9d1d374068c1c05afb"} Jul 06 22:20:42.840246 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:42.840058 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-4x6pf" Jul 06 22:20:42.841531 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:42.841506 2571 patch_prober.go:28] interesting pod/console-operator-68b757fb67-4x6pf container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" start-of-body= Jul 06 22:20:42.841639 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:42.841550 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-68b757fb67-4x6pf" podUID="5796a8ea-adcb-4819-bb18-0b85e1ef4681" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" Jul 06 22:20:43.387523 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:43.387488 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/63d474a2-5a9a-4c1d-8711-bbac91ebfee2-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-829dv\" (UID: \"63d474a2-5a9a-4c1d-8711-bbac91ebfee2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-829dv" Jul 06 22:20:43.387696 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:43.387679 2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 22:20:43.387776 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:43.387763 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63d474a2-5a9a-4c1d-8711-bbac91ebfee2-samples-operator-tls podName:63d474a2-5a9a-4c1d-8711-bbac91ebfee2 nodeName:}" failed. No retries permitted until 2026-07-06 22:20:47.387740456 +0000 UTC m=+54.305983290 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/63d474a2-5a9a-4c1d-8711-bbac91ebfee2-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-829dv" (UID: "63d474a2-5a9a-4c1d-8711-bbac91ebfee2") : secret "samples-operator-tls" not found Jul 06 22:20:43.438316 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:43.437955 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-68b757fb67-4x6pf" podStartSLOduration=1.882499972 podStartE2EDuration="4.437932047s" totalImagesPullingTime="2.555432075s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:20:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:20:42.858148974 +0000 UTC m=+49.776391834" watchObservedRunningTime="2026-07-06 22:20:43.437932047 +0000 UTC m=+50.356174903" Jul 06 22:20:43.439110 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:43.439083 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-s8bk9"] Jul 06 22:20:43.442234 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:20:43.442186 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3734aee6_8f88_4a2d_b6bb_0687f4c4dba2.slice/crio-4545f873cdbf081a81eccb1b090f41d55c836193f9430082d8ac9be93b7b28e9 WatchSource:0}: Error finding container 4545f873cdbf081a81eccb1b090f41d55c836193f9430082d8ac9be93b7b28e9: Status 404 returned error can't find the container with id 4545f873cdbf081a81eccb1b090f41d55c836193f9430082d8ac9be93b7b28e9 Jul 06 22:20:43.843769 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:43.843733 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-fn24h" event={"ID":"9c4f4c11-7942-49f9-9473-c14ea497795d","Type":"ContainerStarted","Data":"95624c7dc2c6098fef54f9182f6c52d017e091436580bf43e921284e0e81f70d"} Jul 06 22:20:43.845019 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:43.844989 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-s8bk9" event={"ID":"3734aee6-8f88-4a2d-b6bb-0687f4c4dba2","Type":"ContainerStarted","Data":"4545f873cdbf081a81eccb1b090f41d55c836193f9430082d8ac9be93b7b28e9"} Jul 06 22:20:43.846541 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:43.846523 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-4x6pf_5796a8ea-adcb-4819-bb18-0b85e1ef4681/console-operator/0.log" Jul 06 22:20:43.846662 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:43.846557 2571 generic.go:356] "Generic (PLEG): container finished" podID="5796a8ea-adcb-4819-bb18-0b85e1ef4681" containerID="48c0ddc4762f63ec17b06ac36bec407b3941d000cd5a6f9d1d374068c1c05afb" exitCode=255 Jul 06 22:20:43.846662 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:43.846592 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-4x6pf" event={"ID":"5796a8ea-adcb-4819-bb18-0b85e1ef4681","Type":"ContainerDied","Data":"48c0ddc4762f63ec17b06ac36bec407b3941d000cd5a6f9d1d374068c1c05afb"} Jul 06 22:20:43.846918 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:43.846799 2571 scope.go:117] "RemoveContainer" containerID="48c0ddc4762f63ec17b06ac36bec407b3941d000cd5a6f9d1d374068c1c05afb" Jul 06 22:20:43.864517 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:43.864466 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-fn24h" podStartSLOduration=1.250939176 podStartE2EDuration="3.864449926s" totalImagesPullingTime="2.61351075s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:20:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:20:43.863048724 +0000 UTC m=+50.781291593" watchObservedRunningTime="2026-07-06 22:20:43.864449926 +0000 UTC m=+50.782692781" Jul 06 22:20:44.049978 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:44.049940 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:20:44.149787 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:44.149698 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:20:44.195644 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:44.195600 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/13cb55e3-2959-4b92-acff-2d8750c2dc62-metrics-tls\") pod \"dns-default-xfxdc\" (UID: \"13cb55e3-2959-4b92-acff-2d8750c2dc62\") " pod="openshift-dns/dns-default-xfxdc" Jul 06 22:20:44.195842 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:44.195670 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8acf7e0c-2b66-4c40-a249-58f6fccba39e-registry-tls\") pod \"image-registry-7675579c6b-dg9ls\" (UID: \"8acf7e0c-2b66-4c40-a249-58f6fccba39e\") " pod="openshift-image-registry/image-registry-7675579c6b-dg9ls" Jul 06 22:20:44.195842 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:44.195776 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 22:20:44.195842 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:44.195826 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 22:20:44.196007 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:44.195846 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7675579c6b-dg9ls: secret "image-registry-tls" not found Jul 06 22:20:44.196007 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:44.195848 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13cb55e3-2959-4b92-acff-2d8750c2dc62-metrics-tls podName:13cb55e3-2959-4b92-acff-2d8750c2dc62 nodeName:}" failed. No retries permitted until 2026-07-06 22:21:00.195827311 +0000 UTC m=+67.114070167 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/13cb55e3-2959-4b92-acff-2d8750c2dc62-metrics-tls") pod "dns-default-xfxdc" (UID: "13cb55e3-2959-4b92-acff-2d8750c2dc62") : secret "dns-default-metrics-tls" not found Jul 06 22:20:44.196007 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:44.195910 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8acf7e0c-2b66-4c40-a249-58f6fccba39e-registry-tls podName:8acf7e0c-2b66-4c40-a249-58f6fccba39e nodeName:}" failed. No retries permitted until 2026-07-06 22:21:00.195891584 +0000 UTC m=+67.114134426 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8acf7e0c-2b66-4c40-a249-58f6fccba39e-registry-tls") pod "image-registry-7675579c6b-dg9ls" (UID: "8acf7e0c-2b66-4c40-a249-58f6fccba39e") : secret "image-registry-tls" not found Jul 06 22:20:44.267325 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:44.267292 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:20:44.776729 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:44.776693 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:20:44.851150 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:44.851035 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:20:44.851150 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:44.851133 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-4x6pf_5796a8ea-adcb-4819-bb18-0b85e1ef4681/console-operator/1.log" Jul 06 22:20:44.851695 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:44.851619 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-4x6pf_5796a8ea-adcb-4819-bb18-0b85e1ef4681/console-operator/0.log" Jul 06 22:20:44.851695 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:44.851662 2571 generic.go:356] "Generic (PLEG): container finished" podID="5796a8ea-adcb-4819-bb18-0b85e1ef4681" containerID="0055ac975bdf22a2fe997a6301b1d6fc68c764873e76edf76cbe9273752f5b37" exitCode=255 Jul 06 22:20:44.851805 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:44.851758 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-4x6pf" event={"ID":"5796a8ea-adcb-4819-bb18-0b85e1ef4681","Type":"ContainerDied","Data":"0055ac975bdf22a2fe997a6301b1d6fc68c764873e76edf76cbe9273752f5b37"} Jul 06 22:20:44.851857 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:44.851823 2571 scope.go:117] "RemoveContainer" containerID="48c0ddc4762f63ec17b06ac36bec407b3941d000cd5a6f9d1d374068c1c05afb" Jul 06 22:20:44.852081 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:44.852065 2571 scope.go:117] "RemoveContainer" containerID="0055ac975bdf22a2fe997a6301b1d6fc68c764873e76edf76cbe9273752f5b37" Jul 06 22:20:44.852324 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:44.852298 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-4x6pf_openshift-console-operator(5796a8ea-adcb-4819-bb18-0b85e1ef4681)\"" pod="openshift-console-operator/console-operator-68b757fb67-4x6pf" podUID="5796a8ea-adcb-4819-bb18-0b85e1ef4681" Jul 06 22:20:44.907048 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:44.907020 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:20:45.188375 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:45.188340 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2ktvb"] Jul 06 22:20:45.216246 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:45.216168 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:20:45.220335 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:45.220306 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2ktvb"] Jul 06 22:20:45.220516 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:45.220411 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2ktvb" Jul 06 22:20:45.222740 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:45.222711 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 06 22:20:45.222870 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:45.222780 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 06 22:20:45.222870 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:45.222839 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 06 22:20:45.222984 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:45.222727 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-dhrwd\"" Jul 06 22:20:45.223111 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:45.223096 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 06 22:20:45.278494 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:45.278385 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:20:45.305572 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:45.305535 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/46b9db84-34e1-414a-a394-2809f8f0fa0b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2ktvb\" (UID: \"46b9db84-34e1-414a-a394-2809f8f0fa0b\") " pod="openshift-insights/insights-runtime-extractor-2ktvb" Jul 06 22:20:45.305753 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:45.305623 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/46b9db84-34e1-414a-a394-2809f8f0fa0b-crio-socket\") pod \"insights-runtime-extractor-2ktvb\" (UID: \"46b9db84-34e1-414a-a394-2809f8f0fa0b\") " pod="openshift-insights/insights-runtime-extractor-2ktvb" Jul 06 22:20:45.305753 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:45.305661 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l79dk\" (UniqueName: \"kubernetes.io/projected/46b9db84-34e1-414a-a394-2809f8f0fa0b-kube-api-access-l79dk\") pod \"insights-runtime-extractor-2ktvb\" (UID: \"46b9db84-34e1-414a-a394-2809f8f0fa0b\") " pod="openshift-insights/insights-runtime-extractor-2ktvb" Jul 06 22:20:45.305753 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:45.305687 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/46b9db84-34e1-414a-a394-2809f8f0fa0b-data-volume\") pod \"insights-runtime-extractor-2ktvb\" (UID: \"46b9db84-34e1-414a-a394-2809f8f0fa0b\") " pod="openshift-insights/insights-runtime-extractor-2ktvb" Jul 06 22:20:45.305753 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:45.305721 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/46b9db84-34e1-414a-a394-2809f8f0fa0b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2ktvb\" (UID: \"46b9db84-34e1-414a-a394-2809f8f0fa0b\") " pod="openshift-insights/insights-runtime-extractor-2ktvb" Jul 06 22:20:45.407050 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:45.407011 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/46b9db84-34e1-414a-a394-2809f8f0fa0b-crio-socket\") pod \"insights-runtime-extractor-2ktvb\" (UID: \"46b9db84-34e1-414a-a394-2809f8f0fa0b\") " pod="openshift-insights/insights-runtime-extractor-2ktvb" Jul 06 22:20:45.407239 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:45.407087 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l79dk\" (UniqueName: \"kubernetes.io/projected/46b9db84-34e1-414a-a394-2809f8f0fa0b-kube-api-access-l79dk\") pod \"insights-runtime-extractor-2ktvb\" (UID: \"46b9db84-34e1-414a-a394-2809f8f0fa0b\") " pod="openshift-insights/insights-runtime-extractor-2ktvb" Jul 06 22:20:45.407239 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:45.407121 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/46b9db84-34e1-414a-a394-2809f8f0fa0b-data-volume\") pod \"insights-runtime-extractor-2ktvb\" (UID: \"46b9db84-34e1-414a-a394-2809f8f0fa0b\") " pod="openshift-insights/insights-runtime-extractor-2ktvb" Jul 06 22:20:45.407239 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:45.407145 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/46b9db84-34e1-414a-a394-2809f8f0fa0b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2ktvb\" (UID: \"46b9db84-34e1-414a-a394-2809f8f0fa0b\") " pod="openshift-insights/insights-runtime-extractor-2ktvb" Jul 06 22:20:45.407239 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:45.407228 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/46b9db84-34e1-414a-a394-2809f8f0fa0b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2ktvb\" (UID: \"46b9db84-34e1-414a-a394-2809f8f0fa0b\") " pod="openshift-insights/insights-runtime-extractor-2ktvb" Jul 06 22:20:45.407439 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:45.407271 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/46b9db84-34e1-414a-a394-2809f8f0fa0b-crio-socket\") pod \"insights-runtime-extractor-2ktvb\" (UID: \"46b9db84-34e1-414a-a394-2809f8f0fa0b\") " pod="openshift-insights/insights-runtime-extractor-2ktvb" Jul 06 22:20:45.407439 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:45.407375 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 22:20:45.407580 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:45.407529 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/46b9db84-34e1-414a-a394-2809f8f0fa0b-data-volume\") pod \"insights-runtime-extractor-2ktvb\" (UID: \"46b9db84-34e1-414a-a394-2809f8f0fa0b\") " pod="openshift-insights/insights-runtime-extractor-2ktvb" Jul 06 22:20:45.407580 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:45.407548 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46b9db84-34e1-414a-a394-2809f8f0fa0b-insights-runtime-extractor-tls podName:46b9db84-34e1-414a-a394-2809f8f0fa0b nodeName:}" failed. No retries permitted until 2026-07-06 22:20:45.907525061 +0000 UTC m=+52.825767898 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/46b9db84-34e1-414a-a394-2809f8f0fa0b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2ktvb" (UID: "46b9db84-34e1-414a-a394-2809f8f0fa0b") : secret "insights-runtime-extractor-tls" not found Jul 06 22:20:45.407717 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:45.407592 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 22:20:45.407717 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:45.407659 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/46b9db84-34e1-414a-a394-2809f8f0fa0b-kube-rbac-proxy-cm podName:46b9db84-34e1-414a-a394-2809f8f0fa0b nodeName:}" failed. No retries permitted until 2026-07-06 22:20:45.907641894 +0000 UTC m=+52.825884743 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/46b9db84-34e1-414a-a394-2809f8f0fa0b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2ktvb" (UID: "46b9db84-34e1-414a-a394-2809f8f0fa0b") : configmap "kube-rbac-proxy" not found Jul 06 22:20:45.417056 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:45.417022 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l79dk\" (UniqueName: \"kubernetes.io/projected/46b9db84-34e1-414a-a394-2809f8f0fa0b-kube-api-access-l79dk\") pod \"insights-runtime-extractor-2ktvb\" (UID: \"46b9db84-34e1-414a-a394-2809f8f0fa0b\") " pod="openshift-insights/insights-runtime-extractor-2ktvb" Jul 06 22:20:45.856652 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:45.856606 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-4x6pf_5796a8ea-adcb-4819-bb18-0b85e1ef4681/console-operator/1.log" Jul 06 22:20:45.857095 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:45.856994 2571 scope.go:117] "RemoveContainer" containerID="0055ac975bdf22a2fe997a6301b1d6fc68c764873e76edf76cbe9273752f5b37" Jul 06 22:20:45.857199 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:45.857178 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-4x6pf_openshift-console-operator(5796a8ea-adcb-4819-bb18-0b85e1ef4681)\"" pod="openshift-console-operator/console-operator-68b757fb67-4x6pf" podUID="5796a8ea-adcb-4819-bb18-0b85e1ef4681" Jul 06 22:20:45.911511 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:45.911469 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/46b9db84-34e1-414a-a394-2809f8f0fa0b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2ktvb\" (UID: \"46b9db84-34e1-414a-a394-2809f8f0fa0b\") " pod="openshift-insights/insights-runtime-extractor-2ktvb" Jul 06 22:20:45.911704 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:45.911595 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/46b9db84-34e1-414a-a394-2809f8f0fa0b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2ktvb\" (UID: \"46b9db84-34e1-414a-a394-2809f8f0fa0b\") " pod="openshift-insights/insights-runtime-extractor-2ktvb" Jul 06 22:20:45.911704 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:45.911630 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 22:20:45.911704 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:45.911697 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 22:20:45.911827 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:45.911709 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/46b9db84-34e1-414a-a394-2809f8f0fa0b-kube-rbac-proxy-cm podName:46b9db84-34e1-414a-a394-2809f8f0fa0b nodeName:}" failed. No retries permitted until 2026-07-06 22:20:46.911689042 +0000 UTC m=+53.829931878 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/46b9db84-34e1-414a-a394-2809f8f0fa0b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2ktvb" (UID: "46b9db84-34e1-414a-a394-2809f8f0fa0b") : configmap "kube-rbac-proxy" not found Jul 06 22:20:45.911827 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:45.911752 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46b9db84-34e1-414a-a394-2809f8f0fa0b-insights-runtime-extractor-tls podName:46b9db84-34e1-414a-a394-2809f8f0fa0b nodeName:}" failed. No retries permitted until 2026-07-06 22:20:46.911735787 +0000 UTC m=+53.829978642 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/46b9db84-34e1-414a-a394-2809f8f0fa0b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2ktvb" (UID: "46b9db84-34e1-414a-a394-2809f8f0fa0b") : secret "insights-runtime-extractor-tls" not found Jul 06 22:20:46.921336 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:46.921290 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/46b9db84-34e1-414a-a394-2809f8f0fa0b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2ktvb\" (UID: \"46b9db84-34e1-414a-a394-2809f8f0fa0b\") " pod="openshift-insights/insights-runtime-extractor-2ktvb" Jul 06 22:20:46.921876 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:46.921392 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/46b9db84-34e1-414a-a394-2809f8f0fa0b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2ktvb\" (UID: \"46b9db84-34e1-414a-a394-2809f8f0fa0b\") " pod="openshift-insights/insights-runtime-extractor-2ktvb" Jul 06 22:20:46.921876 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:46.921447 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 22:20:46.921876 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:46.921522 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 22:20:46.921876 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:46.921531 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46b9db84-34e1-414a-a394-2809f8f0fa0b-insights-runtime-extractor-tls podName:46b9db84-34e1-414a-a394-2809f8f0fa0b nodeName:}" failed. No retries permitted until 2026-07-06 22:20:48.921511271 +0000 UTC m=+55.839754128 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/46b9db84-34e1-414a-a394-2809f8f0fa0b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2ktvb" (UID: "46b9db84-34e1-414a-a394-2809f8f0fa0b") : secret "insights-runtime-extractor-tls" not found Jul 06 22:20:46.921876 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:46.921597 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/46b9db84-34e1-414a-a394-2809f8f0fa0b-kube-rbac-proxy-cm podName:46b9db84-34e1-414a-a394-2809f8f0fa0b nodeName:}" failed. No retries permitted until 2026-07-06 22:20:48.921579542 +0000 UTC m=+55.839822381 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/46b9db84-34e1-414a-a394-2809f8f0fa0b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2ktvb" (UID: "46b9db84-34e1-414a-a394-2809f8f0fa0b") : configmap "kube-rbac-proxy" not found Jul 06 22:20:47.425813 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:47.425767 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/63d474a2-5a9a-4c1d-8711-bbac91ebfee2-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-829dv\" (UID: \"63d474a2-5a9a-4c1d-8711-bbac91ebfee2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-829dv" Jul 06 22:20:47.425994 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:47.425916 2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 22:20:47.425994 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:47.425983 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63d474a2-5a9a-4c1d-8711-bbac91ebfee2-samples-operator-tls podName:63d474a2-5a9a-4c1d-8711-bbac91ebfee2 nodeName:}" failed. No retries permitted until 2026-07-06 22:20:55.425967182 +0000 UTC m=+62.344210017 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/63d474a2-5a9a-4c1d-8711-bbac91ebfee2-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-829dv" (UID: "63d474a2-5a9a-4c1d-8711-bbac91ebfee2") : secret "samples-operator-tls" not found Jul 06 22:20:47.667315 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:47.667276 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:20:47.765774 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:47.765669 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:20:47.864552 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:47.864512 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-s8bk9" event={"ID":"3734aee6-8f88-4a2d-b6bb-0687f4c4dba2","Type":"ContainerStarted","Data":"f9618c6f21b96d1de8465f289d62a95fe49e6249d6947eae18974af6286593cf"} Jul 06 22:20:47.883434 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:47.883379 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-s8bk9" podStartSLOduration=2.210335013 podStartE2EDuration="5.88335914s" totalImagesPullingTime="3.673024127s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:20:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:20:47.883197948 +0000 UTC m=+54.801440803" watchObservedRunningTime="2026-07-06 22:20:47.88335914 +0000 UTC m=+54.801602003" Jul 06 22:20:48.051111 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:48.051019 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:20:48.112777 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:48.112736 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:20:48.182733 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:48.182700 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-l7h9q_e472a07e-8c98-4b7c-9da5-13a2240aaa4c/dns-node-resolver/0.log" Jul 06 22:20:48.477246 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:48.477201 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:20:48.569191 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:48.569157 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:20:48.757600 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:48.757505 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:20:48.850747 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:48.850715 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:20:48.938395 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:48.938355 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/46b9db84-34e1-414a-a394-2809f8f0fa0b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2ktvb\" (UID: \"46b9db84-34e1-414a-a394-2809f8f0fa0b\") " pod="openshift-insights/insights-runtime-extractor-2ktvb" Jul 06 22:20:48.938560 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:48.938448 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/46b9db84-34e1-414a-a394-2809f8f0fa0b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2ktvb\" (UID: \"46b9db84-34e1-414a-a394-2809f8f0fa0b\") " pod="openshift-insights/insights-runtime-extractor-2ktvb" Jul 06 22:20:48.938560 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:48.938513 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 22:20:48.938647 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:48.938574 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 22:20:48.938647 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:48.938579 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46b9db84-34e1-414a-a394-2809f8f0fa0b-insights-runtime-extractor-tls podName:46b9db84-34e1-414a-a394-2809f8f0fa0b nodeName:}" failed. No retries permitted until 2026-07-06 22:20:52.938562317 +0000 UTC m=+59.856805160 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/46b9db84-34e1-414a-a394-2809f8f0fa0b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2ktvb" (UID: "46b9db84-34e1-414a-a394-2809f8f0fa0b") : secret "insights-runtime-extractor-tls" not found Jul 06 22:20:48.938730 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:48.938715 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/46b9db84-34e1-414a-a394-2809f8f0fa0b-kube-rbac-proxy-cm podName:46b9db84-34e1-414a-a394-2809f8f0fa0b nodeName:}" failed. No retries permitted until 2026-07-06 22:20:52.938699383 +0000 UTC m=+59.856942216 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/46b9db84-34e1-414a-a394-2809f8f0fa0b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2ktvb" (UID: "46b9db84-34e1-414a-a394-2809f8f0fa0b") : configmap "kube-rbac-proxy" not found Jul 06 22:20:48.982588 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:48.982554 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tj7dx_cddcda7d-c04f-4fca-a269-9c034ad1e2a1/node-ca/0.log" Jul 06 22:20:49.482202 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:49.482169 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:20:49.582559 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:49.582529 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:20:49.971870 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:49.971832 2571 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-4x6pf" Jul 06 22:20:49.972303 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:49.972288 2571 scope.go:117] "RemoveContainer" containerID="0055ac975bdf22a2fe997a6301b1d6fc68c764873e76edf76cbe9273752f5b37" Jul 06 22:20:49.972461 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:49.972442 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-4x6pf_openshift-console-operator(5796a8ea-adcb-4819-bb18-0b85e1ef4681)\"" pod="openshift-console-operator/console-operator-68b757fb67-4x6pf" podUID="5796a8ea-adcb-4819-bb18-0b85e1ef4681" Jul 06 22:20:49.982997 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:49.982965 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7hlrt_45c9cd76-e64c-4050-a5d2-88d69415cb36/serve-healthcheck-canary/0.log" Jul 06 22:20:50.400341 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:50.400307 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:20:50.454308 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:50.454279 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:20:50.582914 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:50.582888 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-fn24h_9c4f4c11-7942-49f9-9473-c14ea497795d/kube-storage-version-migrator-operator/0.log" Jul 06 22:20:51.519612 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:51.519570 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:20:52.839969 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:52.839926 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-4x6pf" Jul 06 22:20:52.840379 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:52.840310 2571 scope.go:117] "RemoveContainer" containerID="0055ac975bdf22a2fe997a6301b1d6fc68c764873e76edf76cbe9273752f5b37" Jul 06 22:20:52.840483 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:52.840463 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-4x6pf_openshift-console-operator(5796a8ea-adcb-4819-bb18-0b85e1ef4681)\"" pod="openshift-console-operator/console-operator-68b757fb67-4x6pf" podUID="5796a8ea-adcb-4819-bb18-0b85e1ef4681" Jul 06 22:20:52.972167 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:52.972131 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/46b9db84-34e1-414a-a394-2809f8f0fa0b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2ktvb\" (UID: \"46b9db84-34e1-414a-a394-2809f8f0fa0b\") " pod="openshift-insights/insights-runtime-extractor-2ktvb" Jul 06 22:20:52.972390 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:52.972239 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/46b9db84-34e1-414a-a394-2809f8f0fa0b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2ktvb\" (UID: \"46b9db84-34e1-414a-a394-2809f8f0fa0b\") " pod="openshift-insights/insights-runtime-extractor-2ktvb" Jul 06 22:20:52.972390 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:52.972312 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 22:20:52.972390 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:52.972324 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 22:20:52.972390 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:52.972387 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46b9db84-34e1-414a-a394-2809f8f0fa0b-insights-runtime-extractor-tls podName:46b9db84-34e1-414a-a394-2809f8f0fa0b nodeName:}" failed. No retries permitted until 2026-07-06 22:21:00.972367905 +0000 UTC m=+67.890610739 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/46b9db84-34e1-414a-a394-2809f8f0fa0b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2ktvb" (UID: "46b9db84-34e1-414a-a394-2809f8f0fa0b") : secret "insights-runtime-extractor-tls" not found Jul 06 22:20:52.972553 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:52.972408 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/46b9db84-34e1-414a-a394-2809f8f0fa0b-kube-rbac-proxy-cm podName:46b9db84-34e1-414a-a394-2809f8f0fa0b nodeName:}" failed. No retries permitted until 2026-07-06 22:21:00.97240076 +0000 UTC m=+67.890643593 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/46b9db84-34e1-414a-a394-2809f8f0fa0b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2ktvb" (UID: "46b9db84-34e1-414a-a394-2809f8f0fa0b") : configmap "kube-rbac-proxy" not found Jul 06 22:20:53.317511 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:53.317475 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:20:53.777632 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:53.777603 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qbbrs" Jul 06 22:20:54.498155 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:54.498122 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:20:54.699990 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:54.699958 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:20:54.798680 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:54.798600 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:20:54.953958 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:54.953923 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:20:55.054046 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:55.053957 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:20:55.154651 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:55.154611 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:20:55.252753 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:55.252718 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:20:55.494877 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:55.494846 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/63d474a2-5a9a-4c1d-8711-bbac91ebfee2-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-829dv\" (UID: \"63d474a2-5a9a-4c1d-8711-bbac91ebfee2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-829dv" Jul 06 22:20:55.497359 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:55.497342 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/63d474a2-5a9a-4c1d-8711-bbac91ebfee2-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-829dv\" (UID: \"63d474a2-5a9a-4c1d-8711-bbac91ebfee2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-829dv" Jul 06 22:20:55.783522 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:55.783441 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-v96t6\"" Jul 06 22:20:55.791762 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:55.791736 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-829dv" Jul 06 22:20:55.918668 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:55.918635 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-829dv"] Jul 06 22:20:56.203964 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:56.203932 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:20:56.399202 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:56.399169 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:20:56.809234 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:56.809065 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:20:56.890264 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:56.890228 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-829dv" event={"ID":"63d474a2-5a9a-4c1d-8711-bbac91ebfee2","Type":"ContainerStarted","Data":"6474193bfc7abf7ec7e969a30fa7f7ea884a9fb209985b1f745fb49fe4abe26f"} Jul 06 22:20:57.000024 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:56.999977 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:20:57.614433 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:57.614400 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:20:57.844522 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:57.844439 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:20:57.895406 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:57.895371 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-829dv" event={"ID":"63d474a2-5a9a-4c1d-8711-bbac91ebfee2","Type":"ContainerStarted","Data":"6f478a4ebd14898365f07c5d1883ca2b14131bb03b82c5652b1fb7b2b7618fe6"} Jul 06 22:20:57.895572 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:57.895427 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-829dv" event={"ID":"63d474a2-5a9a-4c1d-8711-bbac91ebfee2","Type":"ContainerStarted","Data":"4a7ce7d831469083e62913def3067b67517ddd923a0aecdba70b402292412672"} Jul 06 22:20:57.915942 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:57.915886 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-829dv" podStartSLOduration=17.305263676 podStartE2EDuration="18.915870053s" totalImagesPullingTime="1.610606377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:20:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:20:57.914685235 +0000 UTC m=+64.832928091" watchObservedRunningTime="2026-07-06 22:20:57.915870053 +0000 UTC m=+64.834112973" Jul 06 22:20:57.920561 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:57.920517 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:20:58.209054 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:58.209008 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:20:58.285727 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:58.285680 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:20:58.627453 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:58.627363 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:20:58.802106 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:58.802063 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:20:59.201610 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:59.201575 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:20:59.273195 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:59.273158 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:20:59.329690 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:59.329656 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfc24edf-03da-480b-b0d2-528c9098f001-metrics-certs\") pod \"network-metrics-daemon-pwdsc\" (UID: \"cfc24edf-03da-480b-b0d2-528c9098f001\") " pod="openshift-multus/network-metrics-daemon-pwdsc" Jul 06 22:20:59.332698 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:59.332680 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 22:20:59.340811 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:59.340786 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 06 22:20:59.340874 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:20:59.340862 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfc24edf-03da-480b-b0d2-528c9098f001-metrics-certs podName:cfc24edf-03da-480b-b0d2-528c9098f001 nodeName:}" failed. No retries permitted until 2026-07-06 22:22:03.340843405 +0000 UTC m=+130.259086240 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cfc24edf-03da-480b-b0d2-528c9098f001-metrics-certs") pod "network-metrics-daemon-pwdsc" (UID: "cfc24edf-03da-480b-b0d2-528c9098f001") : secret "metrics-daemon-secret" not found Jul 06 22:20:59.430785 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:59.430746 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d115b038-4f94-41df-82e3-6d42109603e0-original-pull-secret\") pod \"global-pull-secret-syncer-ccgsb\" (UID: \"d115b038-4f94-41df-82e3-6d42109603e0\") " pod="kube-system/global-pull-secret-syncer-ccgsb" Jul 06 22:20:59.433863 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:59.433841 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 22:20:59.443371 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:59.443346 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d115b038-4f94-41df-82e3-6d42109603e0-original-pull-secret\") pod \"global-pull-secret-syncer-ccgsb\" (UID: \"d115b038-4f94-41df-82e3-6d42109603e0\") " pod="kube-system/global-pull-secret-syncer-ccgsb" Jul 06 22:20:59.531998 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:59.531890 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbw84\" (UniqueName: \"kubernetes.io/projected/5f44d847-874a-4f0a-b3ab-6dcae0173f2d-kube-api-access-vbw84\") pod \"network-check-target-fg2lc\" (UID: \"5f44d847-874a-4f0a-b3ab-6dcae0173f2d\") " pod="openshift-network-diagnostics/network-check-target-fg2lc" Jul 06 22:20:59.534457 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:59.534436 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbw84\" (UniqueName: \"kubernetes.io/projected/5f44d847-874a-4f0a-b3ab-6dcae0173f2d-kube-api-access-vbw84\") pod \"network-check-target-fg2lc\" (UID: \"5f44d847-874a-4f0a-b3ab-6dcae0173f2d\") " pod="openshift-network-diagnostics/network-check-target-fg2lc" Jul 06 22:20:59.603494 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:59.603461 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:20:59.643980 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:59.643953 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ccgsb" Jul 06 22:20:59.655037 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:59.655011 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zplcn\"" Jul 06 22:20:59.662547 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:59.662506 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fg2lc" Jul 06 22:20:59.793195 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:59.793121 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ccgsb"] Jul 06 22:20:59.795828 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:20:59.795799 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd115b038_4f94_41df_82e3_6d42109603e0.slice/crio-732d993481bdd5e3c10d8a63dd620862b188d3ebe20172e9d4c96bf2e0dcb9bd WatchSource:0}: Error finding container 732d993481bdd5e3c10d8a63dd620862b188d3ebe20172e9d4c96bf2e0dcb9bd: Status 404 returned error can't find the container with id 732d993481bdd5e3c10d8a63dd620862b188d3ebe20172e9d4c96bf2e0dcb9bd Jul 06 22:20:59.809602 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:59.809571 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fg2lc"] Jul 06 22:20:59.813716 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:20:59.813680 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f44d847_874a_4f0a_b3ab_6dcae0173f2d.slice/crio-50c895dd301eb9aced80010159cba6b9fff14451789c59ee7da85c6ceadea032 WatchSource:0}: Error finding container 50c895dd301eb9aced80010159cba6b9fff14451789c59ee7da85c6ceadea032: Status 404 returned error can't find the container with id 50c895dd301eb9aced80010159cba6b9fff14451789c59ee7da85c6ceadea032 Jul 06 22:20:59.903289 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:59.903250 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ccgsb" event={"ID":"d115b038-4f94-41df-82e3-6d42109603e0","Type":"ContainerStarted","Data":"732d993481bdd5e3c10d8a63dd620862b188d3ebe20172e9d4c96bf2e0dcb9bd"} Jul 06 22:20:59.904688 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:59.904658 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fg2lc" event={"ID":"5f44d847-874a-4f0a-b3ab-6dcae0173f2d","Type":"ContainerStarted","Data":"f9afb1f921d363fed177d8283599b5c311a51233c74cdc32e7e27c8a23a65366"} Jul 06 22:20:59.904842 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:59.904699 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fg2lc" event={"ID":"5f44d847-874a-4f0a-b3ab-6dcae0173f2d","Type":"ContainerStarted","Data":"50c895dd301eb9aced80010159cba6b9fff14451789c59ee7da85c6ceadea032"} Jul 06 22:20:59.904842 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:59.904794 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-fg2lc" Jul 06 22:20:59.924078 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:20:59.924023 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-fg2lc" podStartSLOduration=65.924003694 podStartE2EDuration="1m5.924003694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:19:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:20:59.923379847 +0000 UTC m=+66.841622715" watchObservedRunningTime="2026-07-06 22:20:59.924003694 +0000 UTC m=+66.842246552" Jul 06 22:21:00.013595 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:00.013538 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:00.237877 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:00.237829 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8acf7e0c-2b66-4c40-a249-58f6fccba39e-registry-tls\") pod \"image-registry-7675579c6b-dg9ls\" (UID: \"8acf7e0c-2b66-4c40-a249-58f6fccba39e\") " pod="openshift-image-registry/image-registry-7675579c6b-dg9ls" Jul 06 22:21:00.237877 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:00.237888 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/13cb55e3-2959-4b92-acff-2d8750c2dc62-metrics-tls\") pod \"dns-default-xfxdc\" (UID: \"13cb55e3-2959-4b92-acff-2d8750c2dc62\") " pod="openshift-dns/dns-default-xfxdc" Jul 06 22:21:00.240197 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:00.240163 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/13cb55e3-2959-4b92-acff-2d8750c2dc62-metrics-tls\") pod \"dns-default-xfxdc\" (UID: \"13cb55e3-2959-4b92-acff-2d8750c2dc62\") " pod="openshift-dns/dns-default-xfxdc" Jul 06 22:21:00.240311 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:00.240280 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8acf7e0c-2b66-4c40-a249-58f6fccba39e-registry-tls\") pod \"image-registry-7675579c6b-dg9ls\" (UID: \"8acf7e0c-2b66-4c40-a249-58f6fccba39e\") " pod="openshift-image-registry/image-registry-7675579c6b-dg9ls" Jul 06 22:21:00.403130 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:00.403093 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:00.479365 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:00.479334 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-mnwqj\"" Jul 06 22:21:00.486843 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:00.486820 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7675579c6b-dg9ls" Jul 06 22:21:00.496770 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:00.496748 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2q2fk\"" Jul 06 22:21:00.505293 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:00.505267 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xfxdc" Jul 06 22:21:00.649114 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:00.649082 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7675579c6b-dg9ls"] Jul 06 22:21:00.653249 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:21:00.653219 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8acf7e0c_2b66_4c40_a249_58f6fccba39e.slice/crio-4778b952896215b77d2e57cc2ff6df0532d0f0a41342545cac0a8ce4628acaf3 WatchSource:0}: Error finding container 4778b952896215b77d2e57cc2ff6df0532d0f0a41342545cac0a8ce4628acaf3: Status 404 returned error can't find the container with id 4778b952896215b77d2e57cc2ff6df0532d0f0a41342545cac0a8ce4628acaf3 Jul 06 22:21:00.663853 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:00.663829 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xfxdc"] Jul 06 22:21:00.667926 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:21:00.667902 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13cb55e3_2959_4b92_acff_2d8750c2dc62.slice/crio-f71f9c97f833f186dff15f24217a33fc56c0571c5a532e31031516e568a49e36 WatchSource:0}: Error finding container f71f9c97f833f186dff15f24217a33fc56c0571c5a532e31031516e568a49e36: Status 404 returned error can't find the container with id f71f9c97f833f186dff15f24217a33fc56c0571c5a532e31031516e568a49e36 Jul 06 22:21:00.804715 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:00.804621 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:00.903883 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:00.903844 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:00.909807 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:00.909729 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xfxdc" event={"ID":"13cb55e3-2959-4b92-acff-2d8750c2dc62","Type":"ContainerStarted","Data":"f71f9c97f833f186dff15f24217a33fc56c0571c5a532e31031516e568a49e36"} Jul 06 22:21:00.911785 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:00.911748 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7675579c6b-dg9ls" event={"ID":"8acf7e0c-2b66-4c40-a249-58f6fccba39e","Type":"ContainerStarted","Data":"dca84f4efae78fe994106b6aff83db7e20bf8c12337175be87755d0b3cc72ff3"} Jul 06 22:21:00.911939 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:00.911807 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7675579c6b-dg9ls" event={"ID":"8acf7e0c-2b66-4c40-a249-58f6fccba39e","Type":"ContainerStarted","Data":"4778b952896215b77d2e57cc2ff6df0532d0f0a41342545cac0a8ce4628acaf3"} Jul 06 22:21:00.911939 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:00.911849 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7675579c6b-dg9ls" Jul 06 22:21:00.934199 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:00.934133 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7675579c6b-dg9ls" podStartSLOduration=66.934109752 podStartE2EDuration="1m6.934109752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:19:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:21:00.933661944 +0000 UTC m=+67.851904815" watchObservedRunningTime="2026-07-06 22:21:00.934109752 +0000 UTC m=+67.852352609" Jul 06 22:21:01.047883 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:01.047846 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/46b9db84-34e1-414a-a394-2809f8f0fa0b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2ktvb\" (UID: \"46b9db84-34e1-414a-a394-2809f8f0fa0b\") " pod="openshift-insights/insights-runtime-extractor-2ktvb" Jul 06 22:21:01.048142 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:21:01.047990 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 22:21:01.048142 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:01.048108 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/46b9db84-34e1-414a-a394-2809f8f0fa0b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2ktvb\" (UID: \"46b9db84-34e1-414a-a394-2809f8f0fa0b\") " pod="openshift-insights/insights-runtime-extractor-2ktvb" Jul 06 22:21:01.048450 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:21:01.048430 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46b9db84-34e1-414a-a394-2809f8f0fa0b-insights-runtime-extractor-tls podName:46b9db84-34e1-414a-a394-2809f8f0fa0b nodeName:}" failed. No retries permitted until 2026-07-06 22:21:17.048409845 +0000 UTC m=+83.966652683 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/46b9db84-34e1-414a-a394-2809f8f0fa0b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2ktvb" (UID: "46b9db84-34e1-414a-a394-2809f8f0fa0b") : secret "insights-runtime-extractor-tls" not found Jul 06 22:21:01.048937 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:01.048919 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/46b9db84-34e1-414a-a394-2809f8f0fa0b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2ktvb\" (UID: \"46b9db84-34e1-414a-a394-2809f8f0fa0b\") " pod="openshift-insights/insights-runtime-extractor-2ktvb" Jul 06 22:21:01.203278 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:01.203198 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:01.605015 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:01.604902 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:01.999705 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:01.999665 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:02.216367 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:02.216324 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:02.599925 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:02.599890 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:03.201754 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:03.201709 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:03.600675 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:03.600584 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:03.998689 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:03.998594 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:04.209355 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:04.209327 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:04.604039 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:04.603925 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:04.933153 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:04.933110 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ccgsb" event={"ID":"d115b038-4f94-41df-82e3-6d42109603e0","Type":"ContainerStarted","Data":"df63cca35689a6ef7333b71786226aa50a05d39944336085b841ec4cf551e9ce"} Jul 06 22:21:04.934922 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:04.934897 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xfxdc" event={"ID":"13cb55e3-2959-4b92-acff-2d8750c2dc62","Type":"ContainerStarted","Data":"4b82e90ca007f403a4cb7a09cbf42c93005fc135b2c56e2f548449d212bf2617"} Jul 06 22:21:04.935055 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:04.934940 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xfxdc" event={"ID":"13cb55e3-2959-4b92-acff-2d8750c2dc62","Type":"ContainerStarted","Data":"28db4d110498e28b9dd5622f08af310fe5e38c663e9369be60602cf08b003111"} Jul 06 22:21:04.935055 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:04.935028 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xfxdc" Jul 06 22:21:04.950569 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:04.950523 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-ccgsb" podStartSLOduration=66.525745854 podStartE2EDuration="1m10.950507168s" totalImagesPullingTime="4.424761314s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:19:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:21:04.949840074 +0000 UTC m=+71.868082933" watchObservedRunningTime="2026-07-06 22:21:04.950507168 +0000 UTC m=+71.868750023" Jul 06 22:21:04.996201 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:04.996168 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:05.201366 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:05.201282 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:05.629159 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:05.629079 2571 scope.go:117] "RemoveContainer" containerID="0055ac975bdf22a2fe997a6301b1d6fc68c764873e76edf76cbe9273752f5b37" Jul 06 22:21:05.942200 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:05.942175 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-4x6pf_5796a8ea-adcb-4819-bb18-0b85e1ef4681/console-operator/2.log" Jul 06 22:21:05.942593 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:05.942576 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-4x6pf_5796a8ea-adcb-4819-bb18-0b85e1ef4681/console-operator/1.log" Jul 06 22:21:05.942654 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:05.942613 2571 generic.go:356] "Generic (PLEG): container finished" podID="5796a8ea-adcb-4819-bb18-0b85e1ef4681" containerID="9f100c9994866ecc57030c7a512a9fe2c5afad134f0a9c35f2ee7bdba28a896e" exitCode=255 Jul 06 22:21:05.942703 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:05.942685 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-4x6pf" event={"ID":"5796a8ea-adcb-4819-bb18-0b85e1ef4681","Type":"ContainerDied","Data":"9f100c9994866ecc57030c7a512a9fe2c5afad134f0a9c35f2ee7bdba28a896e"} Jul 06 22:21:05.942757 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:05.942743 2571 scope.go:117] "RemoveContainer" containerID="0055ac975bdf22a2fe997a6301b1d6fc68c764873e76edf76cbe9273752f5b37" Jul 06 22:21:05.943229 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:05.943191 2571 scope.go:117] "RemoveContainer" containerID="9f100c9994866ecc57030c7a512a9fe2c5afad134f0a9c35f2ee7bdba28a896e" Jul 06 22:21:05.943384 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:21:05.943363 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-68b757fb67-4x6pf_openshift-console-operator(5796a8ea-adcb-4819-bb18-0b85e1ef4681)\"" pod="openshift-console-operator/console-operator-68b757fb67-4x6pf" podUID="5796a8ea-adcb-4819-bb18-0b85e1ef4681" Jul 06 22:21:05.962798 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:05.962747 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-xfxdc" podStartSLOduration=34.422291607 podStartE2EDuration="37.962729865s" totalImagesPullingTime="3.540438258s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:20:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:21:04.968838115 +0000 UTC m=+71.887080971" watchObservedRunningTime="2026-07-06 22:21:05.962729865 +0000 UTC m=+72.880972712" Jul 06 22:21:06.947963 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:06.947935 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-4x6pf_5796a8ea-adcb-4819-bb18-0b85e1ef4681/console-operator/2.log" Jul 06 22:21:09.352699 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:09.352663 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:09.412328 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:09.412294 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:09.970932 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:09.970898 2571 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-4x6pf" Jul 06 22:21:09.971327 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:09.971311 2571 scope.go:117] "RemoveContainer" containerID="9f100c9994866ecc57030c7a512a9fe2c5afad134f0a9c35f2ee7bdba28a896e" Jul 06 22:21:09.971510 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:21:09.971488 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-68b757fb67-4x6pf_openshift-console-operator(5796a8ea-adcb-4819-bb18-0b85e1ef4681)\"" pod="openshift-console-operator/console-operator-68b757fb67-4x6pf" podUID="5796a8ea-adcb-4819-bb18-0b85e1ef4681" Jul 06 22:21:11.695225 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:11.695169 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-854cb59bbd-2fjjn"] Jul 06 22:21:11.697950 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:11.697920 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-854cb59bbd-2fjjn" Jul 06 22:21:11.701787 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:11.701761 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 06 22:21:11.703032 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:11.703010 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 06 22:21:11.704053 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:11.703892 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 06 22:21:11.704053 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:11.703916 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 06 22:21:11.730378 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:11.730348 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-854cb59bbd-2fjjn"] Jul 06 22:21:11.840290 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:11.840252 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q6hl\" (UniqueName: \"kubernetes.io/projected/70a86022-dd4c-49e7-8f28-1c83dcfa9656-kube-api-access-8q6hl\") pod \"klusterlet-addon-workmgr-854cb59bbd-2fjjn\" (UID: \"70a86022-dd4c-49e7-8f28-1c83dcfa9656\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-854cb59bbd-2fjjn" Jul 06 22:21:11.840518 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:11.840300 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/70a86022-dd4c-49e7-8f28-1c83dcfa9656-klusterlet-config\") pod \"klusterlet-addon-workmgr-854cb59bbd-2fjjn\" (UID: \"70a86022-dd4c-49e7-8f28-1c83dcfa9656\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-854cb59bbd-2fjjn" Jul 06 22:21:11.840518 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:11.840400 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/70a86022-dd4c-49e7-8f28-1c83dcfa9656-tmp\") pod \"klusterlet-addon-workmgr-854cb59bbd-2fjjn\" (UID: \"70a86022-dd4c-49e7-8f28-1c83dcfa9656\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-854cb59bbd-2fjjn" Jul 06 22:21:11.860541 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:11.860509 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:11.940931 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:11.940892 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/70a86022-dd4c-49e7-8f28-1c83dcfa9656-tmp\") pod \"klusterlet-addon-workmgr-854cb59bbd-2fjjn\" (UID: \"70a86022-dd4c-49e7-8f28-1c83dcfa9656\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-854cb59bbd-2fjjn" Jul 06 22:21:11.941124 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:11.940989 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q6hl\" (UniqueName: \"kubernetes.io/projected/70a86022-dd4c-49e7-8f28-1c83dcfa9656-kube-api-access-8q6hl\") pod \"klusterlet-addon-workmgr-854cb59bbd-2fjjn\" (UID: \"70a86022-dd4c-49e7-8f28-1c83dcfa9656\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-854cb59bbd-2fjjn" Jul 06 22:21:11.941124 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:11.941018 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/70a86022-dd4c-49e7-8f28-1c83dcfa9656-klusterlet-config\") pod \"klusterlet-addon-workmgr-854cb59bbd-2fjjn\" (UID: \"70a86022-dd4c-49e7-8f28-1c83dcfa9656\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-854cb59bbd-2fjjn" Jul 06 22:21:11.941418 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:11.941383 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/70a86022-dd4c-49e7-8f28-1c83dcfa9656-tmp\") pod \"klusterlet-addon-workmgr-854cb59bbd-2fjjn\" (UID: \"70a86022-dd4c-49e7-8f28-1c83dcfa9656\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-854cb59bbd-2fjjn" Jul 06 22:21:11.943490 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:11.943470 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/70a86022-dd4c-49e7-8f28-1c83dcfa9656-klusterlet-config\") pod \"klusterlet-addon-workmgr-854cb59bbd-2fjjn\" (UID: \"70a86022-dd4c-49e7-8f28-1c83dcfa9656\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-854cb59bbd-2fjjn" Jul 06 22:21:11.964657 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:11.964579 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:11.966024 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:11.965999 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q6hl\" (UniqueName: \"kubernetes.io/projected/70a86022-dd4c-49e7-8f28-1c83dcfa9656-kube-api-access-8q6hl\") pod \"klusterlet-addon-workmgr-854cb59bbd-2fjjn\" (UID: \"70a86022-dd4c-49e7-8f28-1c83dcfa9656\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-854cb59bbd-2fjjn" Jul 06 22:21:12.006966 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:12.006927 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-854cb59bbd-2fjjn" Jul 06 22:21:12.053378 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:12.053346 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:12.131727 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:12.131687 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:12.135116 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:12.135089 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-854cb59bbd-2fjjn"] Jul 06 22:21:12.138106 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:21:12.138081 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70a86022_dd4c_49e7_8f28_1c83dcfa9656.slice/crio-e2658bc8ebc43ca3216090e15d4848ec561da6cd64e2967dbca606e80a8036d0 WatchSource:0}: Error finding container e2658bc8ebc43ca3216090e15d4848ec561da6cd64e2967dbca606e80a8036d0: Status 404 returned error can't find the container with id e2658bc8ebc43ca3216090e15d4848ec561da6cd64e2967dbca606e80a8036d0 Jul 06 22:21:12.840514 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:12.840476 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-4x6pf" Jul 06 22:21:12.840989 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:12.840940 2571 scope.go:117] "RemoveContainer" containerID="9f100c9994866ecc57030c7a512a9fe2c5afad134f0a9c35f2ee7bdba28a896e" Jul 06 22:21:12.841137 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:21:12.841111 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-68b757fb67-4x6pf_openshift-console-operator(5796a8ea-adcb-4819-bb18-0b85e1ef4681)\"" pod="openshift-console-operator/console-operator-68b757fb67-4x6pf" podUID="5796a8ea-adcb-4819-bb18-0b85e1ef4681" Jul 06 22:21:12.970521 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:12.970449 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-854cb59bbd-2fjjn" event={"ID":"70a86022-dd4c-49e7-8f28-1c83dcfa9656","Type":"ContainerStarted","Data":"e2658bc8ebc43ca3216090e15d4848ec561da6cd64e2967dbca606e80a8036d0"} Jul 06 22:21:14.945523 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:14.945482 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xfxdc" Jul 06 22:21:14.999906 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:14.999853 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:15.195933 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:15.195752 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:15.278570 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:15.278533 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:15.395320 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:15.395283 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:15.983466 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:15.983428 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-854cb59bbd-2fjjn" event={"ID":"70a86022-dd4c-49e7-8f28-1c83dcfa9656","Type":"ContainerStarted","Data":"9c69581ecda2cff8c5cb137d9967ac6a54416aa73d7944831fa0593c09baffdf"} Jul 06 22:21:15.983933 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:15.983635 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-854cb59bbd-2fjjn" Jul 06 22:21:15.985492 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:15.985473 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-854cb59bbd-2fjjn" Jul 06 22:21:16.002035 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:16.001979 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-854cb59bbd-2fjjn" podStartSLOduration=1.368114919 podStartE2EDuration="5.001965512s" totalImagesPullingTime="3.633850593s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:21:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:21:16.001048527 +0000 UTC m=+82.919291403" watchObservedRunningTime="2026-07-06 22:21:16.001965512 +0000 UTC m=+82.920208368" Jul 06 22:21:17.084046 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:17.083997 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/46b9db84-34e1-414a-a394-2809f8f0fa0b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2ktvb\" (UID: \"46b9db84-34e1-414a-a394-2809f8f0fa0b\") " pod="openshift-insights/insights-runtime-extractor-2ktvb" Jul 06 22:21:17.086430 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:17.086404 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/46b9db84-34e1-414a-a394-2809f8f0fa0b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2ktvb\" (UID: \"46b9db84-34e1-414a-a394-2809f8f0fa0b\") " pod="openshift-insights/insights-runtime-extractor-2ktvb" Jul 06 22:21:17.333951 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:17.333920 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-dhrwd\"" Jul 06 22:21:17.342118 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:17.342059 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2ktvb" Jul 06 22:21:17.472431 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:17.472399 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2ktvb"] Jul 06 22:21:17.475338 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:21:17.475310 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46b9db84_34e1_414a_a394_2809f8f0fa0b.slice/crio-112b4ef0a47c30d4f051b1808b9fd7d4fdaa65592b368c975c33c26a3e7c553c WatchSource:0}: Error finding container 112b4ef0a47c30d4f051b1808b9fd7d4fdaa65592b368c975c33c26a3e7c553c: Status 404 returned error can't find the container with id 112b4ef0a47c30d4f051b1808b9fd7d4fdaa65592b368c975c33c26a3e7c553c Jul 06 22:21:17.994125 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:17.993255 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2ktvb" event={"ID":"46b9db84-34e1-414a-a394-2809f8f0fa0b","Type":"ContainerStarted","Data":"112b4ef0a47c30d4f051b1808b9fd7d4fdaa65592b368c975c33c26a3e7c553c"} Jul 06 22:21:18.264836 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:18.264716 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:18.347118 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:18.347078 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:18.808538 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:18.808492 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:18.908569 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:18.908532 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:18.980436 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:18.980399 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:20.492403 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:20.492364 2571 patch_prober.go:28] interesting pod/image-registry-7675579c6b-dg9ls container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 06 22:21:20.492843 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:20.492433 2571 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7675579c6b-dg9ls" podUID="8acf7e0c-2b66-4c40-a249-58f6fccba39e" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 22:21:20.961616 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:20.961583 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:21.033422 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:21.033384 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:21.920733 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:21.920704 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7675579c6b-dg9ls" Jul 06 22:21:22.011826 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:22.011735 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2ktvb" event={"ID":"46b9db84-34e1-414a-a394-2809f8f0fa0b","Type":"ContainerStarted","Data":"724ca209c099ea18fa57f1749e807e5acc2a6338d09b92d362a05fdc664b4dcc"} Jul 06 22:21:22.011826 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:22.011793 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2ktvb" event={"ID":"46b9db84-34e1-414a-a394-2809f8f0fa0b","Type":"ContainerStarted","Data":"d8345f3db00d07b89fbea80924afe78259ad1bf98d390bf893b290eadbc4be82"} Jul 06 22:21:22.118084 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:22.118047 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:22.181147 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:22.181105 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:22.797194 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:22.797157 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:22.899273 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:22.899232 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:23.100307 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:23.100196 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:24.479123 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:24.479078 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:24.767864 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:24.767723 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:24.894737 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:24.894701 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:24.958997 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:24.958957 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-7h2fd"] Jul 06 22:21:24.962188 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:24.962162 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-m2svj"] Jul 06 22:21:24.962350 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:24.962321 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7h2fd" Jul 06 22:21:24.964829 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:24.964807 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 06 22:21:24.964951 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:24.964940 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 06 22:21:24.965323 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:24.965306 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8fwgn"] Jul 06 22:21:24.965456 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:24.965439 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-m2svj" Jul 06 22:21:24.966052 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:24.966030 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-wwz4x\"" Jul 06 22:21:24.966155 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:24.966085 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 06 22:21:24.966155 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:24.966097 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 06 22:21:24.966272 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:24.966184 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 06 22:21:24.968138 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:21:24.968117 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"node-exporter-tls\" is forbidden: User \"system:node:ip-10-0-137-147.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-137-147.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" type="*v1.Secret" Jul 06 22:21:24.968454 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:21:24.968429 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"node-exporter-dockercfg-qgwgf\" is forbidden: User \"system:node:ip-10-0-137-147.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-137-147.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qgwgf\"" type="*v1.Secret" Jul 06 22:21:24.968541 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:24.968486 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8fwgn" Jul 06 22:21:24.968628 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:21:24.968609 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"node-exporter-accelerators-collector-config\" is forbidden: User \"system:node:ip-10-0-137-147.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-137-147.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" type="*v1.ConfigMap" Jul 06 22:21:24.968697 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:21:24.968668 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"node-exporter-kube-rbac-proxy-config\" is forbidden: User \"system:node:ip-10-0-137-147.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-137-147.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" type="*v1.Secret" Jul 06 22:21:24.971918 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:24.971896 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 06 22:21:24.972015 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:24.972002 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-xktzm\"" Jul 06 22:21:24.972445 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:24.972429 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 06 22:21:24.973028 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:24.973012 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 06 22:21:24.981348 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:24.981325 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-7h2fd"] Jul 06 22:21:24.985783 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:24.985761 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8fwgn"] Jul 06 22:21:25.021293 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.021193 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:25.025049 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.025016 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2ktvb" event={"ID":"46b9db84-34e1-414a-a394-2809f8f0fa0b","Type":"ContainerStarted","Data":"f9429cf1e92a71a63d1f07bdec14710eee3fe3b26dccdee68835008be49a23e3"} Jul 06 22:21:25.051138 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.051106 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b2a1a1e7-c4da-473f-8f3b-0949d9ef885a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-7h2fd\" (UID: \"b2a1a1e7-c4da-473f-8f3b-0949d9ef885a\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7h2fd" Jul 06 22:21:25.051329 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.051144 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cf6e4bbe-5fd6-4bec-84de-7bb8e7a68247-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-8fwgn\" (UID: \"cf6e4bbe-5fd6-4bec-84de-7bb8e7a68247\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8fwgn" Jul 06 22:21:25.051329 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.051180 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cb280153-8e0b-4776-8c56-8c47339afb67-node-exporter-textfile\") pod \"node-exporter-m2svj\" (UID: \"cb280153-8e0b-4776-8c56-8c47339afb67\") " pod="openshift-monitoring/node-exporter-m2svj" Jul 06 22:21:25.051329 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.051245 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cb280153-8e0b-4776-8c56-8c47339afb67-node-exporter-wtmp\") pod \"node-exporter-m2svj\" (UID: \"cb280153-8e0b-4776-8c56-8c47339afb67\") " pod="openshift-monitoring/node-exporter-m2svj" Jul 06 22:21:25.051329 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.051280 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cf6e4bbe-5fd6-4bec-84de-7bb8e7a68247-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-8fwgn\" (UID: \"cf6e4bbe-5fd6-4bec-84de-7bb8e7a68247\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8fwgn" Jul 06 22:21:25.051329 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.051302 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cb280153-8e0b-4776-8c56-8c47339afb67-node-exporter-tls\") pod \"node-exporter-m2svj\" (UID: \"cb280153-8e0b-4776-8c56-8c47339afb67\") " pod="openshift-monitoring/node-exporter-m2svj" Jul 06 22:21:25.051507 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.051337 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cb280153-8e0b-4776-8c56-8c47339afb67-node-exporter-accelerators-collector-config\") pod \"node-exporter-m2svj\" (UID: \"cb280153-8e0b-4776-8c56-8c47339afb67\") " pod="openshift-monitoring/node-exporter-m2svj" Jul 06 22:21:25.051507 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.051388 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb280153-8e0b-4776-8c56-8c47339afb67-sys\") pod \"node-exporter-m2svj\" (UID: \"cb280153-8e0b-4776-8c56-8c47339afb67\") " pod="openshift-monitoring/node-exporter-m2svj" Jul 06 22:21:25.051507 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.051406 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/cf6e4bbe-5fd6-4bec-84de-7bb8e7a68247-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-8fwgn\" (UID: \"cf6e4bbe-5fd6-4bec-84de-7bb8e7a68247\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8fwgn" Jul 06 22:21:25.051507 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.051471 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2a1a1e7-c4da-473f-8f3b-0949d9ef885a-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-7h2fd\" (UID: \"b2a1a1e7-c4da-473f-8f3b-0949d9ef885a\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7h2fd" Jul 06 22:21:25.051507 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.051485 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cb280153-8e0b-4776-8c56-8c47339afb67-root\") pod \"node-exporter-m2svj\" (UID: \"cb280153-8e0b-4776-8c56-8c47339afb67\") " pod="openshift-monitoring/node-exporter-m2svj" Jul 06 22:21:25.051507 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.051504 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl6h6\" (UniqueName: \"kubernetes.io/projected/b2a1a1e7-c4da-473f-8f3b-0949d9ef885a-kube-api-access-tl6h6\") pod \"openshift-state-metrics-54cf96dfcf-7h2fd\" (UID: \"b2a1a1e7-c4da-473f-8f3b-0949d9ef885a\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7h2fd" Jul 06 22:21:25.051767 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.051527 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tssbt\" (UniqueName: \"kubernetes.io/projected/cb280153-8e0b-4776-8c56-8c47339afb67-kube-api-access-tssbt\") pod \"node-exporter-m2svj\" (UID: \"cb280153-8e0b-4776-8c56-8c47339afb67\") " pod="openshift-monitoring/node-exporter-m2svj" Jul 06 22:21:25.051767 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.051552 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cb280153-8e0b-4776-8c56-8c47339afb67-metrics-client-ca\") pod \"node-exporter-m2svj\" (UID: \"cb280153-8e0b-4776-8c56-8c47339afb67\") " pod="openshift-monitoring/node-exporter-m2svj" Jul 06 22:21:25.051767 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.051576 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf6e4bbe-5fd6-4bec-84de-7bb8e7a68247-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-8fwgn\" (UID: \"cf6e4bbe-5fd6-4bec-84de-7bb8e7a68247\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8fwgn" Jul 06 22:21:25.051767 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.051603 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4rwh\" (UniqueName: \"kubernetes.io/projected/cf6e4bbe-5fd6-4bec-84de-7bb8e7a68247-kube-api-access-b4rwh\") pod \"kube-state-metrics-7c9dbb8b7d-8fwgn\" (UID: \"cf6e4bbe-5fd6-4bec-84de-7bb8e7a68247\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8fwgn" Jul 06 22:21:25.051767 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.051661 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cb280153-8e0b-4776-8c56-8c47339afb67-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m2svj\" (UID: \"cb280153-8e0b-4776-8c56-8c47339afb67\") " pod="openshift-monitoring/node-exporter-m2svj" Jul 06 22:21:25.051767 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.051729 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b2a1a1e7-c4da-473f-8f3b-0949d9ef885a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-7h2fd\" (UID: \"b2a1a1e7-c4da-473f-8f3b-0949d9ef885a\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7h2fd" Jul 06 22:21:25.051767 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.051764 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/cf6e4bbe-5fd6-4bec-84de-7bb8e7a68247-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-8fwgn\" (UID: \"cf6e4bbe-5fd6-4bec-84de-7bb8e7a68247\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8fwgn" Jul 06 22:21:25.081519 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.081301 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-2ktvb" podStartSLOduration=33.432216513 podStartE2EDuration="40.08128244s" totalImagesPullingTime="6.649065927s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:20:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:21:25.081257274 +0000 UTC m=+91.999500131" watchObservedRunningTime="2026-07-06 22:21:25.08128244 +0000 UTC m=+91.999525295" Jul 06 22:21:25.104583 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.104548 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:25.152744 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.152712 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cb280153-8e0b-4776-8c56-8c47339afb67-node-exporter-textfile\") pod \"node-exporter-m2svj\" (UID: \"cb280153-8e0b-4776-8c56-8c47339afb67\") " pod="openshift-monitoring/node-exporter-m2svj" Jul 06 22:21:25.152744 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.152748 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cb280153-8e0b-4776-8c56-8c47339afb67-node-exporter-wtmp\") pod \"node-exporter-m2svj\" (UID: \"cb280153-8e0b-4776-8c56-8c47339afb67\") " pod="openshift-monitoring/node-exporter-m2svj" Jul 06 22:21:25.153052 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.152770 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cf6e4bbe-5fd6-4bec-84de-7bb8e7a68247-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-8fwgn\" (UID: \"cf6e4bbe-5fd6-4bec-84de-7bb8e7a68247\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8fwgn" Jul 06 22:21:25.153052 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.152796 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cb280153-8e0b-4776-8c56-8c47339afb67-node-exporter-tls\") pod \"node-exporter-m2svj\" (UID: \"cb280153-8e0b-4776-8c56-8c47339afb67\") " pod="openshift-monitoring/node-exporter-m2svj" Jul 06 22:21:25.153052 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.152913 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cb280153-8e0b-4776-8c56-8c47339afb67-node-exporter-wtmp\") pod \"node-exporter-m2svj\" (UID: \"cb280153-8e0b-4776-8c56-8c47339afb67\") " pod="openshift-monitoring/node-exporter-m2svj" Jul 06 22:21:25.153052 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.152922 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cb280153-8e0b-4776-8c56-8c47339afb67-node-exporter-accelerators-collector-config\") pod \"node-exporter-m2svj\" (UID: \"cb280153-8e0b-4776-8c56-8c47339afb67\") " pod="openshift-monitoring/node-exporter-m2svj" Jul 06 22:21:25.153052 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.152974 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb280153-8e0b-4776-8c56-8c47339afb67-sys\") pod \"node-exporter-m2svj\" (UID: \"cb280153-8e0b-4776-8c56-8c47339afb67\") " pod="openshift-monitoring/node-exporter-m2svj" Jul 06 22:21:25.153052 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.153002 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/cf6e4bbe-5fd6-4bec-84de-7bb8e7a68247-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-8fwgn\" (UID: \"cf6e4bbe-5fd6-4bec-84de-7bb8e7a68247\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8fwgn" Jul 06 22:21:25.153052 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.153032 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb280153-8e0b-4776-8c56-8c47339afb67-sys\") pod \"node-exporter-m2svj\" (UID: \"cb280153-8e0b-4776-8c56-8c47339afb67\") " pod="openshift-monitoring/node-exporter-m2svj" Jul 06 22:21:25.153431 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.153055 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cb280153-8e0b-4776-8c56-8c47339afb67-node-exporter-textfile\") pod \"node-exporter-m2svj\" (UID: \"cb280153-8e0b-4776-8c56-8c47339afb67\") " pod="openshift-monitoring/node-exporter-m2svj" Jul 06 22:21:25.153431 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.153120 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2a1a1e7-c4da-473f-8f3b-0949d9ef885a-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-7h2fd\" (UID: \"b2a1a1e7-c4da-473f-8f3b-0949d9ef885a\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7h2fd" Jul 06 22:21:25.153431 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.153147 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cb280153-8e0b-4776-8c56-8c47339afb67-root\") pod \"node-exporter-m2svj\" (UID: \"cb280153-8e0b-4776-8c56-8c47339afb67\") " pod="openshift-monitoring/node-exporter-m2svj" Jul 06 22:21:25.153431 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.153176 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl6h6\" (UniqueName: \"kubernetes.io/projected/b2a1a1e7-c4da-473f-8f3b-0949d9ef885a-kube-api-access-tl6h6\") pod \"openshift-state-metrics-54cf96dfcf-7h2fd\" (UID: \"b2a1a1e7-c4da-473f-8f3b-0949d9ef885a\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7h2fd" Jul 06 22:21:25.153431 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.153203 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tssbt\" (UniqueName: \"kubernetes.io/projected/cb280153-8e0b-4776-8c56-8c47339afb67-kube-api-access-tssbt\") pod \"node-exporter-m2svj\" (UID: \"cb280153-8e0b-4776-8c56-8c47339afb67\") " pod="openshift-monitoring/node-exporter-m2svj" Jul 06 22:21:25.153431 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.153249 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cb280153-8e0b-4776-8c56-8c47339afb67-root\") pod \"node-exporter-m2svj\" (UID: \"cb280153-8e0b-4776-8c56-8c47339afb67\") " pod="openshift-monitoring/node-exporter-m2svj" Jul 06 22:21:25.153431 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.153274 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cb280153-8e0b-4776-8c56-8c47339afb67-metrics-client-ca\") pod \"node-exporter-m2svj\" (UID: \"cb280153-8e0b-4776-8c56-8c47339afb67\") " pod="openshift-monitoring/node-exporter-m2svj" Jul 06 22:21:25.153431 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.153302 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf6e4bbe-5fd6-4bec-84de-7bb8e7a68247-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-8fwgn\" (UID: \"cf6e4bbe-5fd6-4bec-84de-7bb8e7a68247\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8fwgn" Jul 06 22:21:25.153431 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.153338 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4rwh\" (UniqueName: \"kubernetes.io/projected/cf6e4bbe-5fd6-4bec-84de-7bb8e7a68247-kube-api-access-b4rwh\") pod \"kube-state-metrics-7c9dbb8b7d-8fwgn\" (UID: \"cf6e4bbe-5fd6-4bec-84de-7bb8e7a68247\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8fwgn" Jul 06 22:21:25.153431 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.153341 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/cf6e4bbe-5fd6-4bec-84de-7bb8e7a68247-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-8fwgn\" (UID: \"cf6e4bbe-5fd6-4bec-84de-7bb8e7a68247\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8fwgn" Jul 06 22:21:25.153431 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.153372 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cb280153-8e0b-4776-8c56-8c47339afb67-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m2svj\" (UID: \"cb280153-8e0b-4776-8c56-8c47339afb67\") " pod="openshift-monitoring/node-exporter-m2svj" Jul 06 22:21:25.153897 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.153537 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b2a1a1e7-c4da-473f-8f3b-0949d9ef885a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-7h2fd\" (UID: \"b2a1a1e7-c4da-473f-8f3b-0949d9ef885a\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7h2fd" Jul 06 22:21:25.153897 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.153583 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/cf6e4bbe-5fd6-4bec-84de-7bb8e7a68247-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-8fwgn\" (UID: \"cf6e4bbe-5fd6-4bec-84de-7bb8e7a68247\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8fwgn" Jul 06 22:21:25.153897 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.153617 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b2a1a1e7-c4da-473f-8f3b-0949d9ef885a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-7h2fd\" (UID: \"b2a1a1e7-c4da-473f-8f3b-0949d9ef885a\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7h2fd" Jul 06 22:21:25.153897 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:21:25.153644 2571 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 06 22:21:25.153897 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.153658 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cf6e4bbe-5fd6-4bec-84de-7bb8e7a68247-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-8fwgn\" (UID: \"cf6e4bbe-5fd6-4bec-84de-7bb8e7a68247\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8fwgn" Jul 06 22:21:25.153897 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:21:25.153703 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2a1a1e7-c4da-473f-8f3b-0949d9ef885a-openshift-state-metrics-tls podName:b2a1a1e7-c4da-473f-8f3b-0949d9ef885a nodeName:}" failed. No retries permitted until 2026-07-06 22:21:25.653685631 +0000 UTC m=+92.571928482 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/b2a1a1e7-c4da-473f-8f3b-0949d9ef885a-openshift-state-metrics-tls") pod "openshift-state-metrics-54cf96dfcf-7h2fd" (UID: "b2a1a1e7-c4da-473f-8f3b-0949d9ef885a") : secret "openshift-state-metrics-tls" not found Jul 06 22:21:25.153897 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.153806 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cb280153-8e0b-4776-8c56-8c47339afb67-metrics-client-ca\") pod \"node-exporter-m2svj\" (UID: \"cb280153-8e0b-4776-8c56-8c47339afb67\") " pod="openshift-monitoring/node-exporter-m2svj" Jul 06 22:21:25.154287 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.154030 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf6e4bbe-5fd6-4bec-84de-7bb8e7a68247-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-8fwgn\" (UID: \"cf6e4bbe-5fd6-4bec-84de-7bb8e7a68247\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8fwgn" Jul 06 22:21:25.154287 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.154115 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2a1a1e7-c4da-473f-8f3b-0949d9ef885a-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-7h2fd\" (UID: \"b2a1a1e7-c4da-473f-8f3b-0949d9ef885a\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7h2fd" Jul 06 22:21:25.154415 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.154389 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/cf6e4bbe-5fd6-4bec-84de-7bb8e7a68247-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-8fwgn\" (UID: \"cf6e4bbe-5fd6-4bec-84de-7bb8e7a68247\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8fwgn" Jul 06 22:21:25.155591 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.155561 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cf6e4bbe-5fd6-4bec-84de-7bb8e7a68247-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-8fwgn\" (UID: \"cf6e4bbe-5fd6-4bec-84de-7bb8e7a68247\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8fwgn" Jul 06 22:21:25.155909 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.155890 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cf6e4bbe-5fd6-4bec-84de-7bb8e7a68247-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-8fwgn\" (UID: \"cf6e4bbe-5fd6-4bec-84de-7bb8e7a68247\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8fwgn" Jul 06 22:21:25.156070 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.156052 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b2a1a1e7-c4da-473f-8f3b-0949d9ef885a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-7h2fd\" (UID: \"b2a1a1e7-c4da-473f-8f3b-0949d9ef885a\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7h2fd" Jul 06 22:21:25.169805 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.169770 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl6h6\" (UniqueName: \"kubernetes.io/projected/b2a1a1e7-c4da-473f-8f3b-0949d9ef885a-kube-api-access-tl6h6\") pod \"openshift-state-metrics-54cf96dfcf-7h2fd\" (UID: \"b2a1a1e7-c4da-473f-8f3b-0949d9ef885a\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7h2fd" Jul 06 22:21:25.173716 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.173688 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tssbt\" (UniqueName: \"kubernetes.io/projected/cb280153-8e0b-4776-8c56-8c47339afb67-kube-api-access-tssbt\") pod \"node-exporter-m2svj\" (UID: \"cb280153-8e0b-4776-8c56-8c47339afb67\") " pod="openshift-monitoring/node-exporter-m2svj" Jul 06 22:21:25.174268 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.174242 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4rwh\" (UniqueName: \"kubernetes.io/projected/cf6e4bbe-5fd6-4bec-84de-7bb8e7a68247-kube-api-access-b4rwh\") pod \"kube-state-metrics-7c9dbb8b7d-8fwgn\" (UID: \"cf6e4bbe-5fd6-4bec-84de-7bb8e7a68247\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8fwgn" Jul 06 22:21:25.211512 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.211480 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:25.287810 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.287717 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8fwgn" Jul 06 22:21:25.307471 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.307440 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:25.404760 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.404723 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:25.437960 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.437929 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8fwgn"] Jul 06 22:21:25.441369 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:21:25.441336 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf6e4bbe_5fd6_4bec_84de_7bb8e7a68247.slice/crio-7de9d95be4fcefff29c66fecb6bd0e97ab31f7931f6802512eaa86245e8d3734 WatchSource:0}: Error finding container 7de9d95be4fcefff29c66fecb6bd0e97ab31f7931f6802512eaa86245e8d3734: Status 404 returned error can't find the container with id 7de9d95be4fcefff29c66fecb6bd0e97ab31f7931f6802512eaa86245e8d3734 Jul 06 22:21:25.520183 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.520143 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:25.611122 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.611028 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:25.657107 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.657074 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b2a1a1e7-c4da-473f-8f3b-0949d9ef885a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-7h2fd\" (UID: \"b2a1a1e7-c4da-473f-8f3b-0949d9ef885a\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7h2fd" Jul 06 22:21:25.659449 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.659426 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b2a1a1e7-c4da-473f-8f3b-0949d9ef885a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-7h2fd\" (UID: \"b2a1a1e7-c4da-473f-8f3b-0949d9ef885a\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7h2fd" Jul 06 22:21:25.770831 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.770791 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:25.848685 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.848651 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qgwgf\"" Jul 06 22:21:25.873951 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.873867 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7h2fd" Jul 06 22:21:25.878499 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:25.878468 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:26.003365 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:26.003332 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-7h2fd"] Jul 06 22:21:26.006188 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:21:26.006159 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2a1a1e7_c4da_473f_8f3b_0949d9ef885a.slice/crio-3e76f3954a230c1733dddc48c31fc9c0dd390ba5f41f774823f9f146c19c197e WatchSource:0}: Error finding container 3e76f3954a230c1733dddc48c31fc9c0dd390ba5f41f774823f9f146c19c197e: Status 404 returned error can't find the container with id 3e76f3954a230c1733dddc48c31fc9c0dd390ba5f41f774823f9f146c19c197e Jul 06 22:21:26.028985 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:26.028946 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7h2fd" event={"ID":"b2a1a1e7-c4da-473f-8f3b-0949d9ef885a","Type":"ContainerStarted","Data":"3e76f3954a230c1733dddc48c31fc9c0dd390ba5f41f774823f9f146c19c197e"} Jul 06 22:21:26.030114 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:26.030093 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8fwgn" event={"ID":"cf6e4bbe-5fd6-4bec-84de-7bb8e7a68247","Type":"ContainerStarted","Data":"7de9d95be4fcefff29c66fecb6bd0e97ab31f7931f6802512eaa86245e8d3734"} Jul 06 22:21:26.074286 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:26.074264 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 06 22:21:26.080087 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:26.080065 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 06 22:21:26.085347 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:26.085317 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cb280153-8e0b-4776-8c56-8c47339afb67-node-exporter-tls\") pod \"node-exporter-m2svj\" (UID: \"cb280153-8e0b-4776-8c56-8c47339afb67\") " pod="openshift-monitoring/node-exporter-m2svj" Jul 06 22:21:26.086644 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:26.086531 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cb280153-8e0b-4776-8c56-8c47339afb67-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m2svj\" (UID: \"cb280153-8e0b-4776-8c56-8c47339afb67\") " pod="openshift-monitoring/node-exporter-m2svj" Jul 06 22:21:26.153753 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:21:26.153713 2571 configmap.go:193] Couldn't get configMap openshift-monitoring/node-exporter-accelerators-collector-config: failed to sync configmap cache: timed out waiting for the condition Jul 06 22:21:26.153911 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:21:26.153816 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cb280153-8e0b-4776-8c56-8c47339afb67-node-exporter-accelerators-collector-config podName:cb280153-8e0b-4776-8c56-8c47339afb67 nodeName:}" failed. No retries permitted until 2026-07-06 22:21:26.653792704 +0000 UTC m=+93.572035558 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-accelerators-collector-config" (UniqueName: "kubernetes.io/configmap/cb280153-8e0b-4776-8c56-8c47339afb67-node-exporter-accelerators-collector-config") pod "node-exporter-m2svj" (UID: "cb280153-8e0b-4776-8c56-8c47339afb67") : failed to sync configmap cache: timed out waiting for the condition Jul 06 22:21:26.199028 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:26.198995 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:26.270657 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:26.270628 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 06 22:21:26.628827 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:26.628793 2571 scope.go:117] "RemoveContainer" containerID="9f100c9994866ecc57030c7a512a9fe2c5afad134f0a9c35f2ee7bdba28a896e" Jul 06 22:21:26.667538 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:26.667403 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cb280153-8e0b-4776-8c56-8c47339afb67-node-exporter-accelerators-collector-config\") pod \"node-exporter-m2svj\" (UID: \"cb280153-8e0b-4776-8c56-8c47339afb67\") " pod="openshift-monitoring/node-exporter-m2svj" Jul 06 22:21:26.668424 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:26.668398 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cb280153-8e0b-4776-8c56-8c47339afb67-node-exporter-accelerators-collector-config\") pod \"node-exporter-m2svj\" (UID: \"cb280153-8e0b-4776-8c56-8c47339afb67\") " pod="openshift-monitoring/node-exporter-m2svj" Jul 06 22:21:26.780324 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:26.780283 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-m2svj" Jul 06 22:21:26.793863 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:21:26.793824 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb280153_8e0b_4776_8c56_8c47339afb67.slice/crio-63ead86e4eaa5b0483a2a8250389ce5826f4d138b369fa451d0d031ad5f4a75f WatchSource:0}: Error finding container 63ead86e4eaa5b0483a2a8250389ce5826f4d138b369fa451d0d031ad5f4a75f: Status 404 returned error can't find the container with id 63ead86e4eaa5b0483a2a8250389ce5826f4d138b369fa451d0d031ad5f4a75f Jul 06 22:21:27.036660 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:27.036622 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m2svj" event={"ID":"cb280153-8e0b-4776-8c56-8c47339afb67","Type":"ContainerStarted","Data":"63ead86e4eaa5b0483a2a8250389ce5826f4d138b369fa451d0d031ad5f4a75f"} Jul 06 22:21:27.039354 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:27.039326 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-4x6pf_5796a8ea-adcb-4819-bb18-0b85e1ef4681/console-operator/2.log" Jul 06 22:21:27.039528 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:27.039433 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-4x6pf" event={"ID":"5796a8ea-adcb-4819-bb18-0b85e1ef4681","Type":"ContainerStarted","Data":"ab74b516199c3aafd4d463cf82a5e0bbb2ea70281e78b874430d1be69e4c7a98"} Jul 06 22:21:27.040311 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:27.040026 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-4x6pf" Jul 06 22:21:27.042315 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:27.042284 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7h2fd" event={"ID":"b2a1a1e7-c4da-473f-8f3b-0949d9ef885a","Type":"ContainerStarted","Data":"ae5e33402237cbeb187fabac82701a86e4eb9c400555a0eef132840686b07dc4"} Jul 06 22:21:27.042474 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:27.042344 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7h2fd" event={"ID":"b2a1a1e7-c4da-473f-8f3b-0949d9ef885a","Type":"ContainerStarted","Data":"895e5ceb6aeaa93271de509681738f48360174153c24c39518d8ebf440ca6f26"} Jul 06 22:21:27.673145 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:27.673115 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-68b757fb67-4x6pf" Jul 06 22:21:27.908780 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:27.908736 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:27.985636 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:27.985541 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:28.050124 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:28.049818 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7h2fd" event={"ID":"b2a1a1e7-c4da-473f-8f3b-0949d9ef885a","Type":"ContainerStarted","Data":"cfe41ebc8cae0f2aa157efa0ae800d44afd57692ac15c61073898d08e0be7f96"} Jul 06 22:21:28.052717 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:28.052651 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8fwgn" event={"ID":"cf6e4bbe-5fd6-4bec-84de-7bb8e7a68247","Type":"ContainerStarted","Data":"3b3b1dd99944426a73dd128b76d4875f0f5fc1254af4bd0fa4bb0bd48a8a0d02"} Jul 06 22:21:28.052845 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:28.052722 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8fwgn" event={"ID":"cf6e4bbe-5fd6-4bec-84de-7bb8e7a68247","Type":"ContainerStarted","Data":"82e2522071c41c94e666063a30c842547534258f7f82753149151c8b6eb4c86f"} Jul 06 22:21:28.078949 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:28.078896 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7h2fd" podStartSLOduration=2.585497213 podStartE2EDuration="4.07887352s" totalImagesPullingTime="1.493376307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:21:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:21:28.078272136 +0000 UTC m=+94.996514993" watchObservedRunningTime="2026-07-06 22:21:28.07887352 +0000 UTC m=+94.997116356" Jul 06 22:21:28.091824 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:28.091792 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:28.165175 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:28.165130 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:28.265626 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:28.265588 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:28.369468 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:28.369418 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:28.587890 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:28.587805 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:28.647334 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:28.647302 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:28.763263 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:28.763230 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:28.853402 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:28.853320 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:29.057763 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:29.057729 2571 generic.go:356] "Generic (PLEG): container finished" podID="cb280153-8e0b-4776-8c56-8c47339afb67" containerID="df118fef4311ec2bde1dc757e5565a2d1d412fc998c88c74cbbaf14c4379bc8f" exitCode=0 Jul 06 22:21:29.057954 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:29.057813 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m2svj" event={"ID":"cb280153-8e0b-4776-8c56-8c47339afb67","Type":"ContainerDied","Data":"df118fef4311ec2bde1dc757e5565a2d1d412fc998c88c74cbbaf14c4379bc8f"} Jul 06 22:21:29.060249 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:29.060202 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8fwgn" event={"ID":"cf6e4bbe-5fd6-4bec-84de-7bb8e7a68247","Type":"ContainerStarted","Data":"aa30405f51594864b019f3f6d2688b35692fc2ade825321028476fafe674bf3a"} Jul 06 22:21:29.105709 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:29.105614 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8fwgn" podStartSLOduration=2.9303671060000003 podStartE2EDuration="5.105590951s" totalImagesPullingTime="2.175223845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:21:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:21:29.104785004 +0000 UTC m=+96.023027862" watchObservedRunningTime="2026-07-06 22:21:29.105590951 +0000 UTC m=+96.023833808" Jul 06 22:21:29.370991 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:29.370898 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:29.489167 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:29.489133 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:29.553179 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:29.553141 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:29.611291 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:29.611259 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:29.761820 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:29.761772 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:29.861142 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:29.861113 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:30.062419 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:30.062298 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:30.066325 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:30.066297 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m2svj" event={"ID":"cb280153-8e0b-4776-8c56-8c47339afb67","Type":"ContainerStarted","Data":"f643e5b5190c138ed2d480bfa36694d0f596c57a7a84cc11253f4eed3fda48b4"} Jul 06 22:21:30.066470 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:30.066358 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m2svj" event={"ID":"cb280153-8e0b-4776-8c56-8c47339afb67","Type":"ContainerStarted","Data":"db33c52ab3434efb187625dd7181fbede2129158b6f73af4f669299e0bc8c07a"} Jul 06 22:21:30.097759 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:30.097700 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-m2svj" podStartSLOduration=4.828857284 podStartE2EDuration="6.09768349s" totalImagesPullingTime="1.196843487s" totalInitContainerRuntime="71.982719ms" isStatefulPod=true podCreationTimestamp="2026-07-06 22:21:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:21:30.096919609 +0000 UTC m=+97.015162467" watchObservedRunningTime="2026-07-06 22:21:30.09768349 +0000 UTC m=+97.015926346" Jul 06 22:21:30.172492 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:30.172454 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:30.914489 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:30.914453 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-fg2lc" Jul 06 22:21:31.038436 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.038390 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:31.139422 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.139377 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:31.313519 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.313415 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:31.318348 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.318322 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 22:21:31.324688 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.324662 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:21:31.327179 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.327151 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 06 22:21:31.328265 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.328024 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 06 22:21:31.328265 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.328120 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 06 22:21:31.328265 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.328152 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-22ftm\"" Jul 06 22:21:31.328265 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.328172 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 06 22:21:31.328265 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.328120 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 06 22:21:31.328676 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.328174 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 06 22:21:31.328676 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.328120 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 06 22:21:31.328676 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.328616 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 06 22:21:31.328897 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.328680 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 06 22:21:31.328897 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.328716 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 06 22:21:31.328897 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.328831 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 06 22:21:31.328897 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.328861 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-2728132d4c1c4\"" Jul 06 22:21:31.328897 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.328886 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 06 22:21:31.329232 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.328923 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 06 22:21:31.336445 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.336419 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 06 22:21:31.339625 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.339597 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 06 22:21:31.341584 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.341562 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 22:21:31.413611 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.413571 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:21:31.413611 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.413610 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-config-out\") pod \"prometheus-k8s-0\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:21:31.413819 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.413631 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:21:31.413819 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.413652 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:21:31.413819 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.413672 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:21:31.413819 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.413744 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:21:31.413819 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.413781 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:21:31.413819 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.413810 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-config\") pod \"prometheus-k8s-0\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:21:31.413998 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.413834 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-web-config\") pod \"prometheus-k8s-0\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:21:31.413998 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.413849 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:21:31.413998 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.413879 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:21:31.413998 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.413893 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:21:31.413998 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.413906 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:21:31.413998 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.413926 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:21:31.413998 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.413961 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:21:31.413998 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.413982 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:21:31.413998 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.413998 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:21:31.414299 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.414011 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s9kq\" (UniqueName: \"kubernetes.io/projected/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-kube-api-access-4s9kq\") pod \"prometheus-k8s-0\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:21:31.414299 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.414029 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:21:31.414299 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.414044 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:21:31.515278 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.515241 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-config\") pod \"prometheus-k8s-0\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:21:31.515278 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.515279 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-web-config\") pod \"prometheus-k8s-0\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:21:31.515473 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.515303 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:21:31.515512 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.515490 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:21:31.515555 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.515537 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:21:31.515603 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.515569 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:21:31.515650 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.515599 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:21:31.515707 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.515648 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:21:31.515707 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.515688 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:21:31.515812 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.515716 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:21:31.515812 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.515741 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s9kq\" (UniqueName: \"kubernetes.io/projected/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-kube-api-access-4s9kq\") pod \"prometheus-k8s-0\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:21:31.515812 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.515769 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:21:31.515812 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.515794 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:21:31.516020 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.515856 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:21:31.516020 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.515889 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-config-out\") pod \"prometheus-k8s-0\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:21:31.516020 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.515922 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:21:31.516020 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.515967 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:21:31.516020 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.515998 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:21:31.516311 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.516033 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:21:31.516311 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.516067 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:21:31.516414 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.516330 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:21:31.516766 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.516740 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:21:31.517502 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.517472 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:21:31.518670 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.518132 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:21:31.519110 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.518804 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:21:31.519195 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.519129 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:21:31.519341 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.519319 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:21:31.519493 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.519468 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-config\") pod \"prometheus-k8s-0\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:21:31.519994 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.519971 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:21:31.520176 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.520083 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-web-config\") pod \"prometheus-k8s-0\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:21:31.520608 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.520585 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:21:31.520778 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.520750 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:21:31.520889 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.520867 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:21:31.521009 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.520989 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:21:31.521186 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.521164 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-config-out\") pod \"prometheus-k8s-0\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:21:31.521323 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.521297 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:21:31.521511 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.521490 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:21:31.521684 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.521664 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:21:31.522163 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.522146 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:21:31.530544 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.530513 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s9kq\" (UniqueName: \"kubernetes.io/projected/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-kube-api-access-4s9kq\") pod \"prometheus-k8s-0\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:21:31.635923 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.635893 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:21:31.787016 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.786974 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 22:21:31.790644 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:21:31.790616 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ca65cf7_655d_4fbb_b949_04b0e65cc2f5.slice/crio-0e505c2ad35a123c0b2f42a13db99db046afe5d73d4abf222b541ac0a1191409 WatchSource:0}: Error finding container 0e505c2ad35a123c0b2f42a13db99db046afe5d73d4abf222b541ac0a1191409: Status 404 returned error can't find the container with id 0e505c2ad35a123c0b2f42a13db99db046afe5d73d4abf222b541ac0a1191409 Jul 06 22:21:31.916197 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:31.916080 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:32.075479 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:32.075442 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5","Type":"ContainerStarted","Data":"0e505c2ad35a123c0b2f42a13db99db046afe5d73d4abf222b541ac0a1191409"} Jul 06 22:21:33.080557 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:33.080473 2571 generic.go:356] "Generic (PLEG): container finished" podID="2ca65cf7-655d-4fbb-b949-04b0e65cc2f5" containerID="1dfcc9c7fa34bbcb4790bf6eaa528530cd8a4a6ed718d0f828136b27c716fcba" exitCode=0 Jul 06 22:21:33.080557 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:33.080512 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5","Type":"ContainerDied","Data":"1dfcc9c7fa34bbcb4790bf6eaa528530cd8a4a6ed718d0f828136b27c716fcba"} Jul 06 22:21:34.080306 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:34.080266 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7675579c6b-dg9ls"] Jul 06 22:21:34.190942 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:34.190901 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:34.277095 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:34.277006 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:37.097354 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:37.097317 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5","Type":"ContainerStarted","Data":"5b153f04549dcac389ce67c6dfe9e62bd634ed422b4d7b4eb52d566cca9083c8"} Jul 06 22:21:37.097722 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:37.097385 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5","Type":"ContainerStarted","Data":"169b1b5cbf78c4f82e1efd81dcea434a33024262717c593575c9c65b5e730cf9"} Jul 06 22:21:39.108477 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:39.108441 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5","Type":"ContainerStarted","Data":"04d909b6e4ac12fdc78887eba1f8f05dbdde3b8b10ad209c0043302ef231c43c"} Jul 06 22:21:39.108797 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:39.108547 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5","Type":"ContainerStarted","Data":"64fc4a25032bbf026619c9437beac1aefa126bdf92a75737b6147cfd7226b82f"} Jul 06 22:21:39.108797 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:39.108575 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5","Type":"ContainerStarted","Data":"b538067839c626941e5dc101b229e4615958ed16b4acaf378270e825caccd8ed"} Jul 06 22:21:39.365992 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:39.365880 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:39.426242 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:39.426187 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:40.117380 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:40.117345 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5","Type":"ContainerStarted","Data":"f2b7ebd773d80323cb3057b27a75b5edb511a07a5e334d5182d78ffdb18bccdf"} Jul 06 22:21:40.149603 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:40.149522 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.628074221 podStartE2EDuration="9.149508073s" totalImagesPullingTime="6.502966151s" totalInitContainerRuntime="18.467701ms" isStatefulPod=false podCreationTimestamp="2026-07-06 22:21:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:21:40.148373588 +0000 UTC m=+107.066616444" watchObservedRunningTime="2026-07-06 22:21:40.149508073 +0000 UTC m=+107.067750928" Jul 06 22:21:41.636742 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:41.636700 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:21:41.903717 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:41.903626 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:41.962687 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:41.962658 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:43.164643 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:43.164606 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:43.266633 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:43.266585 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:43.798833 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:43.798791 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:43.859472 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:43.859434 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:45.017093 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:45.017063 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:45.101814 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:45.101759 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:46.753367 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:46.753331 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:46.860568 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:46.860531 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:46.962250 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:46.962192 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:49.586633 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:49.586590 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:49.691539 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:49.691504 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:54.170330 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:54.170291 2571 generic.go:356] "Generic (PLEG): container finished" podID="9c4f4c11-7942-49f9-9473-c14ea497795d" containerID="95624c7dc2c6098fef54f9182f6c52d017e091436580bf43e921284e0e81f70d" exitCode=0 Jul 06 22:21:54.170757 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:54.170368 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-fn24h" event={"ID":"9c4f4c11-7942-49f9-9473-c14ea497795d","Type":"ContainerDied","Data":"95624c7dc2c6098fef54f9182f6c52d017e091436580bf43e921284e0e81f70d"} Jul 06 22:21:54.170757 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:54.170711 2571 scope.go:117] "RemoveContainer" containerID="95624c7dc2c6098fef54f9182f6c52d017e091436580bf43e921284e0e81f70d" Jul 06 22:21:54.417313 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:54.417276 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:54.475508 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:54.475405 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:21:55.176268 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:55.176230 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-fn24h" event={"ID":"9c4f4c11-7942-49f9-9473-c14ea497795d","Type":"ContainerStarted","Data":"6684115a1d83de3324759c440e3940c4eaee9d0e4ec868fdb4a87b1970ca2ce2"} Jul 06 22:21:59.103854 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:59.103815 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7675579c6b-dg9ls" podUID="8acf7e0c-2b66-4c40-a249-58f6fccba39e" containerName="registry" containerID="cri-o://dca84f4efae78fe994106b6aff83db7e20bf8c12337175be87755d0b3cc72ff3" gracePeriod=30 Jul 06 22:21:59.347623 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:59.347595 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7675579c6b-dg9ls" Jul 06 22:21:59.480003 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:59.479961 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8acf7e0c-2b66-4c40-a249-58f6fccba39e-trusted-ca\") pod \"8acf7e0c-2b66-4c40-a249-58f6fccba39e\" (UID: \"8acf7e0c-2b66-4c40-a249-58f6fccba39e\") " Jul 06 22:21:59.480188 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:59.480019 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8acf7e0c-2b66-4c40-a249-58f6fccba39e-registry-certificates\") pod \"8acf7e0c-2b66-4c40-a249-58f6fccba39e\" (UID: \"8acf7e0c-2b66-4c40-a249-58f6fccba39e\") " Jul 06 22:21:59.480188 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:59.480042 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8acf7e0c-2b66-4c40-a249-58f6fccba39e-ca-trust-extracted\") pod \"8acf7e0c-2b66-4c40-a249-58f6fccba39e\" (UID: \"8acf7e0c-2b66-4c40-a249-58f6fccba39e\") " Jul 06 22:21:59.480188 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:59.480085 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8acf7e0c-2b66-4c40-a249-58f6fccba39e-bound-sa-token\") pod \"8acf7e0c-2b66-4c40-a249-58f6fccba39e\" (UID: \"8acf7e0c-2b66-4c40-a249-58f6fccba39e\") " Jul 06 22:21:59.480188 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:59.480105 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8acf7e0c-2b66-4c40-a249-58f6fccba39e-installation-pull-secrets\") pod \"8acf7e0c-2b66-4c40-a249-58f6fccba39e\" (UID: \"8acf7e0c-2b66-4c40-a249-58f6fccba39e\") " Jul 06 22:21:59.480188 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:59.480146 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8acf7e0c-2b66-4c40-a249-58f6fccba39e-image-registry-private-configuration\") pod \"8acf7e0c-2b66-4c40-a249-58f6fccba39e\" (UID: \"8acf7e0c-2b66-4c40-a249-58f6fccba39e\") " Jul 06 22:21:59.480188 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:59.480168 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8acf7e0c-2b66-4c40-a249-58f6fccba39e-registry-tls\") pod \"8acf7e0c-2b66-4c40-a249-58f6fccba39e\" (UID: \"8acf7e0c-2b66-4c40-a249-58f6fccba39e\") " Jul 06 22:21:59.480505 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:59.480192 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5kh4q\" (UniqueName: \"kubernetes.io/projected/8acf7e0c-2b66-4c40-a249-58f6fccba39e-kube-api-access-5kh4q\") pod \"8acf7e0c-2b66-4c40-a249-58f6fccba39e\" (UID: \"8acf7e0c-2b66-4c40-a249-58f6fccba39e\") " Jul 06 22:21:59.480562 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:59.480530 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8acf7e0c-2b66-4c40-a249-58f6fccba39e-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "8acf7e0c-2b66-4c40-a249-58f6fccba39e" (UID: "8acf7e0c-2b66-4c40-a249-58f6fccba39e"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:21:59.480623 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:59.480560 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8acf7e0c-2b66-4c40-a249-58f6fccba39e-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "8acf7e0c-2b66-4c40-a249-58f6fccba39e" (UID: "8acf7e0c-2b66-4c40-a249-58f6fccba39e"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:21:59.482687 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:59.482629 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8acf7e0c-2b66-4c40-a249-58f6fccba39e-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "8acf7e0c-2b66-4c40-a249-58f6fccba39e" (UID: "8acf7e0c-2b66-4c40-a249-58f6fccba39e"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:21:59.482826 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:59.482758 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8acf7e0c-2b66-4c40-a249-58f6fccba39e-kube-api-access-5kh4q" (OuterVolumeSpecName: "kube-api-access-5kh4q") pod "8acf7e0c-2b66-4c40-a249-58f6fccba39e" (UID: "8acf7e0c-2b66-4c40-a249-58f6fccba39e"). InnerVolumeSpecName "kube-api-access-5kh4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:21:59.482888 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:59.482846 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8acf7e0c-2b66-4c40-a249-58f6fccba39e-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "8acf7e0c-2b66-4c40-a249-58f6fccba39e" (UID: "8acf7e0c-2b66-4c40-a249-58f6fccba39e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:21:59.483069 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:59.483037 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8acf7e0c-2b66-4c40-a249-58f6fccba39e-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "8acf7e0c-2b66-4c40-a249-58f6fccba39e" (UID: "8acf7e0c-2b66-4c40-a249-58f6fccba39e"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:21:59.483112 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:59.483087 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8acf7e0c-2b66-4c40-a249-58f6fccba39e-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "8acf7e0c-2b66-4c40-a249-58f6fccba39e" (UID: "8acf7e0c-2b66-4c40-a249-58f6fccba39e"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:21:59.489837 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:59.489808 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8acf7e0c-2b66-4c40-a249-58f6fccba39e-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "8acf7e0c-2b66-4c40-a249-58f6fccba39e" (UID: "8acf7e0c-2b66-4c40-a249-58f6fccba39e"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:21:59.581528 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:59.581483 2571 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8acf7e0c-2b66-4c40-a249-58f6fccba39e-trusted-ca\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:21:59.581528 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:59.581520 2571 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8acf7e0c-2b66-4c40-a249-58f6fccba39e-registry-certificates\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:21:59.581528 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:59.581531 2571 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8acf7e0c-2b66-4c40-a249-58f6fccba39e-ca-trust-extracted\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:21:59.581528 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:59.581540 2571 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8acf7e0c-2b66-4c40-a249-58f6fccba39e-bound-sa-token\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:21:59.581789 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:59.581549 2571 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8acf7e0c-2b66-4c40-a249-58f6fccba39e-installation-pull-secrets\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:21:59.581789 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:59.581558 2571 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8acf7e0c-2b66-4c40-a249-58f6fccba39e-image-registry-private-configuration\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:21:59.581789 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:59.581566 2571 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8acf7e0c-2b66-4c40-a249-58f6fccba39e-registry-tls\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:21:59.581789 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:21:59.581574 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kh4q\" (UniqueName: \"kubernetes.io/projected/8acf7e0c-2b66-4c40-a249-58f6fccba39e-kube-api-access-5kh4q\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:22:00.197138 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:00.197100 2571 generic.go:356] "Generic (PLEG): container finished" podID="8acf7e0c-2b66-4c40-a249-58f6fccba39e" containerID="dca84f4efae78fe994106b6aff83db7e20bf8c12337175be87755d0b3cc72ff3" exitCode=0 Jul 06 22:22:00.197603 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:00.197149 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7675579c6b-dg9ls" Jul 06 22:22:00.197603 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:00.197154 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7675579c6b-dg9ls" event={"ID":"8acf7e0c-2b66-4c40-a249-58f6fccba39e","Type":"ContainerDied","Data":"dca84f4efae78fe994106b6aff83db7e20bf8c12337175be87755d0b3cc72ff3"} Jul 06 22:22:00.197603 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:00.197197 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7675579c6b-dg9ls" event={"ID":"8acf7e0c-2b66-4c40-a249-58f6fccba39e","Type":"ContainerDied","Data":"4778b952896215b77d2e57cc2ff6df0532d0f0a41342545cac0a8ce4628acaf3"} Jul 06 22:22:00.197603 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:00.197246 2571 scope.go:117] "RemoveContainer" containerID="dca84f4efae78fe994106b6aff83db7e20bf8c12337175be87755d0b3cc72ff3" Jul 06 22:22:00.205426 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:00.205408 2571 scope.go:117] "RemoveContainer" containerID="dca84f4efae78fe994106b6aff83db7e20bf8c12337175be87755d0b3cc72ff3" Jul 06 22:22:00.205702 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:22:00.205684 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dca84f4efae78fe994106b6aff83db7e20bf8c12337175be87755d0b3cc72ff3\": container with ID starting with dca84f4efae78fe994106b6aff83db7e20bf8c12337175be87755d0b3cc72ff3 not found: ID does not exist" containerID="dca84f4efae78fe994106b6aff83db7e20bf8c12337175be87755d0b3cc72ff3" Jul 06 22:22:00.205750 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:00.205710 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dca84f4efae78fe994106b6aff83db7e20bf8c12337175be87755d0b3cc72ff3"} err="failed to get container status \"dca84f4efae78fe994106b6aff83db7e20bf8c12337175be87755d0b3cc72ff3\": rpc error: code = NotFound desc = could not find container \"dca84f4efae78fe994106b6aff83db7e20bf8c12337175be87755d0b3cc72ff3\": container with ID starting with dca84f4efae78fe994106b6aff83db7e20bf8c12337175be87755d0b3cc72ff3 not found: ID does not exist" Jul 06 22:22:00.217670 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:00.217641 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7675579c6b-dg9ls"] Jul 06 22:22:00.222113 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:00.222086 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7675579c6b-dg9ls"] Jul 06 22:22:01.632931 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:01.632893 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8acf7e0c-2b66-4c40-a249-58f6fccba39e" path="/var/lib/kubelet/pods/8acf7e0c-2b66-4c40-a249-58f6fccba39e/volumes" Jul 06 22:22:03.109928 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:03.109893 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:22:03.169036 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:03.169003 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:22:03.414425 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:03.414389 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfc24edf-03da-480b-b0d2-528c9098f001-metrics-certs\") pod \"network-metrics-daemon-pwdsc\" (UID: \"cfc24edf-03da-480b-b0d2-528c9098f001\") " pod="openshift-multus/network-metrics-daemon-pwdsc" Jul 06 22:22:03.416774 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:03.416739 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfc24edf-03da-480b-b0d2-528c9098f001-metrics-certs\") pod \"network-metrics-daemon-pwdsc\" (UID: \"cfc24edf-03da-480b-b0d2-528c9098f001\") " pod="openshift-multus/network-metrics-daemon-pwdsc" Jul 06 22:22:03.561725 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:03.561693 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-t8ckr\"" Jul 06 22:22:03.569869 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:03.569832 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pwdsc" Jul 06 22:22:03.736860 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:03.736819 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pwdsc"] Jul 06 22:22:03.741312 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:22:03.741273 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfc24edf_03da_480b_b0d2_528c9098f001.slice/crio-fc6667d587055cc7e4ee0edcf9256da5151aa0ed33f2d7d7dc5454bad7e1b14b WatchSource:0}: Error finding container fc6667d587055cc7e4ee0edcf9256da5151aa0ed33f2d7d7dc5454bad7e1b14b: Status 404 returned error can't find the container with id fc6667d587055cc7e4ee0edcf9256da5151aa0ed33f2d7d7dc5454bad7e1b14b Jul 06 22:22:04.214253 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:04.214217 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pwdsc" event={"ID":"cfc24edf-03da-480b-b0d2-528c9098f001","Type":"ContainerStarted","Data":"fc6667d587055cc7e4ee0edcf9256da5151aa0ed33f2d7d7dc5454bad7e1b14b"} Jul 06 22:22:05.219560 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:05.219524 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pwdsc" event={"ID":"cfc24edf-03da-480b-b0d2-528c9098f001","Type":"ContainerStarted","Data":"f599cd985c71f0338b3515feb6c166f16521dd4b486d2af8ec8ce1e7eaf241b8"} Jul 06 22:22:05.219929 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:05.219575 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pwdsc" event={"ID":"cfc24edf-03da-480b-b0d2-528c9098f001","Type":"ContainerStarted","Data":"5c7f9d00374d74a8f559c6a0c583901e05744c8252d56b7e834f391c7d9d8c64"} Jul 06 22:22:05.239689 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:05.239640 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-pwdsc" podStartSLOduration=131.250659214 podStartE2EDuration="2m12.239623282s" totalImagesPullingTime="988.964068ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:19:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:22:05.239607971 +0000 UTC m=+132.157850825" watchObservedRunningTime="2026-07-06 22:22:05.239623282 +0000 UTC m=+132.157866138" Jul 06 22:22:05.340451 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:05.340411 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:22:05.394400 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:05.394348 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:22:09.370315 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:09.370268 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:22:09.428106 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:09.428063 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:22:24.364262 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:24.364190 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:22:24.463835 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:24.463796 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:22:31.636517 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:31.636481 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:31.655448 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:31.655420 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:32.338836 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:32.338805 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:41.112105 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:41.112068 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:22:41.179381 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:41.179340 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:22:43.586424 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:43.586388 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:22:43.687057 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:43.687025 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:22:43.887022 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:43.886992 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:22:44.025994 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:44.025962 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:22:44.082755 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:44.082708 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:22:44.629001 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:44.628968 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:22:44.690592 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:44.690559 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:22:49.711278 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:49.710504 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 22:22:49.711278 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:49.711078 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2ca65cf7-655d-4fbb-b949-04b0e65cc2f5" containerName="prometheus" containerID="cri-o://169b1b5cbf78c4f82e1efd81dcea434a33024262717c593575c9c65b5e730cf9" gracePeriod=600 Jul 06 22:22:49.711777 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:49.711345 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2ca65cf7-655d-4fbb-b949-04b0e65cc2f5" containerName="kube-rbac-proxy" containerID="cri-o://04d909b6e4ac12fdc78887eba1f8f05dbdde3b8b10ad209c0043302ef231c43c" gracePeriod=600 Jul 06 22:22:49.711777 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:49.711433 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2ca65cf7-655d-4fbb-b949-04b0e65cc2f5" containerName="thanos-sidecar" containerID="cri-o://b538067839c626941e5dc101b229e4615958ed16b4acaf378270e825caccd8ed" gracePeriod=600 Jul 06 22:22:49.711777 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:49.711654 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2ca65cf7-655d-4fbb-b949-04b0e65cc2f5" containerName="kube-rbac-proxy-web" containerID="cri-o://64fc4a25032bbf026619c9437beac1aefa126bdf92a75737b6147cfd7226b82f" gracePeriod=600 Jul 06 22:22:49.711920 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:49.711787 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2ca65cf7-655d-4fbb-b949-04b0e65cc2f5" containerName="kube-rbac-proxy-thanos" containerID="cri-o://f2b7ebd773d80323cb3057b27a75b5edb511a07a5e334d5182d78ffdb18bccdf" gracePeriod=600 Jul 06 22:22:49.714946 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:49.712367 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2ca65cf7-655d-4fbb-b949-04b0e65cc2f5" containerName="config-reloader" containerID="cri-o://5b153f04549dcac389ce67c6dfe9e62bd634ed422b4d7b4eb52d566cca9083c8" gracePeriod=600 Jul 06 22:22:49.953940 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:49.953913 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:50.094633 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.094540 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-config\") pod \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " Jul 06 22:22:50.094633 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.094599 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-secret-kube-rbac-proxy\") pod \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " Jul 06 22:22:50.094843 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.094644 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4s9kq\" (UniqueName: \"kubernetes.io/projected/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-kube-api-access-4s9kq\") pod \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " Jul 06 22:22:50.094843 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.094677 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-prometheus-k8s-rulefiles-2\") pod \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " Jul 06 22:22:50.094843 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.094708 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-prometheus-trusted-ca-bundle\") pod \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " Jul 06 22:22:50.094843 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.094734 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " Jul 06 22:22:50.094843 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.094761 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-secret-grpc-tls\") pod \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " Jul 06 22:22:50.094843 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.094787 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-secret-prometheus-k8s-tls\") pod \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " Jul 06 22:22:50.094843 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.094819 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-tls-assets\") pod \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " Jul 06 22:22:50.095201 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.095042 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "2ca65cf7-655d-4fbb-b949-04b0e65cc2f5" (UID: "2ca65cf7-655d-4fbb-b949-04b0e65cc2f5"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:22:50.096465 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.095432 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-prometheus-k8s-rulefiles-0\") pod \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " Jul 06 22:22:50.096465 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.095476 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-configmap-metrics-client-ca\") pod \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " Jul 06 22:22:50.096465 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.095513 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-prometheus-k8s-db\") pod \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " Jul 06 22:22:50.096465 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.095540 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-configmap-kubelet-serving-ca-bundle\") pod \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " Jul 06 22:22:50.096465 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.095573 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-config-out\") pod \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " Jul 06 22:22:50.096465 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.095616 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-prometheus-k8s-rulefiles-1\") pod \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " Jul 06 22:22:50.096465 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.095650 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-thanos-prometheus-http-client-file\") pod \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " Jul 06 22:22:50.096465 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.095678 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-secret-metrics-client-certs\") pod \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " Jul 06 22:22:50.096465 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.095709 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-web-config\") pod \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " Jul 06 22:22:50.096465 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.095741 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-configmap-serving-certs-ca-bundle\") pod \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " Jul 06 22:22:50.096465 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.095771 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\" (UID: \"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5\") " Jul 06 22:22:50.096465 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.095812 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "2ca65cf7-655d-4fbb-b949-04b0e65cc2f5" (UID: "2ca65cf7-655d-4fbb-b949-04b0e65cc2f5"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:22:50.096465 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.096061 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "2ca65cf7-655d-4fbb-b949-04b0e65cc2f5" (UID: "2ca65cf7-655d-4fbb-b949-04b0e65cc2f5"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:22:50.096465 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.096073 2571 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:22:50.096465 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.096111 2571 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-prometheus-trusted-ca-bundle\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:22:50.097325 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.096569 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "2ca65cf7-655d-4fbb-b949-04b0e65cc2f5" (UID: "2ca65cf7-655d-4fbb-b949-04b0e65cc2f5"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:22:50.097325 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.097238 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "2ca65cf7-655d-4fbb-b949-04b0e65cc2f5" (UID: "2ca65cf7-655d-4fbb-b949-04b0e65cc2f5"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:22:50.097447 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.097339 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "2ca65cf7-655d-4fbb-b949-04b0e65cc2f5" (UID: "2ca65cf7-655d-4fbb-b949-04b0e65cc2f5"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:22:50.098160 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.097866 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "2ca65cf7-655d-4fbb-b949-04b0e65cc2f5" (UID: "2ca65cf7-655d-4fbb-b949-04b0e65cc2f5"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:22:50.098160 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.098069 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-kube-api-access-4s9kq" (OuterVolumeSpecName: "kube-api-access-4s9kq") pod "2ca65cf7-655d-4fbb-b949-04b0e65cc2f5" (UID: "2ca65cf7-655d-4fbb-b949-04b0e65cc2f5"). InnerVolumeSpecName "kube-api-access-4s9kq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:22:50.098160 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.098097 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "2ca65cf7-655d-4fbb-b949-04b0e65cc2f5" (UID: "2ca65cf7-655d-4fbb-b949-04b0e65cc2f5"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:22:50.098160 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.098132 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "2ca65cf7-655d-4fbb-b949-04b0e65cc2f5" (UID: "2ca65cf7-655d-4fbb-b949-04b0e65cc2f5"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:22:50.098470 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.098405 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "2ca65cf7-655d-4fbb-b949-04b0e65cc2f5" (UID: "2ca65cf7-655d-4fbb-b949-04b0e65cc2f5"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:22:50.099061 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.099021 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-config" (OuterVolumeSpecName: "config") pod "2ca65cf7-655d-4fbb-b949-04b0e65cc2f5" (UID: "2ca65cf7-655d-4fbb-b949-04b0e65cc2f5"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:22:50.099401 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.099375 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "2ca65cf7-655d-4fbb-b949-04b0e65cc2f5" (UID: "2ca65cf7-655d-4fbb-b949-04b0e65cc2f5"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:22:50.099521 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.099477 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "2ca65cf7-655d-4fbb-b949-04b0e65cc2f5" (UID: "2ca65cf7-655d-4fbb-b949-04b0e65cc2f5"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:22:50.099705 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.099682 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "2ca65cf7-655d-4fbb-b949-04b0e65cc2f5" (UID: "2ca65cf7-655d-4fbb-b949-04b0e65cc2f5"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:22:50.099786 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.099757 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-config-out" (OuterVolumeSpecName: "config-out") pod "2ca65cf7-655d-4fbb-b949-04b0e65cc2f5" (UID: "2ca65cf7-655d-4fbb-b949-04b0e65cc2f5"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:22:50.099959 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.099937 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "2ca65cf7-655d-4fbb-b949-04b0e65cc2f5" (UID: "2ca65cf7-655d-4fbb-b949-04b0e65cc2f5"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:22:50.100327 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.100308 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "2ca65cf7-655d-4fbb-b949-04b0e65cc2f5" (UID: "2ca65cf7-655d-4fbb-b949-04b0e65cc2f5"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:22:50.100900 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.100881 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "2ca65cf7-655d-4fbb-b949-04b0e65cc2f5" (UID: "2ca65cf7-655d-4fbb-b949-04b0e65cc2f5"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:22:50.108901 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.108878 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-web-config" (OuterVolumeSpecName: "web-config") pod "2ca65cf7-655d-4fbb-b949-04b0e65cc2f5" (UID: "2ca65cf7-655d-4fbb-b949-04b0e65cc2f5"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:22:50.197344 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.197306 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s9kq\" (UniqueName: \"kubernetes.io/projected/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-kube-api-access-4s9kq\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:22:50.197344 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.197340 2571 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:22:50.197344 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.197352 2571 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-secret-grpc-tls\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:22:50.197579 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.197360 2571 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-secret-prometheus-k8s-tls\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:22:50.197579 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.197371 2571 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-tls-assets\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:22:50.197579 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.197379 2571 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:22:50.197579 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.197387 2571 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-configmap-metrics-client-ca\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:22:50.197579 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.197395 2571 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-prometheus-k8s-db\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:22:50.197579 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.197404 2571 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:22:50.197579 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.197412 2571 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-config-out\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:22:50.197579 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.197421 2571 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:22:50.197579 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.197429 2571 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-thanos-prometheus-http-client-file\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:22:50.197579 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.197437 2571 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-secret-metrics-client-certs\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:22:50.197579 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.197445 2571 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-web-config\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:22:50.197579 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.197453 2571 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:22:50.197579 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.197462 2571 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:22:50.197579 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.197472 2571 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-config\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:22:50.197579 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.197480 2571 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5-secret-kube-rbac-proxy\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:22:50.392327 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.392291 2571 generic.go:356] "Generic (PLEG): container finished" podID="2ca65cf7-655d-4fbb-b949-04b0e65cc2f5" containerID="f2b7ebd773d80323cb3057b27a75b5edb511a07a5e334d5182d78ffdb18bccdf" exitCode=0 Jul 06 22:22:50.392327 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.392321 2571 generic.go:356] "Generic (PLEG): container finished" podID="2ca65cf7-655d-4fbb-b949-04b0e65cc2f5" containerID="04d909b6e4ac12fdc78887eba1f8f05dbdde3b8b10ad209c0043302ef231c43c" exitCode=0 Jul 06 22:22:50.392327 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.392330 2571 generic.go:356] "Generic (PLEG): container finished" podID="2ca65cf7-655d-4fbb-b949-04b0e65cc2f5" containerID="64fc4a25032bbf026619c9437beac1aefa126bdf92a75737b6147cfd7226b82f" exitCode=0 Jul 06 22:22:50.392566 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.392337 2571 generic.go:356] "Generic (PLEG): container finished" podID="2ca65cf7-655d-4fbb-b949-04b0e65cc2f5" containerID="b538067839c626941e5dc101b229e4615958ed16b4acaf378270e825caccd8ed" exitCode=0 Jul 06 22:22:50.392566 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.392346 2571 generic.go:356] "Generic (PLEG): container finished" podID="2ca65cf7-655d-4fbb-b949-04b0e65cc2f5" containerID="5b153f04549dcac389ce67c6dfe9e62bd634ed422b4d7b4eb52d566cca9083c8" exitCode=0 Jul 06 22:22:50.392566 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.392353 2571 generic.go:356] "Generic (PLEG): container finished" podID="2ca65cf7-655d-4fbb-b949-04b0e65cc2f5" containerID="169b1b5cbf78c4f82e1efd81dcea434a33024262717c593575c9c65b5e730cf9" exitCode=0 Jul 06 22:22:50.392566 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.392368 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5","Type":"ContainerDied","Data":"f2b7ebd773d80323cb3057b27a75b5edb511a07a5e334d5182d78ffdb18bccdf"} Jul 06 22:22:50.392566 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.392403 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:50.392566 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.392438 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5","Type":"ContainerDied","Data":"04d909b6e4ac12fdc78887eba1f8f05dbdde3b8b10ad209c0043302ef231c43c"} Jul 06 22:22:50.392566 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.392466 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5","Type":"ContainerDied","Data":"64fc4a25032bbf026619c9437beac1aefa126bdf92a75737b6147cfd7226b82f"} Jul 06 22:22:50.392566 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.392492 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5","Type":"ContainerDied","Data":"b538067839c626941e5dc101b229e4615958ed16b4acaf378270e825caccd8ed"} Jul 06 22:22:50.392566 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.392508 2571 scope.go:117] "RemoveContainer" containerID="f2b7ebd773d80323cb3057b27a75b5edb511a07a5e334d5182d78ffdb18bccdf" Jul 06 22:22:50.392566 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.392514 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5","Type":"ContainerDied","Data":"5b153f04549dcac389ce67c6dfe9e62bd634ed422b4d7b4eb52d566cca9083c8"} Jul 06 22:22:50.392566 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.392539 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5","Type":"ContainerDied","Data":"169b1b5cbf78c4f82e1efd81dcea434a33024262717c593575c9c65b5e730cf9"} Jul 06 22:22:50.392566 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.392562 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2ca65cf7-655d-4fbb-b949-04b0e65cc2f5","Type":"ContainerDied","Data":"0e505c2ad35a123c0b2f42a13db99db046afe5d73d4abf222b541ac0a1191409"} Jul 06 22:22:50.400718 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.400698 2571 scope.go:117] "RemoveContainer" containerID="04d909b6e4ac12fdc78887eba1f8f05dbdde3b8b10ad209c0043302ef231c43c" Jul 06 22:22:50.408342 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.408323 2571 scope.go:117] "RemoveContainer" containerID="64fc4a25032bbf026619c9437beac1aefa126bdf92a75737b6147cfd7226b82f" Jul 06 22:22:50.415265 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.415249 2571 scope.go:117] "RemoveContainer" containerID="b538067839c626941e5dc101b229e4615958ed16b4acaf378270e825caccd8ed" Jul 06 22:22:50.421227 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.421183 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 22:22:50.422818 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.422797 2571 scope.go:117] "RemoveContainer" containerID="5b153f04549dcac389ce67c6dfe9e62bd634ed422b4d7b4eb52d566cca9083c8" Jul 06 22:22:50.426797 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.426770 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 22:22:50.430898 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.430872 2571 scope.go:117] "RemoveContainer" containerID="169b1b5cbf78c4f82e1efd81dcea434a33024262717c593575c9c65b5e730cf9" Jul 06 22:22:50.438587 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.438566 2571 scope.go:117] "RemoveContainer" containerID="1dfcc9c7fa34bbcb4790bf6eaa528530cd8a4a6ed718d0f828136b27c716fcba" Jul 06 22:22:50.445459 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.445437 2571 scope.go:117] "RemoveContainer" containerID="f2b7ebd773d80323cb3057b27a75b5edb511a07a5e334d5182d78ffdb18bccdf" Jul 06 22:22:50.445753 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:22:50.445727 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2b7ebd773d80323cb3057b27a75b5edb511a07a5e334d5182d78ffdb18bccdf\": container with ID starting with f2b7ebd773d80323cb3057b27a75b5edb511a07a5e334d5182d78ffdb18bccdf not found: ID does not exist" containerID="f2b7ebd773d80323cb3057b27a75b5edb511a07a5e334d5182d78ffdb18bccdf" Jul 06 22:22:50.445809 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.445756 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2b7ebd773d80323cb3057b27a75b5edb511a07a5e334d5182d78ffdb18bccdf"} err="failed to get container status \"f2b7ebd773d80323cb3057b27a75b5edb511a07a5e334d5182d78ffdb18bccdf\": rpc error: code = NotFound desc = could not find container \"f2b7ebd773d80323cb3057b27a75b5edb511a07a5e334d5182d78ffdb18bccdf\": container with ID starting with f2b7ebd773d80323cb3057b27a75b5edb511a07a5e334d5182d78ffdb18bccdf not found: ID does not exist" Jul 06 22:22:50.445809 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.445779 2571 scope.go:117] "RemoveContainer" containerID="04d909b6e4ac12fdc78887eba1f8f05dbdde3b8b10ad209c0043302ef231c43c" Jul 06 22:22:50.446010 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:22:50.445988 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04d909b6e4ac12fdc78887eba1f8f05dbdde3b8b10ad209c0043302ef231c43c\": container with ID starting with 04d909b6e4ac12fdc78887eba1f8f05dbdde3b8b10ad209c0043302ef231c43c not found: ID does not exist" containerID="04d909b6e4ac12fdc78887eba1f8f05dbdde3b8b10ad209c0043302ef231c43c" Jul 06 22:22:50.446076 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.446020 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04d909b6e4ac12fdc78887eba1f8f05dbdde3b8b10ad209c0043302ef231c43c"} err="failed to get container status \"04d909b6e4ac12fdc78887eba1f8f05dbdde3b8b10ad209c0043302ef231c43c\": rpc error: code = NotFound desc = could not find container \"04d909b6e4ac12fdc78887eba1f8f05dbdde3b8b10ad209c0043302ef231c43c\": container with ID starting with 04d909b6e4ac12fdc78887eba1f8f05dbdde3b8b10ad209c0043302ef231c43c not found: ID does not exist" Jul 06 22:22:50.446076 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.446046 2571 scope.go:117] "RemoveContainer" containerID="64fc4a25032bbf026619c9437beac1aefa126bdf92a75737b6147cfd7226b82f" Jul 06 22:22:50.446292 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:22:50.446275 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64fc4a25032bbf026619c9437beac1aefa126bdf92a75737b6147cfd7226b82f\": container with ID starting with 64fc4a25032bbf026619c9437beac1aefa126bdf92a75737b6147cfd7226b82f not found: ID does not exist" containerID="64fc4a25032bbf026619c9437beac1aefa126bdf92a75737b6147cfd7226b82f" Jul 06 22:22:50.446347 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.446299 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64fc4a25032bbf026619c9437beac1aefa126bdf92a75737b6147cfd7226b82f"} err="failed to get container status \"64fc4a25032bbf026619c9437beac1aefa126bdf92a75737b6147cfd7226b82f\": rpc error: code = NotFound desc = could not find container \"64fc4a25032bbf026619c9437beac1aefa126bdf92a75737b6147cfd7226b82f\": container with ID starting with 64fc4a25032bbf026619c9437beac1aefa126bdf92a75737b6147cfd7226b82f not found: ID does not exist" Jul 06 22:22:50.446347 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.446315 2571 scope.go:117] "RemoveContainer" containerID="b538067839c626941e5dc101b229e4615958ed16b4acaf378270e825caccd8ed" Jul 06 22:22:50.446550 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:22:50.446530 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b538067839c626941e5dc101b229e4615958ed16b4acaf378270e825caccd8ed\": container with ID starting with b538067839c626941e5dc101b229e4615958ed16b4acaf378270e825caccd8ed not found: ID does not exist" containerID="b538067839c626941e5dc101b229e4615958ed16b4acaf378270e825caccd8ed" Jul 06 22:22:50.446624 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.446557 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b538067839c626941e5dc101b229e4615958ed16b4acaf378270e825caccd8ed"} err="failed to get container status \"b538067839c626941e5dc101b229e4615958ed16b4acaf378270e825caccd8ed\": rpc error: code = NotFound desc = could not find container \"b538067839c626941e5dc101b229e4615958ed16b4acaf378270e825caccd8ed\": container with ID starting with b538067839c626941e5dc101b229e4615958ed16b4acaf378270e825caccd8ed not found: ID does not exist" Jul 06 22:22:50.446624 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.446577 2571 scope.go:117] "RemoveContainer" containerID="5b153f04549dcac389ce67c6dfe9e62bd634ed422b4d7b4eb52d566cca9083c8" Jul 06 22:22:50.446795 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:22:50.446779 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b153f04549dcac389ce67c6dfe9e62bd634ed422b4d7b4eb52d566cca9083c8\": container with ID starting with 5b153f04549dcac389ce67c6dfe9e62bd634ed422b4d7b4eb52d566cca9083c8 not found: ID does not exist" containerID="5b153f04549dcac389ce67c6dfe9e62bd634ed422b4d7b4eb52d566cca9083c8" Jul 06 22:22:50.446832 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.446799 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b153f04549dcac389ce67c6dfe9e62bd634ed422b4d7b4eb52d566cca9083c8"} err="failed to get container status \"5b153f04549dcac389ce67c6dfe9e62bd634ed422b4d7b4eb52d566cca9083c8\": rpc error: code = NotFound desc = could not find container \"5b153f04549dcac389ce67c6dfe9e62bd634ed422b4d7b4eb52d566cca9083c8\": container with ID starting with 5b153f04549dcac389ce67c6dfe9e62bd634ed422b4d7b4eb52d566cca9083c8 not found: ID does not exist" Jul 06 22:22:50.446832 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.446812 2571 scope.go:117] "RemoveContainer" containerID="169b1b5cbf78c4f82e1efd81dcea434a33024262717c593575c9c65b5e730cf9" Jul 06 22:22:50.447009 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:22:50.446994 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"169b1b5cbf78c4f82e1efd81dcea434a33024262717c593575c9c65b5e730cf9\": container with ID starting with 169b1b5cbf78c4f82e1efd81dcea434a33024262717c593575c9c65b5e730cf9 not found: ID does not exist" containerID="169b1b5cbf78c4f82e1efd81dcea434a33024262717c593575c9c65b5e730cf9" Jul 06 22:22:50.447056 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.447011 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"169b1b5cbf78c4f82e1efd81dcea434a33024262717c593575c9c65b5e730cf9"} err="failed to get container status \"169b1b5cbf78c4f82e1efd81dcea434a33024262717c593575c9c65b5e730cf9\": rpc error: code = NotFound desc = could not find container \"169b1b5cbf78c4f82e1efd81dcea434a33024262717c593575c9c65b5e730cf9\": container with ID starting with 169b1b5cbf78c4f82e1efd81dcea434a33024262717c593575c9c65b5e730cf9 not found: ID does not exist" Jul 06 22:22:50.447056 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.447025 2571 scope.go:117] "RemoveContainer" containerID="1dfcc9c7fa34bbcb4790bf6eaa528530cd8a4a6ed718d0f828136b27c716fcba" Jul 06 22:22:50.447249 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:22:50.447234 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dfcc9c7fa34bbcb4790bf6eaa528530cd8a4a6ed718d0f828136b27c716fcba\": container with ID starting with 1dfcc9c7fa34bbcb4790bf6eaa528530cd8a4a6ed718d0f828136b27c716fcba not found: ID does not exist" containerID="1dfcc9c7fa34bbcb4790bf6eaa528530cd8a4a6ed718d0f828136b27c716fcba" Jul 06 22:22:50.447315 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.447257 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dfcc9c7fa34bbcb4790bf6eaa528530cd8a4a6ed718d0f828136b27c716fcba"} err="failed to get container status \"1dfcc9c7fa34bbcb4790bf6eaa528530cd8a4a6ed718d0f828136b27c716fcba\": rpc error: code = NotFound desc = could not find container \"1dfcc9c7fa34bbcb4790bf6eaa528530cd8a4a6ed718d0f828136b27c716fcba\": container with ID starting with 1dfcc9c7fa34bbcb4790bf6eaa528530cd8a4a6ed718d0f828136b27c716fcba not found: ID does not exist" Jul 06 22:22:50.447315 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.447276 2571 scope.go:117] "RemoveContainer" containerID="f2b7ebd773d80323cb3057b27a75b5edb511a07a5e334d5182d78ffdb18bccdf" Jul 06 22:22:50.447479 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.447463 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2b7ebd773d80323cb3057b27a75b5edb511a07a5e334d5182d78ffdb18bccdf"} err="failed to get container status \"f2b7ebd773d80323cb3057b27a75b5edb511a07a5e334d5182d78ffdb18bccdf\": rpc error: code = NotFound desc = could not find container \"f2b7ebd773d80323cb3057b27a75b5edb511a07a5e334d5182d78ffdb18bccdf\": container with ID starting with f2b7ebd773d80323cb3057b27a75b5edb511a07a5e334d5182d78ffdb18bccdf not found: ID does not exist" Jul 06 22:22:50.447530 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.447480 2571 scope.go:117] "RemoveContainer" containerID="04d909b6e4ac12fdc78887eba1f8f05dbdde3b8b10ad209c0043302ef231c43c" Jul 06 22:22:50.447666 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.447649 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04d909b6e4ac12fdc78887eba1f8f05dbdde3b8b10ad209c0043302ef231c43c"} err="failed to get container status \"04d909b6e4ac12fdc78887eba1f8f05dbdde3b8b10ad209c0043302ef231c43c\": rpc error: code = NotFound desc = could not find container \"04d909b6e4ac12fdc78887eba1f8f05dbdde3b8b10ad209c0043302ef231c43c\": container with ID starting with 04d909b6e4ac12fdc78887eba1f8f05dbdde3b8b10ad209c0043302ef231c43c not found: ID does not exist" Jul 06 22:22:50.447744 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.447666 2571 scope.go:117] "RemoveContainer" containerID="64fc4a25032bbf026619c9437beac1aefa126bdf92a75737b6147cfd7226b82f" Jul 06 22:22:50.447876 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.447856 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64fc4a25032bbf026619c9437beac1aefa126bdf92a75737b6147cfd7226b82f"} err="failed to get container status \"64fc4a25032bbf026619c9437beac1aefa126bdf92a75737b6147cfd7226b82f\": rpc error: code = NotFound desc = could not find container \"64fc4a25032bbf026619c9437beac1aefa126bdf92a75737b6147cfd7226b82f\": container with ID starting with 64fc4a25032bbf026619c9437beac1aefa126bdf92a75737b6147cfd7226b82f not found: ID does not exist" Jul 06 22:22:50.447932 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.447877 2571 scope.go:117] "RemoveContainer" containerID="b538067839c626941e5dc101b229e4615958ed16b4acaf378270e825caccd8ed" Jul 06 22:22:50.448134 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.448098 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b538067839c626941e5dc101b229e4615958ed16b4acaf378270e825caccd8ed"} err="failed to get container status \"b538067839c626941e5dc101b229e4615958ed16b4acaf378270e825caccd8ed\": rpc error: code = NotFound desc = could not find container \"b538067839c626941e5dc101b229e4615958ed16b4acaf378270e825caccd8ed\": container with ID starting with b538067839c626941e5dc101b229e4615958ed16b4acaf378270e825caccd8ed not found: ID does not exist" Jul 06 22:22:50.448196 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.448136 2571 scope.go:117] "RemoveContainer" containerID="5b153f04549dcac389ce67c6dfe9e62bd634ed422b4d7b4eb52d566cca9083c8" Jul 06 22:22:50.448471 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.448439 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b153f04549dcac389ce67c6dfe9e62bd634ed422b4d7b4eb52d566cca9083c8"} err="failed to get container status \"5b153f04549dcac389ce67c6dfe9e62bd634ed422b4d7b4eb52d566cca9083c8\": rpc error: code = NotFound desc = could not find container \"5b153f04549dcac389ce67c6dfe9e62bd634ed422b4d7b4eb52d566cca9083c8\": container with ID starting with 5b153f04549dcac389ce67c6dfe9e62bd634ed422b4d7b4eb52d566cca9083c8 not found: ID does not exist" Jul 06 22:22:50.448610 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.448590 2571 scope.go:117] "RemoveContainer" containerID="169b1b5cbf78c4f82e1efd81dcea434a33024262717c593575c9c65b5e730cf9" Jul 06 22:22:50.448874 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.448853 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"169b1b5cbf78c4f82e1efd81dcea434a33024262717c593575c9c65b5e730cf9"} err="failed to get container status \"169b1b5cbf78c4f82e1efd81dcea434a33024262717c593575c9c65b5e730cf9\": rpc error: code = NotFound desc = could not find container \"169b1b5cbf78c4f82e1efd81dcea434a33024262717c593575c9c65b5e730cf9\": container with ID starting with 169b1b5cbf78c4f82e1efd81dcea434a33024262717c593575c9c65b5e730cf9 not found: ID does not exist" Jul 06 22:22:50.448937 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.448875 2571 scope.go:117] "RemoveContainer" containerID="1dfcc9c7fa34bbcb4790bf6eaa528530cd8a4a6ed718d0f828136b27c716fcba" Jul 06 22:22:50.449094 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.449076 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dfcc9c7fa34bbcb4790bf6eaa528530cd8a4a6ed718d0f828136b27c716fcba"} err="failed to get container status \"1dfcc9c7fa34bbcb4790bf6eaa528530cd8a4a6ed718d0f828136b27c716fcba\": rpc error: code = NotFound desc = could not find container \"1dfcc9c7fa34bbcb4790bf6eaa528530cd8a4a6ed718d0f828136b27c716fcba\": container with ID starting with 1dfcc9c7fa34bbcb4790bf6eaa528530cd8a4a6ed718d0f828136b27c716fcba not found: ID does not exist" Jul 06 22:22:50.449166 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.449098 2571 scope.go:117] "RemoveContainer" containerID="f2b7ebd773d80323cb3057b27a75b5edb511a07a5e334d5182d78ffdb18bccdf" Jul 06 22:22:50.449377 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.449355 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2b7ebd773d80323cb3057b27a75b5edb511a07a5e334d5182d78ffdb18bccdf"} err="failed to get container status \"f2b7ebd773d80323cb3057b27a75b5edb511a07a5e334d5182d78ffdb18bccdf\": rpc error: code = NotFound desc = could not find container \"f2b7ebd773d80323cb3057b27a75b5edb511a07a5e334d5182d78ffdb18bccdf\": container with ID starting with f2b7ebd773d80323cb3057b27a75b5edb511a07a5e334d5182d78ffdb18bccdf not found: ID does not exist" Jul 06 22:22:50.449423 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.449378 2571 scope.go:117] "RemoveContainer" containerID="04d909b6e4ac12fdc78887eba1f8f05dbdde3b8b10ad209c0043302ef231c43c" Jul 06 22:22:50.449620 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.449602 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04d909b6e4ac12fdc78887eba1f8f05dbdde3b8b10ad209c0043302ef231c43c"} err="failed to get container status \"04d909b6e4ac12fdc78887eba1f8f05dbdde3b8b10ad209c0043302ef231c43c\": rpc error: code = NotFound desc = could not find container \"04d909b6e4ac12fdc78887eba1f8f05dbdde3b8b10ad209c0043302ef231c43c\": container with ID starting with 04d909b6e4ac12fdc78887eba1f8f05dbdde3b8b10ad209c0043302ef231c43c not found: ID does not exist" Jul 06 22:22:50.449681 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.449620 2571 scope.go:117] "RemoveContainer" containerID="64fc4a25032bbf026619c9437beac1aefa126bdf92a75737b6147cfd7226b82f" Jul 06 22:22:50.449841 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.449825 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64fc4a25032bbf026619c9437beac1aefa126bdf92a75737b6147cfd7226b82f"} err="failed to get container status \"64fc4a25032bbf026619c9437beac1aefa126bdf92a75737b6147cfd7226b82f\": rpc error: code = NotFound desc = could not find container \"64fc4a25032bbf026619c9437beac1aefa126bdf92a75737b6147cfd7226b82f\": container with ID starting with 64fc4a25032bbf026619c9437beac1aefa126bdf92a75737b6147cfd7226b82f not found: ID does not exist" Jul 06 22:22:50.449886 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.449841 2571 scope.go:117] "RemoveContainer" containerID="b538067839c626941e5dc101b229e4615958ed16b4acaf378270e825caccd8ed" Jul 06 22:22:50.450043 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.450025 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b538067839c626941e5dc101b229e4615958ed16b4acaf378270e825caccd8ed"} err="failed to get container status \"b538067839c626941e5dc101b229e4615958ed16b4acaf378270e825caccd8ed\": rpc error: code = NotFound desc = could not find container \"b538067839c626941e5dc101b229e4615958ed16b4acaf378270e825caccd8ed\": container with ID starting with b538067839c626941e5dc101b229e4615958ed16b4acaf378270e825caccd8ed not found: ID does not exist" Jul 06 22:22:50.450112 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.450046 2571 scope.go:117] "RemoveContainer" containerID="5b153f04549dcac389ce67c6dfe9e62bd634ed422b4d7b4eb52d566cca9083c8" Jul 06 22:22:50.450283 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.450264 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b153f04549dcac389ce67c6dfe9e62bd634ed422b4d7b4eb52d566cca9083c8"} err="failed to get container status \"5b153f04549dcac389ce67c6dfe9e62bd634ed422b4d7b4eb52d566cca9083c8\": rpc error: code = NotFound desc = could not find container \"5b153f04549dcac389ce67c6dfe9e62bd634ed422b4d7b4eb52d566cca9083c8\": container with ID starting with 5b153f04549dcac389ce67c6dfe9e62bd634ed422b4d7b4eb52d566cca9083c8 not found: ID does not exist" Jul 06 22:22:50.450357 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.450284 2571 scope.go:117] "RemoveContainer" containerID="169b1b5cbf78c4f82e1efd81dcea434a33024262717c593575c9c65b5e730cf9" Jul 06 22:22:50.450489 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.450471 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"169b1b5cbf78c4f82e1efd81dcea434a33024262717c593575c9c65b5e730cf9"} err="failed to get container status \"169b1b5cbf78c4f82e1efd81dcea434a33024262717c593575c9c65b5e730cf9\": rpc error: code = NotFound desc = could not find container \"169b1b5cbf78c4f82e1efd81dcea434a33024262717c593575c9c65b5e730cf9\": container with ID starting with 169b1b5cbf78c4f82e1efd81dcea434a33024262717c593575c9c65b5e730cf9 not found: ID does not exist" Jul 06 22:22:50.450527 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.450490 2571 scope.go:117] "RemoveContainer" containerID="1dfcc9c7fa34bbcb4790bf6eaa528530cd8a4a6ed718d0f828136b27c716fcba" Jul 06 22:22:50.450663 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.450645 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dfcc9c7fa34bbcb4790bf6eaa528530cd8a4a6ed718d0f828136b27c716fcba"} err="failed to get container status \"1dfcc9c7fa34bbcb4790bf6eaa528530cd8a4a6ed718d0f828136b27c716fcba\": rpc error: code = NotFound desc = could not find container \"1dfcc9c7fa34bbcb4790bf6eaa528530cd8a4a6ed718d0f828136b27c716fcba\": container with ID starting with 1dfcc9c7fa34bbcb4790bf6eaa528530cd8a4a6ed718d0f828136b27c716fcba not found: ID does not exist" Jul 06 22:22:50.450728 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.450664 2571 scope.go:117] "RemoveContainer" containerID="f2b7ebd773d80323cb3057b27a75b5edb511a07a5e334d5182d78ffdb18bccdf" Jul 06 22:22:50.450869 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.450851 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2b7ebd773d80323cb3057b27a75b5edb511a07a5e334d5182d78ffdb18bccdf"} err="failed to get container status \"f2b7ebd773d80323cb3057b27a75b5edb511a07a5e334d5182d78ffdb18bccdf\": rpc error: code = NotFound desc = could not find container \"f2b7ebd773d80323cb3057b27a75b5edb511a07a5e334d5182d78ffdb18bccdf\": container with ID starting with f2b7ebd773d80323cb3057b27a75b5edb511a07a5e334d5182d78ffdb18bccdf not found: ID does not exist" Jul 06 22:22:50.450916 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.450870 2571 scope.go:117] "RemoveContainer" containerID="04d909b6e4ac12fdc78887eba1f8f05dbdde3b8b10ad209c0043302ef231c43c" Jul 06 22:22:50.451047 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.451030 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04d909b6e4ac12fdc78887eba1f8f05dbdde3b8b10ad209c0043302ef231c43c"} err="failed to get container status \"04d909b6e4ac12fdc78887eba1f8f05dbdde3b8b10ad209c0043302ef231c43c\": rpc error: code = NotFound desc = could not find container \"04d909b6e4ac12fdc78887eba1f8f05dbdde3b8b10ad209c0043302ef231c43c\": container with ID starting with 04d909b6e4ac12fdc78887eba1f8f05dbdde3b8b10ad209c0043302ef231c43c not found: ID does not exist" Jul 06 22:22:50.451089 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.451047 2571 scope.go:117] "RemoveContainer" containerID="64fc4a25032bbf026619c9437beac1aefa126bdf92a75737b6147cfd7226b82f" Jul 06 22:22:50.451281 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.451264 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64fc4a25032bbf026619c9437beac1aefa126bdf92a75737b6147cfd7226b82f"} err="failed to get container status \"64fc4a25032bbf026619c9437beac1aefa126bdf92a75737b6147cfd7226b82f\": rpc error: code = NotFound desc = could not find container \"64fc4a25032bbf026619c9437beac1aefa126bdf92a75737b6147cfd7226b82f\": container with ID starting with 64fc4a25032bbf026619c9437beac1aefa126bdf92a75737b6147cfd7226b82f not found: ID does not exist" Jul 06 22:22:50.451333 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.451284 2571 scope.go:117] "RemoveContainer" containerID="b538067839c626941e5dc101b229e4615958ed16b4acaf378270e825caccd8ed" Jul 06 22:22:50.451476 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.451459 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b538067839c626941e5dc101b229e4615958ed16b4acaf378270e825caccd8ed"} err="failed to get container status \"b538067839c626941e5dc101b229e4615958ed16b4acaf378270e825caccd8ed\": rpc error: code = NotFound desc = could not find container \"b538067839c626941e5dc101b229e4615958ed16b4acaf378270e825caccd8ed\": container with ID starting with b538067839c626941e5dc101b229e4615958ed16b4acaf378270e825caccd8ed not found: ID does not exist" Jul 06 22:22:50.451532 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.451476 2571 scope.go:117] "RemoveContainer" containerID="5b153f04549dcac389ce67c6dfe9e62bd634ed422b4d7b4eb52d566cca9083c8" Jul 06 22:22:50.451673 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.451646 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b153f04549dcac389ce67c6dfe9e62bd634ed422b4d7b4eb52d566cca9083c8"} err="failed to get container status \"5b153f04549dcac389ce67c6dfe9e62bd634ed422b4d7b4eb52d566cca9083c8\": rpc error: code = NotFound desc = could not find container \"5b153f04549dcac389ce67c6dfe9e62bd634ed422b4d7b4eb52d566cca9083c8\": container with ID starting with 5b153f04549dcac389ce67c6dfe9e62bd634ed422b4d7b4eb52d566cca9083c8 not found: ID does not exist" Jul 06 22:22:50.451673 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.451667 2571 scope.go:117] "RemoveContainer" containerID="169b1b5cbf78c4f82e1efd81dcea434a33024262717c593575c9c65b5e730cf9" Jul 06 22:22:50.451874 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.451852 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"169b1b5cbf78c4f82e1efd81dcea434a33024262717c593575c9c65b5e730cf9"} err="failed to get container status \"169b1b5cbf78c4f82e1efd81dcea434a33024262717c593575c9c65b5e730cf9\": rpc error: code = NotFound desc = could not find container \"169b1b5cbf78c4f82e1efd81dcea434a33024262717c593575c9c65b5e730cf9\": container with ID starting with 169b1b5cbf78c4f82e1efd81dcea434a33024262717c593575c9c65b5e730cf9 not found: ID does not exist" Jul 06 22:22:50.451943 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.451875 2571 scope.go:117] "RemoveContainer" containerID="1dfcc9c7fa34bbcb4790bf6eaa528530cd8a4a6ed718d0f828136b27c716fcba" Jul 06 22:22:50.452091 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.452075 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dfcc9c7fa34bbcb4790bf6eaa528530cd8a4a6ed718d0f828136b27c716fcba"} err="failed to get container status \"1dfcc9c7fa34bbcb4790bf6eaa528530cd8a4a6ed718d0f828136b27c716fcba\": rpc error: code = NotFound desc = could not find container \"1dfcc9c7fa34bbcb4790bf6eaa528530cd8a4a6ed718d0f828136b27c716fcba\": container with ID starting with 1dfcc9c7fa34bbcb4790bf6eaa528530cd8a4a6ed718d0f828136b27c716fcba not found: ID does not exist" Jul 06 22:22:50.452145 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.452091 2571 scope.go:117] "RemoveContainer" containerID="f2b7ebd773d80323cb3057b27a75b5edb511a07a5e334d5182d78ffdb18bccdf" Jul 06 22:22:50.452328 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.452307 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2b7ebd773d80323cb3057b27a75b5edb511a07a5e334d5182d78ffdb18bccdf"} err="failed to get container status \"f2b7ebd773d80323cb3057b27a75b5edb511a07a5e334d5182d78ffdb18bccdf\": rpc error: code = NotFound desc = could not find container \"f2b7ebd773d80323cb3057b27a75b5edb511a07a5e334d5182d78ffdb18bccdf\": container with ID starting with f2b7ebd773d80323cb3057b27a75b5edb511a07a5e334d5182d78ffdb18bccdf not found: ID does not exist" Jul 06 22:22:50.452400 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.452329 2571 scope.go:117] "RemoveContainer" containerID="04d909b6e4ac12fdc78887eba1f8f05dbdde3b8b10ad209c0043302ef231c43c" Jul 06 22:22:50.452527 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.452511 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04d909b6e4ac12fdc78887eba1f8f05dbdde3b8b10ad209c0043302ef231c43c"} err="failed to get container status \"04d909b6e4ac12fdc78887eba1f8f05dbdde3b8b10ad209c0043302ef231c43c\": rpc error: code = NotFound desc = could not find container \"04d909b6e4ac12fdc78887eba1f8f05dbdde3b8b10ad209c0043302ef231c43c\": container with ID starting with 04d909b6e4ac12fdc78887eba1f8f05dbdde3b8b10ad209c0043302ef231c43c not found: ID does not exist" Jul 06 22:22:50.452593 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.452528 2571 scope.go:117] "RemoveContainer" containerID="64fc4a25032bbf026619c9437beac1aefa126bdf92a75737b6147cfd7226b82f" Jul 06 22:22:50.452721 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.452704 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64fc4a25032bbf026619c9437beac1aefa126bdf92a75737b6147cfd7226b82f"} err="failed to get container status \"64fc4a25032bbf026619c9437beac1aefa126bdf92a75737b6147cfd7226b82f\": rpc error: code = NotFound desc = could not find container \"64fc4a25032bbf026619c9437beac1aefa126bdf92a75737b6147cfd7226b82f\": container with ID starting with 64fc4a25032bbf026619c9437beac1aefa126bdf92a75737b6147cfd7226b82f not found: ID does not exist" Jul 06 22:22:50.452783 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.452722 2571 scope.go:117] "RemoveContainer" containerID="b538067839c626941e5dc101b229e4615958ed16b4acaf378270e825caccd8ed" Jul 06 22:22:50.452924 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.452909 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b538067839c626941e5dc101b229e4615958ed16b4acaf378270e825caccd8ed"} err="failed to get container status \"b538067839c626941e5dc101b229e4615958ed16b4acaf378270e825caccd8ed\": rpc error: code = NotFound desc = could not find container \"b538067839c626941e5dc101b229e4615958ed16b4acaf378270e825caccd8ed\": container with ID starting with b538067839c626941e5dc101b229e4615958ed16b4acaf378270e825caccd8ed not found: ID does not exist" Jul 06 22:22:50.452973 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.452924 2571 scope.go:117] "RemoveContainer" containerID="5b153f04549dcac389ce67c6dfe9e62bd634ed422b4d7b4eb52d566cca9083c8" Jul 06 22:22:50.453102 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.453085 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b153f04549dcac389ce67c6dfe9e62bd634ed422b4d7b4eb52d566cca9083c8"} err="failed to get container status \"5b153f04549dcac389ce67c6dfe9e62bd634ed422b4d7b4eb52d566cca9083c8\": rpc error: code = NotFound desc = could not find container \"5b153f04549dcac389ce67c6dfe9e62bd634ed422b4d7b4eb52d566cca9083c8\": container with ID starting with 5b153f04549dcac389ce67c6dfe9e62bd634ed422b4d7b4eb52d566cca9083c8 not found: ID does not exist" Jul 06 22:22:50.453169 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.453103 2571 scope.go:117] "RemoveContainer" containerID="169b1b5cbf78c4f82e1efd81dcea434a33024262717c593575c9c65b5e730cf9" Jul 06 22:22:50.453306 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.453287 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"169b1b5cbf78c4f82e1efd81dcea434a33024262717c593575c9c65b5e730cf9"} err="failed to get container status \"169b1b5cbf78c4f82e1efd81dcea434a33024262717c593575c9c65b5e730cf9\": rpc error: code = NotFound desc = could not find container \"169b1b5cbf78c4f82e1efd81dcea434a33024262717c593575c9c65b5e730cf9\": container with ID starting with 169b1b5cbf78c4f82e1efd81dcea434a33024262717c593575c9c65b5e730cf9 not found: ID does not exist" Jul 06 22:22:50.453376 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.453306 2571 scope.go:117] "RemoveContainer" containerID="1dfcc9c7fa34bbcb4790bf6eaa528530cd8a4a6ed718d0f828136b27c716fcba" Jul 06 22:22:50.453547 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.453529 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dfcc9c7fa34bbcb4790bf6eaa528530cd8a4a6ed718d0f828136b27c716fcba"} err="failed to get container status \"1dfcc9c7fa34bbcb4790bf6eaa528530cd8a4a6ed718d0f828136b27c716fcba\": rpc error: code = NotFound desc = could not find container \"1dfcc9c7fa34bbcb4790bf6eaa528530cd8a4a6ed718d0f828136b27c716fcba\": container with ID starting with 1dfcc9c7fa34bbcb4790bf6eaa528530cd8a4a6ed718d0f828136b27c716fcba not found: ID does not exist" Jul 06 22:22:50.453547 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.453545 2571 scope.go:117] "RemoveContainer" containerID="f2b7ebd773d80323cb3057b27a75b5edb511a07a5e334d5182d78ffdb18bccdf" Jul 06 22:22:50.453741 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.453726 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2b7ebd773d80323cb3057b27a75b5edb511a07a5e334d5182d78ffdb18bccdf"} err="failed to get container status \"f2b7ebd773d80323cb3057b27a75b5edb511a07a5e334d5182d78ffdb18bccdf\": rpc error: code = NotFound desc = could not find container \"f2b7ebd773d80323cb3057b27a75b5edb511a07a5e334d5182d78ffdb18bccdf\": container with ID starting with f2b7ebd773d80323cb3057b27a75b5edb511a07a5e334d5182d78ffdb18bccdf not found: ID does not exist" Jul 06 22:22:50.453741 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.453741 2571 scope.go:117] "RemoveContainer" containerID="04d909b6e4ac12fdc78887eba1f8f05dbdde3b8b10ad209c0043302ef231c43c" Jul 06 22:22:50.453929 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.453913 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04d909b6e4ac12fdc78887eba1f8f05dbdde3b8b10ad209c0043302ef231c43c"} err="failed to get container status \"04d909b6e4ac12fdc78887eba1f8f05dbdde3b8b10ad209c0043302ef231c43c\": rpc error: code = NotFound desc = could not find container \"04d909b6e4ac12fdc78887eba1f8f05dbdde3b8b10ad209c0043302ef231c43c\": container with ID starting with 04d909b6e4ac12fdc78887eba1f8f05dbdde3b8b10ad209c0043302ef231c43c not found: ID does not exist" Jul 06 22:22:50.453978 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.453929 2571 scope.go:117] "RemoveContainer" containerID="64fc4a25032bbf026619c9437beac1aefa126bdf92a75737b6147cfd7226b82f" Jul 06 22:22:50.454096 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.454081 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64fc4a25032bbf026619c9437beac1aefa126bdf92a75737b6147cfd7226b82f"} err="failed to get container status \"64fc4a25032bbf026619c9437beac1aefa126bdf92a75737b6147cfd7226b82f\": rpc error: code = NotFound desc = could not find container \"64fc4a25032bbf026619c9437beac1aefa126bdf92a75737b6147cfd7226b82f\": container with ID starting with 64fc4a25032bbf026619c9437beac1aefa126bdf92a75737b6147cfd7226b82f not found: ID does not exist" Jul 06 22:22:50.454137 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.454096 2571 scope.go:117] "RemoveContainer" containerID="b538067839c626941e5dc101b229e4615958ed16b4acaf378270e825caccd8ed" Jul 06 22:22:50.454291 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.454275 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b538067839c626941e5dc101b229e4615958ed16b4acaf378270e825caccd8ed"} err="failed to get container status \"b538067839c626941e5dc101b229e4615958ed16b4acaf378270e825caccd8ed\": rpc error: code = NotFound desc = could not find container \"b538067839c626941e5dc101b229e4615958ed16b4acaf378270e825caccd8ed\": container with ID starting with b538067839c626941e5dc101b229e4615958ed16b4acaf378270e825caccd8ed not found: ID does not exist" Jul 06 22:22:50.454338 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.454292 2571 scope.go:117] "RemoveContainer" containerID="5b153f04549dcac389ce67c6dfe9e62bd634ed422b4d7b4eb52d566cca9083c8" Jul 06 22:22:50.454478 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.454462 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b153f04549dcac389ce67c6dfe9e62bd634ed422b4d7b4eb52d566cca9083c8"} err="failed to get container status \"5b153f04549dcac389ce67c6dfe9e62bd634ed422b4d7b4eb52d566cca9083c8\": rpc error: code = NotFound desc = could not find container \"5b153f04549dcac389ce67c6dfe9e62bd634ed422b4d7b4eb52d566cca9083c8\": container with ID starting with 5b153f04549dcac389ce67c6dfe9e62bd634ed422b4d7b4eb52d566cca9083c8 not found: ID does not exist" Jul 06 22:22:50.454478 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.454478 2571 scope.go:117] "RemoveContainer" containerID="169b1b5cbf78c4f82e1efd81dcea434a33024262717c593575c9c65b5e730cf9" Jul 06 22:22:50.454651 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.454631 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"169b1b5cbf78c4f82e1efd81dcea434a33024262717c593575c9c65b5e730cf9"} err="failed to get container status \"169b1b5cbf78c4f82e1efd81dcea434a33024262717c593575c9c65b5e730cf9\": rpc error: code = NotFound desc = could not find container \"169b1b5cbf78c4f82e1efd81dcea434a33024262717c593575c9c65b5e730cf9\": container with ID starting with 169b1b5cbf78c4f82e1efd81dcea434a33024262717c593575c9c65b5e730cf9 not found: ID does not exist" Jul 06 22:22:50.454693 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.454651 2571 scope.go:117] "RemoveContainer" containerID="1dfcc9c7fa34bbcb4790bf6eaa528530cd8a4a6ed718d0f828136b27c716fcba" Jul 06 22:22:50.454844 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.454822 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dfcc9c7fa34bbcb4790bf6eaa528530cd8a4a6ed718d0f828136b27c716fcba"} err="failed to get container status \"1dfcc9c7fa34bbcb4790bf6eaa528530cd8a4a6ed718d0f828136b27c716fcba\": rpc error: code = NotFound desc = could not find container \"1dfcc9c7fa34bbcb4790bf6eaa528530cd8a4a6ed718d0f828136b27c716fcba\": container with ID starting with 1dfcc9c7fa34bbcb4790bf6eaa528530cd8a4a6ed718d0f828136b27c716fcba not found: ID does not exist" Jul 06 22:22:50.457909 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.457879 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 22:22:50.458133 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.458121 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ca65cf7-655d-4fbb-b949-04b0e65cc2f5" containerName="prometheus" Jul 06 22:22:50.458174 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.458135 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ca65cf7-655d-4fbb-b949-04b0e65cc2f5" containerName="prometheus" Jul 06 22:22:50.458174 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.458146 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ca65cf7-655d-4fbb-b949-04b0e65cc2f5" containerName="init-config-reloader" Jul 06 22:22:50.458174 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.458152 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ca65cf7-655d-4fbb-b949-04b0e65cc2f5" containerName="init-config-reloader" Jul 06 22:22:50.458174 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.458159 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ca65cf7-655d-4fbb-b949-04b0e65cc2f5" containerName="kube-rbac-proxy-web" Jul 06 22:22:50.458174 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.458165 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ca65cf7-655d-4fbb-b949-04b0e65cc2f5" containerName="kube-rbac-proxy-web" Jul 06 22:22:50.458174 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.458171 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ca65cf7-655d-4fbb-b949-04b0e65cc2f5" containerName="config-reloader" Jul 06 22:22:50.458368 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.458176 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ca65cf7-655d-4fbb-b949-04b0e65cc2f5" containerName="config-reloader" Jul 06 22:22:50.458368 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.458191 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ca65cf7-655d-4fbb-b949-04b0e65cc2f5" containerName="kube-rbac-proxy-thanos" Jul 06 22:22:50.458368 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.458196 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ca65cf7-655d-4fbb-b949-04b0e65cc2f5" containerName="kube-rbac-proxy-thanos" Jul 06 22:22:50.458368 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.458202 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8acf7e0c-2b66-4c40-a249-58f6fccba39e" containerName="registry" Jul 06 22:22:50.458368 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.458226 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="8acf7e0c-2b66-4c40-a249-58f6fccba39e" containerName="registry" Jul 06 22:22:50.458368 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.458237 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ca65cf7-655d-4fbb-b949-04b0e65cc2f5" containerName="thanos-sidecar" Jul 06 22:22:50.458368 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.458242 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ca65cf7-655d-4fbb-b949-04b0e65cc2f5" containerName="thanos-sidecar" Jul 06 22:22:50.458368 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.458247 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ca65cf7-655d-4fbb-b949-04b0e65cc2f5" containerName="kube-rbac-proxy" Jul 06 22:22:50.458368 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.458252 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ca65cf7-655d-4fbb-b949-04b0e65cc2f5" containerName="kube-rbac-proxy" Jul 06 22:22:50.458368 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.458292 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ca65cf7-655d-4fbb-b949-04b0e65cc2f5" containerName="prometheus" Jul 06 22:22:50.458368 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.458300 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="8acf7e0c-2b66-4c40-a249-58f6fccba39e" containerName="registry" Jul 06 22:22:50.458368 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.458305 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ca65cf7-655d-4fbb-b949-04b0e65cc2f5" containerName="thanos-sidecar" Jul 06 22:22:50.458368 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.458311 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ca65cf7-655d-4fbb-b949-04b0e65cc2f5" containerName="kube-rbac-proxy-web" Jul 06 22:22:50.458368 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.458315 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ca65cf7-655d-4fbb-b949-04b0e65cc2f5" containerName="kube-rbac-proxy" Jul 06 22:22:50.458368 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.458319 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ca65cf7-655d-4fbb-b949-04b0e65cc2f5" containerName="kube-rbac-proxy-thanos" Jul 06 22:22:50.458368 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.458325 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ca65cf7-655d-4fbb-b949-04b0e65cc2f5" containerName="config-reloader" Jul 06 22:22:50.463529 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.463512 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:50.466275 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.466249 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 06 22:22:50.466365 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.466254 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 06 22:22:50.466421 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.466379 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 06 22:22:50.466421 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.466400 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 06 22:22:50.466536 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.466448 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 06 22:22:50.466536 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.466467 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-22ftm\"" Jul 06 22:22:50.466536 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.466473 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 06 22:22:50.466536 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.466401 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 06 22:22:50.466536 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.466467 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 06 22:22:50.466768 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.466751 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 06 22:22:50.467013 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.466995 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 06 22:22:50.467094 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.466996 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 06 22:22:50.467094 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.466999 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 06 22:22:50.467094 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.467002 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 06 22:22:50.467531 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.467516 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-2728132d4c1c4\"" Jul 06 22:22:50.470725 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.470637 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 06 22:22:50.474090 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.474074 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 06 22:22:50.477947 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.477927 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 22:22:50.600443 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.600407 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ce8044dc-7328-4898-a3bf-357ce89db933-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ce8044dc-7328-4898-a3bf-357ce89db933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:50.600443 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.600450 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ce8044dc-7328-4898-a3bf-357ce89db933-web-config\") pod \"prometheus-k8s-0\" (UID: \"ce8044dc-7328-4898-a3bf-357ce89db933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:50.600659 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.600471 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce8044dc-7328-4898-a3bf-357ce89db933-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ce8044dc-7328-4898-a3bf-357ce89db933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:50.600659 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.600500 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ce8044dc-7328-4898-a3bf-357ce89db933-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ce8044dc-7328-4898-a3bf-357ce89db933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:50.600659 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.600547 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ce8044dc-7328-4898-a3bf-357ce89db933-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ce8044dc-7328-4898-a3bf-357ce89db933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:50.600659 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.600565 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ce8044dc-7328-4898-a3bf-357ce89db933-config-out\") pod \"prometheus-k8s-0\" (UID: \"ce8044dc-7328-4898-a3bf-357ce89db933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:50.600659 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.600586 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ce8044dc-7328-4898-a3bf-357ce89db933-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ce8044dc-7328-4898-a3bf-357ce89db933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:50.600659 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.600601 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ce8044dc-7328-4898-a3bf-357ce89db933-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ce8044dc-7328-4898-a3bf-357ce89db933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:50.600659 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.600618 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce8044dc-7328-4898-a3bf-357ce89db933-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ce8044dc-7328-4898-a3bf-357ce89db933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:50.600659 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.600645 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pczrh\" (UniqueName: \"kubernetes.io/projected/ce8044dc-7328-4898-a3bf-357ce89db933-kube-api-access-pczrh\") pod \"prometheus-k8s-0\" (UID: \"ce8044dc-7328-4898-a3bf-357ce89db933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:50.600920 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.600698 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ce8044dc-7328-4898-a3bf-357ce89db933-config\") pod \"prometheus-k8s-0\" (UID: \"ce8044dc-7328-4898-a3bf-357ce89db933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:50.600920 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.600721 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ce8044dc-7328-4898-a3bf-357ce89db933-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ce8044dc-7328-4898-a3bf-357ce89db933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:50.600920 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.600745 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ce8044dc-7328-4898-a3bf-357ce89db933-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ce8044dc-7328-4898-a3bf-357ce89db933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:50.600920 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.600767 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ce8044dc-7328-4898-a3bf-357ce89db933-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ce8044dc-7328-4898-a3bf-357ce89db933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:50.600920 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.600789 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce8044dc-7328-4898-a3bf-357ce89db933-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ce8044dc-7328-4898-a3bf-357ce89db933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:50.600920 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.600819 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce8044dc-7328-4898-a3bf-357ce89db933-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ce8044dc-7328-4898-a3bf-357ce89db933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:50.600920 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.600837 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ce8044dc-7328-4898-a3bf-357ce89db933-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ce8044dc-7328-4898-a3bf-357ce89db933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:50.600920 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.600858 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ce8044dc-7328-4898-a3bf-357ce89db933-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ce8044dc-7328-4898-a3bf-357ce89db933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:50.600920 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.600878 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ce8044dc-7328-4898-a3bf-357ce89db933-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ce8044dc-7328-4898-a3bf-357ce89db933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:50.600920 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.600895 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ce8044dc-7328-4898-a3bf-357ce89db933-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ce8044dc-7328-4898-a3bf-357ce89db933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:50.701957 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.701876 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ce8044dc-7328-4898-a3bf-357ce89db933-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ce8044dc-7328-4898-a3bf-357ce89db933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:50.701957 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.701920 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce8044dc-7328-4898-a3bf-357ce89db933-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ce8044dc-7328-4898-a3bf-357ce89db933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:50.702108 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.701970 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce8044dc-7328-4898-a3bf-357ce89db933-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ce8044dc-7328-4898-a3bf-357ce89db933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:50.702108 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.701995 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ce8044dc-7328-4898-a3bf-357ce89db933-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ce8044dc-7328-4898-a3bf-357ce89db933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:50.702108 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.702022 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ce8044dc-7328-4898-a3bf-357ce89db933-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ce8044dc-7328-4898-a3bf-357ce89db933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:50.702108 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.702054 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ce8044dc-7328-4898-a3bf-357ce89db933-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ce8044dc-7328-4898-a3bf-357ce89db933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:50.702316 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.702230 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ce8044dc-7328-4898-a3bf-357ce89db933-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ce8044dc-7328-4898-a3bf-357ce89db933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:50.702316 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.702274 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ce8044dc-7328-4898-a3bf-357ce89db933-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ce8044dc-7328-4898-a3bf-357ce89db933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:50.702384 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.702319 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ce8044dc-7328-4898-a3bf-357ce89db933-web-config\") pod \"prometheus-k8s-0\" (UID: \"ce8044dc-7328-4898-a3bf-357ce89db933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:50.702384 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.702351 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce8044dc-7328-4898-a3bf-357ce89db933-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ce8044dc-7328-4898-a3bf-357ce89db933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:50.702384 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.702379 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ce8044dc-7328-4898-a3bf-357ce89db933-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ce8044dc-7328-4898-a3bf-357ce89db933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:50.702471 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.702416 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ce8044dc-7328-4898-a3bf-357ce89db933-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ce8044dc-7328-4898-a3bf-357ce89db933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:50.702471 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.702440 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ce8044dc-7328-4898-a3bf-357ce89db933-config-out\") pod \"prometheus-k8s-0\" (UID: \"ce8044dc-7328-4898-a3bf-357ce89db933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:50.702528 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.702468 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ce8044dc-7328-4898-a3bf-357ce89db933-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ce8044dc-7328-4898-a3bf-357ce89db933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:50.702528 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.702489 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ce8044dc-7328-4898-a3bf-357ce89db933-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ce8044dc-7328-4898-a3bf-357ce89db933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:50.702528 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.702513 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce8044dc-7328-4898-a3bf-357ce89db933-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ce8044dc-7328-4898-a3bf-357ce89db933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:50.702617 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.702545 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pczrh\" (UniqueName: \"kubernetes.io/projected/ce8044dc-7328-4898-a3bf-357ce89db933-kube-api-access-pczrh\") pod \"prometheus-k8s-0\" (UID: \"ce8044dc-7328-4898-a3bf-357ce89db933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:50.702617 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.702598 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ce8044dc-7328-4898-a3bf-357ce89db933-config\") pod \"prometheus-k8s-0\" (UID: \"ce8044dc-7328-4898-a3bf-357ce89db933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:50.702674 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.702624 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ce8044dc-7328-4898-a3bf-357ce89db933-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ce8044dc-7328-4898-a3bf-357ce89db933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:50.702674 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.702656 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ce8044dc-7328-4898-a3bf-357ce89db933-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ce8044dc-7328-4898-a3bf-357ce89db933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:50.702733 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.702705 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ce8044dc-7328-4898-a3bf-357ce89db933-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ce8044dc-7328-4898-a3bf-357ce89db933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:50.702767 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.702740 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ce8044dc-7328-4898-a3bf-357ce89db933-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ce8044dc-7328-4898-a3bf-357ce89db933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:50.702819 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.702797 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce8044dc-7328-4898-a3bf-357ce89db933-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ce8044dc-7328-4898-a3bf-357ce89db933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:50.702854 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.702822 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce8044dc-7328-4898-a3bf-357ce89db933-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ce8044dc-7328-4898-a3bf-357ce89db933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:50.703712 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.703531 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce8044dc-7328-4898-a3bf-357ce89db933-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ce8044dc-7328-4898-a3bf-357ce89db933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:50.703712 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.703688 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce8044dc-7328-4898-a3bf-357ce89db933-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ce8044dc-7328-4898-a3bf-357ce89db933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:50.704864 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.703947 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ce8044dc-7328-4898-a3bf-357ce89db933-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ce8044dc-7328-4898-a3bf-357ce89db933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:50.705743 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.705706 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ce8044dc-7328-4898-a3bf-357ce89db933-config-out\") pod \"prometheus-k8s-0\" (UID: \"ce8044dc-7328-4898-a3bf-357ce89db933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:50.705886 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.705864 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ce8044dc-7328-4898-a3bf-357ce89db933-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ce8044dc-7328-4898-a3bf-357ce89db933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:50.705954 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.705874 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ce8044dc-7328-4898-a3bf-357ce89db933-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ce8044dc-7328-4898-a3bf-357ce89db933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:50.705954 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.705899 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ce8044dc-7328-4898-a3bf-357ce89db933-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ce8044dc-7328-4898-a3bf-357ce89db933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:50.706073 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.706047 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ce8044dc-7328-4898-a3bf-357ce89db933-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ce8044dc-7328-4898-a3bf-357ce89db933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:50.706428 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.706385 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ce8044dc-7328-4898-a3bf-357ce89db933-web-config\") pod \"prometheus-k8s-0\" (UID: \"ce8044dc-7328-4898-a3bf-357ce89db933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:50.706753 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.706733 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ce8044dc-7328-4898-a3bf-357ce89db933-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ce8044dc-7328-4898-a3bf-357ce89db933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:50.707978 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.707948 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ce8044dc-7328-4898-a3bf-357ce89db933-config\") pod \"prometheus-k8s-0\" (UID: \"ce8044dc-7328-4898-a3bf-357ce89db933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:50.708316 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.708295 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ce8044dc-7328-4898-a3bf-357ce89db933-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ce8044dc-7328-4898-a3bf-357ce89db933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:50.708369 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.708324 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ce8044dc-7328-4898-a3bf-357ce89db933-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ce8044dc-7328-4898-a3bf-357ce89db933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:50.708462 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.708441 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ce8044dc-7328-4898-a3bf-357ce89db933-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ce8044dc-7328-4898-a3bf-357ce89db933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:50.708548 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.708531 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ce8044dc-7328-4898-a3bf-357ce89db933-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ce8044dc-7328-4898-a3bf-357ce89db933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:50.716100 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.716075 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pczrh\" (UniqueName: \"kubernetes.io/projected/ce8044dc-7328-4898-a3bf-357ce89db933-kube-api-access-pczrh\") pod \"prometheus-k8s-0\" (UID: \"ce8044dc-7328-4898-a3bf-357ce89db933\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:50.773656 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.773605 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:50.916581 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:50.916546 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 22:22:50.919747 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:22:50.919717 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce8044dc_7328_4898_a3bf_357ce89db933.slice/crio-f6fa4cd468fb61e8e95c9050ae6e29181e650b09cfa92c03d54e3face651f323 WatchSource:0}: Error finding container f6fa4cd468fb61e8e95c9050ae6e29181e650b09cfa92c03d54e3face651f323: Status 404 returned error can't find the container with id f6fa4cd468fb61e8e95c9050ae6e29181e650b09cfa92c03d54e3face651f323 Jul 06 22:22:51.402323 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:51.402288 2571 generic.go:356] "Generic (PLEG): container finished" podID="ce8044dc-7328-4898-a3bf-357ce89db933" containerID="8325f15dff0adff388149593ae5757b6bd2256f295ac0d93ab76539c53c2aca5" exitCode=0 Jul 06 22:22:51.402477 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:51.402343 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ce8044dc-7328-4898-a3bf-357ce89db933","Type":"ContainerDied","Data":"8325f15dff0adff388149593ae5757b6bd2256f295ac0d93ab76539c53c2aca5"} Jul 06 22:22:51.402477 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:51.402421 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ce8044dc-7328-4898-a3bf-357ce89db933","Type":"ContainerStarted","Data":"f6fa4cd468fb61e8e95c9050ae6e29181e650b09cfa92c03d54e3face651f323"} Jul 06 22:22:51.635440 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:51.635408 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ca65cf7-655d-4fbb-b949-04b0e65cc2f5" path="/var/lib/kubelet/pods/2ca65cf7-655d-4fbb-b949-04b0e65cc2f5/volumes" Jul 06 22:22:52.409746 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:52.409706 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ce8044dc-7328-4898-a3bf-357ce89db933","Type":"ContainerStarted","Data":"4ab68eaa3f2927fb99fc8f628d70d5aed797fcbf88621e5a1179b140028f185c"} Jul 06 22:22:52.410127 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:52.409780 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ce8044dc-7328-4898-a3bf-357ce89db933","Type":"ContainerStarted","Data":"ea589abc3f4dd10966c9bba0ffa9a9ad95037344ee7f48399ec9966d251857d2"} Jul 06 22:22:52.410127 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:52.409813 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ce8044dc-7328-4898-a3bf-357ce89db933","Type":"ContainerStarted","Data":"bfb2f8594a691836fc8fb08463e22b19b15345a86a8cf8df59d40b97d2c8ca35"} Jul 06 22:22:52.410127 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:52.409845 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ce8044dc-7328-4898-a3bf-357ce89db933","Type":"ContainerStarted","Data":"b060d96c68f5c901fcb35377b6d1592d94802d7aeb3bc369953f0cf2ce64870f"} Jul 06 22:22:52.410127 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:52.409882 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ce8044dc-7328-4898-a3bf-357ce89db933","Type":"ContainerStarted","Data":"c4df255b06c656774e1c7385864f10ec43049d5d99f3a43bdbde226473c9770f"} Jul 06 22:22:52.410127 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:52.409924 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ce8044dc-7328-4898-a3bf-357ce89db933","Type":"ContainerStarted","Data":"1c5f131e7532747c12d3304b87a7fd4c582724582d75351bab6c1b8d1797a81b"} Jul 06 22:22:52.446846 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:52.446785 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.426795066 podStartE2EDuration="2.446766501s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.971435ms" isStatefulPod=false podCreationTimestamp="2026-07-06 22:22:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:22:52.444121596 +0000 UTC m=+179.362364443" watchObservedRunningTime="2026-07-06 22:22:52.446766501 +0000 UTC m=+179.365009356" Jul 06 22:22:52.897063 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:52.897032 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:22:52.958923 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:52.958881 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:22:54.744129 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:54.744080 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:22:54.845767 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:54.845724 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:22:54.946445 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:54.946406 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:22:54.998992 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:54.998907 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:22:55.774376 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:55.774322 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:22:56.206217 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:22:56.206184 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:23:09.356642 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:23:09.356600 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:23:09.454326 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:23:09.454289 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:23:09.508060 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:23:09.508024 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:23:26.145137 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:23:26.145099 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:23:26.237077 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:23:26.237040 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:23:27.417643 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:23:27.417606 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:23:41.851789 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:23:41.851754 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:23:41.905291 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:23:41.905253 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:23:50.774782 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:23:50.774735 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:23:50.790519 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:23:50.790489 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:23:51.644083 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:23:51.644050 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:23:52.680125 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:23:52.680067 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:23:53.004486 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:23:53.004364 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:23:56.293554 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:23:56.293520 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:23:56.394526 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:23:56.394491 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:23:57.692619 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:23:57.692582 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:23:57.787840 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:23:57.787804 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:23:58.486960 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:23:58.486926 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:23:58.574969 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:23:58.574934 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:00.760604 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:00.760571 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:00.905738 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:00.905705 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:01.120103 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:01.120024 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-gpbjg"] Jul 06 22:24:01.123518 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:01.123498 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-gpbjg" Jul 06 22:24:01.126198 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:01.126173 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-6k6r8\"" Jul 06 22:24:01.126360 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:01.126173 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 06 22:24:01.127059 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:01.127044 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 06 22:24:01.135130 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:01.135104 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-gpbjg"] Jul 06 22:24:01.182929 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:01.182896 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stw84\" (UniqueName: \"kubernetes.io/projected/87437991-7866-4abf-840b-868ecfd035c8-kube-api-access-stw84\") pod \"cert-manager-cainjector-559cdd8c87-gpbjg\" (UID: \"87437991-7866-4abf-840b-868ecfd035c8\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gpbjg" Jul 06 22:24:01.182929 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:01.182935 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/87437991-7866-4abf-840b-868ecfd035c8-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-gpbjg\" (UID: \"87437991-7866-4abf-840b-868ecfd035c8\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gpbjg" Jul 06 22:24:01.283591 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:01.283556 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stw84\" (UniqueName: \"kubernetes.io/projected/87437991-7866-4abf-840b-868ecfd035c8-kube-api-access-stw84\") pod \"cert-manager-cainjector-559cdd8c87-gpbjg\" (UID: \"87437991-7866-4abf-840b-868ecfd035c8\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gpbjg" Jul 06 22:24:01.283761 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:01.283598 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/87437991-7866-4abf-840b-868ecfd035c8-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-gpbjg\" (UID: \"87437991-7866-4abf-840b-868ecfd035c8\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gpbjg" Jul 06 22:24:01.297800 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:01.297768 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/87437991-7866-4abf-840b-868ecfd035c8-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-gpbjg\" (UID: \"87437991-7866-4abf-840b-868ecfd035c8\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gpbjg" Jul 06 22:24:01.297931 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:01.297771 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stw84\" (UniqueName: \"kubernetes.io/projected/87437991-7866-4abf-840b-868ecfd035c8-kube-api-access-stw84\") pod \"cert-manager-cainjector-559cdd8c87-gpbjg\" (UID: \"87437991-7866-4abf-840b-868ecfd035c8\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gpbjg" Jul 06 22:24:01.444177 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:01.444143 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-gpbjg" Jul 06 22:24:01.574922 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:01.574741 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-gpbjg"] Jul 06 22:24:01.577705 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:24:01.577675 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod87437991_7866_4abf_840b_868ecfd035c8.slice/crio-da1024b5817166865aef7452064d263a45270fd376b8f8494bc10f374426ceb0 WatchSource:0}: Error finding container da1024b5817166865aef7452064d263a45270fd376b8f8494bc10f374426ceb0: Status 404 returned error can't find the container with id da1024b5817166865aef7452064d263a45270fd376b8f8494bc10f374426ceb0 Jul 06 22:24:01.669361 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:01.669324 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-gpbjg" event={"ID":"87437991-7866-4abf-840b-868ecfd035c8","Type":"ContainerStarted","Data":"da1024b5817166865aef7452064d263a45270fd376b8f8494bc10f374426ceb0"} Jul 06 22:24:03.079555 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:03.079487 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:03.132929 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:03.132890 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:05.687879 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:05.687840 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-gpbjg" event={"ID":"87437991-7866-4abf-840b-868ecfd035c8","Type":"ContainerStarted","Data":"f49fa1db47469dedc91e072bed66b95237d4dda27acfbf251f6e15f2bfd2be96"} Jul 06 22:24:05.711304 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:05.711251 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-gpbjg" podStartSLOduration=1.460019667 podStartE2EDuration="4.711233998s" totalImagesPullingTime="3.251214331s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:24:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:24:05.709788349 +0000 UTC m=+252.628031345" watchObservedRunningTime="2026-07-06 22:24:05.711233998 +0000 UTC m=+252.629476856" Jul 06 22:24:07.301905 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:07.301868 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:07.402436 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:07.402401 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:08.908112 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:08.908081 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:10.584357 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:10.584310 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:10.681437 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:10.681400 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:16.685476 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:16.685386 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:16.777630 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:16.777595 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:18.612862 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:18.612819 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:19.193197 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:19.193158 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:22.842978 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:22.842932 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:22.914364 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:22.914321 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:22.921633 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:22.921606 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-67b4867ff6-mpbh8"] Jul 06 22:24:22.968346 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:22.968311 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-67b4867ff6-mpbh8"] Jul 06 22:24:22.968510 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:22.968394 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-67b4867ff6-mpbh8" Jul 06 22:24:22.973456 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:22.973435 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 06 22:24:22.975934 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:22.975913 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 06 22:24:22.976053 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:22.975934 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 06 22:24:22.986907 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:22.986880 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-8vjvn\"" Jul 06 22:24:22.987028 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:22.986889 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 06 22:24:22.987137 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:22.987124 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 06 22:24:23.072769 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:23.072726 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/278c0f33-7076-4cb1-9801-cc976024cd5e-manager-config\") pod \"lws-controller-manager-67b4867ff6-mpbh8\" (UID: \"278c0f33-7076-4cb1-9801-cc976024cd5e\") " pod="openshift-lws-operator/lws-controller-manager-67b4867ff6-mpbh8" Jul 06 22:24:23.072987 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:23.072789 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/278c0f33-7076-4cb1-9801-cc976024cd5e-cert\") pod \"lws-controller-manager-67b4867ff6-mpbh8\" (UID: \"278c0f33-7076-4cb1-9801-cc976024cd5e\") " pod="openshift-lws-operator/lws-controller-manager-67b4867ff6-mpbh8" Jul 06 22:24:23.072987 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:23.072905 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6shps\" (UniqueName: \"kubernetes.io/projected/278c0f33-7076-4cb1-9801-cc976024cd5e-kube-api-access-6shps\") pod \"lws-controller-manager-67b4867ff6-mpbh8\" (UID: \"278c0f33-7076-4cb1-9801-cc976024cd5e\") " pod="openshift-lws-operator/lws-controller-manager-67b4867ff6-mpbh8" Jul 06 22:24:23.072987 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:23.072967 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/278c0f33-7076-4cb1-9801-cc976024cd5e-metrics-cert\") pod \"lws-controller-manager-67b4867ff6-mpbh8\" (UID: \"278c0f33-7076-4cb1-9801-cc976024cd5e\") " pod="openshift-lws-operator/lws-controller-manager-67b4867ff6-mpbh8" Jul 06 22:24:23.173720 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:23.173674 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6shps\" (UniqueName: \"kubernetes.io/projected/278c0f33-7076-4cb1-9801-cc976024cd5e-kube-api-access-6shps\") pod \"lws-controller-manager-67b4867ff6-mpbh8\" (UID: \"278c0f33-7076-4cb1-9801-cc976024cd5e\") " pod="openshift-lws-operator/lws-controller-manager-67b4867ff6-mpbh8" Jul 06 22:24:23.173935 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:23.173731 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/278c0f33-7076-4cb1-9801-cc976024cd5e-metrics-cert\") pod \"lws-controller-manager-67b4867ff6-mpbh8\" (UID: \"278c0f33-7076-4cb1-9801-cc976024cd5e\") " pod="openshift-lws-operator/lws-controller-manager-67b4867ff6-mpbh8" Jul 06 22:24:23.173935 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:23.173764 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/278c0f33-7076-4cb1-9801-cc976024cd5e-manager-config\") pod \"lws-controller-manager-67b4867ff6-mpbh8\" (UID: \"278c0f33-7076-4cb1-9801-cc976024cd5e\") " pod="openshift-lws-operator/lws-controller-manager-67b4867ff6-mpbh8" Jul 06 22:24:23.173935 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:23.173795 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/278c0f33-7076-4cb1-9801-cc976024cd5e-cert\") pod \"lws-controller-manager-67b4867ff6-mpbh8\" (UID: \"278c0f33-7076-4cb1-9801-cc976024cd5e\") " pod="openshift-lws-operator/lws-controller-manager-67b4867ff6-mpbh8" Jul 06 22:24:23.174453 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:23.174426 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/278c0f33-7076-4cb1-9801-cc976024cd5e-manager-config\") pod \"lws-controller-manager-67b4867ff6-mpbh8\" (UID: \"278c0f33-7076-4cb1-9801-cc976024cd5e\") " pod="openshift-lws-operator/lws-controller-manager-67b4867ff6-mpbh8" Jul 06 22:24:23.176349 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:23.176328 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/278c0f33-7076-4cb1-9801-cc976024cd5e-metrics-cert\") pod \"lws-controller-manager-67b4867ff6-mpbh8\" (UID: \"278c0f33-7076-4cb1-9801-cc976024cd5e\") " pod="openshift-lws-operator/lws-controller-manager-67b4867ff6-mpbh8" Jul 06 22:24:23.176521 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:23.176499 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/278c0f33-7076-4cb1-9801-cc976024cd5e-cert\") pod \"lws-controller-manager-67b4867ff6-mpbh8\" (UID: \"278c0f33-7076-4cb1-9801-cc976024cd5e\") " pod="openshift-lws-operator/lws-controller-manager-67b4867ff6-mpbh8" Jul 06 22:24:23.184100 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:23.184066 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6shps\" (UniqueName: \"kubernetes.io/projected/278c0f33-7076-4cb1-9801-cc976024cd5e-kube-api-access-6shps\") pod \"lws-controller-manager-67b4867ff6-mpbh8\" (UID: \"278c0f33-7076-4cb1-9801-cc976024cd5e\") " pod="openshift-lws-operator/lws-controller-manager-67b4867ff6-mpbh8" Jul 06 22:24:23.278116 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:23.278078 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-67b4867ff6-mpbh8" Jul 06 22:24:23.419576 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:23.419547 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-67b4867ff6-mpbh8"] Jul 06 22:24:23.422786 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:24:23.422764 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod278c0f33_7076_4cb1_9801_cc976024cd5e.slice/crio-faa0196e81f8c8592eb667795efcd0124efef1397c7cdfada657741666527619 WatchSource:0}: Error finding container faa0196e81f8c8592eb667795efcd0124efef1397c7cdfada657741666527619: Status 404 returned error can't find the container with id faa0196e81f8c8592eb667795efcd0124efef1397c7cdfada657741666527619 Jul 06 22:24:23.758626 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:23.758516 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-67b4867ff6-mpbh8" event={"ID":"278c0f33-7076-4cb1-9801-cc976024cd5e","Type":"ContainerStarted","Data":"faa0196e81f8c8592eb667795efcd0124efef1397c7cdfada657741666527619"} Jul 06 22:24:26.772731 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:26.772696 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-67b4867ff6-mpbh8" event={"ID":"278c0f33-7076-4cb1-9801-cc976024cd5e","Type":"ContainerStarted","Data":"683afcd1b2f570dd7cfe1c2fd153618ae62bf65f4e65c943ab1554020f6def2c"} Jul 06 22:24:26.773157 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:26.772812 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-67b4867ff6-mpbh8" Jul 06 22:24:26.807585 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:26.804361 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-67b4867ff6-mpbh8" podStartSLOduration=2.45731237 podStartE2EDuration="4.804340208s" totalImagesPullingTime="2.347027838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:24:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:24:26.80211091 +0000 UTC m=+273.720353767" watchObservedRunningTime="2026-07-06 22:24:26.804340208 +0000 UTC m=+273.722583064" Jul 06 22:24:34.512878 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:34.512840 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:34.610558 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:34.610527 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:35.312444 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:35.312410 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:35.482486 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:35.482448 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:35.587406 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:35.587320 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:35.697956 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:35.697921 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-c6597d756-dmckl"] Jul 06 22:24:35.702058 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:35.702031 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-c6597d756-dmckl" Jul 06 22:24:35.706117 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:35.706095 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-th8c4\"" Jul 06 22:24:35.706425 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:35.706409 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 06 22:24:35.706622 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:35.706609 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 06 22:24:35.706680 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:35.706635 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 06 22:24:35.707595 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:35.707567 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 06 22:24:35.726838 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:35.726807 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-c6597d756-dmckl"] Jul 06 22:24:35.795953 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:35.795917 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:35.886315 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:35.886283 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkcb9\" (UniqueName: \"kubernetes.io/projected/c180e6ba-1217-4cf6-8e95-bbbf6cc98170-kube-api-access-lkcb9\") pod \"opendatahub-operator-controller-manager-c6597d756-dmckl\" (UID: \"c180e6ba-1217-4cf6-8e95-bbbf6cc98170\") " pod="opendatahub/opendatahub-operator-controller-manager-c6597d756-dmckl" Jul 06 22:24:35.886512 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:35.886339 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c180e6ba-1217-4cf6-8e95-bbbf6cc98170-webhook-cert\") pod \"opendatahub-operator-controller-manager-c6597d756-dmckl\" (UID: \"c180e6ba-1217-4cf6-8e95-bbbf6cc98170\") " pod="opendatahub/opendatahub-operator-controller-manager-c6597d756-dmckl" Jul 06 22:24:35.886512 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:35.886451 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c180e6ba-1217-4cf6-8e95-bbbf6cc98170-tmp\") pod \"opendatahub-operator-controller-manager-c6597d756-dmckl\" (UID: \"c180e6ba-1217-4cf6-8e95-bbbf6cc98170\") " pod="opendatahub/opendatahub-operator-controller-manager-c6597d756-dmckl" Jul 06 22:24:35.886629 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:35.886508 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c180e6ba-1217-4cf6-8e95-bbbf6cc98170-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c6597d756-dmckl\" (UID: \"c180e6ba-1217-4cf6-8e95-bbbf6cc98170\") " pod="opendatahub/opendatahub-operator-controller-manager-c6597d756-dmckl" Jul 06 22:24:35.886629 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:35.886547 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c180e6ba-1217-4cf6-8e95-bbbf6cc98170-manifests\") pod \"opendatahub-operator-controller-manager-c6597d756-dmckl\" (UID: \"c180e6ba-1217-4cf6-8e95-bbbf6cc98170\") " pod="opendatahub/opendatahub-operator-controller-manager-c6597d756-dmckl" Jul 06 22:24:35.897054 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:35.897015 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:35.987332 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:35.987290 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c180e6ba-1217-4cf6-8e95-bbbf6cc98170-tmp\") pod \"opendatahub-operator-controller-manager-c6597d756-dmckl\" (UID: \"c180e6ba-1217-4cf6-8e95-bbbf6cc98170\") " pod="opendatahub/opendatahub-operator-controller-manager-c6597d756-dmckl" Jul 06 22:24:35.987506 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:35.987341 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c180e6ba-1217-4cf6-8e95-bbbf6cc98170-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c6597d756-dmckl\" (UID: \"c180e6ba-1217-4cf6-8e95-bbbf6cc98170\") " pod="opendatahub/opendatahub-operator-controller-manager-c6597d756-dmckl" Jul 06 22:24:35.987506 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:35.987367 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c180e6ba-1217-4cf6-8e95-bbbf6cc98170-manifests\") pod \"opendatahub-operator-controller-manager-c6597d756-dmckl\" (UID: \"c180e6ba-1217-4cf6-8e95-bbbf6cc98170\") " pod="opendatahub/opendatahub-operator-controller-manager-c6597d756-dmckl" Jul 06 22:24:35.987506 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:35.987391 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkcb9\" (UniqueName: \"kubernetes.io/projected/c180e6ba-1217-4cf6-8e95-bbbf6cc98170-kube-api-access-lkcb9\") pod \"opendatahub-operator-controller-manager-c6597d756-dmckl\" (UID: \"c180e6ba-1217-4cf6-8e95-bbbf6cc98170\") " pod="opendatahub/opendatahub-operator-controller-manager-c6597d756-dmckl" Jul 06 22:24:35.987951 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:35.987552 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c180e6ba-1217-4cf6-8e95-bbbf6cc98170-webhook-cert\") pod \"opendatahub-operator-controller-manager-c6597d756-dmckl\" (UID: \"c180e6ba-1217-4cf6-8e95-bbbf6cc98170\") " pod="opendatahub/opendatahub-operator-controller-manager-c6597d756-dmckl" Jul 06 22:24:35.987951 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:35.987815 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c180e6ba-1217-4cf6-8e95-bbbf6cc98170-manifests\") pod \"opendatahub-operator-controller-manager-c6597d756-dmckl\" (UID: \"c180e6ba-1217-4cf6-8e95-bbbf6cc98170\") " pod="opendatahub/opendatahub-operator-controller-manager-c6597d756-dmckl" Jul 06 22:24:35.989670 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:35.989637 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c180e6ba-1217-4cf6-8e95-bbbf6cc98170-tmp\") pod \"opendatahub-operator-controller-manager-c6597d756-dmckl\" (UID: \"c180e6ba-1217-4cf6-8e95-bbbf6cc98170\") " pod="opendatahub/opendatahub-operator-controller-manager-c6597d756-dmckl" Jul 06 22:24:35.989842 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:35.989821 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c180e6ba-1217-4cf6-8e95-bbbf6cc98170-webhook-cert\") pod \"opendatahub-operator-controller-manager-c6597d756-dmckl\" (UID: \"c180e6ba-1217-4cf6-8e95-bbbf6cc98170\") " pod="opendatahub/opendatahub-operator-controller-manager-c6597d756-dmckl" Jul 06 22:24:35.989967 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:35.989948 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c180e6ba-1217-4cf6-8e95-bbbf6cc98170-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c6597d756-dmckl\" (UID: \"c180e6ba-1217-4cf6-8e95-bbbf6cc98170\") " pod="opendatahub/opendatahub-operator-controller-manager-c6597d756-dmckl" Jul 06 22:24:35.997649 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:35.997612 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkcb9\" (UniqueName: \"kubernetes.io/projected/c180e6ba-1217-4cf6-8e95-bbbf6cc98170-kube-api-access-lkcb9\") pod \"opendatahub-operator-controller-manager-c6597d756-dmckl\" (UID: \"c180e6ba-1217-4cf6-8e95-bbbf6cc98170\") " pod="opendatahub/opendatahub-operator-controller-manager-c6597d756-dmckl" Jul 06 22:24:35.998116 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:35.998084 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:36.011977 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:36.011936 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-c6597d756-dmckl" Jul 06 22:24:36.097915 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:36.097875 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:36.157311 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:36.157225 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-c6597d756-dmckl"] Jul 06 22:24:36.160695 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:24:36.160663 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc180e6ba_1217_4cf6_8e95_bbbf6cc98170.slice/crio-3c12c9f2235d09f8c99281bc7ed2a670e9b4eb670fef6a42b0014c08f0db9545 WatchSource:0}: Error finding container 3c12c9f2235d09f8c99281bc7ed2a670e9b4eb670fef6a42b0014c08f0db9545: Status 404 returned error can't find the container with id 3c12c9f2235d09f8c99281bc7ed2a670e9b4eb670fef6a42b0014c08f0db9545 Jul 06 22:24:36.160991 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:36.160964 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:36.234569 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:36.234536 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:36.336749 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:36.336702 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:36.815357 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:36.815313 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c6597d756-dmckl" event={"ID":"c180e6ba-1217-4cf6-8e95-bbbf6cc98170","Type":"ContainerStarted","Data":"3c12c9f2235d09f8c99281bc7ed2a670e9b4eb670fef6a42b0014c08f0db9545"} Jul 06 22:24:37.779591 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:37.779561 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-67b4867ff6-mpbh8" Jul 06 22:24:39.831731 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:39.831690 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c6597d756-dmckl" event={"ID":"c180e6ba-1217-4cf6-8e95-bbbf6cc98170","Type":"ContainerStarted","Data":"ea3524a0d9623a02e5ea1b0037fd0b696587f21eb6f8290c00d0db8b9990378b"} Jul 06 22:24:40.837024 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:40.836984 2571 generic.go:356] "Generic (PLEG): container finished" podID="c180e6ba-1217-4cf6-8e95-bbbf6cc98170" containerID="ea3524a0d9623a02e5ea1b0037fd0b696587f21eb6f8290c00d0db8b9990378b" exitCode=0 Jul 06 22:24:40.837510 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:40.837069 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c6597d756-dmckl" event={"ID":"c180e6ba-1217-4cf6-8e95-bbbf6cc98170","Type":"ContainerDied","Data":"ea3524a0d9623a02e5ea1b0037fd0b696587f21eb6f8290c00d0db8b9990378b"} Jul 06 22:24:41.843450 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:41.843407 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c6597d756-dmckl" event={"ID":"c180e6ba-1217-4cf6-8e95-bbbf6cc98170","Type":"ContainerStarted","Data":"06fbf69e082af0ab7f4dd43f3cc182a2dda75a7a60cde58090f26d050fe7b582"} Jul 06 22:24:41.843837 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:41.843545 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-c6597d756-dmckl" Jul 06 22:24:41.867478 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:41.867400 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-c6597d756-dmckl" podStartSLOduration=2.538451703 podStartE2EDuration="6.867373601s" totalImagesPullingTime="3.231933681s" totalInitContainerRuntime="1.096988217s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:24:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:24:41.865009286 +0000 UTC m=+288.783252142" watchObservedRunningTime="2026-07-06 22:24:41.867373601 +0000 UTC m=+288.785616460" Jul 06 22:24:47.525466 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:47.525423 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:47.640949 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:47.640916 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:52.850099 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:52.850063 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-c6597d756-dmckl" Jul 06 22:24:53.497080 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:53.497052 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-4x6pf_5796a8ea-adcb-4819-bb18-0b85e1ef4681/console-operator/2.log" Jul 06 22:24:53.500604 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:53.500562 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-4x6pf_5796a8ea-adcb-4819-bb18-0b85e1ef4681/console-operator/2.log" Jul 06 22:24:53.501570 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:53.501545 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qbbrs_bf23ef00-a2d0-499c-8c87-5d6b98510ea0/ovn-acl-logging/0.log" Jul 06 22:24:53.505291 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:53.505263 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qbbrs_bf23ef00-a2d0-499c-8c87-5d6b98510ea0/ovn-acl-logging/0.log" Jul 06 22:24:53.508527 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:53.508506 2571 kubelet.go:1648] "Image garbage collection succeeded" Jul 06 22:24:54.748755 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:54.748714 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:54.847394 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:54.847357 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:55.288570 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:55.288524 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:55.384847 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:55.384808 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7588b845d6-brn5g"] Jul 06 22:24:55.398505 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:55.398475 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7588b845d6-brn5g" Jul 06 22:24:55.400978 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:55.400947 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7588b845d6-brn5g"] Jul 06 22:24:55.401448 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:55.401423 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 06 22:24:55.401598 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:55.401511 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 06 22:24:55.401791 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:55.401774 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-gbm95\"" Jul 06 22:24:55.401864 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:55.401777 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 06 22:24:55.402262 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:55.402244 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 06 22:24:55.452393 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:55.452357 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a00e36ef-1dc2-403d-9cca-4298c2eb5561-tmp\") pod \"kube-auth-proxy-7588b845d6-brn5g\" (UID: \"a00e36ef-1dc2-403d-9cca-4298c2eb5561\") " pod="openshift-ingress/kube-auth-proxy-7588b845d6-brn5g" Jul 06 22:24:55.452393 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:55.452393 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j4t7\" (UniqueName: \"kubernetes.io/projected/a00e36ef-1dc2-403d-9cca-4298c2eb5561-kube-api-access-5j4t7\") pod \"kube-auth-proxy-7588b845d6-brn5g\" (UID: \"a00e36ef-1dc2-403d-9cca-4298c2eb5561\") " pod="openshift-ingress/kube-auth-proxy-7588b845d6-brn5g" Jul 06 22:24:55.452606 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:55.452414 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a00e36ef-1dc2-403d-9cca-4298c2eb5561-tls-certs\") pod \"kube-auth-proxy-7588b845d6-brn5g\" (UID: \"a00e36ef-1dc2-403d-9cca-4298c2eb5561\") " pod="openshift-ingress/kube-auth-proxy-7588b845d6-brn5g" Jul 06 22:24:55.492414 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:55.492381 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:55.553160 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:55.553070 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a00e36ef-1dc2-403d-9cca-4298c2eb5561-tmp\") pod \"kube-auth-proxy-7588b845d6-brn5g\" (UID: \"a00e36ef-1dc2-403d-9cca-4298c2eb5561\") " pod="openshift-ingress/kube-auth-proxy-7588b845d6-brn5g" Jul 06 22:24:55.553160 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:55.553114 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j4t7\" (UniqueName: \"kubernetes.io/projected/a00e36ef-1dc2-403d-9cca-4298c2eb5561-kube-api-access-5j4t7\") pod \"kube-auth-proxy-7588b845d6-brn5g\" (UID: \"a00e36ef-1dc2-403d-9cca-4298c2eb5561\") " pod="openshift-ingress/kube-auth-proxy-7588b845d6-brn5g" Jul 06 22:24:55.553160 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:55.553142 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a00e36ef-1dc2-403d-9cca-4298c2eb5561-tls-certs\") pod \"kube-auth-proxy-7588b845d6-brn5g\" (UID: \"a00e36ef-1dc2-403d-9cca-4298c2eb5561\") " pod="openshift-ingress/kube-auth-proxy-7588b845d6-brn5g" Jul 06 22:24:55.555337 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:55.555305 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a00e36ef-1dc2-403d-9cca-4298c2eb5561-tmp\") pod \"kube-auth-proxy-7588b845d6-brn5g\" (UID: \"a00e36ef-1dc2-403d-9cca-4298c2eb5561\") " pod="openshift-ingress/kube-auth-proxy-7588b845d6-brn5g" Jul 06 22:24:55.555646 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:55.555623 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a00e36ef-1dc2-403d-9cca-4298c2eb5561-tls-certs\") pod \"kube-auth-proxy-7588b845d6-brn5g\" (UID: \"a00e36ef-1dc2-403d-9cca-4298c2eb5561\") " pod="openshift-ingress/kube-auth-proxy-7588b845d6-brn5g" Jul 06 22:24:55.556724 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:55.556646 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:55.564744 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:55.564721 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j4t7\" (UniqueName: \"kubernetes.io/projected/a00e36ef-1dc2-403d-9cca-4298c2eb5561-kube-api-access-5j4t7\") pod \"kube-auth-proxy-7588b845d6-brn5g\" (UID: \"a00e36ef-1dc2-403d-9cca-4298c2eb5561\") " pod="openshift-ingress/kube-auth-proxy-7588b845d6-brn5g" Jul 06 22:24:55.655514 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:55.655479 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:55.710152 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:55.710120 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7588b845d6-brn5g" Jul 06 22:24:55.757691 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:55.757652 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:55.910490 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:55.910465 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7588b845d6-brn5g"] Jul 06 22:24:55.913350 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:24:55.913318 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda00e36ef_1dc2_403d_9cca_4298c2eb5561.slice/crio-277f5a37aaface064dfe5c2e443ecdd0a69050fc120f43150ee850d0f83da0a8 WatchSource:0}: Error finding container 277f5a37aaface064dfe5c2e443ecdd0a69050fc120f43150ee850d0f83da0a8: Status 404 returned error can't find the container with id 277f5a37aaface064dfe5c2e443ecdd0a69050fc120f43150ee850d0f83da0a8 Jul 06 22:24:55.915525 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:55.915505 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 22:24:56.284529 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:56.284495 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:56.447982 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:56.447946 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:56.573729 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:56.573618 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:56.673147 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:56.673096 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:56.861444 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:56.861353 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:56.907764 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:56.907731 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7588b845d6-brn5g" event={"ID":"a00e36ef-1dc2-403d-9cca-4298c2eb5561","Type":"ContainerStarted","Data":"277f5a37aaface064dfe5c2e443ecdd0a69050fc120f43150ee850d0f83da0a8"} Jul 06 22:24:57.587896 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:57.587852 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:58.917602 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:58.917551 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7588b845d6-brn5g" event={"ID":"a00e36ef-1dc2-403d-9cca-4298c2eb5561","Type":"ContainerStarted","Data":"e22d52764e733e8a761efb2f931702254eca30861600c955d7764bc18c829396"} Jul 06 22:24:58.936131 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:24:58.936074 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7588b845d6-brn5g" podStartSLOduration=1.142166775 podStartE2EDuration="3.936057394s" totalImagesPullingTime="2.793890619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:24:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:24:58.934738126 +0000 UTC m=+305.852980981" watchObservedRunningTime="2026-07-06 22:24:58.936057394 +0000 UTC m=+305.854300249" Jul 06 22:25:02.879786 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:25:02.879742 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:04.312390 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:25:04.312355 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:07.060408 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:25:07.060372 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:07.129238 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:25:07.129181 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:07.340012 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:25:07.339925 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:07.516179 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:25:07.516139 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:07.596864 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:25:07.596773 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:07.766759 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:25:07.766725 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:07.954382 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:25:07.954330 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:10.291731 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:25:10.291685 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:10.402831 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:25:10.402798 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:12.511511 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:25:12.511467 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:12.581925 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:25:12.581875 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:56.142622 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:25:56.142527 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:56.221343 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:25:56.221301 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:56.378091 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:25:56.378057 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:57.075150 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:25:57.075107 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:57.284283 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:25:57.284245 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:26:05.848383 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:05.848348 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:26:25.058292 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:25.058254 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:26:25.858366 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:25.858316 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:26:30.471288 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:30.471250 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:26:32.535288 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:32.535241 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:26:32.630956 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:32.630925 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:26:32.863507 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:32.863412 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:26:35.104030 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:35.103989 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:26:36.893875 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:36.893839 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:26:36.963593 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:36.963555 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-djhsp"] Jul 06 22:26:36.966740 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:36.966720 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-djhsp" Jul 06 22:26:36.969355 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:36.969336 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-kpqhd\"" Jul 06 22:26:36.969597 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:36.969584 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 06 22:26:36.969650 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:36.969619 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 06 22:26:36.979877 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:36.979852 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-djhsp"] Jul 06 22:26:37.017474 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:37.017434 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf4ch\" (UniqueName: \"kubernetes.io/projected/c201181f-7595-4f82-b801-f0afe3da76f7-kube-api-access-gf4ch\") pod \"kuadrant-operator-controller-manager-8684c9ff49-djhsp\" (UID: \"c201181f-7595-4f82-b801-f0afe3da76f7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-djhsp" Jul 06 22:26:37.017653 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:37.017512 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c201181f-7595-4f82-b801-f0afe3da76f7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-djhsp\" (UID: \"c201181f-7595-4f82-b801-f0afe3da76f7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-djhsp" Jul 06 22:26:37.118414 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:37.118373 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf4ch\" (UniqueName: \"kubernetes.io/projected/c201181f-7595-4f82-b801-f0afe3da76f7-kube-api-access-gf4ch\") pod \"kuadrant-operator-controller-manager-8684c9ff49-djhsp\" (UID: \"c201181f-7595-4f82-b801-f0afe3da76f7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-djhsp" Jul 06 22:26:37.118621 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:37.118474 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c201181f-7595-4f82-b801-f0afe3da76f7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-djhsp\" (UID: \"c201181f-7595-4f82-b801-f0afe3da76f7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-djhsp" Jul 06 22:26:37.118854 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:37.118833 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c201181f-7595-4f82-b801-f0afe3da76f7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-djhsp\" (UID: \"c201181f-7595-4f82-b801-f0afe3da76f7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-djhsp" Jul 06 22:26:37.128623 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:37.128600 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf4ch\" (UniqueName: \"kubernetes.io/projected/c201181f-7595-4f82-b801-f0afe3da76f7-kube-api-access-gf4ch\") pod \"kuadrant-operator-controller-manager-8684c9ff49-djhsp\" (UID: \"c201181f-7595-4f82-b801-f0afe3da76f7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-djhsp" Jul 06 22:26:37.276687 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:37.276587 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-djhsp" Jul 06 22:26:37.420155 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:37.420107 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-djhsp"] Jul 06 22:26:37.423222 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:26:37.423181 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc201181f_7595_4f82_b801_f0afe3da76f7.slice/crio-40833f5d25196416366300bb4fc343a6a6868d2d9e66f49e5228d62444a024cd WatchSource:0}: Error finding container 40833f5d25196416366300bb4fc343a6a6868d2d9e66f49e5228d62444a024cd: Status 404 returned error can't find the container with id 40833f5d25196416366300bb4fc343a6a6868d2d9e66f49e5228d62444a024cd Jul 06 22:26:38.073438 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:38.073395 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:26:38.343674 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:38.343595 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-djhsp" event={"ID":"c201181f-7595-4f82-b801-f0afe3da76f7","Type":"ContainerStarted","Data":"40833f5d25196416366300bb4fc343a6a6868d2d9e66f49e5228d62444a024cd"} Jul 06 22:26:39.901705 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:39.901661 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:26:41.463075 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:41.463032 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:26:41.521071 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:41.521008 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:26:41.868056 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:41.867964 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:26:41.928476 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:41.928443 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:26:44.376949 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:44.376911 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-djhsp" event={"ID":"c201181f-7595-4f82-b801-f0afe3da76f7","Type":"ContainerStarted","Data":"b40e8202a7983497edcb7e593a810e7fdb1fb9f5b0374f42e93ba9a72117b339"} Jul 06 22:26:44.377461 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:44.377049 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-djhsp" Jul 06 22:26:44.400387 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:44.400337 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-djhsp" podStartSLOduration=1.673176893 podStartE2EDuration="8.40032113s" totalImagesPullingTime="6.727144237s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:26:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:26:44.398780198 +0000 UTC m=+411.317023056" watchObservedRunningTime="2026-07-06 22:26:44.40032113 +0000 UTC m=+411.318563986" Jul 06 22:26:45.463928 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:45.463888 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:26:45.565282 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:45.565244 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:26:47.024389 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:47.024351 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:26:47.309551 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:47.309423 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:26:47.958786 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:47.958755 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:26:48.863765 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:48.863731 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:26:48.964505 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:48.964469 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:26:53.602010 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:53.601976 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:26:53.664733 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:53.664696 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:26:54.584946 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:54.584910 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:26:54.693027 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:54.692986 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:26:55.383740 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:55.383709 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-djhsp" Jul 06 22:26:57.344928 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:57.344896 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-djhsp"] Jul 06 22:26:57.345366 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:57.345155 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-djhsp" podUID="c201181f-7595-4f82-b801-f0afe3da76f7" containerName="manager" containerID="cri-o://b40e8202a7983497edcb7e593a810e7fdb1fb9f5b0374f42e93ba9a72117b339" gracePeriod=2 Jul 06 22:26:57.350882 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:57.350765 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-djhsp"] Jul 06 22:26:57.381447 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:57.381399 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wzzbn"] Jul 06 22:26:57.381839 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:57.381816 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c201181f-7595-4f82-b801-f0afe3da76f7" containerName="manager" Jul 06 22:26:57.381839 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:57.381829 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c201181f-7595-4f82-b801-f0afe3da76f7" containerName="manager" Jul 06 22:26:57.382031 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:57.381912 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="c201181f-7595-4f82-b801-f0afe3da76f7" containerName="manager" Jul 06 22:26:57.384876 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:57.384855 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wzzbn" Jul 06 22:26:57.408799 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:57.408568 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wzzbn"] Jul 06 22:26:57.428557 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:26:57.428520 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-djhsp\" is forbidden: User \"system:node:ip-10-0-137-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-147.ec2.internal' and this object" podUID="c201181f-7595-4f82-b801-f0afe3da76f7" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-djhsp" Jul 06 22:26:57.480819 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:57.480786 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7f013012-0f5b-46d3-bf3c-37ceb1546ff9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wzzbn\" (UID: \"7f013012-0f5b-46d3-bf3c-37ceb1546ff9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wzzbn" Jul 06 22:26:57.480985 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:57.480861 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz9pg\" (UniqueName: \"kubernetes.io/projected/7f013012-0f5b-46d3-bf3c-37ceb1546ff9-kube-api-access-zz9pg\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wzzbn\" (UID: \"7f013012-0f5b-46d3-bf3c-37ceb1546ff9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wzzbn" Jul 06 22:26:57.581375 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:57.581339 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7f013012-0f5b-46d3-bf3c-37ceb1546ff9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wzzbn\" (UID: \"7f013012-0f5b-46d3-bf3c-37ceb1546ff9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wzzbn" Jul 06 22:26:57.581521 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:57.581441 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz9pg\" (UniqueName: \"kubernetes.io/projected/7f013012-0f5b-46d3-bf3c-37ceb1546ff9-kube-api-access-zz9pg\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wzzbn\" (UID: \"7f013012-0f5b-46d3-bf3c-37ceb1546ff9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wzzbn" Jul 06 22:26:57.581835 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:57.581801 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7f013012-0f5b-46d3-bf3c-37ceb1546ff9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wzzbn\" (UID: \"7f013012-0f5b-46d3-bf3c-37ceb1546ff9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wzzbn" Jul 06 22:26:57.589249 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:57.589226 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-djhsp" Jul 06 22:26:57.591530 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:26:57.591496 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-djhsp\" is forbidden: User \"system:node:ip-10-0-137-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-147.ec2.internal' and this object" podUID="c201181f-7595-4f82-b801-f0afe3da76f7" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-djhsp" Jul 06 22:26:57.591655 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:57.591575 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz9pg\" (UniqueName: \"kubernetes.io/projected/7f013012-0f5b-46d3-bf3c-37ceb1546ff9-kube-api-access-zz9pg\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wzzbn\" (UID: \"7f013012-0f5b-46d3-bf3c-37ceb1546ff9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wzzbn" Jul 06 22:26:57.682170 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:57.682135 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c201181f-7595-4f82-b801-f0afe3da76f7-extensions-socket-volume\") pod \"c201181f-7595-4f82-b801-f0afe3da76f7\" (UID: \"c201181f-7595-4f82-b801-f0afe3da76f7\") " Jul 06 22:26:57.682382 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:57.682234 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gf4ch\" (UniqueName: \"kubernetes.io/projected/c201181f-7595-4f82-b801-f0afe3da76f7-kube-api-access-gf4ch\") pod \"c201181f-7595-4f82-b801-f0afe3da76f7\" (UID: \"c201181f-7595-4f82-b801-f0afe3da76f7\") " Jul 06 22:26:57.682598 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:57.682566 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c201181f-7595-4f82-b801-f0afe3da76f7-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "c201181f-7595-4f82-b801-f0afe3da76f7" (UID: "c201181f-7595-4f82-b801-f0afe3da76f7"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:26:57.684469 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:57.684446 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c201181f-7595-4f82-b801-f0afe3da76f7-kube-api-access-gf4ch" (OuterVolumeSpecName: "kube-api-access-gf4ch") pod "c201181f-7595-4f82-b801-f0afe3da76f7" (UID: "c201181f-7595-4f82-b801-f0afe3da76f7"). InnerVolumeSpecName "kube-api-access-gf4ch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:26:57.712803 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:57.712773 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wzzbn" Jul 06 22:26:57.783462 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:57.783424 2571 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c201181f-7595-4f82-b801-f0afe3da76f7-extensions-socket-volume\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:26:57.783462 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:57.783461 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf4ch\" (UniqueName: \"kubernetes.io/projected/c201181f-7595-4f82-b801-f0afe3da76f7-kube-api-access-gf4ch\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:26:57.858304 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:57.858277 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wzzbn"] Jul 06 22:26:57.860648 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:26:57.860623 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f013012_0f5b_46d3_bf3c_37ceb1546ff9.slice/crio-deac02ff9fbfbb97c2beb6ec35566f55d3a3dae7c8a636704e0ae1bce800d591 WatchSource:0}: Error finding container deac02ff9fbfbb97c2beb6ec35566f55d3a3dae7c8a636704e0ae1bce800d591: Status 404 returned error can't find the container with id deac02ff9fbfbb97c2beb6ec35566f55d3a3dae7c8a636704e0ae1bce800d591 Jul 06 22:26:58.442278 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:58.442237 2571 generic.go:356] "Generic (PLEG): container finished" podID="c201181f-7595-4f82-b801-f0afe3da76f7" containerID="b40e8202a7983497edcb7e593a810e7fdb1fb9f5b0374f42e93ba9a72117b339" exitCode=0 Jul 06 22:26:58.442761 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:58.442304 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-djhsp" Jul 06 22:26:58.442761 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:58.442340 2571 scope.go:117] "RemoveContainer" containerID="b40e8202a7983497edcb7e593a810e7fdb1fb9f5b0374f42e93ba9a72117b339" Jul 06 22:26:58.444262 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:58.444237 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wzzbn" event={"ID":"7f013012-0f5b-46d3-bf3c-37ceb1546ff9","Type":"ContainerStarted","Data":"6c9aae35ee2a0a6419da4bc9a1cb2361f9d9314d146221016e1aab71f306cefe"} Jul 06 22:26:58.444364 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:58.444295 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wzzbn" event={"ID":"7f013012-0f5b-46d3-bf3c-37ceb1546ff9","Type":"ContainerStarted","Data":"deac02ff9fbfbb97c2beb6ec35566f55d3a3dae7c8a636704e0ae1bce800d591"} Jul 06 22:26:58.444435 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:58.444378 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wzzbn" Jul 06 22:26:58.451959 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:58.451938 2571 scope.go:117] "RemoveContainer" containerID="b40e8202a7983497edcb7e593a810e7fdb1fb9f5b0374f42e93ba9a72117b339" Jul 06 22:26:58.452249 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:26:58.452229 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b40e8202a7983497edcb7e593a810e7fdb1fb9f5b0374f42e93ba9a72117b339\": container with ID starting with b40e8202a7983497edcb7e593a810e7fdb1fb9f5b0374f42e93ba9a72117b339 not found: ID does not exist" containerID="b40e8202a7983497edcb7e593a810e7fdb1fb9f5b0374f42e93ba9a72117b339" Jul 06 22:26:58.452312 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:58.452258 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b40e8202a7983497edcb7e593a810e7fdb1fb9f5b0374f42e93ba9a72117b339"} err="failed to get container status \"b40e8202a7983497edcb7e593a810e7fdb1fb9f5b0374f42e93ba9a72117b339\": rpc error: code = NotFound desc = could not find container \"b40e8202a7983497edcb7e593a810e7fdb1fb9f5b0374f42e93ba9a72117b339\": container with ID starting with b40e8202a7983497edcb7e593a810e7fdb1fb9f5b0374f42e93ba9a72117b339 not found: ID does not exist" Jul 06 22:26:58.474722 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:58.474656 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wzzbn" podStartSLOduration=1.474639156 podStartE2EDuration="1.474639156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:26:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:26:58.472435762 +0000 UTC m=+425.390678629" watchObservedRunningTime="2026-07-06 22:26:58.474639156 +0000 UTC m=+425.392882012" Jul 06 22:26:58.941309 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:58.941279 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:26:59.051337 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:59.051297 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:26:59.146808 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:59.146777 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:26:59.633769 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:26:59.633738 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c201181f-7595-4f82-b801-f0afe3da76f7" path="/var/lib/kubelet/pods/c201181f-7595-4f82-b801-f0afe3da76f7/volumes" Jul 06 22:27:00.085271 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:00.085186 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:27:00.185792 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:00.185762 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:27:00.481860 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:00.481829 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:27:00.581962 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:00.581928 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:27:04.078601 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:04.078564 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:27:04.178161 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:04.178124 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:27:07.913033 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:07.912992 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:27:08.006340 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:08.006302 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:27:09.452351 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:09.452320 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wzzbn" Jul 06 22:27:14.556139 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:14.556101 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:27:14.657184 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:14.657152 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:27:28.383663 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:28.383585 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wzzbn"] Jul 06 22:27:28.384082 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:28.383867 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wzzbn" podUID="7f013012-0f5b-46d3-bf3c-37ceb1546ff9" containerName="manager" containerID="cri-o://6c9aae35ee2a0a6419da4bc9a1cb2361f9d9314d146221016e1aab71f306cefe" gracePeriod=10 Jul 06 22:27:28.581419 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:28.581387 2571 generic.go:356] "Generic (PLEG): container finished" podID="7f013012-0f5b-46d3-bf3c-37ceb1546ff9" containerID="6c9aae35ee2a0a6419da4bc9a1cb2361f9d9314d146221016e1aab71f306cefe" exitCode=0 Jul 06 22:27:28.581572 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:28.581443 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wzzbn" event={"ID":"7f013012-0f5b-46d3-bf3c-37ceb1546ff9","Type":"ContainerDied","Data":"6c9aae35ee2a0a6419da4bc9a1cb2361f9d9314d146221016e1aab71f306cefe"} Jul 06 22:27:28.636495 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:28.636415 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hxnqr"] Jul 06 22:27:28.637221 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:28.637179 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wzzbn" Jul 06 22:27:28.640120 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:28.640098 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hxnqr" Jul 06 22:27:28.656467 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:28.656435 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hxnqr"] Jul 06 22:27:28.657140 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:28.657120 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zz9pg\" (UniqueName: \"kubernetes.io/projected/7f013012-0f5b-46d3-bf3c-37ceb1546ff9-kube-api-access-zz9pg\") pod \"7f013012-0f5b-46d3-bf3c-37ceb1546ff9\" (UID: \"7f013012-0f5b-46d3-bf3c-37ceb1546ff9\") " Jul 06 22:27:28.658321 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:28.657236 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7f013012-0f5b-46d3-bf3c-37ceb1546ff9-extensions-socket-volume\") pod \"7f013012-0f5b-46d3-bf3c-37ceb1546ff9\" (UID: \"7f013012-0f5b-46d3-bf3c-37ceb1546ff9\") " Jul 06 22:27:28.658321 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:28.657488 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ef6cd197-77c7-4335-9f33-02ba163be40c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-hxnqr\" (UID: \"ef6cd197-77c7-4335-9f33-02ba163be40c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hxnqr" Jul 06 22:27:28.658321 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:28.657623 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2js5c\" (UniqueName: \"kubernetes.io/projected/ef6cd197-77c7-4335-9f33-02ba163be40c-kube-api-access-2js5c\") pod \"kuadrant-operator-controller-manager-86878c769f-hxnqr\" (UID: \"ef6cd197-77c7-4335-9f33-02ba163be40c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hxnqr" Jul 06 22:27:28.658321 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:28.657991 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f013012-0f5b-46d3-bf3c-37ceb1546ff9-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "7f013012-0f5b-46d3-bf3c-37ceb1546ff9" (UID: "7f013012-0f5b-46d3-bf3c-37ceb1546ff9"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:27:28.660623 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:28.660592 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f013012-0f5b-46d3-bf3c-37ceb1546ff9-kube-api-access-zz9pg" (OuterVolumeSpecName: "kube-api-access-zz9pg") pod "7f013012-0f5b-46d3-bf3c-37ceb1546ff9" (UID: "7f013012-0f5b-46d3-bf3c-37ceb1546ff9"). InnerVolumeSpecName "kube-api-access-zz9pg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:27:28.758149 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:28.758108 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2js5c\" (UniqueName: \"kubernetes.io/projected/ef6cd197-77c7-4335-9f33-02ba163be40c-kube-api-access-2js5c\") pod \"kuadrant-operator-controller-manager-86878c769f-hxnqr\" (UID: \"ef6cd197-77c7-4335-9f33-02ba163be40c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hxnqr" Jul 06 22:27:28.758348 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:28.758165 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ef6cd197-77c7-4335-9f33-02ba163be40c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-hxnqr\" (UID: \"ef6cd197-77c7-4335-9f33-02ba163be40c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hxnqr" Jul 06 22:27:28.758348 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:28.758202 2571 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7f013012-0f5b-46d3-bf3c-37ceb1546ff9-extensions-socket-volume\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:27:28.758348 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:28.758243 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zz9pg\" (UniqueName: \"kubernetes.io/projected/7f013012-0f5b-46d3-bf3c-37ceb1546ff9-kube-api-access-zz9pg\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:27:28.758555 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:28.758538 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ef6cd197-77c7-4335-9f33-02ba163be40c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-hxnqr\" (UID: \"ef6cd197-77c7-4335-9f33-02ba163be40c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hxnqr" Jul 06 22:27:28.773964 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:28.773928 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2js5c\" (UniqueName: \"kubernetes.io/projected/ef6cd197-77c7-4335-9f33-02ba163be40c-kube-api-access-2js5c\") pod \"kuadrant-operator-controller-manager-86878c769f-hxnqr\" (UID: \"ef6cd197-77c7-4335-9f33-02ba163be40c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hxnqr" Jul 06 22:27:28.950316 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:28.950197 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hxnqr" Jul 06 22:27:29.088729 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:29.088695 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hxnqr"] Jul 06 22:27:29.092288 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:27:29.092260 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef6cd197_77c7_4335_9f33_02ba163be40c.slice/crio-81d775d08c0d76f0e1dbf4675d29a170b4739362f3f80243b37c1f04e25df9c8 WatchSource:0}: Error finding container 81d775d08c0d76f0e1dbf4675d29a170b4739362f3f80243b37c1f04e25df9c8: Status 404 returned error can't find the container with id 81d775d08c0d76f0e1dbf4675d29a170b4739362f3f80243b37c1f04e25df9c8 Jul 06 22:27:29.586691 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:29.586656 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wzzbn" event={"ID":"7f013012-0f5b-46d3-bf3c-37ceb1546ff9","Type":"ContainerDied","Data":"deac02ff9fbfbb97c2beb6ec35566f55d3a3dae7c8a636704e0ae1bce800d591"} Jul 06 22:27:29.587142 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:29.586670 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wzzbn" Jul 06 22:27:29.587142 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:29.586715 2571 scope.go:117] "RemoveContainer" containerID="6c9aae35ee2a0a6419da4bc9a1cb2361f9d9314d146221016e1aab71f306cefe" Jul 06 22:27:29.588378 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:29.588350 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hxnqr" event={"ID":"ef6cd197-77c7-4335-9f33-02ba163be40c","Type":"ContainerStarted","Data":"00fab0ae896d7cecf4ec19c571f6cb049aac7845644eeb130d70d46270c2813a"} Jul 06 22:27:29.588478 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:29.588404 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hxnqr" event={"ID":"ef6cd197-77c7-4335-9f33-02ba163be40c","Type":"ContainerStarted","Data":"81d775d08c0d76f0e1dbf4675d29a170b4739362f3f80243b37c1f04e25df9c8"} Jul 06 22:27:29.588478 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:29.588437 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hxnqr" Jul 06 22:27:29.624160 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:29.624098 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hxnqr" podStartSLOduration=1.6240799369999999 podStartE2EDuration="1.624079937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:27:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:27:29.622855663 +0000 UTC m=+456.541098532" watchObservedRunningTime="2026-07-06 22:27:29.624079937 +0000 UTC m=+456.542322793" Jul 06 22:27:29.640367 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:29.640335 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wzzbn"] Jul 06 22:27:29.645473 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:29.645447 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wzzbn"] Jul 06 22:27:31.634892 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:31.634860 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f013012-0f5b-46d3-bf3c-37ceb1546ff9" path="/var/lib/kubelet/pods/7f013012-0f5b-46d3-bf3c-37ceb1546ff9/volumes" Jul 06 22:27:40.597105 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:40.597068 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hxnqr" Jul 06 22:27:44.807920 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:44.807882 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:27:45.169418 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:45.169375 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:27:49.637094 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:49.637057 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:27:49.750288 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:49.750249 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:27:49.850322 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:49.850276 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:27:50.602037 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:50.602005 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-4pjbq"] Jul 06 22:27:50.602353 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:50.602339 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f013012-0f5b-46d3-bf3c-37ceb1546ff9" containerName="manager" Jul 06 22:27:50.602353 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:50.602354 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f013012-0f5b-46d3-bf3c-37ceb1546ff9" containerName="manager" Jul 06 22:27:50.602448 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:50.602409 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f013012-0f5b-46d3-bf3c-37ceb1546ff9" containerName="manager" Jul 06 22:27:50.605355 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:50.605332 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-4pjbq" Jul 06 22:27:50.609337 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:50.609310 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-f9fwm\"" Jul 06 22:27:50.622473 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:50.619139 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-4pjbq"] Jul 06 22:27:50.645392 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:50.645356 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:27:50.748722 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:50.748668 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:27:50.757992 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:50.757939 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgnfx\" (UniqueName: \"kubernetes.io/projected/4f8ad399-b2ed-46c3-b8e8-d01b78059da4-kube-api-access-dgnfx\") pod \"authorino-58cd955675-4pjbq\" (UID: \"4f8ad399-b2ed-46c3-b8e8-d01b78059da4\") " pod="kuadrant-system/authorino-58cd955675-4pjbq" Jul 06 22:27:50.805087 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:50.805052 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-wxjh7"] Jul 06 22:27:50.808289 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:50.808270 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-wxjh7" Jul 06 22:27:50.817251 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:50.817202 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-wxjh7"] Jul 06 22:27:50.859071 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:50.858985 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgnfx\" (UniqueName: \"kubernetes.io/projected/4f8ad399-b2ed-46c3-b8e8-d01b78059da4-kube-api-access-dgnfx\") pod \"authorino-58cd955675-4pjbq\" (UID: \"4f8ad399-b2ed-46c3-b8e8-d01b78059da4\") " pod="kuadrant-system/authorino-58cd955675-4pjbq" Jul 06 22:27:50.870042 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:50.869999 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgnfx\" (UniqueName: \"kubernetes.io/projected/4f8ad399-b2ed-46c3-b8e8-d01b78059da4-kube-api-access-dgnfx\") pod \"authorino-58cd955675-4pjbq\" (UID: \"4f8ad399-b2ed-46c3-b8e8-d01b78059da4\") " pod="kuadrant-system/authorino-58cd955675-4pjbq" Jul 06 22:27:50.914356 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:50.914320 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-4pjbq" Jul 06 22:27:50.959553 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:50.959519 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzhmp\" (UniqueName: \"kubernetes.io/projected/7824ab91-2449-48f0-aca3-9cf3376a6deb-kube-api-access-rzhmp\") pod \"authorino-75d556bd77-wxjh7\" (UID: \"7824ab91-2449-48f0-aca3-9cf3376a6deb\") " pod="kuadrant-system/authorino-75d556bd77-wxjh7" Jul 06 22:27:51.054143 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:51.054113 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-4pjbq"] Jul 06 22:27:51.056633 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:27:51.056602 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4f8ad399_b2ed_46c3_b8e8_d01b78059da4.slice/crio-9984a05822468fe3789a5b913e4602fa10a951fe9274ec82fd1f41739bd795ac WatchSource:0}: Error finding container 9984a05822468fe3789a5b913e4602fa10a951fe9274ec82fd1f41739bd795ac: Status 404 returned error can't find the container with id 9984a05822468fe3789a5b913e4602fa10a951fe9274ec82fd1f41739bd795ac Jul 06 22:27:51.060166 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:51.060120 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzhmp\" (UniqueName: \"kubernetes.io/projected/7824ab91-2449-48f0-aca3-9cf3376a6deb-kube-api-access-rzhmp\") pod \"authorino-75d556bd77-wxjh7\" (UID: \"7824ab91-2449-48f0-aca3-9cf3376a6deb\") " pod="kuadrant-system/authorino-75d556bd77-wxjh7" Jul 06 22:27:51.076309 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:51.076276 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzhmp\" (UniqueName: \"kubernetes.io/projected/7824ab91-2449-48f0-aca3-9cf3376a6deb-kube-api-access-rzhmp\") pod \"authorino-75d556bd77-wxjh7\" (UID: \"7824ab91-2449-48f0-aca3-9cf3376a6deb\") " pod="kuadrant-system/authorino-75d556bd77-wxjh7" Jul 06 22:27:51.118264 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:51.118171 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-wxjh7" Jul 06 22:27:51.274717 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:51.274682 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-wxjh7"] Jul 06 22:27:51.276840 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:27:51.276812 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7824ab91_2449_48f0_aca3_9cf3376a6deb.slice/crio-7f1acdffc3131b92303797e25d82642a130fac445f34a2d0a001dbd52a3d7ef3 WatchSource:0}: Error finding container 7f1acdffc3131b92303797e25d82642a130fac445f34a2d0a001dbd52a3d7ef3: Status 404 returned error can't find the container with id 7f1acdffc3131b92303797e25d82642a130fac445f34a2d0a001dbd52a3d7ef3 Jul 06 22:27:51.694398 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:51.694316 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-wxjh7" event={"ID":"7824ab91-2449-48f0-aca3-9cf3376a6deb","Type":"ContainerStarted","Data":"7f1acdffc3131b92303797e25d82642a130fac445f34a2d0a001dbd52a3d7ef3"} Jul 06 22:27:51.697278 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:51.697243 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-4pjbq" event={"ID":"4f8ad399-b2ed-46c3-b8e8-d01b78059da4","Type":"ContainerStarted","Data":"9984a05822468fe3789a5b913e4602fa10a951fe9274ec82fd1f41739bd795ac"} Jul 06 22:27:52.582421 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:52.582241 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:27:54.724352 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:54.724301 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-4pjbq" event={"ID":"4f8ad399-b2ed-46c3-b8e8-d01b78059da4","Type":"ContainerStarted","Data":"450070c97a4bc90b715d2a20b91b4a02871f11f44fdd54663987a2809e834d73"} Jul 06 22:27:54.726373 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:54.726324 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-wxjh7" event={"ID":"7824ab91-2449-48f0-aca3-9cf3376a6deb","Type":"ContainerStarted","Data":"4a4a29ecec38cf0f82b22bb05d03338df43d5dcc08ab5f64ebaedb599e36f214"} Jul 06 22:27:54.744989 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:54.744908 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-4pjbq" podStartSLOduration=1.214050045 podStartE2EDuration="4.744888983s" totalImagesPullingTime="3.530838938s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:27:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:27:54.744063449 +0000 UTC m=+481.662306321" watchObservedRunningTime="2026-07-06 22:27:54.744888983 +0000 UTC m=+481.663131840" Jul 06 22:27:54.761939 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:54.761881 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-wxjh7" podStartSLOduration=1.4399328040000001 podStartE2EDuration="4.761859954s" totalImagesPullingTime="3.32192715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:27:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:27:54.761146935 +0000 UTC m=+481.679389793" watchObservedRunningTime="2026-07-06 22:27:54.761859954 +0000 UTC m=+481.680102804" Jul 06 22:27:54.787867 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:54.787828 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-4pjbq"] Jul 06 22:27:56.734814 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:56.734766 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-4pjbq" podUID="4f8ad399-b2ed-46c3-b8e8-d01b78059da4" containerName="authorino" containerID="cri-o://450070c97a4bc90b715d2a20b91b4a02871f11f44fdd54663987a2809e834d73" gracePeriod=30 Jul 06 22:27:56.982791 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:56.982763 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-4pjbq" Jul 06 22:27:57.118046 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:57.117944 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dgnfx\" (UniqueName: \"kubernetes.io/projected/4f8ad399-b2ed-46c3-b8e8-d01b78059da4-kube-api-access-dgnfx\") pod \"4f8ad399-b2ed-46c3-b8e8-d01b78059da4\" (UID: \"4f8ad399-b2ed-46c3-b8e8-d01b78059da4\") " Jul 06 22:27:57.120069 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:57.120038 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f8ad399-b2ed-46c3-b8e8-d01b78059da4-kube-api-access-dgnfx" (OuterVolumeSpecName: "kube-api-access-dgnfx") pod "4f8ad399-b2ed-46c3-b8e8-d01b78059da4" (UID: "4f8ad399-b2ed-46c3-b8e8-d01b78059da4"). InnerVolumeSpecName "kube-api-access-dgnfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:27:57.218713 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:57.218673 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgnfx\" (UniqueName: \"kubernetes.io/projected/4f8ad399-b2ed-46c3-b8e8-d01b78059da4-kube-api-access-dgnfx\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:27:57.741342 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:57.741302 2571 generic.go:356] "Generic (PLEG): container finished" podID="4f8ad399-b2ed-46c3-b8e8-d01b78059da4" containerID="450070c97a4bc90b715d2a20b91b4a02871f11f44fdd54663987a2809e834d73" exitCode=0 Jul 06 22:27:57.741787 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:57.741350 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-4pjbq" event={"ID":"4f8ad399-b2ed-46c3-b8e8-d01b78059da4","Type":"ContainerDied","Data":"450070c97a4bc90b715d2a20b91b4a02871f11f44fdd54663987a2809e834d73"} Jul 06 22:27:57.741787 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:57.741365 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-4pjbq" Jul 06 22:27:57.741787 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:57.741391 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-4pjbq" event={"ID":"4f8ad399-b2ed-46c3-b8e8-d01b78059da4","Type":"ContainerDied","Data":"9984a05822468fe3789a5b913e4602fa10a951fe9274ec82fd1f41739bd795ac"} Jul 06 22:27:57.741787 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:57.741412 2571 scope.go:117] "RemoveContainer" containerID="450070c97a4bc90b715d2a20b91b4a02871f11f44fdd54663987a2809e834d73" Jul 06 22:27:57.750362 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:57.750339 2571 scope.go:117] "RemoveContainer" containerID="450070c97a4bc90b715d2a20b91b4a02871f11f44fdd54663987a2809e834d73" Jul 06 22:27:57.750697 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:27:57.750674 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"450070c97a4bc90b715d2a20b91b4a02871f11f44fdd54663987a2809e834d73\": container with ID starting with 450070c97a4bc90b715d2a20b91b4a02871f11f44fdd54663987a2809e834d73 not found: ID does not exist" containerID="450070c97a4bc90b715d2a20b91b4a02871f11f44fdd54663987a2809e834d73" Jul 06 22:27:57.750776 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:57.750702 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"450070c97a4bc90b715d2a20b91b4a02871f11f44fdd54663987a2809e834d73"} err="failed to get container status \"450070c97a4bc90b715d2a20b91b4a02871f11f44fdd54663987a2809e834d73\": rpc error: code = NotFound desc = could not find container \"450070c97a4bc90b715d2a20b91b4a02871f11f44fdd54663987a2809e834d73\": container with ID starting with 450070c97a4bc90b715d2a20b91b4a02871f11f44fdd54663987a2809e834d73 not found: ID does not exist" Jul 06 22:27:57.763033 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:57.763006 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-4pjbq"] Jul 06 22:27:57.769980 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:57.769955 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-4pjbq"] Jul 06 22:27:59.634525 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:27:59.634488 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f8ad399-b2ed-46c3-b8e8-d01b78059da4" path="/var/lib/kubelet/pods/4f8ad399-b2ed-46c3-b8e8-d01b78059da4/volumes" Jul 06 22:28:11.193823 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:11.193785 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:28:23.007990 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:23.007957 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:28:23.274622 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:23.274541 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-chdtn"] Jul 06 22:28:23.274880 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:23.274868 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f8ad399-b2ed-46c3-b8e8-d01b78059da4" containerName="authorino" Jul 06 22:28:23.274922 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:23.274882 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f8ad399-b2ed-46c3-b8e8-d01b78059da4" containerName="authorino" Jul 06 22:28:23.274956 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:23.274940 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f8ad399-b2ed-46c3-b8e8-d01b78059da4" containerName="authorino" Jul 06 22:28:23.278884 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:23.278862 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-chdtn" Jul 06 22:28:23.295123 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:23.295092 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-chdtn"] Jul 06 22:28:23.323550 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:23.323513 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f6g5\" (UniqueName: \"kubernetes.io/projected/d817998c-107c-48c4-8325-79b5195d329b-kube-api-access-6f6g5\") pod \"authorino-5d4b67c446-chdtn\" (UID: \"d817998c-107c-48c4-8325-79b5195d329b\") " pod="kuadrant-system/authorino-5d4b67c446-chdtn" Jul 06 22:28:23.365385 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:23.365346 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:28:23.424813 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:23.424778 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f6g5\" (UniqueName: \"kubernetes.io/projected/d817998c-107c-48c4-8325-79b5195d329b-kube-api-access-6f6g5\") pod \"authorino-5d4b67c446-chdtn\" (UID: \"d817998c-107c-48c4-8325-79b5195d329b\") " pod="kuadrant-system/authorino-5d4b67c446-chdtn" Jul 06 22:28:23.434940 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:23.434911 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f6g5\" (UniqueName: \"kubernetes.io/projected/d817998c-107c-48c4-8325-79b5195d329b-kube-api-access-6f6g5\") pod \"authorino-5d4b67c446-chdtn\" (UID: \"d817998c-107c-48c4-8325-79b5195d329b\") " pod="kuadrant-system/authorino-5d4b67c446-chdtn" Jul 06 22:28:23.529579 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:23.529472 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-chdtn"] Jul 06 22:28:23.529767 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:23.529750 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-chdtn" Jul 06 22:28:23.665795 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:23.665763 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-chdtn"] Jul 06 22:28:23.667934 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:28:23.667897 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd817998c_107c_48c4_8325_79b5195d329b.slice/crio-aef97163d1ecb5abcec10cf089d2c95a3c0b0358f710fadb9e7fc86c4ecff84a WatchSource:0}: Error finding container aef97163d1ecb5abcec10cf089d2c95a3c0b0358f710fadb9e7fc86c4ecff84a: Status 404 returned error can't find the container with id aef97163d1ecb5abcec10cf089d2c95a3c0b0358f710fadb9e7fc86c4ecff84a Jul 06 22:28:23.856799 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:23.856709 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-chdtn" event={"ID":"d817998c-107c-48c4-8325-79b5195d329b","Type":"ContainerStarted","Data":"aef97163d1ecb5abcec10cf089d2c95a3c0b0358f710fadb9e7fc86c4ecff84a"} Jul 06 22:28:24.863075 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:24.863033 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-chdtn" event={"ID":"d817998c-107c-48c4-8325-79b5195d329b","Type":"ContainerStarted","Data":"2ea4f4efecf87f4269abb61d019b8eaab811f81a1ae0658f1b0e8d35e29ce849"} Jul 06 22:28:24.863609 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:24.863124 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-chdtn" podUID="d817998c-107c-48c4-8325-79b5195d329b" containerName="authorino" containerID="cri-o://2ea4f4efecf87f4269abb61d019b8eaab811f81a1ae0658f1b0e8d35e29ce849" gracePeriod=30 Jul 06 22:28:24.883279 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:24.883231 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-chdtn" podStartSLOduration=1.51291389 podStartE2EDuration="1.883198043s" totalImagesPullingTime="370.284153ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:28:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:28:24.882747732 +0000 UTC m=+511.800990588" watchObservedRunningTime="2026-07-06 22:28:24.883198043 +0000 UTC m=+511.801440898" Jul 06 22:28:25.111730 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:25.111701 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-chdtn" Jul 06 22:28:25.139628 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:25.139587 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6f6g5\" (UniqueName: \"kubernetes.io/projected/d817998c-107c-48c4-8325-79b5195d329b-kube-api-access-6f6g5\") pod \"d817998c-107c-48c4-8325-79b5195d329b\" (UID: \"d817998c-107c-48c4-8325-79b5195d329b\") " Jul 06 22:28:25.141742 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:25.141710 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d817998c-107c-48c4-8325-79b5195d329b-kube-api-access-6f6g5" (OuterVolumeSpecName: "kube-api-access-6f6g5") pod "d817998c-107c-48c4-8325-79b5195d329b" (UID: "d817998c-107c-48c4-8325-79b5195d329b"). InnerVolumeSpecName "kube-api-access-6f6g5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:28:25.241141 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:25.241097 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6f6g5\" (UniqueName: \"kubernetes.io/projected/d817998c-107c-48c4-8325-79b5195d329b-kube-api-access-6f6g5\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:28:25.869286 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:25.869250 2571 generic.go:356] "Generic (PLEG): container finished" podID="d817998c-107c-48c4-8325-79b5195d329b" containerID="2ea4f4efecf87f4269abb61d019b8eaab811f81a1ae0658f1b0e8d35e29ce849" exitCode=0 Jul 06 22:28:25.869720 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:25.869293 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-chdtn" Jul 06 22:28:25.869720 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:25.869315 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-chdtn" event={"ID":"d817998c-107c-48c4-8325-79b5195d329b","Type":"ContainerDied","Data":"2ea4f4efecf87f4269abb61d019b8eaab811f81a1ae0658f1b0e8d35e29ce849"} Jul 06 22:28:25.869720 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:25.869350 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-chdtn" event={"ID":"d817998c-107c-48c4-8325-79b5195d329b","Type":"ContainerDied","Data":"aef97163d1ecb5abcec10cf089d2c95a3c0b0358f710fadb9e7fc86c4ecff84a"} Jul 06 22:28:25.869720 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:25.869366 2571 scope.go:117] "RemoveContainer" containerID="2ea4f4efecf87f4269abb61d019b8eaab811f81a1ae0658f1b0e8d35e29ce849" Jul 06 22:28:25.877339 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:25.877318 2571 scope.go:117] "RemoveContainer" containerID="2ea4f4efecf87f4269abb61d019b8eaab811f81a1ae0658f1b0e8d35e29ce849" Jul 06 22:28:25.877597 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:28:25.877578 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ea4f4efecf87f4269abb61d019b8eaab811f81a1ae0658f1b0e8d35e29ce849\": container with ID starting with 2ea4f4efecf87f4269abb61d019b8eaab811f81a1ae0658f1b0e8d35e29ce849 not found: ID does not exist" containerID="2ea4f4efecf87f4269abb61d019b8eaab811f81a1ae0658f1b0e8d35e29ce849" Jul 06 22:28:25.877662 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:25.877604 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ea4f4efecf87f4269abb61d019b8eaab811f81a1ae0658f1b0e8d35e29ce849"} err="failed to get container status \"2ea4f4efecf87f4269abb61d019b8eaab811f81a1ae0658f1b0e8d35e29ce849\": rpc error: code = NotFound desc = could not find container \"2ea4f4efecf87f4269abb61d019b8eaab811f81a1ae0658f1b0e8d35e29ce849\": container with ID starting with 2ea4f4efecf87f4269abb61d019b8eaab811f81a1ae0658f1b0e8d35e29ce849 not found: ID does not exist" Jul 06 22:28:25.895665 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:25.895588 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-chdtn"] Jul 06 22:28:25.902915 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:25.902887 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-chdtn"] Jul 06 22:28:27.633557 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:27.633527 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d817998c-107c-48c4-8325-79b5195d329b" path="/var/lib/kubelet/pods/d817998c-107c-48c4-8325-79b5195d329b/volumes" Jul 06 22:28:27.767751 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:27.767712 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-wxjh7"] Jul 06 22:28:27.768004 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:27.767941 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-wxjh7" podUID="7824ab91-2449-48f0-aca3-9cf3376a6deb" containerName="authorino" containerID="cri-o://4a4a29ecec38cf0f82b22bb05d03338df43d5dcc08ab5f64ebaedb599e36f214" gracePeriod=30 Jul 06 22:28:28.021216 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:28.021169 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-wxjh7" Jul 06 22:28:28.064282 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:28.064245 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rzhmp\" (UniqueName: \"kubernetes.io/projected/7824ab91-2449-48f0-aca3-9cf3376a6deb-kube-api-access-rzhmp\") pod \"7824ab91-2449-48f0-aca3-9cf3376a6deb\" (UID: \"7824ab91-2449-48f0-aca3-9cf3376a6deb\") " Jul 06 22:28:28.066271 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:28.066240 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7824ab91-2449-48f0-aca3-9cf3376a6deb-kube-api-access-rzhmp" (OuterVolumeSpecName: "kube-api-access-rzhmp") pod "7824ab91-2449-48f0-aca3-9cf3376a6deb" (UID: "7824ab91-2449-48f0-aca3-9cf3376a6deb"). InnerVolumeSpecName "kube-api-access-rzhmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:28:28.165020 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:28.164932 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzhmp\" (UniqueName: \"kubernetes.io/projected/7824ab91-2449-48f0-aca3-9cf3376a6deb-kube-api-access-rzhmp\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:28:28.884203 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:28.884167 2571 generic.go:356] "Generic (PLEG): container finished" podID="7824ab91-2449-48f0-aca3-9cf3376a6deb" containerID="4a4a29ecec38cf0f82b22bb05d03338df43d5dcc08ab5f64ebaedb599e36f214" exitCode=0 Jul 06 22:28:28.884694 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:28.884246 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-wxjh7" Jul 06 22:28:28.884694 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:28.884258 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-wxjh7" event={"ID":"7824ab91-2449-48f0-aca3-9cf3376a6deb","Type":"ContainerDied","Data":"4a4a29ecec38cf0f82b22bb05d03338df43d5dcc08ab5f64ebaedb599e36f214"} Jul 06 22:28:28.884694 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:28.884312 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-wxjh7" event={"ID":"7824ab91-2449-48f0-aca3-9cf3376a6deb","Type":"ContainerDied","Data":"7f1acdffc3131b92303797e25d82642a130fac445f34a2d0a001dbd52a3d7ef3"} Jul 06 22:28:28.884694 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:28.884336 2571 scope.go:117] "RemoveContainer" containerID="4a4a29ecec38cf0f82b22bb05d03338df43d5dcc08ab5f64ebaedb599e36f214" Jul 06 22:28:28.892641 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:28.892624 2571 scope.go:117] "RemoveContainer" containerID="4a4a29ecec38cf0f82b22bb05d03338df43d5dcc08ab5f64ebaedb599e36f214" Jul 06 22:28:28.892906 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:28:28.892886 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a4a29ecec38cf0f82b22bb05d03338df43d5dcc08ab5f64ebaedb599e36f214\": container with ID starting with 4a4a29ecec38cf0f82b22bb05d03338df43d5dcc08ab5f64ebaedb599e36f214 not found: ID does not exist" containerID="4a4a29ecec38cf0f82b22bb05d03338df43d5dcc08ab5f64ebaedb599e36f214" Jul 06 22:28:28.892953 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:28.892915 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a4a29ecec38cf0f82b22bb05d03338df43d5dcc08ab5f64ebaedb599e36f214"} err="failed to get container status \"4a4a29ecec38cf0f82b22bb05d03338df43d5dcc08ab5f64ebaedb599e36f214\": rpc error: code = NotFound desc = could not find container \"4a4a29ecec38cf0f82b22bb05d03338df43d5dcc08ab5f64ebaedb599e36f214\": container with ID starting with 4a4a29ecec38cf0f82b22bb05d03338df43d5dcc08ab5f64ebaedb599e36f214 not found: ID does not exist" Jul 06 22:28:28.909998 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:28.909967 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-wxjh7"] Jul 06 22:28:28.914720 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:28.914701 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-wxjh7"] Jul 06 22:28:29.633835 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:29.633798 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7824ab91-2449-48f0-aca3-9cf3376a6deb" path="/var/lib/kubelet/pods/7824ab91-2449-48f0-aca3-9cf3376a6deb/volumes" Jul 06 22:28:32.151874 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:32.151836 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:28:32.158698 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:32.158671 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7cd9bffc57-rv67m"] Jul 06 22:28:32.158982 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:32.158970 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d817998c-107c-48c4-8325-79b5195d329b" containerName="authorino" Jul 06 22:28:32.159023 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:32.158997 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d817998c-107c-48c4-8325-79b5195d329b" containerName="authorino" Jul 06 22:28:32.159023 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:32.159011 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7824ab91-2449-48f0-aca3-9cf3376a6deb" containerName="authorino" Jul 06 22:28:32.159023 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:32.159016 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="7824ab91-2449-48f0-aca3-9cf3376a6deb" containerName="authorino" Jul 06 22:28:32.159115 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:32.159072 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="7824ab91-2449-48f0-aca3-9cf3376a6deb" containerName="authorino" Jul 06 22:28:32.159115 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:32.159084 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="d817998c-107c-48c4-8325-79b5195d329b" containerName="authorino" Jul 06 22:28:32.163465 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:32.163444 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7cd9bffc57-rv67m" Jul 06 22:28:32.166186 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:32.166166 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 06 22:28:32.166317 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:32.166171 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 06 22:28:32.166317 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:32.166249 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-mgg8c\"" Jul 06 22:28:32.172623 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:32.172601 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7cd9bffc57-rv67m"] Jul 06 22:28:32.202985 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:32.202952 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz9tj\" (UniqueName: \"kubernetes.io/projected/c2ec2ff5-7410-420c-a9df-ccbfc3f4228a-kube-api-access-pz9tj\") pod \"maas-controller-7cd9bffc57-rv67m\" (UID: \"c2ec2ff5-7410-420c-a9df-ccbfc3f4228a\") " pod="opendatahub/maas-controller-7cd9bffc57-rv67m" Jul 06 22:28:32.202985 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:32.202995 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c2ec2ff5-7410-420c-a9df-ccbfc3f4228a-webhook-cert\") pod \"maas-controller-7cd9bffc57-rv67m\" (UID: \"c2ec2ff5-7410-420c-a9df-ccbfc3f4228a\") " pod="opendatahub/maas-controller-7cd9bffc57-rv67m" Jul 06 22:28:32.251595 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:32.251560 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:28:32.303811 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:32.303768 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz9tj\" (UniqueName: \"kubernetes.io/projected/c2ec2ff5-7410-420c-a9df-ccbfc3f4228a-kube-api-access-pz9tj\") pod \"maas-controller-7cd9bffc57-rv67m\" (UID: \"c2ec2ff5-7410-420c-a9df-ccbfc3f4228a\") " pod="opendatahub/maas-controller-7cd9bffc57-rv67m" Jul 06 22:28:32.303959 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:32.303829 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c2ec2ff5-7410-420c-a9df-ccbfc3f4228a-webhook-cert\") pod \"maas-controller-7cd9bffc57-rv67m\" (UID: \"c2ec2ff5-7410-420c-a9df-ccbfc3f4228a\") " pod="opendatahub/maas-controller-7cd9bffc57-rv67m" Jul 06 22:28:32.303959 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:28:32.303922 2571 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 06 22:28:32.304070 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:28:32.303992 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2ec2ff5-7410-420c-a9df-ccbfc3f4228a-webhook-cert podName:c2ec2ff5-7410-420c-a9df-ccbfc3f4228a nodeName:}" failed. No retries permitted until 2026-07-06 22:28:32.803976142 +0000 UTC m=+519.722218975 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/c2ec2ff5-7410-420c-a9df-ccbfc3f4228a-webhook-cert") pod "maas-controller-7cd9bffc57-rv67m" (UID: "c2ec2ff5-7410-420c-a9df-ccbfc3f4228a") : secret "maas-controller-webhook-cert" not found Jul 06 22:28:32.317051 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:32.317010 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:28:32.319425 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:32.319399 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz9tj\" (UniqueName: \"kubernetes.io/projected/c2ec2ff5-7410-420c-a9df-ccbfc3f4228a-kube-api-access-pz9tj\") pod \"maas-controller-7cd9bffc57-rv67m\" (UID: \"c2ec2ff5-7410-420c-a9df-ccbfc3f4228a\") " pod="opendatahub/maas-controller-7cd9bffc57-rv67m" Jul 06 22:28:32.356160 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:32.356118 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5b4776d6c4-hb4m6"] Jul 06 22:28:32.359497 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:32.359478 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b4776d6c4-hb4m6" Jul 06 22:28:32.372092 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:32.372062 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b4776d6c4-hb4m6"] Jul 06 22:28:32.405281 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:32.405164 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bf8e72e9-cae2-4d09-a789-0ff3cee3af17-webhook-cert\") pod \"maas-controller-5b4776d6c4-hb4m6\" (UID: \"bf8e72e9-cae2-4d09-a789-0ff3cee3af17\") " pod="opendatahub/maas-controller-5b4776d6c4-hb4m6" Jul 06 22:28:32.405281 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:32.405203 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4xqn\" (UniqueName: \"kubernetes.io/projected/bf8e72e9-cae2-4d09-a789-0ff3cee3af17-kube-api-access-c4xqn\") pod \"maas-controller-5b4776d6c4-hb4m6\" (UID: \"bf8e72e9-cae2-4d09-a789-0ff3cee3af17\") " pod="opendatahub/maas-controller-5b4776d6c4-hb4m6" Jul 06 22:28:32.481714 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:32.481678 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:28:32.506409 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:32.506367 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bf8e72e9-cae2-4d09-a789-0ff3cee3af17-webhook-cert\") pod \"maas-controller-5b4776d6c4-hb4m6\" (UID: \"bf8e72e9-cae2-4d09-a789-0ff3cee3af17\") " pod="opendatahub/maas-controller-5b4776d6c4-hb4m6" Jul 06 22:28:32.506572 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:32.506416 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4xqn\" (UniqueName: \"kubernetes.io/projected/bf8e72e9-cae2-4d09-a789-0ff3cee3af17-kube-api-access-c4xqn\") pod \"maas-controller-5b4776d6c4-hb4m6\" (UID: \"bf8e72e9-cae2-4d09-a789-0ff3cee3af17\") " pod="opendatahub/maas-controller-5b4776d6c4-hb4m6" Jul 06 22:28:32.508822 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:32.508803 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bf8e72e9-cae2-4d09-a789-0ff3cee3af17-webhook-cert\") pod \"maas-controller-5b4776d6c4-hb4m6\" (UID: \"bf8e72e9-cae2-4d09-a789-0ff3cee3af17\") " pod="opendatahub/maas-controller-5b4776d6c4-hb4m6" Jul 06 22:28:32.516968 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:32.516938 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4xqn\" (UniqueName: \"kubernetes.io/projected/bf8e72e9-cae2-4d09-a789-0ff3cee3af17-kube-api-access-c4xqn\") pod \"maas-controller-5b4776d6c4-hb4m6\" (UID: \"bf8e72e9-cae2-4d09-a789-0ff3cee3af17\") " pod="opendatahub/maas-controller-5b4776d6c4-hb4m6" Jul 06 22:28:32.670620 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:32.670520 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b4776d6c4-hb4m6" Jul 06 22:28:32.808625 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:32.808588 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c2ec2ff5-7410-420c-a9df-ccbfc3f4228a-webhook-cert\") pod \"maas-controller-7cd9bffc57-rv67m\" (UID: \"c2ec2ff5-7410-420c-a9df-ccbfc3f4228a\") " pod="opendatahub/maas-controller-7cd9bffc57-rv67m" Jul 06 22:28:32.809485 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:32.809458 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b4776d6c4-hb4m6"] Jul 06 22:28:32.811334 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:32.811295 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c2ec2ff5-7410-420c-a9df-ccbfc3f4228a-webhook-cert\") pod \"maas-controller-7cd9bffc57-rv67m\" (UID: \"c2ec2ff5-7410-420c-a9df-ccbfc3f4228a\") " pod="opendatahub/maas-controller-7cd9bffc57-rv67m" Jul 06 22:28:32.813404 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:28:32.813379 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf8e72e9_cae2_4d09_a789_0ff3cee3af17.slice/crio-e043822ba2d2120a4ba65101ef9c44f0175c20a27641b55c36285a9608d77039 WatchSource:0}: Error finding container e043822ba2d2120a4ba65101ef9c44f0175c20a27641b55c36285a9608d77039: Status 404 returned error can't find the container with id e043822ba2d2120a4ba65101ef9c44f0175c20a27641b55c36285a9608d77039 Jul 06 22:28:32.905711 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:32.905679 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b4776d6c4-hb4m6" event={"ID":"bf8e72e9-cae2-4d09-a789-0ff3cee3af17","Type":"ContainerStarted","Data":"e043822ba2d2120a4ba65101ef9c44f0175c20a27641b55c36285a9608d77039"} Jul 06 22:28:33.074474 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:33.074377 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7cd9bffc57-rv67m" Jul 06 22:28:33.208703 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:33.208533 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7cd9bffc57-rv67m"] Jul 06 22:28:33.211306 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:28:33.211282 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2ec2ff5_7410_420c_a9df_ccbfc3f4228a.slice/crio-c2c8562489fecdf5746017d76ac423f4a4dca3842d2d5a5294d70f7f02482678 WatchSource:0}: Error finding container c2c8562489fecdf5746017d76ac423f4a4dca3842d2d5a5294d70f7f02482678: Status 404 returned error can't find the container with id c2c8562489fecdf5746017d76ac423f4a4dca3842d2d5a5294d70f7f02482678 Jul 06 22:28:33.893917 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:33.893879 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:28:33.917069 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:33.916900 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7cd9bffc57-rv67m" event={"ID":"c2ec2ff5-7410-420c-a9df-ccbfc3f4228a","Type":"ContainerStarted","Data":"c2c8562489fecdf5746017d76ac423f4a4dca3842d2d5a5294d70f7f02482678"} Jul 06 22:28:33.959288 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:33.959145 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:28:35.057071 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:35.057037 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:28:35.152764 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:35.152728 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:28:35.928671 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:35.928632 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7cd9bffc57-rv67m" event={"ID":"c2ec2ff5-7410-420c-a9df-ccbfc3f4228a","Type":"ContainerStarted","Data":"8a99dbd27092ce79e3938cb888e178501ec869b21e100c85e10b513a3468bff8"} Jul 06 22:28:35.928847 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:35.928729 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7cd9bffc57-rv67m" Jul 06 22:28:35.930093 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:35.930066 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b4776d6c4-hb4m6" event={"ID":"bf8e72e9-cae2-4d09-a789-0ff3cee3af17","Type":"ContainerStarted","Data":"45b932ad762ad8063cf5ace7b229fba81da940918271311bed3b3e7a6274dcd4"} Jul 06 22:28:35.930199 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:35.930147 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5b4776d6c4-hb4m6" Jul 06 22:28:35.951553 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:35.951499 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7cd9bffc57-rv67m" podStartSLOduration=2.291027977 podStartE2EDuration="3.951484105s" totalImagesPullingTime="1.660456128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:28:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:28:35.948877847 +0000 UTC m=+522.867120705" watchObservedRunningTime="2026-07-06 22:28:35.951484105 +0000 UTC m=+522.869726960" Jul 06 22:28:35.971303 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:35.971255 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5b4776d6c4-hb4m6" podStartSLOduration=1.923533497 podStartE2EDuration="3.971238976s" totalImagesPullingTime="2.047705479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:28:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:28:35.969838272 +0000 UTC m=+522.888081157" watchObservedRunningTime="2026-07-06 22:28:35.971238976 +0000 UTC m=+522.889481831" Jul 06 22:28:45.297442 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:45.297358 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:28:46.945960 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:46.945931 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7cd9bffc57-rv67m" Jul 06 22:28:46.946359 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:46.946002 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5b4776d6c4-hb4m6" Jul 06 22:28:47.042723 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:47.042688 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7cd9bffc57-rv67m"] Jul 06 22:28:47.042927 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:47.042882 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-7cd9bffc57-rv67m" podUID="c2ec2ff5-7410-420c-a9df-ccbfc3f4228a" containerName="manager" containerID="cri-o://8a99dbd27092ce79e3938cb888e178501ec869b21e100c85e10b513a3468bff8" gracePeriod=30 Jul 06 22:28:47.296054 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:47.296023 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7cd9bffc57-rv67m" Jul 06 22:28:47.445983 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:47.445938 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pz9tj\" (UniqueName: \"kubernetes.io/projected/c2ec2ff5-7410-420c-a9df-ccbfc3f4228a-kube-api-access-pz9tj\") pod \"c2ec2ff5-7410-420c-a9df-ccbfc3f4228a\" (UID: \"c2ec2ff5-7410-420c-a9df-ccbfc3f4228a\") " Jul 06 22:28:47.445983 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:47.445982 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c2ec2ff5-7410-420c-a9df-ccbfc3f4228a-webhook-cert\") pod \"c2ec2ff5-7410-420c-a9df-ccbfc3f4228a\" (UID: \"c2ec2ff5-7410-420c-a9df-ccbfc3f4228a\") " Jul 06 22:28:47.448217 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:47.448167 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2ec2ff5-7410-420c-a9df-ccbfc3f4228a-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "c2ec2ff5-7410-420c-a9df-ccbfc3f4228a" (UID: "c2ec2ff5-7410-420c-a9df-ccbfc3f4228a"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:28:47.448326 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:47.448174 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2ec2ff5-7410-420c-a9df-ccbfc3f4228a-kube-api-access-pz9tj" (OuterVolumeSpecName: "kube-api-access-pz9tj") pod "c2ec2ff5-7410-420c-a9df-ccbfc3f4228a" (UID: "c2ec2ff5-7410-420c-a9df-ccbfc3f4228a"). InnerVolumeSpecName "kube-api-access-pz9tj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:28:47.546800 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:47.546702 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pz9tj\" (UniqueName: \"kubernetes.io/projected/c2ec2ff5-7410-420c-a9df-ccbfc3f4228a-kube-api-access-pz9tj\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:28:47.546800 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:47.546739 2571 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c2ec2ff5-7410-420c-a9df-ccbfc3f4228a-webhook-cert\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:28:47.987539 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:47.987499 2571 generic.go:356] "Generic (PLEG): container finished" podID="c2ec2ff5-7410-420c-a9df-ccbfc3f4228a" containerID="8a99dbd27092ce79e3938cb888e178501ec869b21e100c85e10b513a3468bff8" exitCode=0 Jul 06 22:28:47.987539 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:47.987536 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7cd9bffc57-rv67m" Jul 06 22:28:47.988083 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:47.987577 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7cd9bffc57-rv67m" event={"ID":"c2ec2ff5-7410-420c-a9df-ccbfc3f4228a","Type":"ContainerDied","Data":"8a99dbd27092ce79e3938cb888e178501ec869b21e100c85e10b513a3468bff8"} Jul 06 22:28:47.988083 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:47.987630 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7cd9bffc57-rv67m" event={"ID":"c2ec2ff5-7410-420c-a9df-ccbfc3f4228a","Type":"ContainerDied","Data":"c2c8562489fecdf5746017d76ac423f4a4dca3842d2d5a5294d70f7f02482678"} Jul 06 22:28:47.988083 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:47.987652 2571 scope.go:117] "RemoveContainer" containerID="8a99dbd27092ce79e3938cb888e178501ec869b21e100c85e10b513a3468bff8" Jul 06 22:28:47.995867 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:47.995848 2571 scope.go:117] "RemoveContainer" containerID="8a99dbd27092ce79e3938cb888e178501ec869b21e100c85e10b513a3468bff8" Jul 06 22:28:47.996112 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:28:47.996094 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a99dbd27092ce79e3938cb888e178501ec869b21e100c85e10b513a3468bff8\": container with ID starting with 8a99dbd27092ce79e3938cb888e178501ec869b21e100c85e10b513a3468bff8 not found: ID does not exist" containerID="8a99dbd27092ce79e3938cb888e178501ec869b21e100c85e10b513a3468bff8" Jul 06 22:28:47.996167 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:47.996128 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a99dbd27092ce79e3938cb888e178501ec869b21e100c85e10b513a3468bff8"} err="failed to get container status \"8a99dbd27092ce79e3938cb888e178501ec869b21e100c85e10b513a3468bff8\": rpc error: code = NotFound desc = could not find container \"8a99dbd27092ce79e3938cb888e178501ec869b21e100c85e10b513a3468bff8\": container with ID starting with 8a99dbd27092ce79e3938cb888e178501ec869b21e100c85e10b513a3468bff8 not found: ID does not exist" Jul 06 22:28:48.009981 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:48.009949 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7cd9bffc57-rv67m"] Jul 06 22:28:48.015708 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:48.015669 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7cd9bffc57-rv67m"] Jul 06 22:28:49.634133 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:28:49.634098 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2ec2ff5-7410-420c-a9df-ccbfc3f4228a" path="/var/lib/kubelet/pods/c2ec2ff5-7410-420c-a9df-ccbfc3f4228a/volumes" Jul 06 22:29:09.632815 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:09.632775 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:29:10.132537 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:10.132491 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:29:10.176818 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:10.176783 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-bdbcbfbbd-cz42d"] Jul 06 22:29:10.177101 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:10.177089 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2ec2ff5-7410-420c-a9df-ccbfc3f4228a" containerName="manager" Jul 06 22:29:10.177152 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:10.177103 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2ec2ff5-7410-420c-a9df-ccbfc3f4228a" containerName="manager" Jul 06 22:29:10.177233 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:10.177219 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2ec2ff5-7410-420c-a9df-ccbfc3f4228a" containerName="manager" Jul 06 22:29:10.182966 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:10.182942 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-bdbcbfbbd-cz42d" Jul 06 22:29:10.185928 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:10.185904 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-serving-cert\"" Jul 06 22:29:10.192179 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:10.192150 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-bdbcbfbbd-cz42d"] Jul 06 22:29:10.242841 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:10.242807 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmtjf\" (UniqueName: \"kubernetes.io/projected/e9108810-4ebe-41de-ad13-f23455bb516d-kube-api-access-mmtjf\") pod \"maas-api-bdbcbfbbd-cz42d\" (UID: \"e9108810-4ebe-41de-ad13-f23455bb516d\") " pod="opendatahub/maas-api-bdbcbfbbd-cz42d" Jul 06 22:29:10.243021 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:10.242883 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9108810-4ebe-41de-ad13-f23455bb516d-openshift-service-ca\") pod \"maas-api-bdbcbfbbd-cz42d\" (UID: \"e9108810-4ebe-41de-ad13-f23455bb516d\") " pod="opendatahub/maas-api-bdbcbfbbd-cz42d" Jul 06 22:29:10.243021 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:10.242905 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9108810-4ebe-41de-ad13-f23455bb516d-maas-api-tls\") pod \"maas-api-bdbcbfbbd-cz42d\" (UID: \"e9108810-4ebe-41de-ad13-f23455bb516d\") " pod="opendatahub/maas-api-bdbcbfbbd-cz42d" Jul 06 22:29:10.245633 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:10.245597 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:29:10.343941 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:10.343906 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmtjf\" (UniqueName: \"kubernetes.io/projected/e9108810-4ebe-41de-ad13-f23455bb516d-kube-api-access-mmtjf\") pod \"maas-api-bdbcbfbbd-cz42d\" (UID: \"e9108810-4ebe-41de-ad13-f23455bb516d\") " pod="opendatahub/maas-api-bdbcbfbbd-cz42d" Jul 06 22:29:10.344143 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:10.343978 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9108810-4ebe-41de-ad13-f23455bb516d-openshift-service-ca\") pod \"maas-api-bdbcbfbbd-cz42d\" (UID: \"e9108810-4ebe-41de-ad13-f23455bb516d\") " pod="opendatahub/maas-api-bdbcbfbbd-cz42d" Jul 06 22:29:10.344143 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:10.343998 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9108810-4ebe-41de-ad13-f23455bb516d-maas-api-tls\") pod \"maas-api-bdbcbfbbd-cz42d\" (UID: \"e9108810-4ebe-41de-ad13-f23455bb516d\") " pod="opendatahub/maas-api-bdbcbfbbd-cz42d" Jul 06 22:29:10.344728 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:10.344701 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9108810-4ebe-41de-ad13-f23455bb516d-openshift-service-ca\") pod \"maas-api-bdbcbfbbd-cz42d\" (UID: \"e9108810-4ebe-41de-ad13-f23455bb516d\") " pod="opendatahub/maas-api-bdbcbfbbd-cz42d" Jul 06 22:29:10.346439 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:10.346416 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9108810-4ebe-41de-ad13-f23455bb516d-maas-api-tls\") pod \"maas-api-bdbcbfbbd-cz42d\" (UID: \"e9108810-4ebe-41de-ad13-f23455bb516d\") " pod="opendatahub/maas-api-bdbcbfbbd-cz42d" Jul 06 22:29:10.355012 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:10.354989 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmtjf\" (UniqueName: \"kubernetes.io/projected/e9108810-4ebe-41de-ad13-f23455bb516d-kube-api-access-mmtjf\") pod \"maas-api-bdbcbfbbd-cz42d\" (UID: \"e9108810-4ebe-41de-ad13-f23455bb516d\") " pod="opendatahub/maas-api-bdbcbfbbd-cz42d" Jul 06 22:29:10.495516 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:10.495473 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-bdbcbfbbd-cz42d" Jul 06 22:29:10.632479 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:10.632449 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-bdbcbfbbd-cz42d"] Jul 06 22:29:10.635177 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:29:10.635146 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9108810_4ebe_41de_ad13_f23455bb516d.slice/crio-520a38e0d6a418d3b5b34112f130733586db9478f99fb9afcc2d510208743a7d WatchSource:0}: Error finding container 520a38e0d6a418d3b5b34112f130733586db9478f99fb9afcc2d510208743a7d: Status 404 returned error can't find the container with id 520a38e0d6a418d3b5b34112f130733586db9478f99fb9afcc2d510208743a7d Jul 06 22:29:10.663310 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:10.663278 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:29:10.764625 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:10.764536 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:29:10.960596 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:10.960559 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-cmgpv"] Jul 06 22:29:10.965979 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:10.965953 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-cmgpv" Jul 06 22:29:10.968778 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:10.968755 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 06 22:29:10.969030 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:10.969015 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-8b5qj\"" Jul 06 22:29:10.977053 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:10.976990 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-cmgpv"] Jul 06 22:29:11.051620 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:11.051525 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcpqx\" (UniqueName: \"kubernetes.io/projected/afb88cc6-f49f-41d0-9634-115396264273-kube-api-access-jcpqx\") pod \"payload-processing-58f86c7798-cmgpv\" (UID: \"afb88cc6-f49f-41d0-9634-115396264273\") " pod="openshift-ingress/payload-processing-58f86c7798-cmgpv" Jul 06 22:29:11.051620 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:11.051600 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afb88cc6-f49f-41d0-9634-115396264273-plugins-config-volume\") pod \"payload-processing-58f86c7798-cmgpv\" (UID: \"afb88cc6-f49f-41d0-9634-115396264273\") " pod="openshift-ingress/payload-processing-58f86c7798-cmgpv" Jul 06 22:29:11.094910 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:11.094878 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-bdbcbfbbd-cz42d" event={"ID":"e9108810-4ebe-41de-ad13-f23455bb516d","Type":"ContainerStarted","Data":"520a38e0d6a418d3b5b34112f130733586db9478f99fb9afcc2d510208743a7d"} Jul 06 22:29:11.152402 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:11.152364 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcpqx\" (UniqueName: \"kubernetes.io/projected/afb88cc6-f49f-41d0-9634-115396264273-kube-api-access-jcpqx\") pod \"payload-processing-58f86c7798-cmgpv\" (UID: \"afb88cc6-f49f-41d0-9634-115396264273\") " pod="openshift-ingress/payload-processing-58f86c7798-cmgpv" Jul 06 22:29:11.152601 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:11.152413 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afb88cc6-f49f-41d0-9634-115396264273-plugins-config-volume\") pod \"payload-processing-58f86c7798-cmgpv\" (UID: \"afb88cc6-f49f-41d0-9634-115396264273\") " pod="openshift-ingress/payload-processing-58f86c7798-cmgpv" Jul 06 22:29:11.153071 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:11.153050 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afb88cc6-f49f-41d0-9634-115396264273-plugins-config-volume\") pod \"payload-processing-58f86c7798-cmgpv\" (UID: \"afb88cc6-f49f-41d0-9634-115396264273\") " pod="openshift-ingress/payload-processing-58f86c7798-cmgpv" Jul 06 22:29:11.162166 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:11.162144 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcpqx\" (UniqueName: \"kubernetes.io/projected/afb88cc6-f49f-41d0-9634-115396264273-kube-api-access-jcpqx\") pod \"payload-processing-58f86c7798-cmgpv\" (UID: \"afb88cc6-f49f-41d0-9634-115396264273\") " pod="openshift-ingress/payload-processing-58f86c7798-cmgpv" Jul 06 22:29:11.277745 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:11.277700 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-cmgpv" Jul 06 22:29:11.430567 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:11.430529 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-cmgpv"] Jul 06 22:29:11.434347 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:29:11.434316 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafb88cc6_f49f_41d0_9634_115396264273.slice/crio-7346a55ffcf8cba09b93952b8399f0db6761e2caf9470bea1fa709886fc89454 WatchSource:0}: Error finding container 7346a55ffcf8cba09b93952b8399f0db6761e2caf9470bea1fa709886fc89454: Status 404 returned error can't find the container with id 7346a55ffcf8cba09b93952b8399f0db6761e2caf9470bea1fa709886fc89454 Jul 06 22:29:12.101009 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:12.100968 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-cmgpv" event={"ID":"afb88cc6-f49f-41d0-9634-115396264273","Type":"ContainerStarted","Data":"7346a55ffcf8cba09b93952b8399f0db6761e2caf9470bea1fa709886fc89454"} Jul 06 22:29:13.106490 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:13.106454 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-cmgpv" event={"ID":"afb88cc6-f49f-41d0-9634-115396264273","Type":"ContainerStarted","Data":"71013a18397e1d9b69240f23f9e5b08380f152d904739d5512c0709521850068"} Jul 06 22:29:13.106965 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:13.106623 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58f86c7798-cmgpv" Jul 06 22:29:13.108016 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:13.107994 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-bdbcbfbbd-cz42d" event={"ID":"e9108810-4ebe-41de-ad13-f23455bb516d","Type":"ContainerStarted","Data":"92e34e07bb59d3c50a4771deee06dddac8673580818c76ab9a672ac87bb5bcc1"} Jul 06 22:29:13.108088 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:13.108045 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-bdbcbfbbd-cz42d" Jul 06 22:29:13.127857 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:13.127794 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58f86c7798-cmgpv" podStartSLOduration=1.623553606 podStartE2EDuration="3.127777117s" totalImagesPullingTime="1.504223511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:29:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:29:13.126441403 +0000 UTC m=+560.044684272" watchObservedRunningTime="2026-07-06 22:29:13.127777117 +0000 UTC m=+560.046019973" Jul 06 22:29:13.149120 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:13.149052 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-bdbcbfbbd-cz42d" podStartSLOduration=1.567719931 podStartE2EDuration="3.149030506s" totalImagesPullingTime="1.581310575s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:29:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:29:13.146499572 +0000 UTC m=+560.064742430" watchObservedRunningTime="2026-07-06 22:29:13.149030506 +0000 UTC m=+560.067273365" Jul 06 22:29:19.118008 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:19.117973 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-bdbcbfbbd-cz42d" Jul 06 22:29:20.106327 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:20.106294 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:29:20.204101 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:20.204055 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:29:24.115043 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:24.115017 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58f86c7798-cmgpv" Jul 06 22:29:28.146711 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:28.146673 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lcllt"] Jul 06 22:29:28.150359 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:28.150332 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lcllt" Jul 06 22:29:28.153136 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:28.153107 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-j9vrr\"" Jul 06 22:29:28.153760 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:28.153740 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 06 22:29:28.154393 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:28.154370 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 06 22:29:28.154496 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:28.154440 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 06 22:29:28.162682 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:28.162659 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lcllt"] Jul 06 22:29:28.190610 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:28.190574 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:29:28.209941 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:28.209906 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d058f5dd-1fdd-49d3-aad1-0e6908b00b4f-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lcllt\" (UID: \"d058f5dd-1fdd-49d3-aad1-0e6908b00b4f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lcllt" Jul 06 22:29:28.209941 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:28.209942 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d058f5dd-1fdd-49d3-aad1-0e6908b00b4f-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lcllt\" (UID: \"d058f5dd-1fdd-49d3-aad1-0e6908b00b4f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lcllt" Jul 06 22:29:28.210159 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:28.209981 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d058f5dd-1fdd-49d3-aad1-0e6908b00b4f-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lcllt\" (UID: \"d058f5dd-1fdd-49d3-aad1-0e6908b00b4f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lcllt" Jul 06 22:29:28.210159 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:28.210003 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d058f5dd-1fdd-49d3-aad1-0e6908b00b4f-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lcllt\" (UID: \"d058f5dd-1fdd-49d3-aad1-0e6908b00b4f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lcllt" Jul 06 22:29:28.210159 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:28.210048 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbkxw\" (UniqueName: \"kubernetes.io/projected/d058f5dd-1fdd-49d3-aad1-0e6908b00b4f-kube-api-access-zbkxw\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lcllt\" (UID: \"d058f5dd-1fdd-49d3-aad1-0e6908b00b4f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lcllt" Jul 06 22:29:28.210159 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:28.210099 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d058f5dd-1fdd-49d3-aad1-0e6908b00b4f-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lcllt\" (UID: \"d058f5dd-1fdd-49d3-aad1-0e6908b00b4f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lcllt" Jul 06 22:29:28.311529 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:28.311495 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d058f5dd-1fdd-49d3-aad1-0e6908b00b4f-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lcllt\" (UID: \"d058f5dd-1fdd-49d3-aad1-0e6908b00b4f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lcllt" Jul 06 22:29:28.311529 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:28.311535 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d058f5dd-1fdd-49d3-aad1-0e6908b00b4f-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lcllt\" (UID: \"d058f5dd-1fdd-49d3-aad1-0e6908b00b4f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lcllt" Jul 06 22:29:28.311788 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:28.311581 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbkxw\" (UniqueName: \"kubernetes.io/projected/d058f5dd-1fdd-49d3-aad1-0e6908b00b4f-kube-api-access-zbkxw\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lcllt\" (UID: \"d058f5dd-1fdd-49d3-aad1-0e6908b00b4f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lcllt" Jul 06 22:29:28.311788 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:28.311605 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d058f5dd-1fdd-49d3-aad1-0e6908b00b4f-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lcllt\" (UID: \"d058f5dd-1fdd-49d3-aad1-0e6908b00b4f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lcllt" Jul 06 22:29:28.311788 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:28.311654 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d058f5dd-1fdd-49d3-aad1-0e6908b00b4f-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lcllt\" (UID: \"d058f5dd-1fdd-49d3-aad1-0e6908b00b4f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lcllt" Jul 06 22:29:28.311788 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:28.311671 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d058f5dd-1fdd-49d3-aad1-0e6908b00b4f-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lcllt\" (UID: \"d058f5dd-1fdd-49d3-aad1-0e6908b00b4f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lcllt" Jul 06 22:29:28.312151 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:28.312119 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d058f5dd-1fdd-49d3-aad1-0e6908b00b4f-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lcllt\" (UID: \"d058f5dd-1fdd-49d3-aad1-0e6908b00b4f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lcllt" Jul 06 22:29:28.312151 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:28.312140 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d058f5dd-1fdd-49d3-aad1-0e6908b00b4f-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lcllt\" (UID: \"d058f5dd-1fdd-49d3-aad1-0e6908b00b4f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lcllt" Jul 06 22:29:28.312365 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:28.312192 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d058f5dd-1fdd-49d3-aad1-0e6908b00b4f-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lcllt\" (UID: \"d058f5dd-1fdd-49d3-aad1-0e6908b00b4f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lcllt" Jul 06 22:29:28.313970 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:28.313949 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d058f5dd-1fdd-49d3-aad1-0e6908b00b4f-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lcllt\" (UID: \"d058f5dd-1fdd-49d3-aad1-0e6908b00b4f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lcllt" Jul 06 22:29:28.314417 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:28.314399 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d058f5dd-1fdd-49d3-aad1-0e6908b00b4f-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lcllt\" (UID: \"d058f5dd-1fdd-49d3-aad1-0e6908b00b4f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lcllt" Jul 06 22:29:28.323771 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:28.323739 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbkxw\" (UniqueName: \"kubernetes.io/projected/d058f5dd-1fdd-49d3-aad1-0e6908b00b4f-kube-api-access-zbkxw\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lcllt\" (UID: \"d058f5dd-1fdd-49d3-aad1-0e6908b00b4f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lcllt" Jul 06 22:29:28.462821 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:28.462728 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lcllt" Jul 06 22:29:28.612920 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:28.612885 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lcllt"] Jul 06 22:29:28.616801 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:29:28.616769 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd058f5dd_1fdd_49d3_aad1_0e6908b00b4f.slice/crio-ccd21cefd5d493caf462581e1d9a68549237d8ddc8431b629b3259f45b3858fb WatchSource:0}: Error finding container ccd21cefd5d493caf462581e1d9a68549237d8ddc8431b629b3259f45b3858fb: Status 404 returned error can't find the container with id ccd21cefd5d493caf462581e1d9a68549237d8ddc8431b629b3259f45b3858fb Jul 06 22:29:29.180961 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:29.180928 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lcllt" event={"ID":"d058f5dd-1fdd-49d3-aad1-0e6908b00b4f","Type":"ContainerStarted","Data":"ccd21cefd5d493caf462581e1d9a68549237d8ddc8431b629b3259f45b3858fb"} Jul 06 22:29:30.187490 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:30.187450 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lcllt" event={"ID":"d058f5dd-1fdd-49d3-aad1-0e6908b00b4f","Type":"ContainerStarted","Data":"ae08c9a0f314ca15e0369a0762fa549ac70238d94fced784a5f5e928b675425f"} Jul 06 22:29:30.215285 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:30.215232 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lcllt" podStartSLOduration=1.031934535 podStartE2EDuration="2.21520295s" totalImagesPullingTime="1.183268415s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:29:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:29:30.213835151 +0000 UTC m=+577.132078004" watchObservedRunningTime="2026-07-06 22:29:30.21520295 +0000 UTC m=+577.133445805" Jul 06 22:29:31.446006 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:31.445975 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:29:38.463534 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:38.463476 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lcllt" Jul 06 22:29:38.463534 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:38.463551 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lcllt" Jul 06 22:29:38.475980 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:38.475944 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lcllt" Jul 06 22:29:39.240013 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:39.239981 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lcllt" Jul 06 22:29:39.655121 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:39.655090 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-xtcpb"] Jul 06 22:29:39.658655 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:39.658635 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xtcpb" Jul 06 22:29:39.661151 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:39.661131 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 06 22:29:39.670362 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:39.670338 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-xtcpb"] Jul 06 22:29:39.674194 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:39.674167 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:29:39.813683 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:39.813631 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbe2b1ea-53bf-45a2-8cef-71f2afa45cde-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xtcpb\" (UID: \"fbe2b1ea-53bf-45a2-8cef-71f2afa45cde\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xtcpb" Jul 06 22:29:39.813683 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:39.813682 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbe2b1ea-53bf-45a2-8cef-71f2afa45cde-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xtcpb\" (UID: \"fbe2b1ea-53bf-45a2-8cef-71f2afa45cde\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xtcpb" Jul 06 22:29:39.813961 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:39.813799 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbe2b1ea-53bf-45a2-8cef-71f2afa45cde-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xtcpb\" (UID: \"fbe2b1ea-53bf-45a2-8cef-71f2afa45cde\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xtcpb" Jul 06 22:29:39.813961 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:39.813871 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbe2b1ea-53bf-45a2-8cef-71f2afa45cde-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xtcpb\" (UID: \"fbe2b1ea-53bf-45a2-8cef-71f2afa45cde\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xtcpb" Jul 06 22:29:39.813961 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:39.813899 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbe2b1ea-53bf-45a2-8cef-71f2afa45cde-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xtcpb\" (UID: \"fbe2b1ea-53bf-45a2-8cef-71f2afa45cde\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xtcpb" Jul 06 22:29:39.813961 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:39.813940 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgfw6\" (UniqueName: \"kubernetes.io/projected/fbe2b1ea-53bf-45a2-8cef-71f2afa45cde-kube-api-access-tgfw6\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xtcpb\" (UID: \"fbe2b1ea-53bf-45a2-8cef-71f2afa45cde\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xtcpb" Jul 06 22:29:39.914889 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:39.914774 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbe2b1ea-53bf-45a2-8cef-71f2afa45cde-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xtcpb\" (UID: \"fbe2b1ea-53bf-45a2-8cef-71f2afa45cde\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xtcpb" Jul 06 22:29:39.914889 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:39.914826 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbe2b1ea-53bf-45a2-8cef-71f2afa45cde-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xtcpb\" (UID: \"fbe2b1ea-53bf-45a2-8cef-71f2afa45cde\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xtcpb" Jul 06 22:29:39.914889 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:39.914863 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbe2b1ea-53bf-45a2-8cef-71f2afa45cde-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xtcpb\" (UID: \"fbe2b1ea-53bf-45a2-8cef-71f2afa45cde\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xtcpb" Jul 06 22:29:39.914889 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:39.914890 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbe2b1ea-53bf-45a2-8cef-71f2afa45cde-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xtcpb\" (UID: \"fbe2b1ea-53bf-45a2-8cef-71f2afa45cde\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xtcpb" Jul 06 22:29:39.915274 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:39.914906 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbe2b1ea-53bf-45a2-8cef-71f2afa45cde-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xtcpb\" (UID: \"fbe2b1ea-53bf-45a2-8cef-71f2afa45cde\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xtcpb" Jul 06 22:29:39.915274 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:39.915068 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgfw6\" (UniqueName: \"kubernetes.io/projected/fbe2b1ea-53bf-45a2-8cef-71f2afa45cde-kube-api-access-tgfw6\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xtcpb\" (UID: \"fbe2b1ea-53bf-45a2-8cef-71f2afa45cde\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xtcpb" Jul 06 22:29:39.915379 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:39.915297 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbe2b1ea-53bf-45a2-8cef-71f2afa45cde-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xtcpb\" (UID: \"fbe2b1ea-53bf-45a2-8cef-71f2afa45cde\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xtcpb" Jul 06 22:29:39.915417 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:39.915371 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbe2b1ea-53bf-45a2-8cef-71f2afa45cde-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xtcpb\" (UID: \"fbe2b1ea-53bf-45a2-8cef-71f2afa45cde\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xtcpb" Jul 06 22:29:39.915417 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:39.915366 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbe2b1ea-53bf-45a2-8cef-71f2afa45cde-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xtcpb\" (UID: \"fbe2b1ea-53bf-45a2-8cef-71f2afa45cde\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xtcpb" Jul 06 22:29:39.917282 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:39.917264 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbe2b1ea-53bf-45a2-8cef-71f2afa45cde-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xtcpb\" (UID: \"fbe2b1ea-53bf-45a2-8cef-71f2afa45cde\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xtcpb" Jul 06 22:29:39.917470 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:39.917451 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbe2b1ea-53bf-45a2-8cef-71f2afa45cde-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xtcpb\" (UID: \"fbe2b1ea-53bf-45a2-8cef-71f2afa45cde\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xtcpb" Jul 06 22:29:39.925965 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:39.925933 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgfw6\" (UniqueName: \"kubernetes.io/projected/fbe2b1ea-53bf-45a2-8cef-71f2afa45cde-kube-api-access-tgfw6\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xtcpb\" (UID: \"fbe2b1ea-53bf-45a2-8cef-71f2afa45cde\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xtcpb" Jul 06 22:29:39.969367 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:39.969333 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xtcpb" Jul 06 22:29:40.113713 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:40.113688 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-xtcpb"] Jul 06 22:29:40.116145 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:29:40.116096 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbe2b1ea_53bf_45a2_8cef_71f2afa45cde.slice/crio-de572b51f943133ec771d66393afef1831c05702c8e3c16ae6559477bd5dcc02 WatchSource:0}: Error finding container de572b51f943133ec771d66393afef1831c05702c8e3c16ae6559477bd5dcc02: Status 404 returned error can't find the container with id de572b51f943133ec771d66393afef1831c05702c8e3c16ae6559477bd5dcc02 Jul 06 22:29:40.233651 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:40.233546 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xtcpb" event={"ID":"fbe2b1ea-53bf-45a2-8cef-71f2afa45cde","Type":"ContainerStarted","Data":"de572b51f943133ec771d66393afef1831c05702c8e3c16ae6559477bd5dcc02"} Jul 06 22:29:41.240179 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:41.240145 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xtcpb" event={"ID":"fbe2b1ea-53bf-45a2-8cef-71f2afa45cde","Type":"ContainerStarted","Data":"1e3198c5affee2a89a1111592619ea390eadd004d6e94c856117603af70ffa4b"} Jul 06 22:29:41.261535 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:41.261478 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xtcpb" podStartSLOduration=2.083867103 podStartE2EDuration="2.261458424s" totalImagesPullingTime="177.591321ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:29:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:29:41.260242407 +0000 UTC m=+588.178485264" watchObservedRunningTime="2026-07-06 22:29:41.261458424 +0000 UTC m=+588.179701279" Jul 06 22:29:49.970187 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:49.970138 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xtcpb" Jul 06 22:29:49.970645 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:49.970243 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xtcpb" Jul 06 22:29:49.982836 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:49.982808 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xtcpb" Jul 06 22:29:50.173125 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:50.173082 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:29:50.292542 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:50.292465 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xtcpb" Jul 06 22:29:53.531054 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:53.531025 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-4x6pf_5796a8ea-adcb-4819-bb18-0b85e1ef4681/console-operator/2.log" Jul 06 22:29:53.532147 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:53.532118 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-4x6pf_5796a8ea-adcb-4819-bb18-0b85e1ef4681/console-operator/2.log" Jul 06 22:29:53.535151 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:53.535126 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qbbrs_bf23ef00-a2d0-499c-8c87-5d6b98510ea0/ovn-acl-logging/0.log" Jul 06 22:29:53.536201 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:53.536183 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qbbrs_bf23ef00-a2d0-499c-8c87-5d6b98510ea0/ovn-acl-logging/0.log" Jul 06 22:29:57.378000 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:29:57.377965 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:30:00.141759 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:30:00.141719 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29722950-vjkjj"] Jul 06 22:30:00.145313 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:30:00.145289 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722950-vjkjj" Jul 06 22:30:00.148787 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:30:00.148765 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-98x7w\"" Jul 06 22:30:00.156452 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:30:00.156427 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722950-vjkjj"] Jul 06 22:30:00.193607 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:30:00.193568 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c7dc\" (UniqueName: \"kubernetes.io/projected/3491aced-8662-4105-b094-476506976c45-kube-api-access-8c7dc\") pod \"maas-api-key-cleanup-29722950-vjkjj\" (UID: \"3491aced-8662-4105-b094-476506976c45\") " pod="opendatahub/maas-api-key-cleanup-29722950-vjkjj" Jul 06 22:30:00.294236 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:30:00.294172 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c7dc\" (UniqueName: \"kubernetes.io/projected/3491aced-8662-4105-b094-476506976c45-kube-api-access-8c7dc\") pod \"maas-api-key-cleanup-29722950-vjkjj\" (UID: \"3491aced-8662-4105-b094-476506976c45\") " pod="opendatahub/maas-api-key-cleanup-29722950-vjkjj" Jul 06 22:30:00.311158 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:30:00.311118 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c7dc\" (UniqueName: \"kubernetes.io/projected/3491aced-8662-4105-b094-476506976c45-kube-api-access-8c7dc\") pod \"maas-api-key-cleanup-29722950-vjkjj\" (UID: \"3491aced-8662-4105-b094-476506976c45\") " pod="opendatahub/maas-api-key-cleanup-29722950-vjkjj" Jul 06 22:30:00.455981 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:30:00.455894 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722950-vjkjj" Jul 06 22:30:00.594305 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:30:00.594274 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722950-vjkjj"] Jul 06 22:30:00.597089 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:30:00.597060 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3491aced_8662_4105_b094_476506976c45.slice/crio-0a285d51f59c936922fe822911ed83e8abd33584c34dd857982e756299d671c0 WatchSource:0}: Error finding container 0a285d51f59c936922fe822911ed83e8abd33584c34dd857982e756299d671c0: Status 404 returned error can't find the container with id 0a285d51f59c936922fe822911ed83e8abd33584c34dd857982e756299d671c0 Jul 06 22:30:00.599317 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:30:00.599294 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 22:30:01.334755 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:30:01.334701 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722950-vjkjj" event={"ID":"3491aced-8662-4105-b094-476506976c45","Type":"ContainerStarted","Data":"0a285d51f59c936922fe822911ed83e8abd33584c34dd857982e756299d671c0"} Jul 06 22:30:02.057958 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:30:02.057914 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:30:03.347064 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:30:03.346981 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722950-vjkjj" event={"ID":"3491aced-8662-4105-b094-476506976c45","Type":"ContainerStarted","Data":"15a91d169b0f2935c38e961a2cde9b804769913c4b6f8f490b5779fa8a37ceff"} Jul 06 22:30:03.367866 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:30:03.367801 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-29722950-vjkjj" podStartSLOduration=1.279410531 podStartE2EDuration="3.367781183s" totalImagesPullingTime="2.088370652s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:30:03.36667339 +0000 UTC m=+610.284916237" watchObservedRunningTime="2026-07-06 22:30:03.367781183 +0000 UTC m=+610.286024043" Jul 06 22:30:04.353486 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:30:04.353451 2571 generic.go:356] "Generic (PLEG): container finished" podID="3491aced-8662-4105-b094-476506976c45" containerID="15a91d169b0f2935c38e961a2cde9b804769913c4b6f8f490b5779fa8a37ceff" exitCode=0 Jul 06 22:30:04.353885 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:30:04.353538 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722950-vjkjj" event={"ID":"3491aced-8662-4105-b094-476506976c45","Type":"ContainerDied","Data":"15a91d169b0f2935c38e961a2cde9b804769913c4b6f8f490b5779fa8a37ceff"} Jul 06 22:30:05.490765 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:30:05.490733 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722950-vjkjj" Jul 06 22:30:05.541825 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:30:05.541790 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8c7dc\" (UniqueName: \"kubernetes.io/projected/3491aced-8662-4105-b094-476506976c45-kube-api-access-8c7dc\") pod \"3491aced-8662-4105-b094-476506976c45\" (UID: \"3491aced-8662-4105-b094-476506976c45\") " Jul 06 22:30:05.544028 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:30:05.543993 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3491aced-8662-4105-b094-476506976c45-kube-api-access-8c7dc" (OuterVolumeSpecName: "kube-api-access-8c7dc") pod "3491aced-8662-4105-b094-476506976c45" (UID: "3491aced-8662-4105-b094-476506976c45"). InnerVolumeSpecName "kube-api-access-8c7dc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:30:05.642943 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:30:05.642906 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c7dc\" (UniqueName: \"kubernetes.io/projected/3491aced-8662-4105-b094-476506976c45-kube-api-access-8c7dc\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:30:06.364104 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:30:06.364068 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722950-vjkjj" event={"ID":"3491aced-8662-4105-b094-476506976c45","Type":"ContainerDied","Data":"0a285d51f59c936922fe822911ed83e8abd33584c34dd857982e756299d671c0"} Jul 06 22:30:06.364104 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:30:06.364100 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722950-vjkjj" Jul 06 22:30:06.364602 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:30:06.364126 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a285d51f59c936922fe822911ed83e8abd33584c34dd857982e756299d671c0" Jul 06 22:31:17.552517 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:31:17.552480 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d7d6f976b-rgjjr"] Jul 06 22:31:17.552996 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:31:17.552842 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3491aced-8662-4105-b094-476506976c45" containerName="cleanup" Jul 06 22:31:17.552996 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:31:17.552854 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="3491aced-8662-4105-b094-476506976c45" containerName="cleanup" Jul 06 22:31:17.552996 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:31:17.552917 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="3491aced-8662-4105-b094-476506976c45" containerName="cleanup" Jul 06 22:31:17.555999 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:31:17.555982 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d7d6f976b-rgjjr" Jul 06 22:31:17.559562 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:31:17.559541 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 06 22:31:17.559702 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:31:17.559541 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-f9fwm\"" Jul 06 22:31:17.566794 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:31:17.566765 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d7d6f976b-rgjjr"] Jul 06 22:31:17.585912 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:31:17.585862 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r78hg\" (UniqueName: \"kubernetes.io/projected/e9e55f7f-183e-4f3b-975c-188c209a8df7-kube-api-access-r78hg\") pod \"authorino-6d7d6f976b-rgjjr\" (UID: \"e9e55f7f-183e-4f3b-975c-188c209a8df7\") " pod="kuadrant-system/authorino-6d7d6f976b-rgjjr" Jul 06 22:31:17.586100 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:31:17.585970 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e9e55f7f-183e-4f3b-975c-188c209a8df7-tls-cert\") pod \"authorino-6d7d6f976b-rgjjr\" (UID: \"e9e55f7f-183e-4f3b-975c-188c209a8df7\") " pod="kuadrant-system/authorino-6d7d6f976b-rgjjr" Jul 06 22:31:17.687033 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:31:17.686993 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r78hg\" (UniqueName: \"kubernetes.io/projected/e9e55f7f-183e-4f3b-975c-188c209a8df7-kube-api-access-r78hg\") pod \"authorino-6d7d6f976b-rgjjr\" (UID: \"e9e55f7f-183e-4f3b-975c-188c209a8df7\") " pod="kuadrant-system/authorino-6d7d6f976b-rgjjr" Jul 06 22:31:17.687258 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:31:17.687074 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e9e55f7f-183e-4f3b-975c-188c209a8df7-tls-cert\") pod \"authorino-6d7d6f976b-rgjjr\" (UID: \"e9e55f7f-183e-4f3b-975c-188c209a8df7\") " pod="kuadrant-system/authorino-6d7d6f976b-rgjjr" Jul 06 22:31:17.689710 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:31:17.689683 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e9e55f7f-183e-4f3b-975c-188c209a8df7-tls-cert\") pod \"authorino-6d7d6f976b-rgjjr\" (UID: \"e9e55f7f-183e-4f3b-975c-188c209a8df7\") " pod="kuadrant-system/authorino-6d7d6f976b-rgjjr" Jul 06 22:31:17.704400 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:31:17.704362 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r78hg\" (UniqueName: \"kubernetes.io/projected/e9e55f7f-183e-4f3b-975c-188c209a8df7-kube-api-access-r78hg\") pod \"authorino-6d7d6f976b-rgjjr\" (UID: \"e9e55f7f-183e-4f3b-975c-188c209a8df7\") " pod="kuadrant-system/authorino-6d7d6f976b-rgjjr" Jul 06 22:31:17.732029 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:31:17.731991 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:31:17.866278 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:31:17.866173 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d7d6f976b-rgjjr" Jul 06 22:31:18.010661 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:31:18.010626 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d7d6f976b-rgjjr"] Jul 06 22:31:18.014092 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:31:18.014049 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode9e55f7f_183e_4f3b_975c_188c209a8df7.slice/crio-165914ce48d7a42a19d7ed233f108bc2a7a011e9190c8a406ebbeca34bb32b19 WatchSource:0}: Error finding container 165914ce48d7a42a19d7ed233f108bc2a7a011e9190c8a406ebbeca34bb32b19: Status 404 returned error can't find the container with id 165914ce48d7a42a19d7ed233f108bc2a7a011e9190c8a406ebbeca34bb32b19 Jul 06 22:31:18.691409 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:31:18.691369 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d7d6f976b-rgjjr" event={"ID":"e9e55f7f-183e-4f3b-975c-188c209a8df7","Type":"ContainerStarted","Data":"72395bac3d3c6b0abe1d0d51970db770d34d01a7410423e96aae28525ad489c2"} Jul 06 22:31:18.691848 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:31:18.691423 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d7d6f976b-rgjjr" event={"ID":"e9e55f7f-183e-4f3b-975c-188c209a8df7","Type":"ContainerStarted","Data":"165914ce48d7a42a19d7ed233f108bc2a7a011e9190c8a406ebbeca34bb32b19"} Jul 06 22:31:18.713729 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:31:18.713661 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6d7d6f976b-rgjjr" podStartSLOduration=1.359627918 podStartE2EDuration="1.713638388s" totalImagesPullingTime="354.01047ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:31:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:31:18.712926697 +0000 UTC m=+685.631169567" watchObservedRunningTime="2026-07-06 22:31:18.713638388 +0000 UTC m=+685.631881247" Jul 06 22:31:22.632926 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:31:22.632893 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:31:22.904078 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:31:22.903969 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:31:23.000947 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:31:23.000913 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:32:29.536253 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:32:29.536195 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:32:29.634881 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:32:29.634849 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5b4776d6c4-hb4m6"] Jul 06 22:32:29.635068 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:32:29.635044 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-5b4776d6c4-hb4m6" podUID="bf8e72e9-cae2-4d09-a789-0ff3cee3af17" containerName="manager" containerID="cri-o://45b932ad762ad8063cf5ace7b229fba81da940918271311bed3b3e7a6274dcd4" gracePeriod=30 Jul 06 22:32:29.886325 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:32:29.886293 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b4776d6c4-hb4m6" Jul 06 22:32:30.023328 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:32:30.023296 2571 generic.go:356] "Generic (PLEG): container finished" podID="bf8e72e9-cae2-4d09-a789-0ff3cee3af17" containerID="45b932ad762ad8063cf5ace7b229fba81da940918271311bed3b3e7a6274dcd4" exitCode=0 Jul 06 22:32:30.023506 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:32:30.023335 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b4776d6c4-hb4m6" event={"ID":"bf8e72e9-cae2-4d09-a789-0ff3cee3af17","Type":"ContainerDied","Data":"45b932ad762ad8063cf5ace7b229fba81da940918271311bed3b3e7a6274dcd4"} Jul 06 22:32:30.023506 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:32:30.023346 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b4776d6c4-hb4m6" Jul 06 22:32:30.023506 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:32:30.023378 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b4776d6c4-hb4m6" event={"ID":"bf8e72e9-cae2-4d09-a789-0ff3cee3af17","Type":"ContainerDied","Data":"e043822ba2d2120a4ba65101ef9c44f0175c20a27641b55c36285a9608d77039"} Jul 06 22:32:30.023506 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:32:30.023400 2571 scope.go:117] "RemoveContainer" containerID="45b932ad762ad8063cf5ace7b229fba81da940918271311bed3b3e7a6274dcd4" Jul 06 22:32:30.031642 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:32:30.031624 2571 scope.go:117] "RemoveContainer" containerID="45b932ad762ad8063cf5ace7b229fba81da940918271311bed3b3e7a6274dcd4" Jul 06 22:32:30.031918 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:32:30.031891 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45b932ad762ad8063cf5ace7b229fba81da940918271311bed3b3e7a6274dcd4\": container with ID starting with 45b932ad762ad8063cf5ace7b229fba81da940918271311bed3b3e7a6274dcd4 not found: ID does not exist" containerID="45b932ad762ad8063cf5ace7b229fba81da940918271311bed3b3e7a6274dcd4" Jul 06 22:32:30.031991 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:32:30.031931 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45b932ad762ad8063cf5ace7b229fba81da940918271311bed3b3e7a6274dcd4"} err="failed to get container status \"45b932ad762ad8063cf5ace7b229fba81da940918271311bed3b3e7a6274dcd4\": rpc error: code = NotFound desc = could not find container \"45b932ad762ad8063cf5ace7b229fba81da940918271311bed3b3e7a6274dcd4\": container with ID starting with 45b932ad762ad8063cf5ace7b229fba81da940918271311bed3b3e7a6274dcd4 not found: ID does not exist" Jul 06 22:32:30.055773 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:32:30.055690 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c4xqn\" (UniqueName: \"kubernetes.io/projected/bf8e72e9-cae2-4d09-a789-0ff3cee3af17-kube-api-access-c4xqn\") pod \"bf8e72e9-cae2-4d09-a789-0ff3cee3af17\" (UID: \"bf8e72e9-cae2-4d09-a789-0ff3cee3af17\") " Jul 06 22:32:30.055908 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:32:30.055776 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bf8e72e9-cae2-4d09-a789-0ff3cee3af17-webhook-cert\") pod \"bf8e72e9-cae2-4d09-a789-0ff3cee3af17\" (UID: \"bf8e72e9-cae2-4d09-a789-0ff3cee3af17\") " Jul 06 22:32:30.057893 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:32:30.057862 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf8e72e9-cae2-4d09-a789-0ff3cee3af17-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "bf8e72e9-cae2-4d09-a789-0ff3cee3af17" (UID: "bf8e72e9-cae2-4d09-a789-0ff3cee3af17"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:32:30.057893 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:32:30.057879 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf8e72e9-cae2-4d09-a789-0ff3cee3af17-kube-api-access-c4xqn" (OuterVolumeSpecName: "kube-api-access-c4xqn") pod "bf8e72e9-cae2-4d09-a789-0ff3cee3af17" (UID: "bf8e72e9-cae2-4d09-a789-0ff3cee3af17"). InnerVolumeSpecName "kube-api-access-c4xqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:32:30.157296 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:32:30.157261 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4xqn\" (UniqueName: \"kubernetes.io/projected/bf8e72e9-cae2-4d09-a789-0ff3cee3af17-kube-api-access-c4xqn\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:32:30.157296 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:32:30.157289 2571 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bf8e72e9-cae2-4d09-a789-0ff3cee3af17-webhook-cert\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:32:30.347813 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:32:30.347776 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5b4776d6c4-hb4m6"] Jul 06 22:32:30.352793 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:32:30.352767 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5b4776d6c4-hb4m6"] Jul 06 22:32:31.024049 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:32:31.024008 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:32:31.133566 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:32:31.133528 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5b4776d6c4-8zkq6"] Jul 06 22:32:31.133911 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:32:31.133898 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf8e72e9-cae2-4d09-a789-0ff3cee3af17" containerName="manager" Jul 06 22:32:31.133957 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:32:31.133912 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf8e72e9-cae2-4d09-a789-0ff3cee3af17" containerName="manager" Jul 06 22:32:31.133992 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:32:31.133974 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf8e72e9-cae2-4d09-a789-0ff3cee3af17" containerName="manager" Jul 06 22:32:31.138202 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:32:31.138180 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b4776d6c4-8zkq6" Jul 06 22:32:31.140554 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:32:31.140533 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 06 22:32:31.141233 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:32:31.141197 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 06 22:32:31.141302 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:32:31.141221 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-mgg8c\"" Jul 06 22:32:31.147153 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:32:31.147127 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b4776d6c4-8zkq6"] Jul 06 22:32:31.266941 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:32:31.266898 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/17688e6e-694d-424c-a7b7-71f9d4cda9fe-webhook-cert\") pod \"maas-controller-5b4776d6c4-8zkq6\" (UID: \"17688e6e-694d-424c-a7b7-71f9d4cda9fe\") " pod="opendatahub/maas-controller-5b4776d6c4-8zkq6" Jul 06 22:32:31.267125 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:32:31.266971 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkbhs\" (UniqueName: \"kubernetes.io/projected/17688e6e-694d-424c-a7b7-71f9d4cda9fe-kube-api-access-rkbhs\") pod \"maas-controller-5b4776d6c4-8zkq6\" (UID: \"17688e6e-694d-424c-a7b7-71f9d4cda9fe\") " pod="opendatahub/maas-controller-5b4776d6c4-8zkq6" Jul 06 22:32:31.367574 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:32:31.367481 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/17688e6e-694d-424c-a7b7-71f9d4cda9fe-webhook-cert\") pod \"maas-controller-5b4776d6c4-8zkq6\" (UID: \"17688e6e-694d-424c-a7b7-71f9d4cda9fe\") " pod="opendatahub/maas-controller-5b4776d6c4-8zkq6" Jul 06 22:32:31.367574 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:32:31.367521 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkbhs\" (UniqueName: \"kubernetes.io/projected/17688e6e-694d-424c-a7b7-71f9d4cda9fe-kube-api-access-rkbhs\") pod \"maas-controller-5b4776d6c4-8zkq6\" (UID: \"17688e6e-694d-424c-a7b7-71f9d4cda9fe\") " pod="opendatahub/maas-controller-5b4776d6c4-8zkq6" Jul 06 22:32:31.369942 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:32:31.369909 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/17688e6e-694d-424c-a7b7-71f9d4cda9fe-webhook-cert\") pod \"maas-controller-5b4776d6c4-8zkq6\" (UID: \"17688e6e-694d-424c-a7b7-71f9d4cda9fe\") " pod="opendatahub/maas-controller-5b4776d6c4-8zkq6" Jul 06 22:32:31.377166 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:32:31.377143 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkbhs\" (UniqueName: \"kubernetes.io/projected/17688e6e-694d-424c-a7b7-71f9d4cda9fe-kube-api-access-rkbhs\") pod \"maas-controller-5b4776d6c4-8zkq6\" (UID: \"17688e6e-694d-424c-a7b7-71f9d4cda9fe\") " pod="opendatahub/maas-controller-5b4776d6c4-8zkq6" Jul 06 22:32:31.451375 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:32:31.451330 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b4776d6c4-8zkq6" Jul 06 22:32:31.590262 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:32:31.590233 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b4776d6c4-8zkq6"] Jul 06 22:32:31.592390 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:32:31.592348 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17688e6e_694d_424c_a7b7_71f9d4cda9fe.slice/crio-0aaace6a5ab01a83efc933671664dd0d9066021c08d08aa31c68f4752688978f WatchSource:0}: Error finding container 0aaace6a5ab01a83efc933671664dd0d9066021c08d08aa31c68f4752688978f: Status 404 returned error can't find the container with id 0aaace6a5ab01a83efc933671664dd0d9066021c08d08aa31c68f4752688978f Jul 06 22:32:31.634828 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:32:31.634794 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf8e72e9-cae2-4d09-a789-0ff3cee3af17" path="/var/lib/kubelet/pods/bf8e72e9-cae2-4d09-a789-0ff3cee3af17/volumes" Jul 06 22:32:32.036458 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:32:32.036421 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b4776d6c4-8zkq6" event={"ID":"17688e6e-694d-424c-a7b7-71f9d4cda9fe","Type":"ContainerStarted","Data":"5517e69abacc29636b3c8cfccbdf1ffda47d70d025863d9df1181c80e9d8c9e8"} Jul 06 22:32:32.036914 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:32:32.036485 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b4776d6c4-8zkq6" event={"ID":"17688e6e-694d-424c-a7b7-71f9d4cda9fe","Type":"ContainerStarted","Data":"0aaace6a5ab01a83efc933671664dd0d9066021c08d08aa31c68f4752688978f"} Jul 06 22:32:32.036914 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:32:32.036540 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5b4776d6c4-8zkq6" Jul 06 22:32:32.057909 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:32:32.057830 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5b4776d6c4-8zkq6" podStartSLOduration=0.688862925 podStartE2EDuration="1.057806385s" totalImagesPullingTime="368.94346ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:32:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:32:32.055950121 +0000 UTC m=+758.974192990" watchObservedRunningTime="2026-07-06 22:32:32.057806385 +0000 UTC m=+758.976049245" Jul 06 22:32:34.837423 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:32:34.837385 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:32:43.043037 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:32:43.043001 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:32:43.047024 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:32:43.046999 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5b4776d6c4-8zkq6" Jul 06 22:32:43.144074 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:32:43.144037 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:32:46.246621 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:32:46.246580 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:33:11.567447 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:33:11.567412 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:33:11.664521 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:33:11.664486 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:33:11.786719 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:33:11.786677 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:33:17.809769 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:33:17.809687 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:33:17.863284 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:33:17.863242 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:33:17.942194 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:33:17.942149 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:33:18.039984 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:33:18.039953 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:33:18.154178 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:33:18.154144 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:33:24.480000 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:33:24.479970 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:33:24.538235 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:33:24.538180 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:33:37.443913 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:33:37.443831 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:34:21.151423 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:34:21.151378 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:34:21.252325 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:34:21.252286 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:34:33.794138 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:34:33.794100 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:34:53.567485 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:34:53.567458 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-4x6pf_5796a8ea-adcb-4819-bb18-0b85e1ef4681/console-operator/2.log" Jul 06 22:34:53.568024 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:34:53.567663 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-4x6pf_5796a8ea-adcb-4819-bb18-0b85e1ef4681/console-operator/2.log" Jul 06 22:34:53.570866 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:34:53.570844 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qbbrs_bf23ef00-a2d0-499c-8c87-5d6b98510ea0/ovn-acl-logging/0.log" Jul 06 22:34:53.571216 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:34:53.571187 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qbbrs_bf23ef00-a2d0-499c-8c87-5d6b98510ea0/ovn-acl-logging/0.log" Jul 06 22:37:13.721871 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:37:13.721831 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hxnqr"] Jul 06 22:37:13.723911 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:37:13.722051 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hxnqr" podUID="ef6cd197-77c7-4335-9f33-02ba163be40c" containerName="manager" containerID="cri-o://00fab0ae896d7cecf4ec19c571f6cb049aac7845644eeb130d70d46270c2813a" gracePeriod=10 Jul 06 22:37:16.416368 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:37:16.416313 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:37:17.572870 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:37:17.572814 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:37:17.579669 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:37:17.579638 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hxnqr" Jul 06 22:37:17.739836 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:37:17.739801 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ef6cd197-77c7-4335-9f33-02ba163be40c-extensions-socket-volume\") pod \"ef6cd197-77c7-4335-9f33-02ba163be40c\" (UID: \"ef6cd197-77c7-4335-9f33-02ba163be40c\") " Jul 06 22:37:17.740035 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:37:17.739873 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2js5c\" (UniqueName: \"kubernetes.io/projected/ef6cd197-77c7-4335-9f33-02ba163be40c-kube-api-access-2js5c\") pod \"ef6cd197-77c7-4335-9f33-02ba163be40c\" (UID: \"ef6cd197-77c7-4335-9f33-02ba163be40c\") " Jul 06 22:37:17.740285 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:37:17.740262 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef6cd197-77c7-4335-9f33-02ba163be40c-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ef6cd197-77c7-4335-9f33-02ba163be40c" (UID: "ef6cd197-77c7-4335-9f33-02ba163be40c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:37:17.742102 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:37:17.742068 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef6cd197-77c7-4335-9f33-02ba163be40c-kube-api-access-2js5c" (OuterVolumeSpecName: "kube-api-access-2js5c") pod "ef6cd197-77c7-4335-9f33-02ba163be40c" (UID: "ef6cd197-77c7-4335-9f33-02ba163be40c"). InnerVolumeSpecName "kube-api-access-2js5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:37:17.840679 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:37:17.840638 2571 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ef6cd197-77c7-4335-9f33-02ba163be40c-extensions-socket-volume\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:37:17.840679 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:37:17.840670 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2js5c\" (UniqueName: \"kubernetes.io/projected/ef6cd197-77c7-4335-9f33-02ba163be40c-kube-api-access-2js5c\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:37:18.384201 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:37:18.384169 2571 generic.go:356] "Generic (PLEG): container finished" podID="ef6cd197-77c7-4335-9f33-02ba163be40c" containerID="00fab0ae896d7cecf4ec19c571f6cb049aac7845644eeb130d70d46270c2813a" exitCode=0 Jul 06 22:37:18.384389 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:37:18.384237 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hxnqr" Jul 06 22:37:18.384389 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:37:18.384258 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hxnqr" event={"ID":"ef6cd197-77c7-4335-9f33-02ba163be40c","Type":"ContainerDied","Data":"00fab0ae896d7cecf4ec19c571f6cb049aac7845644eeb130d70d46270c2813a"} Jul 06 22:37:18.384389 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:37:18.384313 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hxnqr" event={"ID":"ef6cd197-77c7-4335-9f33-02ba163be40c","Type":"ContainerDied","Data":"81d775d08c0d76f0e1dbf4675d29a170b4739362f3f80243b37c1f04e25df9c8"} Jul 06 22:37:18.384389 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:37:18.384337 2571 scope.go:117] "RemoveContainer" containerID="00fab0ae896d7cecf4ec19c571f6cb049aac7845644eeb130d70d46270c2813a" Jul 06 22:37:18.394378 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:37:18.394357 2571 scope.go:117] "RemoveContainer" containerID="00fab0ae896d7cecf4ec19c571f6cb049aac7845644eeb130d70d46270c2813a" Jul 06 22:37:18.394713 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:37:18.394693 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00fab0ae896d7cecf4ec19c571f6cb049aac7845644eeb130d70d46270c2813a\": container with ID starting with 00fab0ae896d7cecf4ec19c571f6cb049aac7845644eeb130d70d46270c2813a not found: ID does not exist" containerID="00fab0ae896d7cecf4ec19c571f6cb049aac7845644eeb130d70d46270c2813a" Jul 06 22:37:18.394762 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:37:18.394722 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00fab0ae896d7cecf4ec19c571f6cb049aac7845644eeb130d70d46270c2813a"} err="failed to get container status \"00fab0ae896d7cecf4ec19c571f6cb049aac7845644eeb130d70d46270c2813a\": rpc error: code = NotFound desc = could not find container \"00fab0ae896d7cecf4ec19c571f6cb049aac7845644eeb130d70d46270c2813a\": container with ID starting with 00fab0ae896d7cecf4ec19c571f6cb049aac7845644eeb130d70d46270c2813a not found: ID does not exist" Jul 06 22:37:18.407182 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:37:18.407149 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hxnqr"] Jul 06 22:37:18.414131 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:37:18.414105 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hxnqr"] Jul 06 22:37:19.634007 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:37:19.633970 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef6cd197-77c7-4335-9f33-02ba163be40c" path="/var/lib/kubelet/pods/ef6cd197-77c7-4335-9f33-02ba163be40c/volumes" Jul 06 22:38:14.856881 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:38:14.856847 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ng78j"] Jul 06 22:38:14.857383 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:38:14.857251 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef6cd197-77c7-4335-9f33-02ba163be40c" containerName="manager" Jul 06 22:38:14.857383 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:38:14.857263 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef6cd197-77c7-4335-9f33-02ba163be40c" containerName="manager" Jul 06 22:38:14.857383 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:38:14.857323 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef6cd197-77c7-4335-9f33-02ba163be40c" containerName="manager" Jul 06 22:38:14.861666 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:38:14.861640 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ng78j" Jul 06 22:38:14.863967 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:38:14.863948 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-kpqhd\"" Jul 06 22:38:14.871275 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:38:14.871245 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ng78j"] Jul 06 22:38:14.940868 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:38:14.940825 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29h54\" (UniqueName: \"kubernetes.io/projected/5eadc65a-d34e-458a-aa84-5f1a08b89cfc-kube-api-access-29h54\") pod \"kuadrant-operator-controller-manager-86878c769f-ng78j\" (UID: \"5eadc65a-d34e-458a-aa84-5f1a08b89cfc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ng78j" Jul 06 22:38:14.941054 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:38:14.940969 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5eadc65a-d34e-458a-aa84-5f1a08b89cfc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-ng78j\" (UID: \"5eadc65a-d34e-458a-aa84-5f1a08b89cfc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ng78j" Jul 06 22:38:15.041903 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:38:15.041854 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5eadc65a-d34e-458a-aa84-5f1a08b89cfc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-ng78j\" (UID: \"5eadc65a-d34e-458a-aa84-5f1a08b89cfc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ng78j" Jul 06 22:38:15.042097 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:38:15.041923 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29h54\" (UniqueName: \"kubernetes.io/projected/5eadc65a-d34e-458a-aa84-5f1a08b89cfc-kube-api-access-29h54\") pod \"kuadrant-operator-controller-manager-86878c769f-ng78j\" (UID: \"5eadc65a-d34e-458a-aa84-5f1a08b89cfc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ng78j" Jul 06 22:38:15.042325 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:38:15.042304 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5eadc65a-d34e-458a-aa84-5f1a08b89cfc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-ng78j\" (UID: \"5eadc65a-d34e-458a-aa84-5f1a08b89cfc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ng78j" Jul 06 22:38:15.055156 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:38:15.055129 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29h54\" (UniqueName: \"kubernetes.io/projected/5eadc65a-d34e-458a-aa84-5f1a08b89cfc-kube-api-access-29h54\") pod \"kuadrant-operator-controller-manager-86878c769f-ng78j\" (UID: \"5eadc65a-d34e-458a-aa84-5f1a08b89cfc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ng78j" Jul 06 22:38:15.172040 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:38:15.171993 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ng78j" Jul 06 22:38:15.312945 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:38:15.312759 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ng78j"] Jul 06 22:38:15.316000 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:38:15.315970 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eadc65a_d34e_458a_aa84_5f1a08b89cfc.slice/crio-867ee7941cffe61b9c6e802425d96cd64cc4a522577ecca464bb360e08d8600f WatchSource:0}: Error finding container 867ee7941cffe61b9c6e802425d96cd64cc4a522577ecca464bb360e08d8600f: Status 404 returned error can't find the container with id 867ee7941cffe61b9c6e802425d96cd64cc4a522577ecca464bb360e08d8600f Jul 06 22:38:15.651583 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:38:15.651545 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ng78j" event={"ID":"5eadc65a-d34e-458a-aa84-5f1a08b89cfc","Type":"ContainerStarted","Data":"c566db7a41e239aee7e7fae3c2c51b69d5047b0d25bce59e540003fc46941d50"} Jul 06 22:38:15.651764 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:38:15.651596 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ng78j" event={"ID":"5eadc65a-d34e-458a-aa84-5f1a08b89cfc","Type":"ContainerStarted","Data":"867ee7941cffe61b9c6e802425d96cd64cc4a522577ecca464bb360e08d8600f"} Jul 06 22:38:15.651764 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:38:15.651646 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ng78j" Jul 06 22:38:15.674264 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:38:15.674160 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ng78j" podStartSLOduration=1.6741441 podStartE2EDuration="1.6741441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:38:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:38:15.67293073 +0000 UTC m=+1102.591173587" watchObservedRunningTime="2026-07-06 22:38:15.6741441 +0000 UTC m=+1102.592386956" Jul 06 22:38:26.658191 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:38:26.658152 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ng78j" Jul 06 22:39:53.596894 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:39:53.596864 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-4x6pf_5796a8ea-adcb-4819-bb18-0b85e1ef4681/console-operator/2.log" Jul 06 22:39:53.597805 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:39:53.597779 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-4x6pf_5796a8ea-adcb-4819-bb18-0b85e1ef4681/console-operator/2.log" Jul 06 22:39:53.600201 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:39:53.600180 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qbbrs_bf23ef00-a2d0-499c-8c87-5d6b98510ea0/ovn-acl-logging/0.log" Jul 06 22:39:53.601473 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:39:53.601450 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qbbrs_bf23ef00-a2d0-499c-8c87-5d6b98510ea0/ovn-acl-logging/0.log" Jul 06 22:41:52.661487 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:41:52.661457 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:42:26.686004 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:42:26.685908 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:42:45.826318 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:42:45.826281 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:43:41.139471 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:43:41.139430 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:43:41.197311 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:43:41.197269 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:43:51.812081 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:43:51.811994 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:44:13.814133 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:13.814093 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:44:21.263118 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:21.263084 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:44:29.546337 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:29.546297 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:44:34.631635 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:34.631600 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:44:37.474198 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:37.474150 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:44:37.576950 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:37.576916 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:44:37.930324 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:37.930293 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:44:43.928582 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:43.928545 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:44:43.985387 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:43.985355 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:44:44.138993 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:44.138959 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:44:44.195076 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:44.194990 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:44:45.259767 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:45.259731 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:44:45.360535 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:45.360498 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:44:45.506217 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:45.506173 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-c448c87f-849b885cc4-kbggp"] Jul 06 22:44:45.510029 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:45.509969 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-c448c87f-849b885cc4-kbggp" Jul 06 22:44:45.512525 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:45.512498 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-c448c87f-serving-cert\"" Jul 06 22:44:45.512525 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:45.512519 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-98x7w\"" Jul 06 22:44:45.519264 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:45.519240 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-c448c87f-849b885cc4-kbggp"] Jul 06 22:44:45.535505 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:45.535471 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:44:45.555311 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:45.555275 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03a5c7ce-9ab1-431e-a107-92db30199e97-maas-api-tls\") pod \"maas-api-e2e-ait-c448c87f-849b885cc4-kbggp\" (UID: \"03a5c7ce-9ab1-431e-a107-92db30199e97\") " pod="opendatahub/maas-api-e2e-ait-c448c87f-849b885cc4-kbggp" Jul 06 22:44:45.555530 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:45.555456 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03a5c7ce-9ab1-431e-a107-92db30199e97-openshift-service-ca\") pod \"maas-api-e2e-ait-c448c87f-849b885cc4-kbggp\" (UID: \"03a5c7ce-9ab1-431e-a107-92db30199e97\") " pod="opendatahub/maas-api-e2e-ait-c448c87f-849b885cc4-kbggp" Jul 06 22:44:45.555530 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:45.555486 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fv4f\" (UniqueName: \"kubernetes.io/projected/03a5c7ce-9ab1-431e-a107-92db30199e97-kube-api-access-9fv4f\") pod \"maas-api-e2e-ait-c448c87f-849b885cc4-kbggp\" (UID: \"03a5c7ce-9ab1-431e-a107-92db30199e97\") " pod="opendatahub/maas-api-e2e-ait-c448c87f-849b885cc4-kbggp" Jul 06 22:44:45.656371 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:45.656332 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03a5c7ce-9ab1-431e-a107-92db30199e97-openshift-service-ca\") pod \"maas-api-e2e-ait-c448c87f-849b885cc4-kbggp\" (UID: \"03a5c7ce-9ab1-431e-a107-92db30199e97\") " pod="opendatahub/maas-api-e2e-ait-c448c87f-849b885cc4-kbggp" Jul 06 22:44:45.656371 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:45.656370 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fv4f\" (UniqueName: \"kubernetes.io/projected/03a5c7ce-9ab1-431e-a107-92db30199e97-kube-api-access-9fv4f\") pod \"maas-api-e2e-ait-c448c87f-849b885cc4-kbggp\" (UID: \"03a5c7ce-9ab1-431e-a107-92db30199e97\") " pod="opendatahub/maas-api-e2e-ait-c448c87f-849b885cc4-kbggp" Jul 06 22:44:45.656602 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:45.656396 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03a5c7ce-9ab1-431e-a107-92db30199e97-maas-api-tls\") pod \"maas-api-e2e-ait-c448c87f-849b885cc4-kbggp\" (UID: \"03a5c7ce-9ab1-431e-a107-92db30199e97\") " pod="opendatahub/maas-api-e2e-ait-c448c87f-849b885cc4-kbggp" Jul 06 22:44:45.656602 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:44:45.656499 2571 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-ait-c448c87f-serving-cert: secret "maas-api-e2e-ait-c448c87f-serving-cert" not found Jul 06 22:44:45.656602 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:44:45.656566 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03a5c7ce-9ab1-431e-a107-92db30199e97-maas-api-tls podName:03a5c7ce-9ab1-431e-a107-92db30199e97 nodeName:}" failed. No retries permitted until 2026-07-06 22:44:46.156547644 +0000 UTC m=+1493.074790481 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/03a5c7ce-9ab1-431e-a107-92db30199e97-maas-api-tls") pod "maas-api-e2e-ait-c448c87f-849b885cc4-kbggp" (UID: "03a5c7ce-9ab1-431e-a107-92db30199e97") : secret "maas-api-e2e-ait-c448c87f-serving-cert" not found Jul 06 22:44:45.657010 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:45.656989 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03a5c7ce-9ab1-431e-a107-92db30199e97-openshift-service-ca\") pod \"maas-api-e2e-ait-c448c87f-849b885cc4-kbggp\" (UID: \"03a5c7ce-9ab1-431e-a107-92db30199e97\") " pod="opendatahub/maas-api-e2e-ait-c448c87f-849b885cc4-kbggp" Jul 06 22:44:45.666409 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:45.666378 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fv4f\" (UniqueName: \"kubernetes.io/projected/03a5c7ce-9ab1-431e-a107-92db30199e97-kube-api-access-9fv4f\") pod \"maas-api-e2e-ait-c448c87f-849b885cc4-kbggp\" (UID: \"03a5c7ce-9ab1-431e-a107-92db30199e97\") " pod="opendatahub/maas-api-e2e-ait-c448c87f-849b885cc4-kbggp" Jul 06 22:44:45.835244 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:45.835150 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:44:46.160359 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:46.160322 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03a5c7ce-9ab1-431e-a107-92db30199e97-maas-api-tls\") pod \"maas-api-e2e-ait-c448c87f-849b885cc4-kbggp\" (UID: \"03a5c7ce-9ab1-431e-a107-92db30199e97\") " pod="opendatahub/maas-api-e2e-ait-c448c87f-849b885cc4-kbggp" Jul 06 22:44:46.162863 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:46.162834 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03a5c7ce-9ab1-431e-a107-92db30199e97-maas-api-tls\") pod \"maas-api-e2e-ait-c448c87f-849b885cc4-kbggp\" (UID: \"03a5c7ce-9ab1-431e-a107-92db30199e97\") " pod="opendatahub/maas-api-e2e-ait-c448c87f-849b885cc4-kbggp" Jul 06 22:44:46.176593 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:46.176564 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:44:46.273385 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:46.273353 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:44:46.423226 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:46.423115 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-c448c87f-849b885cc4-kbggp" Jul 06 22:44:46.568752 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:46.568721 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-c448c87f-849b885cc4-kbggp"] Jul 06 22:44:46.571562 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:44:46.571528 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03a5c7ce_9ab1_431e_a107_92db30199e97.slice/crio-98aaef258f03823e32ed2fc60d92fa57c39614b5ce95014f9be7203fba6aadd2 WatchSource:0}: Error finding container 98aaef258f03823e32ed2fc60d92fa57c39614b5ce95014f9be7203fba6aadd2: Status 404 returned error can't find the container with id 98aaef258f03823e32ed2fc60d92fa57c39614b5ce95014f9be7203fba6aadd2 Jul 06 22:44:46.572902 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:46.572881 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 22:44:47.528791 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:47.528751 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-c448c87f-849b885cc4-kbggp" event={"ID":"03a5c7ce-9ab1-431e-a107-92db30199e97","Type":"ContainerStarted","Data":"33ab43d5edae6115612e442d7a732fa343173389cb24068b5ea352f096083099"} Jul 06 22:44:47.529238 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:47.528818 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-c448c87f-849b885cc4-kbggp" event={"ID":"03a5c7ce-9ab1-431e-a107-92db30199e97","Type":"ContainerStarted","Data":"98aaef258f03823e32ed2fc60d92fa57c39614b5ce95014f9be7203fba6aadd2"} Jul 06 22:44:47.529238 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:47.528851 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-c448c87f-849b885cc4-kbggp" Jul 06 22:44:47.547159 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:47.547103 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-c448c87f-849b885cc4-kbggp" podStartSLOduration=1.785990742 podStartE2EDuration="2.547084257s" totalImagesPullingTime="761.093515ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:44:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:44:47.545864473 +0000 UTC m=+1494.464107329" watchObservedRunningTime="2026-07-06 22:44:47.547084257 +0000 UTC m=+1494.465327113" Jul 06 22:44:47.794778 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:47.794747 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:44:50.409738 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:50.409699 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:44:50.587967 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:50.587923 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:44:50.707451 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:50.707357 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:44:50.782990 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:50.782928 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-c448c87f-849b885cc4-kbggp"] Jul 06 22:44:50.783588 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:50.783255 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-c448c87f-849b885cc4-kbggp" podUID="03a5c7ce-9ab1-431e-a107-92db30199e97" containerName="maas-api" containerID="cri-o://33ab43d5edae6115612e442d7a732fa343173389cb24068b5ea352f096083099" gracePeriod=30 Jul 06 22:44:50.805065 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:50.805018 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:44:51.036526 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:51.036494 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-c448c87f-849b885cc4-kbggp" Jul 06 22:44:51.099597 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:51.099555 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03a5c7ce-9ab1-431e-a107-92db30199e97-maas-api-tls\") pod \"03a5c7ce-9ab1-431e-a107-92db30199e97\" (UID: \"03a5c7ce-9ab1-431e-a107-92db30199e97\") " Jul 06 22:44:51.099597 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:51.099598 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9fv4f\" (UniqueName: \"kubernetes.io/projected/03a5c7ce-9ab1-431e-a107-92db30199e97-kube-api-access-9fv4f\") pod \"03a5c7ce-9ab1-431e-a107-92db30199e97\" (UID: \"03a5c7ce-9ab1-431e-a107-92db30199e97\") " Jul 06 22:44:51.099822 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:51.099672 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03a5c7ce-9ab1-431e-a107-92db30199e97-openshift-service-ca\") pod \"03a5c7ce-9ab1-431e-a107-92db30199e97\" (UID: \"03a5c7ce-9ab1-431e-a107-92db30199e97\") " Jul 06 22:44:51.100073 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:51.100045 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03a5c7ce-9ab1-431e-a107-92db30199e97-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "03a5c7ce-9ab1-431e-a107-92db30199e97" (UID: "03a5c7ce-9ab1-431e-a107-92db30199e97"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:44:51.101705 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:51.101677 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03a5c7ce-9ab1-431e-a107-92db30199e97-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "03a5c7ce-9ab1-431e-a107-92db30199e97" (UID: "03a5c7ce-9ab1-431e-a107-92db30199e97"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:44:51.101816 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:51.101727 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03a5c7ce-9ab1-431e-a107-92db30199e97-kube-api-access-9fv4f" (OuterVolumeSpecName: "kube-api-access-9fv4f") pod "03a5c7ce-9ab1-431e-a107-92db30199e97" (UID: "03a5c7ce-9ab1-431e-a107-92db30199e97"). InnerVolumeSpecName "kube-api-access-9fv4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:44:51.200869 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:51.200826 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03a5c7ce-9ab1-431e-a107-92db30199e97-maas-api-tls\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:44:51.200869 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:51.200864 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fv4f\" (UniqueName: \"kubernetes.io/projected/03a5c7ce-9ab1-431e-a107-92db30199e97-kube-api-access-9fv4f\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:44:51.200869 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:51.200875 2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03a5c7ce-9ab1-431e-a107-92db30199e97-openshift-service-ca\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:44:51.549251 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:51.549180 2571 generic.go:356] "Generic (PLEG): container finished" podID="03a5c7ce-9ab1-431e-a107-92db30199e97" containerID="33ab43d5edae6115612e442d7a732fa343173389cb24068b5ea352f096083099" exitCode=0 Jul 06 22:44:51.549251 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:51.549243 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-c448c87f-849b885cc4-kbggp" Jul 06 22:44:51.549729 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:51.549268 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-c448c87f-849b885cc4-kbggp" event={"ID":"03a5c7ce-9ab1-431e-a107-92db30199e97","Type":"ContainerDied","Data":"33ab43d5edae6115612e442d7a732fa343173389cb24068b5ea352f096083099"} Jul 06 22:44:51.549729 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:51.549332 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-c448c87f-849b885cc4-kbggp" event={"ID":"03a5c7ce-9ab1-431e-a107-92db30199e97","Type":"ContainerDied","Data":"98aaef258f03823e32ed2fc60d92fa57c39614b5ce95014f9be7203fba6aadd2"} Jul 06 22:44:51.549729 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:51.549356 2571 scope.go:117] "RemoveContainer" containerID="33ab43d5edae6115612e442d7a732fa343173389cb24068b5ea352f096083099" Jul 06 22:44:51.559966 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:51.559946 2571 scope.go:117] "RemoveContainer" containerID="33ab43d5edae6115612e442d7a732fa343173389cb24068b5ea352f096083099" Jul 06 22:44:51.560304 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:44:51.560285 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33ab43d5edae6115612e442d7a732fa343173389cb24068b5ea352f096083099\": container with ID starting with 33ab43d5edae6115612e442d7a732fa343173389cb24068b5ea352f096083099 not found: ID does not exist" containerID="33ab43d5edae6115612e442d7a732fa343173389cb24068b5ea352f096083099" Jul 06 22:44:51.560353 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:51.560315 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33ab43d5edae6115612e442d7a732fa343173389cb24068b5ea352f096083099"} err="failed to get container status \"33ab43d5edae6115612e442d7a732fa343173389cb24068b5ea352f096083099\": rpc error: code = NotFound desc = could not find container \"33ab43d5edae6115612e442d7a732fa343173389cb24068b5ea352f096083099\": container with ID starting with 33ab43d5edae6115612e442d7a732fa343173389cb24068b5ea352f096083099 not found: ID does not exist" Jul 06 22:44:51.573704 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:51.573674 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-c448c87f-849b885cc4-kbggp"] Jul 06 22:44:51.575918 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:51.575892 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-c448c87f-849b885cc4-kbggp"] Jul 06 22:44:51.634626 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:51.634591 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03a5c7ce-9ab1-431e-a107-92db30199e97" path="/var/lib/kubelet/pods/03a5c7ce-9ab1-431e-a107-92db30199e97/volumes" Jul 06 22:44:52.172064 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:52.172028 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:44:53.627682 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:53.627651 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-4x6pf_5796a8ea-adcb-4819-bb18-0b85e1ef4681/console-operator/2.log" Jul 06 22:44:53.630348 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:53.630325 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-4x6pf_5796a8ea-adcb-4819-bb18-0b85e1ef4681/console-operator/2.log" Jul 06 22:44:53.632747 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:53.632722 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qbbrs_bf23ef00-a2d0-499c-8c87-5d6b98510ea0/ovn-acl-logging/0.log" Jul 06 22:44:53.635132 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:53.635110 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qbbrs_bf23ef00-a2d0-499c-8c87-5d6b98510ea0/ovn-acl-logging/0.log" Jul 06 22:44:57.747184 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:57.747118 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:44:57.847837 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:57.847802 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:44:58.115039 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:58.114924 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:44:58.183185 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:58.183151 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:44:58.183426 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:58.183407 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-5980e803-6c659b9fc5-px55z"] Jul 06 22:44:58.183845 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:58.183826 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03a5c7ce-9ab1-431e-a107-92db30199e97" containerName="maas-api" Jul 06 22:44:58.183845 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:58.183843 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="03a5c7ce-9ab1-431e-a107-92db30199e97" containerName="maas-api" Jul 06 22:44:58.184012 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:58.183947 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="03a5c7ce-9ab1-431e-a107-92db30199e97" containerName="maas-api" Jul 06 22:44:58.188264 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:58.188243 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-5980e803-6c659b9fc5-px55z" Jul 06 22:44:58.192408 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:58.192383 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-5980e803-serving-cert\"" Jul 06 22:44:58.192525 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:58.192387 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-98x7w\"" Jul 06 22:44:58.196676 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:58.196648 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-5980e803-6c659b9fc5-px55z"] Jul 06 22:44:58.265627 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:58.265587 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvjvg\" (UniqueName: \"kubernetes.io/projected/20393d99-4a0e-4607-8750-f696637c085e-kube-api-access-dvjvg\") pod \"maas-api-e2e-ait-5980e803-6c659b9fc5-px55z\" (UID: \"20393d99-4a0e-4607-8750-f696637c085e\") " pod="opendatahub/maas-api-e2e-ait-5980e803-6c659b9fc5-px55z" Jul 06 22:44:58.265821 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:58.265656 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20393d99-4a0e-4607-8750-f696637c085e-openshift-service-ca\") pod \"maas-api-e2e-ait-5980e803-6c659b9fc5-px55z\" (UID: \"20393d99-4a0e-4607-8750-f696637c085e\") " pod="opendatahub/maas-api-e2e-ait-5980e803-6c659b9fc5-px55z" Jul 06 22:44:58.265821 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:58.265713 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20393d99-4a0e-4607-8750-f696637c085e-maas-api-tls\") pod \"maas-api-e2e-ait-5980e803-6c659b9fc5-px55z\" (UID: \"20393d99-4a0e-4607-8750-f696637c085e\") " pod="opendatahub/maas-api-e2e-ait-5980e803-6c659b9fc5-px55z" Jul 06 22:44:58.276926 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:58.276898 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:44:58.366818 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:58.366729 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20393d99-4a0e-4607-8750-f696637c085e-maas-api-tls\") pod \"maas-api-e2e-ait-5980e803-6c659b9fc5-px55z\" (UID: \"20393d99-4a0e-4607-8750-f696637c085e\") " pod="opendatahub/maas-api-e2e-ait-5980e803-6c659b9fc5-px55z" Jul 06 22:44:58.366818 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:58.366784 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvjvg\" (UniqueName: \"kubernetes.io/projected/20393d99-4a0e-4607-8750-f696637c085e-kube-api-access-dvjvg\") pod \"maas-api-e2e-ait-5980e803-6c659b9fc5-px55z\" (UID: \"20393d99-4a0e-4607-8750-f696637c085e\") " pod="opendatahub/maas-api-e2e-ait-5980e803-6c659b9fc5-px55z" Jul 06 22:44:58.367007 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:58.366831 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20393d99-4a0e-4607-8750-f696637c085e-openshift-service-ca\") pod \"maas-api-e2e-ait-5980e803-6c659b9fc5-px55z\" (UID: \"20393d99-4a0e-4607-8750-f696637c085e\") " pod="opendatahub/maas-api-e2e-ait-5980e803-6c659b9fc5-px55z" Jul 06 22:44:58.367007 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:44:58.366889 2571 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-ait-5980e803-serving-cert: secret "maas-api-e2e-ait-5980e803-serving-cert" not found Jul 06 22:44:58.367007 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:44:58.366978 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20393d99-4a0e-4607-8750-f696637c085e-maas-api-tls podName:20393d99-4a0e-4607-8750-f696637c085e nodeName:}" failed. No retries permitted until 2026-07-06 22:44:58.866956421 +0000 UTC m=+1505.785199256 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/20393d99-4a0e-4607-8750-f696637c085e-maas-api-tls") pod "maas-api-e2e-ait-5980e803-6c659b9fc5-px55z" (UID: "20393d99-4a0e-4607-8750-f696637c085e") : secret "maas-api-e2e-ait-5980e803-serving-cert" not found Jul 06 22:44:58.367458 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:58.367441 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20393d99-4a0e-4607-8750-f696637c085e-openshift-service-ca\") pod \"maas-api-e2e-ait-5980e803-6c659b9fc5-px55z\" (UID: \"20393d99-4a0e-4607-8750-f696637c085e\") " pod="opendatahub/maas-api-e2e-ait-5980e803-6c659b9fc5-px55z" Jul 06 22:44:58.376416 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:58.376388 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvjvg\" (UniqueName: \"kubernetes.io/projected/20393d99-4a0e-4607-8750-f696637c085e-kube-api-access-dvjvg\") pod \"maas-api-e2e-ait-5980e803-6c659b9fc5-px55z\" (UID: \"20393d99-4a0e-4607-8750-f696637c085e\") " pod="opendatahub/maas-api-e2e-ait-5980e803-6c659b9fc5-px55z" Jul 06 22:44:58.575228 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:58.575170 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:44:58.772487 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:58.772453 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:44:58.871324 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:58.871285 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20393d99-4a0e-4607-8750-f696637c085e-maas-api-tls\") pod \"maas-api-e2e-ait-5980e803-6c659b9fc5-px55z\" (UID: \"20393d99-4a0e-4607-8750-f696637c085e\") " pod="opendatahub/maas-api-e2e-ait-5980e803-6c659b9fc5-px55z" Jul 06 22:44:58.872046 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:58.872007 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:44:58.873843 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:58.873823 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20393d99-4a0e-4607-8750-f696637c085e-maas-api-tls\") pod \"maas-api-e2e-ait-5980e803-6c659b9fc5-px55z\" (UID: \"20393d99-4a0e-4607-8750-f696637c085e\") " pod="opendatahub/maas-api-e2e-ait-5980e803-6c659b9fc5-px55z" Jul 06 22:44:59.101920 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:59.101826 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-5980e803-6c659b9fc5-px55z" Jul 06 22:44:59.249659 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:59.249622 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-5980e803-6c659b9fc5-px55z"] Jul 06 22:44:59.253262 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:44:59.253198 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20393d99_4a0e_4607_8750_f696637c085e.slice/crio-7fc826edd7445a9f048f035f518f6c90fe3f13403ccb726048fc265f8dcbbc9f WatchSource:0}: Error finding container 7fc826edd7445a9f048f035f518f6c90fe3f13403ccb726048fc265f8dcbbc9f: Status 404 returned error can't find the container with id 7fc826edd7445a9f048f035f518f6c90fe3f13403ccb726048fc265f8dcbbc9f Jul 06 22:44:59.589391 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:44:59.589350 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-5980e803-6c659b9fc5-px55z" event={"ID":"20393d99-4a0e-4607-8750-f696637c085e","Type":"ContainerStarted","Data":"7fc826edd7445a9f048f035f518f6c90fe3f13403ccb726048fc265f8dcbbc9f"} Jul 06 22:45:00.196148 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:00.196103 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:45:00.595972 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:00.595882 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-5980e803-6c659b9fc5-px55z" event={"ID":"20393d99-4a0e-4607-8750-f696637c085e","Type":"ContainerStarted","Data":"af6903defa43d39214d006723841d62eb0649f3ad2bdaad230a3c7071927ae89"} Jul 06 22:45:00.595972 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:00.595949 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-5980e803-6c659b9fc5-px55z" Jul 06 22:45:00.619482 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:00.619427 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-5980e803-6c659b9fc5-px55z" podStartSLOduration=2.218396943 podStartE2EDuration="2.619412456s" totalImagesPullingTime="401.015513ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:44:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:45:00.616933065 +0000 UTC m=+1507.535175920" watchObservedRunningTime="2026-07-06 22:45:00.619412456 +0000 UTC m=+1507.537655337" Jul 06 22:45:03.262993 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:03.262956 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:45:03.626967 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:03.626871 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-5980e803-6c659b9fc5-px55z"] Jul 06 22:45:03.627202 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:03.627117 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-5980e803-6c659b9fc5-px55z" podUID="20393d99-4a0e-4607-8750-f696637c085e" containerName="maas-api" containerID="cri-o://af6903defa43d39214d006723841d62eb0649f3ad2bdaad230a3c7071927ae89" gracePeriod=30 Jul 06 22:45:03.654984 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:03.654955 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:45:03.883549 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:03.883527 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-5980e803-6c659b9fc5-px55z" Jul 06 22:45:04.021879 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:04.021840 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20393d99-4a0e-4607-8750-f696637c085e-maas-api-tls\") pod \"20393d99-4a0e-4607-8750-f696637c085e\" (UID: \"20393d99-4a0e-4607-8750-f696637c085e\") " Jul 06 22:45:04.022076 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:04.021893 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dvjvg\" (UniqueName: \"kubernetes.io/projected/20393d99-4a0e-4607-8750-f696637c085e-kube-api-access-dvjvg\") pod \"20393d99-4a0e-4607-8750-f696637c085e\" (UID: \"20393d99-4a0e-4607-8750-f696637c085e\") " Jul 06 22:45:04.022076 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:04.021924 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20393d99-4a0e-4607-8750-f696637c085e-openshift-service-ca\") pod \"20393d99-4a0e-4607-8750-f696637c085e\" (UID: \"20393d99-4a0e-4607-8750-f696637c085e\") " Jul 06 22:45:04.022355 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:04.022320 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20393d99-4a0e-4607-8750-f696637c085e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "20393d99-4a0e-4607-8750-f696637c085e" (UID: "20393d99-4a0e-4607-8750-f696637c085e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:45:04.024084 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:04.024053 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20393d99-4a0e-4607-8750-f696637c085e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "20393d99-4a0e-4607-8750-f696637c085e" (UID: "20393d99-4a0e-4607-8750-f696637c085e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:45:04.024195 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:04.024173 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20393d99-4a0e-4607-8750-f696637c085e-kube-api-access-dvjvg" (OuterVolumeSpecName: "kube-api-access-dvjvg") pod "20393d99-4a0e-4607-8750-f696637c085e" (UID: "20393d99-4a0e-4607-8750-f696637c085e"). InnerVolumeSpecName "kube-api-access-dvjvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:45:04.122642 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:04.122604 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20393d99-4a0e-4607-8750-f696637c085e-maas-api-tls\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:45:04.122642 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:04.122637 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvjvg\" (UniqueName: \"kubernetes.io/projected/20393d99-4a0e-4607-8750-f696637c085e-kube-api-access-dvjvg\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:45:04.122642 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:04.122647 2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20393d99-4a0e-4607-8750-f696637c085e-openshift-service-ca\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:45:04.616059 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:04.616018 2571 generic.go:356] "Generic (PLEG): container finished" podID="20393d99-4a0e-4607-8750-f696637c085e" containerID="af6903defa43d39214d006723841d62eb0649f3ad2bdaad230a3c7071927ae89" exitCode=0 Jul 06 22:45:04.616506 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:04.616086 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-5980e803-6c659b9fc5-px55z" Jul 06 22:45:04.616506 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:04.616102 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-5980e803-6c659b9fc5-px55z" event={"ID":"20393d99-4a0e-4607-8750-f696637c085e","Type":"ContainerDied","Data":"af6903defa43d39214d006723841d62eb0649f3ad2bdaad230a3c7071927ae89"} Jul 06 22:45:04.616506 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:04.616203 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-5980e803-6c659b9fc5-px55z" event={"ID":"20393d99-4a0e-4607-8750-f696637c085e","Type":"ContainerDied","Data":"7fc826edd7445a9f048f035f518f6c90fe3f13403ccb726048fc265f8dcbbc9f"} Jul 06 22:45:04.616506 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:04.616237 2571 scope.go:117] "RemoveContainer" containerID="af6903defa43d39214d006723841d62eb0649f3ad2bdaad230a3c7071927ae89" Jul 06 22:45:04.625973 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:04.625951 2571 scope.go:117] "RemoveContainer" containerID="af6903defa43d39214d006723841d62eb0649f3ad2bdaad230a3c7071927ae89" Jul 06 22:45:04.626301 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:45:04.626276 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af6903defa43d39214d006723841d62eb0649f3ad2bdaad230a3c7071927ae89\": container with ID starting with af6903defa43d39214d006723841d62eb0649f3ad2bdaad230a3c7071927ae89 not found: ID does not exist" containerID="af6903defa43d39214d006723841d62eb0649f3ad2bdaad230a3c7071927ae89" Jul 06 22:45:04.626379 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:04.626308 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af6903defa43d39214d006723841d62eb0649f3ad2bdaad230a3c7071927ae89"} err="failed to get container status \"af6903defa43d39214d006723841d62eb0649f3ad2bdaad230a3c7071927ae89\": rpc error: code = NotFound desc = could not find container \"af6903defa43d39214d006723841d62eb0649f3ad2bdaad230a3c7071927ae89\": container with ID starting with af6903defa43d39214d006723841d62eb0649f3ad2bdaad230a3c7071927ae89 not found: ID does not exist" Jul 06 22:45:04.639289 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:04.639257 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-5980e803-6c659b9fc5-px55z"] Jul 06 22:45:04.643895 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:04.643871 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-5980e803-6c659b9fc5-px55z"] Jul 06 22:45:05.636061 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:05.636020 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20393d99-4a0e-4607-8750-f696637c085e" path="/var/lib/kubelet/pods/20393d99-4a0e-4607-8750-f696637c085e/volumes" Jul 06 22:45:09.337621 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:09.337580 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:45:09.438519 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:09.438485 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:45:11.003375 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:11.003327 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:45:15.809601 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:15.809530 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:45:15.911691 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:15.911656 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:45:16.025201 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:16.025162 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:45:16.120494 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:16.120413 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:45:17.136480 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:17.136448 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:45:17.206283 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:17.206250 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:45:17.210392 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:17.210366 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-derive-3f0ec651-57f6c678c9-7dczw"] Jul 06 22:45:17.210710 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:17.210698 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20393d99-4a0e-4607-8750-f696637c085e" containerName="maas-api" Jul 06 22:45:17.210710 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:17.210711 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="20393d99-4a0e-4607-8750-f696637c085e" containerName="maas-api" Jul 06 22:45:17.210840 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:17.210775 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="20393d99-4a0e-4607-8750-f696637c085e" containerName="maas-api" Jul 06 22:45:17.215568 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:17.215545 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-3f0ec651-57f6c678c9-7dczw" Jul 06 22:45:17.218950 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:17.218926 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-derive-3f0ec651-serving-cert\"" Jul 06 22:45:17.218950 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:17.218945 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-98x7w\"" Jul 06 22:45:17.226607 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:17.226577 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-3f0ec651-57f6c678c9-7dczw"] Jul 06 22:45:17.300178 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:17.300128 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:45:17.343254 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:17.343191 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m492v\" (UniqueName: \"kubernetes.io/projected/08eaeb39-870d-4d4c-9738-7893715c79a6-kube-api-access-m492v\") pod \"maas-api-e2e-derive-3f0ec651-57f6c678c9-7dczw\" (UID: \"08eaeb39-870d-4d4c-9738-7893715c79a6\") " pod="opendatahub/maas-api-e2e-derive-3f0ec651-57f6c678c9-7dczw" Jul 06 22:45:17.343448 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:17.343408 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08eaeb39-870d-4d4c-9738-7893715c79a6-maas-api-tls\") pod \"maas-api-e2e-derive-3f0ec651-57f6c678c9-7dczw\" (UID: \"08eaeb39-870d-4d4c-9738-7893715c79a6\") " pod="opendatahub/maas-api-e2e-derive-3f0ec651-57f6c678c9-7dczw" Jul 06 22:45:17.343515 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:17.343453 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08eaeb39-870d-4d4c-9738-7893715c79a6-openshift-service-ca\") pod \"maas-api-e2e-derive-3f0ec651-57f6c678c9-7dczw\" (UID: \"08eaeb39-870d-4d4c-9738-7893715c79a6\") " pod="opendatahub/maas-api-e2e-derive-3f0ec651-57f6c678c9-7dczw" Jul 06 22:45:17.444491 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:17.444390 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m492v\" (UniqueName: \"kubernetes.io/projected/08eaeb39-870d-4d4c-9738-7893715c79a6-kube-api-access-m492v\") pod \"maas-api-e2e-derive-3f0ec651-57f6c678c9-7dczw\" (UID: \"08eaeb39-870d-4d4c-9738-7893715c79a6\") " pod="opendatahub/maas-api-e2e-derive-3f0ec651-57f6c678c9-7dczw" Jul 06 22:45:17.444491 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:17.444449 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08eaeb39-870d-4d4c-9738-7893715c79a6-maas-api-tls\") pod \"maas-api-e2e-derive-3f0ec651-57f6c678c9-7dczw\" (UID: \"08eaeb39-870d-4d4c-9738-7893715c79a6\") " pod="opendatahub/maas-api-e2e-derive-3f0ec651-57f6c678c9-7dczw" Jul 06 22:45:17.444491 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:17.444479 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08eaeb39-870d-4d4c-9738-7893715c79a6-openshift-service-ca\") pod \"maas-api-e2e-derive-3f0ec651-57f6c678c9-7dczw\" (UID: \"08eaeb39-870d-4d4c-9738-7893715c79a6\") " pod="opendatahub/maas-api-e2e-derive-3f0ec651-57f6c678c9-7dczw" Jul 06 22:45:17.444788 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:45:17.444614 2571 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-derive-3f0ec651-serving-cert: secret "maas-api-e2e-derive-3f0ec651-serving-cert" not found Jul 06 22:45:17.444788 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:45:17.444698 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08eaeb39-870d-4d4c-9738-7893715c79a6-maas-api-tls podName:08eaeb39-870d-4d4c-9738-7893715c79a6 nodeName:}" failed. No retries permitted until 2026-07-06 22:45:17.944676983 +0000 UTC m=+1524.862919832 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/08eaeb39-870d-4d4c-9738-7893715c79a6-maas-api-tls") pod "maas-api-e2e-derive-3f0ec651-57f6c678c9-7dczw" (UID: "08eaeb39-870d-4d4c-9738-7893715c79a6") : secret "maas-api-e2e-derive-3f0ec651-serving-cert" not found Jul 06 22:45:17.445158 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:17.445139 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08eaeb39-870d-4d4c-9738-7893715c79a6-openshift-service-ca\") pod \"maas-api-e2e-derive-3f0ec651-57f6c678c9-7dczw\" (UID: \"08eaeb39-870d-4d4c-9738-7893715c79a6\") " pod="opendatahub/maas-api-e2e-derive-3f0ec651-57f6c678c9-7dczw" Jul 06 22:45:17.453282 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:17.453252 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m492v\" (UniqueName: \"kubernetes.io/projected/08eaeb39-870d-4d4c-9738-7893715c79a6-kube-api-access-m492v\") pod \"maas-api-e2e-derive-3f0ec651-57f6c678c9-7dczw\" (UID: \"08eaeb39-870d-4d4c-9738-7893715c79a6\") " pod="opendatahub/maas-api-e2e-derive-3f0ec651-57f6c678c9-7dczw" Jul 06 22:45:17.724362 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:17.724277 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:45:17.827141 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:17.827104 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:45:17.924376 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:17.924340 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:45:17.950402 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:17.950369 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08eaeb39-870d-4d4c-9738-7893715c79a6-maas-api-tls\") pod \"maas-api-e2e-derive-3f0ec651-57f6c678c9-7dczw\" (UID: \"08eaeb39-870d-4d4c-9738-7893715c79a6\") " pod="opendatahub/maas-api-e2e-derive-3f0ec651-57f6c678c9-7dczw" Jul 06 22:45:17.952876 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:17.952844 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08eaeb39-870d-4d4c-9738-7893715c79a6-maas-api-tls\") pod \"maas-api-e2e-derive-3f0ec651-57f6c678c9-7dczw\" (UID: \"08eaeb39-870d-4d4c-9738-7893715c79a6\") " pod="opendatahub/maas-api-e2e-derive-3f0ec651-57f6c678c9-7dczw" Jul 06 22:45:18.128868 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:18.128765 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-3f0ec651-57f6c678c9-7dczw" Jul 06 22:45:18.271828 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:18.271791 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-3f0ec651-57f6c678c9-7dczw"] Jul 06 22:45:18.275173 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:45:18.275143 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08eaeb39_870d_4d4c_9738_7893715c79a6.slice/crio-ce34dd1a81c79cdb98c1e59e9a642774d0ed7003c466d2cca4b86349fe16c7b7 WatchSource:0}: Error finding container ce34dd1a81c79cdb98c1e59e9a642774d0ed7003c466d2cca4b86349fe16c7b7: Status 404 returned error can't find the container with id ce34dd1a81c79cdb98c1e59e9a642774d0ed7003c466d2cca4b86349fe16c7b7 Jul 06 22:45:18.686571 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:18.686546 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-3f0ec651-57f6c678c9-7dczw" event={"ID":"08eaeb39-870d-4d4c-9738-7893715c79a6","Type":"ContainerStarted","Data":"ce34dd1a81c79cdb98c1e59e9a642774d0ed7003c466d2cca4b86349fe16c7b7"} Jul 06 22:45:19.621419 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:19.621387 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:45:19.692327 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:19.692282 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-3f0ec651-57f6c678c9-7dczw" event={"ID":"08eaeb39-870d-4d4c-9738-7893715c79a6","Type":"ContainerStarted","Data":"d1ba684f9f76a9580fabf73d60208b7ccd15114f0948c44df1c97e34498c1ad6"} Jul 06 22:45:19.692501 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:19.692385 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-derive-3f0ec651-57f6c678c9-7dczw" Jul 06 22:45:19.710755 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:19.710535 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-derive-3f0ec651-57f6c678c9-7dczw" podStartSLOduration=2.353298547 podStartE2EDuration="2.710514718s" totalImagesPullingTime="357.216171ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:45:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:45:19.709985063 +0000 UTC m=+1526.628227920" watchObservedRunningTime="2026-07-06 22:45:19.710514718 +0000 UTC m=+1526.628757579" Jul 06 22:45:21.610534 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:21.610491 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:45:21.782345 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:21.782315 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:45:21.883160 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:21.883121 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:45:21.965182 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:21.965146 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-3f0ec651-57f6c678c9-7dczw"] Jul 06 22:45:21.965466 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:21.965415 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-derive-3f0ec651-57f6c678c9-7dczw" podUID="08eaeb39-870d-4d4c-9738-7893715c79a6" containerName="maas-api" containerID="cri-o://d1ba684f9f76a9580fabf73d60208b7ccd15114f0948c44df1c97e34498c1ad6" gracePeriod=30 Jul 06 22:45:21.991834 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:21.991810 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:45:22.218891 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:22.218866 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-3f0ec651-57f6c678c9-7dczw" Jul 06 22:45:22.389335 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:22.389231 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08eaeb39-870d-4d4c-9738-7893715c79a6-maas-api-tls\") pod \"08eaeb39-870d-4d4c-9738-7893715c79a6\" (UID: \"08eaeb39-870d-4d4c-9738-7893715c79a6\") " Jul 06 22:45:22.389335 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:22.389296 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m492v\" (UniqueName: \"kubernetes.io/projected/08eaeb39-870d-4d4c-9738-7893715c79a6-kube-api-access-m492v\") pod \"08eaeb39-870d-4d4c-9738-7893715c79a6\" (UID: \"08eaeb39-870d-4d4c-9738-7893715c79a6\") " Jul 06 22:45:22.389573 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:22.389408 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08eaeb39-870d-4d4c-9738-7893715c79a6-openshift-service-ca\") pod \"08eaeb39-870d-4d4c-9738-7893715c79a6\" (UID: \"08eaeb39-870d-4d4c-9738-7893715c79a6\") " Jul 06 22:45:22.389726 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:22.389704 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08eaeb39-870d-4d4c-9738-7893715c79a6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "08eaeb39-870d-4d4c-9738-7893715c79a6" (UID: "08eaeb39-870d-4d4c-9738-7893715c79a6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:45:22.391358 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:22.391336 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08eaeb39-870d-4d4c-9738-7893715c79a6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "08eaeb39-870d-4d4c-9738-7893715c79a6" (UID: "08eaeb39-870d-4d4c-9738-7893715c79a6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:45:22.391457 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:22.391393 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08eaeb39-870d-4d4c-9738-7893715c79a6-kube-api-access-m492v" (OuterVolumeSpecName: "kube-api-access-m492v") pod "08eaeb39-870d-4d4c-9738-7893715c79a6" (UID: "08eaeb39-870d-4d4c-9738-7893715c79a6"). InnerVolumeSpecName "kube-api-access-m492v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:45:22.490497 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:22.490444 2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08eaeb39-870d-4d4c-9738-7893715c79a6-openshift-service-ca\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:45:22.490497 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:22.490491 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08eaeb39-870d-4d4c-9738-7893715c79a6-maas-api-tls\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:45:22.490497 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:22.490501 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m492v\" (UniqueName: \"kubernetes.io/projected/08eaeb39-870d-4d4c-9738-7893715c79a6-kube-api-access-m492v\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:45:22.706489 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:22.706400 2571 generic.go:356] "Generic (PLEG): container finished" podID="08eaeb39-870d-4d4c-9738-7893715c79a6" containerID="d1ba684f9f76a9580fabf73d60208b7ccd15114f0948c44df1c97e34498c1ad6" exitCode=0 Jul 06 22:45:22.706489 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:22.706447 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-3f0ec651-57f6c678c9-7dczw" Jul 06 22:45:22.706489 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:22.706470 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-3f0ec651-57f6c678c9-7dczw" event={"ID":"08eaeb39-870d-4d4c-9738-7893715c79a6","Type":"ContainerDied","Data":"d1ba684f9f76a9580fabf73d60208b7ccd15114f0948c44df1c97e34498c1ad6"} Jul 06 22:45:22.706982 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:22.706513 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-3f0ec651-57f6c678c9-7dczw" event={"ID":"08eaeb39-870d-4d4c-9738-7893715c79a6","Type":"ContainerDied","Data":"ce34dd1a81c79cdb98c1e59e9a642774d0ed7003c466d2cca4b86349fe16c7b7"} Jul 06 22:45:22.706982 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:22.706537 2571 scope.go:117] "RemoveContainer" containerID="d1ba684f9f76a9580fabf73d60208b7ccd15114f0948c44df1c97e34498c1ad6" Jul 06 22:45:22.716003 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:22.715982 2571 scope.go:117] "RemoveContainer" containerID="d1ba684f9f76a9580fabf73d60208b7ccd15114f0948c44df1c97e34498c1ad6" Jul 06 22:45:22.716349 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:45:22.716326 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1ba684f9f76a9580fabf73d60208b7ccd15114f0948c44df1c97e34498c1ad6\": container with ID starting with d1ba684f9f76a9580fabf73d60208b7ccd15114f0948c44df1c97e34498c1ad6 not found: ID does not exist" containerID="d1ba684f9f76a9580fabf73d60208b7ccd15114f0948c44df1c97e34498c1ad6" Jul 06 22:45:22.716438 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:22.716354 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1ba684f9f76a9580fabf73d60208b7ccd15114f0948c44df1c97e34498c1ad6"} err="failed to get container status \"d1ba684f9f76a9580fabf73d60208b7ccd15114f0948c44df1c97e34498c1ad6\": rpc error: code = NotFound desc = could not find container \"d1ba684f9f76a9580fabf73d60208b7ccd15114f0948c44df1c97e34498c1ad6\": container with ID starting with d1ba684f9f76a9580fabf73d60208b7ccd15114f0948c44df1c97e34498c1ad6 not found: ID does not exist" Jul 06 22:45:22.731075 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:22.731041 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-3f0ec651-57f6c678c9-7dczw"] Jul 06 22:45:22.738293 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:22.738264 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-derive-3f0ec651-57f6c678c9-7dczw"] Jul 06 22:45:23.572183 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:23.572145 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:45:23.635547 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:23.635513 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08eaeb39-870d-4d4c-9738-7893715c79a6" path="/var/lib/kubelet/pods/08eaeb39-870d-4d4c-9738-7893715c79a6/volumes" Jul 06 22:45:27.973111 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:27.973061 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:45:28.073333 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:28.073295 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:45:28.435903 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:28.435861 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:45:28.535308 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:28.535266 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:45:28.636320 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:28.636277 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:45:30.021344 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:30.021313 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:45:36.398745 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:36.398709 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:45:36.457089 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:36.457055 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:45:36.543948 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:36.543913 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:45:36.665234 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:36.665140 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:45:36.766478 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:36.766432 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:45:37.097474 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:45:37.097389 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:46:03.050611 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:46:03.050576 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:46:09.059978 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:46:09.059946 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:46:09.161516 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:46:09.161483 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:46:09.258591 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:46:09.258559 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:46:25.122153 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:46:25.122115 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:46:31.105578 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:46:31.105542 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:46:31.212609 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:46:31.212575 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:46:31.306269 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:46:31.306236 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:46:41.839345 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:46:41.839306 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:46:42.648515 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:46:42.648485 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:46:48.763651 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:46:48.763568 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:46:48.865430 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:46:48.865389 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:46:48.964251 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:46:48.964192 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:46:49.065094 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:46:49.065017 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:46:49.813127 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:46:49.813086 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:46:49.908525 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:46:49.908485 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:46:50.008696 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:46:50.008669 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:46:51.621532 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:46:51.621498 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:46:57.835105 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:46:57.835070 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:46:57.935482 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:46:57.935443 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:46:58.507396 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:46:58.507359 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:04.726959 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:04.726917 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:04.785723 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:04.785675 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:04.955096 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:04.955051 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:05.307673 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:05.307634 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:05.499221 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:05.499172 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:08.619481 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:08.619446 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:08.697857 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:08.697817 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-29698a60-75f978db7c-z88vr"] Jul 06 22:47:08.698085 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:08.697995 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:08.698375 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:08.698356 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08eaeb39-870d-4d4c-9738-7893715c79a6" containerName="maas-api" Jul 06 22:47:08.698375 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:08.698377 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="08eaeb39-870d-4d4c-9738-7893715c79a6" containerName="maas-api" Jul 06 22:47:08.698499 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:08.698461 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="08eaeb39-870d-4d4c-9738-7893715c79a6" containerName="maas-api" Jul 06 22:47:08.701426 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:08.701402 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-29698a60-75f978db7c-z88vr" Jul 06 22:47:08.703608 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:08.703585 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-29698a60-serving-cert\"" Jul 06 22:47:08.703751 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:08.703610 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-98x7w\"" Jul 06 22:47:08.710412 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:08.710383 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-29698a60-75f978db7c-z88vr"] Jul 06 22:47:08.734265 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:08.734223 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh9p8\" (UniqueName: \"kubernetes.io/projected/c3f08988-0ff1-4891-a0a7-012fc27415c8-kube-api-access-qh9p8\") pod \"maas-api-e2e-mt-29698a60-75f978db7c-z88vr\" (UID: \"c3f08988-0ff1-4891-a0a7-012fc27415c8\") " pod="opendatahub/maas-api-e2e-mt-29698a60-75f978db7c-z88vr" Jul 06 22:47:08.734455 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:08.734287 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3f08988-0ff1-4891-a0a7-012fc27415c8-openshift-service-ca\") pod \"maas-api-e2e-mt-29698a60-75f978db7c-z88vr\" (UID: \"c3f08988-0ff1-4891-a0a7-012fc27415c8\") " pod="opendatahub/maas-api-e2e-mt-29698a60-75f978db7c-z88vr" Jul 06 22:47:08.734455 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:08.734327 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3f08988-0ff1-4891-a0a7-012fc27415c8-maas-api-tls\") pod \"maas-api-e2e-mt-29698a60-75f978db7c-z88vr\" (UID: \"c3f08988-0ff1-4891-a0a7-012fc27415c8\") " pod="opendatahub/maas-api-e2e-mt-29698a60-75f978db7c-z88vr" Jul 06 22:47:08.763393 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:08.763361 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:08.835067 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:08.835021 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3f08988-0ff1-4891-a0a7-012fc27415c8-openshift-service-ca\") pod \"maas-api-e2e-mt-29698a60-75f978db7c-z88vr\" (UID: \"c3f08988-0ff1-4891-a0a7-012fc27415c8\") " pod="opendatahub/maas-api-e2e-mt-29698a60-75f978db7c-z88vr" Jul 06 22:47:08.835276 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:08.835088 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3f08988-0ff1-4891-a0a7-012fc27415c8-maas-api-tls\") pod \"maas-api-e2e-mt-29698a60-75f978db7c-z88vr\" (UID: \"c3f08988-0ff1-4891-a0a7-012fc27415c8\") " pod="opendatahub/maas-api-e2e-mt-29698a60-75f978db7c-z88vr" Jul 06 22:47:08.835276 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:08.835196 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh9p8\" (UniqueName: \"kubernetes.io/projected/c3f08988-0ff1-4891-a0a7-012fc27415c8-kube-api-access-qh9p8\") pod \"maas-api-e2e-mt-29698a60-75f978db7c-z88vr\" (UID: \"c3f08988-0ff1-4891-a0a7-012fc27415c8\") " pod="opendatahub/maas-api-e2e-mt-29698a60-75f978db7c-z88vr" Jul 06 22:47:08.835366 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:47:08.835281 2571 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-mt-29698a60-serving-cert: secret "maas-api-e2e-mt-29698a60-serving-cert" not found Jul 06 22:47:08.835411 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:47:08.835364 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3f08988-0ff1-4891-a0a7-012fc27415c8-maas-api-tls podName:c3f08988-0ff1-4891-a0a7-012fc27415c8 nodeName:}" failed. No retries permitted until 2026-07-06 22:47:09.335343015 +0000 UTC m=+1636.253585869 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c3f08988-0ff1-4891-a0a7-012fc27415c8-maas-api-tls") pod "maas-api-e2e-mt-29698a60-75f978db7c-z88vr" (UID: "c3f08988-0ff1-4891-a0a7-012fc27415c8") : secret "maas-api-e2e-mt-29698a60-serving-cert" not found Jul 06 22:47:08.835717 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:08.835700 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3f08988-0ff1-4891-a0a7-012fc27415c8-openshift-service-ca\") pod \"maas-api-e2e-mt-29698a60-75f978db7c-z88vr\" (UID: \"c3f08988-0ff1-4891-a0a7-012fc27415c8\") " pod="opendatahub/maas-api-e2e-mt-29698a60-75f978db7c-z88vr" Jul 06 22:47:08.846127 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:08.846091 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh9p8\" (UniqueName: \"kubernetes.io/projected/c3f08988-0ff1-4891-a0a7-012fc27415c8-kube-api-access-qh9p8\") pod \"maas-api-e2e-mt-29698a60-75f978db7c-z88vr\" (UID: \"c3f08988-0ff1-4891-a0a7-012fc27415c8\") " pod="opendatahub/maas-api-e2e-mt-29698a60-75f978db7c-z88vr" Jul 06 22:47:08.965449 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:08.965413 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:09.309882 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:09.309799 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:09.340921 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:09.340882 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3f08988-0ff1-4891-a0a7-012fc27415c8-maas-api-tls\") pod \"maas-api-e2e-mt-29698a60-75f978db7c-z88vr\" (UID: \"c3f08988-0ff1-4891-a0a7-012fc27415c8\") " pod="opendatahub/maas-api-e2e-mt-29698a60-75f978db7c-z88vr" Jul 06 22:47:09.343621 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:09.343590 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3f08988-0ff1-4891-a0a7-012fc27415c8-maas-api-tls\") pod \"maas-api-e2e-mt-29698a60-75f978db7c-z88vr\" (UID: \"c3f08988-0ff1-4891-a0a7-012fc27415c8\") " pod="opendatahub/maas-api-e2e-mt-29698a60-75f978db7c-z88vr" Jul 06 22:47:09.409683 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:09.409652 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:09.613244 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:09.613112 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-29698a60-75f978db7c-z88vr" Jul 06 22:47:09.756022 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:09.755960 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-29698a60-75f978db7c-z88vr"] Jul 06 22:47:09.758389 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:47:09.758355 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3f08988_0ff1_4891_a0a7_012fc27415c8.slice/crio-81c21ec19be55371bc640f2ebf7e97d34d7cefaff0570ad2e97aa0ae70d9b5b1 WatchSource:0}: Error finding container 81c21ec19be55371bc640f2ebf7e97d34d7cefaff0570ad2e97aa0ae70d9b5b1: Status 404 returned error can't find the container with id 81c21ec19be55371bc640f2ebf7e97d34d7cefaff0570ad2e97aa0ae70d9b5b1 Jul 06 22:47:10.221048 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:10.221008 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-29698a60-75f978db7c-z88vr" event={"ID":"c3f08988-0ff1-4891-a0a7-012fc27415c8","Type":"ContainerStarted","Data":"81c21ec19be55371bc640f2ebf7e97d34d7cefaff0570ad2e97aa0ae70d9b5b1"} Jul 06 22:47:11.226811 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:11.226776 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-29698a60-75f978db7c-z88vr" event={"ID":"c3f08988-0ff1-4891-a0a7-012fc27415c8","Type":"ContainerStarted","Data":"f4e70659b5a7bcc7ac8198c169cb54e267e6abc8ff4da033e48f25bbf69434ef"} Jul 06 22:47:11.227249 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:11.226906 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-29698a60-75f978db7c-z88vr" Jul 06 22:47:11.245723 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:11.245664 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-29698a60-75f978db7c-z88vr" podStartSLOduration=2.677855697 podStartE2EDuration="3.245648289s" totalImagesPullingTime="567.792592ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:47:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:47:11.244307095 +0000 UTC m=+1638.162549948" watchObservedRunningTime="2026-07-06 22:47:11.245648289 +0000 UTC m=+1638.163891139" Jul 06 22:47:13.104661 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:13.104631 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:13.750851 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:13.750815 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:13.847785 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:13.847750 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:14.186049 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:14.186013 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:17.238394 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:17.238358 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-29698a60-75f978db7c-z88vr" Jul 06 22:47:20.504047 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:20.504013 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:20.604062 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:20.604024 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:20.916971 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:20.916936 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:21.293745 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:21.293666 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-29698a60-75f978db7c-z88vr"] Jul 06 22:47:21.293892 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:21.293864 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-29698a60-75f978db7c-z88vr" podUID="c3f08988-0ff1-4891-a0a7-012fc27415c8" containerName="maas-api" containerID="cri-o://f4e70659b5a7bcc7ac8198c169cb54e267e6abc8ff4da033e48f25bbf69434ef" gracePeriod=30 Jul 06 22:47:21.312605 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:21.312578 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:21.410144 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:21.410108 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:21.546470 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:21.546403 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-29698a60-75f978db7c-z88vr" Jul 06 22:47:21.650347 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:21.648763 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qh9p8\" (UniqueName: \"kubernetes.io/projected/c3f08988-0ff1-4891-a0a7-012fc27415c8-kube-api-access-qh9p8\") pod \"c3f08988-0ff1-4891-a0a7-012fc27415c8\" (UID: \"c3f08988-0ff1-4891-a0a7-012fc27415c8\") " Jul 06 22:47:21.650347 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:21.648843 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3f08988-0ff1-4891-a0a7-012fc27415c8-maas-api-tls\") pod \"c3f08988-0ff1-4891-a0a7-012fc27415c8\" (UID: \"c3f08988-0ff1-4891-a0a7-012fc27415c8\") " Jul 06 22:47:21.650347 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:21.648959 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3f08988-0ff1-4891-a0a7-012fc27415c8-openshift-service-ca\") pod \"c3f08988-0ff1-4891-a0a7-012fc27415c8\" (UID: \"c3f08988-0ff1-4891-a0a7-012fc27415c8\") " Jul 06 22:47:21.650347 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:21.650050 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3f08988-0ff1-4891-a0a7-012fc27415c8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c3f08988-0ff1-4891-a0a7-012fc27415c8" (UID: "c3f08988-0ff1-4891-a0a7-012fc27415c8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:47:21.657079 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:21.657044 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3f08988-0ff1-4891-a0a7-012fc27415c8-kube-api-access-qh9p8" (OuterVolumeSpecName: "kube-api-access-qh9p8") pod "c3f08988-0ff1-4891-a0a7-012fc27415c8" (UID: "c3f08988-0ff1-4891-a0a7-012fc27415c8"). InnerVolumeSpecName "kube-api-access-qh9p8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:47:21.657079 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:21.657068 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3f08988-0ff1-4891-a0a7-012fc27415c8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c3f08988-0ff1-4891-a0a7-012fc27415c8" (UID: "c3f08988-0ff1-4891-a0a7-012fc27415c8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:47:21.750847 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:21.750807 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3f08988-0ff1-4891-a0a7-012fc27415c8-maas-api-tls\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:47:21.750847 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:21.750841 2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3f08988-0ff1-4891-a0a7-012fc27415c8-openshift-service-ca\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:47:21.750847 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:21.750855 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh9p8\" (UniqueName: \"kubernetes.io/projected/c3f08988-0ff1-4891-a0a7-012fc27415c8-kube-api-access-qh9p8\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:47:22.285065 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:22.285028 2571 generic.go:356] "Generic (PLEG): container finished" podID="c3f08988-0ff1-4891-a0a7-012fc27415c8" containerID="f4e70659b5a7bcc7ac8198c169cb54e267e6abc8ff4da033e48f25bbf69434ef" exitCode=0 Jul 06 22:47:22.285303 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:22.285089 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-29698a60-75f978db7c-z88vr" Jul 06 22:47:22.285303 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:22.285126 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-29698a60-75f978db7c-z88vr" event={"ID":"c3f08988-0ff1-4891-a0a7-012fc27415c8","Type":"ContainerDied","Data":"f4e70659b5a7bcc7ac8198c169cb54e267e6abc8ff4da033e48f25bbf69434ef"} Jul 06 22:47:22.285303 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:22.285235 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-29698a60-75f978db7c-z88vr" event={"ID":"c3f08988-0ff1-4891-a0a7-012fc27415c8","Type":"ContainerDied","Data":"81c21ec19be55371bc640f2ebf7e97d34d7cefaff0570ad2e97aa0ae70d9b5b1"} Jul 06 22:47:22.285303 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:22.285269 2571 scope.go:117] "RemoveContainer" containerID="f4e70659b5a7bcc7ac8198c169cb54e267e6abc8ff4da033e48f25bbf69434ef" Jul 06 22:47:22.294245 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:22.294196 2571 scope.go:117] "RemoveContainer" containerID="f4e70659b5a7bcc7ac8198c169cb54e267e6abc8ff4da033e48f25bbf69434ef" Jul 06 22:47:22.294549 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:47:22.294530 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4e70659b5a7bcc7ac8198c169cb54e267e6abc8ff4da033e48f25bbf69434ef\": container with ID starting with f4e70659b5a7bcc7ac8198c169cb54e267e6abc8ff4da033e48f25bbf69434ef not found: ID does not exist" containerID="f4e70659b5a7bcc7ac8198c169cb54e267e6abc8ff4da033e48f25bbf69434ef" Jul 06 22:47:22.294613 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:22.294558 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4e70659b5a7bcc7ac8198c169cb54e267e6abc8ff4da033e48f25bbf69434ef"} err="failed to get container status \"f4e70659b5a7bcc7ac8198c169cb54e267e6abc8ff4da033e48f25bbf69434ef\": rpc error: code = NotFound desc = could not find container \"f4e70659b5a7bcc7ac8198c169cb54e267e6abc8ff4da033e48f25bbf69434ef\": container with ID starting with f4e70659b5a7bcc7ac8198c169cb54e267e6abc8ff4da033e48f25bbf69434ef not found: ID does not exist" Jul 06 22:47:22.308241 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:22.308194 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-29698a60-75f978db7c-z88vr"] Jul 06 22:47:22.314954 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:22.314927 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-29698a60-75f978db7c-z88vr"] Jul 06 22:47:23.634937 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:23.634906 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3f08988-0ff1-4891-a0a7-012fc27415c8" path="/var/lib/kubelet/pods/c3f08988-0ff1-4891-a0a7-012fc27415c8/volumes" Jul 06 22:47:27.357999 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:27.357963 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:27.415587 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:27.415560 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:27.624537 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:27.624448 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:27.853575 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:27.853540 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:27.974190 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:27.974158 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:29.464341 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:29.464306 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:31.445659 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:31.445625 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:34.418509 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:34.418472 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:34.522186 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:34.522131 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:34.618462 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:34.618429 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:34.751475 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:34.751346 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:40.732027 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:40.731999 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:40.787396 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:40.787363 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:41.684872 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:41.684837 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:47.514355 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:47.514314 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:47.824769 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:47.824669 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:48.124513 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:48.124411 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:50.852938 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:50.852893 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:50.927868 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:50.927829 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-09d58787-5b69699c46-7z7f9"] Jul 06 22:47:50.928148 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:50.928043 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:50.928386 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:50.928367 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3f08988-0ff1-4891-a0a7-012fc27415c8" containerName="maas-api" Jul 06 22:47:50.928481 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:50.928388 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3f08988-0ff1-4891-a0a7-012fc27415c8" containerName="maas-api" Jul 06 22:47:50.928535 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:50.928502 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3f08988-0ff1-4891-a0a7-012fc27415c8" containerName="maas-api" Jul 06 22:47:50.931151 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:50.931129 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-09d58787-5b69699c46-7z7f9" Jul 06 22:47:50.933476 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:50.933453 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-09d58787-serving-cert\"" Jul 06 22:47:50.933614 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:50.933593 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-98x7w\"" Jul 06 22:47:50.943102 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:50.943073 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-09d58787-5b69699c46-7z7f9"] Jul 06 22:47:50.992791 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:50.992756 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:51.114392 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:51.114296 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvfgv\" (UniqueName: \"kubernetes.io/projected/47c6f238-a633-4434-a836-382770581ac6-kube-api-access-mvfgv\") pod \"maas-api-e2e-mt-09d58787-5b69699c46-7z7f9\" (UID: \"47c6f238-a633-4434-a836-382770581ac6\") " pod="opendatahub/maas-api-e2e-mt-09d58787-5b69699c46-7z7f9" Jul 06 22:47:51.114392 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:51.114364 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47c6f238-a633-4434-a836-382770581ac6-openshift-service-ca\") pod \"maas-api-e2e-mt-09d58787-5b69699c46-7z7f9\" (UID: \"47c6f238-a633-4434-a836-382770581ac6\") " pod="opendatahub/maas-api-e2e-mt-09d58787-5b69699c46-7z7f9" Jul 06 22:47:51.114660 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:51.114428 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47c6f238-a633-4434-a836-382770581ac6-maas-api-tls\") pod \"maas-api-e2e-mt-09d58787-5b69699c46-7z7f9\" (UID: \"47c6f238-a633-4434-a836-382770581ac6\") " pod="opendatahub/maas-api-e2e-mt-09d58787-5b69699c46-7z7f9" Jul 06 22:47:51.215420 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:51.215366 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvfgv\" (UniqueName: \"kubernetes.io/projected/47c6f238-a633-4434-a836-382770581ac6-kube-api-access-mvfgv\") pod \"maas-api-e2e-mt-09d58787-5b69699c46-7z7f9\" (UID: \"47c6f238-a633-4434-a836-382770581ac6\") " pod="opendatahub/maas-api-e2e-mt-09d58787-5b69699c46-7z7f9" Jul 06 22:47:51.215653 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:51.215448 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47c6f238-a633-4434-a836-382770581ac6-openshift-service-ca\") pod \"maas-api-e2e-mt-09d58787-5b69699c46-7z7f9\" (UID: \"47c6f238-a633-4434-a836-382770581ac6\") " pod="opendatahub/maas-api-e2e-mt-09d58787-5b69699c46-7z7f9" Jul 06 22:47:51.215653 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:51.215468 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47c6f238-a633-4434-a836-382770581ac6-maas-api-tls\") pod \"maas-api-e2e-mt-09d58787-5b69699c46-7z7f9\" (UID: \"47c6f238-a633-4434-a836-382770581ac6\") " pod="opendatahub/maas-api-e2e-mt-09d58787-5b69699c46-7z7f9" Jul 06 22:47:51.215653 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:47:51.215627 2571 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-mt-09d58787-serving-cert: secret "maas-api-e2e-mt-09d58787-serving-cert" not found Jul 06 22:47:51.215845 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:47:51.215716 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47c6f238-a633-4434-a836-382770581ac6-maas-api-tls podName:47c6f238-a633-4434-a836-382770581ac6 nodeName:}" failed. No retries permitted until 2026-07-06 22:47:51.715692201 +0000 UTC m=+1678.633935049 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/47c6f238-a633-4434-a836-382770581ac6-maas-api-tls") pod "maas-api-e2e-mt-09d58787-5b69699c46-7z7f9" (UID: "47c6f238-a633-4434-a836-382770581ac6") : secret "maas-api-e2e-mt-09d58787-serving-cert" not found Jul 06 22:47:51.216197 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:51.216179 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47c6f238-a633-4434-a836-382770581ac6-openshift-service-ca\") pod \"maas-api-e2e-mt-09d58787-5b69699c46-7z7f9\" (UID: \"47c6f238-a633-4434-a836-382770581ac6\") " pod="opendatahub/maas-api-e2e-mt-09d58787-5b69699c46-7z7f9" Jul 06 22:47:51.225294 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:51.225275 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvfgv\" (UniqueName: \"kubernetes.io/projected/47c6f238-a633-4434-a836-382770581ac6-kube-api-access-mvfgv\") pod \"maas-api-e2e-mt-09d58787-5b69699c46-7z7f9\" (UID: \"47c6f238-a633-4434-a836-382770581ac6\") " pod="opendatahub/maas-api-e2e-mt-09d58787-5b69699c46-7z7f9" Jul 06 22:47:51.422677 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:51.422640 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:51.524703 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:51.524665 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:51.625101 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:51.625070 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:51.720685 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:51.720595 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47c6f238-a633-4434-a836-382770581ac6-maas-api-tls\") pod \"maas-api-e2e-mt-09d58787-5b69699c46-7z7f9\" (UID: \"47c6f238-a633-4434-a836-382770581ac6\") " pod="opendatahub/maas-api-e2e-mt-09d58787-5b69699c46-7z7f9" Jul 06 22:47:51.723182 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:51.723148 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47c6f238-a633-4434-a836-382770581ac6-maas-api-tls\") pod \"maas-api-e2e-mt-09d58787-5b69699c46-7z7f9\" (UID: \"47c6f238-a633-4434-a836-382770581ac6\") " pod="opendatahub/maas-api-e2e-mt-09d58787-5b69699c46-7z7f9" Jul 06 22:47:51.844280 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:51.844239 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-09d58787-5b69699c46-7z7f9" Jul 06 22:47:51.991907 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:51.991875 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-09d58787-5b69699c46-7z7f9"] Jul 06 22:47:51.994035 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:47:51.994008 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47c6f238_a633_4434_a836_382770581ac6.slice/crio-0fee3b51c49c714c408d1fe3b76c3aac1654cda00c0eac19c9c4c9cf47c22c62 WatchSource:0}: Error finding container 0fee3b51c49c714c408d1fe3b76c3aac1654cda00c0eac19c9c4c9cf47c22c62: Status 404 returned error can't find the container with id 0fee3b51c49c714c408d1fe3b76c3aac1654cda00c0eac19c9c4c9cf47c22c62 Jul 06 22:47:52.435363 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:52.435321 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-09d58787-5b69699c46-7z7f9" event={"ID":"47c6f238-a633-4434-a836-382770581ac6","Type":"ContainerStarted","Data":"0fee3b51c49c714c408d1fe3b76c3aac1654cda00c0eac19c9c4c9cf47c22c62"} Jul 06 22:47:52.803141 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:52.803051 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:52.901874 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:52.901838 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:53.442171 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:53.442132 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-09d58787-5b69699c46-7z7f9" event={"ID":"47c6f238-a633-4434-a836-382770581ac6","Type":"ContainerStarted","Data":"ca50d112aa431891e3ca9ac36a168fc56e908a9ea1d6df31134a77581245fad7"} Jul 06 22:47:53.442604 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:53.442193 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-09d58787-5b69699c46-7z7f9" Jul 06 22:47:53.459817 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:53.459769 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-09d58787-5b69699c46-7z7f9" podStartSLOduration=3.000331539 podStartE2EDuration="3.4597539s" totalImagesPullingTime="459.422361ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:47:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:47:53.45840287 +0000 UTC m=+1680.376645728" watchObservedRunningTime="2026-07-06 22:47:53.4597539 +0000 UTC m=+1680.377996766" Jul 06 22:47:55.356068 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:55.356028 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:59.452987 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:47:59.452953 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-09d58787-5b69699c46-7z7f9" Jul 06 22:48:12.219266 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:48:12.219232 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:48:12.318891 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:48:12.318858 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:48:18.805920 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:48:18.805841 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-09d58787/e2e-lifecycle-model-09d58787-kserve-f6bcc55d7-cfh42"] Jul 06 22:48:18.809589 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:48:18.809565 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-09d58787/e2e-lifecycle-model-09d58787-kserve-f6bcc55d7-cfh42" Jul 06 22:48:18.811917 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:48:18.811895 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-09d58787\"/\"default-dockercfg-nds6g\"" Jul 06 22:48:18.813140 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:48:18.813121 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-09d58787\"/\"openshift-service-ca.crt\"" Jul 06 22:48:18.813255 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:48:18.813182 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-09d58787\"/\"kube-root-ca.crt\"" Jul 06 22:48:18.813255 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:48:18.813123 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-09d58787\"/\"e2e-lifecycle-model-09d58787-kserve-self-signed-certs\"" Jul 06 22:48:18.819726 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:48:18.819703 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-09d58787/e2e-lifecycle-model-09d58787-kserve-f6bcc55d7-cfh42"] Jul 06 22:48:18.827629 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:48:18.827598 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:48:18.857804 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:48:18.857759 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3552039-a4be-4a33-82f4-423e22b58a6b-home\") pod \"e2e-lifecycle-model-09d58787-kserve-f6bcc55d7-cfh42\" (UID: \"f3552039-a4be-4a33-82f4-423e22b58a6b\") " pod="ai-tenant-e2e-mt-09d58787/e2e-lifecycle-model-09d58787-kserve-f6bcc55d7-cfh42" Jul 06 22:48:18.858019 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:48:18.857831 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3552039-a4be-4a33-82f4-423e22b58a6b-dshm\") pod \"e2e-lifecycle-model-09d58787-kserve-f6bcc55d7-cfh42\" (UID: \"f3552039-a4be-4a33-82f4-423e22b58a6b\") " pod="ai-tenant-e2e-mt-09d58787/e2e-lifecycle-model-09d58787-kserve-f6bcc55d7-cfh42" Jul 06 22:48:18.858019 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:48:18.857861 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3552039-a4be-4a33-82f4-423e22b58a6b-model-cache\") pod \"e2e-lifecycle-model-09d58787-kserve-f6bcc55d7-cfh42\" (UID: \"f3552039-a4be-4a33-82f4-423e22b58a6b\") " pod="ai-tenant-e2e-mt-09d58787/e2e-lifecycle-model-09d58787-kserve-f6bcc55d7-cfh42" Jul 06 22:48:18.858019 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:48:18.857902 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3552039-a4be-4a33-82f4-423e22b58a6b-tls-certs\") pod \"e2e-lifecycle-model-09d58787-kserve-f6bcc55d7-cfh42\" (UID: \"f3552039-a4be-4a33-82f4-423e22b58a6b\") " pod="ai-tenant-e2e-mt-09d58787/e2e-lifecycle-model-09d58787-kserve-f6bcc55d7-cfh42" Jul 06 22:48:18.858019 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:48:18.857984 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3552039-a4be-4a33-82f4-423e22b58a6b-tmp-dir\") pod \"e2e-lifecycle-model-09d58787-kserve-f6bcc55d7-cfh42\" (UID: \"f3552039-a4be-4a33-82f4-423e22b58a6b\") " pod="ai-tenant-e2e-mt-09d58787/e2e-lifecycle-model-09d58787-kserve-f6bcc55d7-cfh42" Jul 06 22:48:18.858019 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:48:18.858009 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h8rl\" (UniqueName: \"kubernetes.io/projected/f3552039-a4be-4a33-82f4-423e22b58a6b-kube-api-access-2h8rl\") pod \"e2e-lifecycle-model-09d58787-kserve-f6bcc55d7-cfh42\" (UID: \"f3552039-a4be-4a33-82f4-423e22b58a6b\") " pod="ai-tenant-e2e-mt-09d58787/e2e-lifecycle-model-09d58787-kserve-f6bcc55d7-cfh42" Jul 06 22:48:18.959314 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:48:18.959275 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3552039-a4be-4a33-82f4-423e22b58a6b-tls-certs\") pod \"e2e-lifecycle-model-09d58787-kserve-f6bcc55d7-cfh42\" (UID: \"f3552039-a4be-4a33-82f4-423e22b58a6b\") " pod="ai-tenant-e2e-mt-09d58787/e2e-lifecycle-model-09d58787-kserve-f6bcc55d7-cfh42" Jul 06 22:48:18.959513 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:48:18.959339 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3552039-a4be-4a33-82f4-423e22b58a6b-tmp-dir\") pod \"e2e-lifecycle-model-09d58787-kserve-f6bcc55d7-cfh42\" (UID: \"f3552039-a4be-4a33-82f4-423e22b58a6b\") " pod="ai-tenant-e2e-mt-09d58787/e2e-lifecycle-model-09d58787-kserve-f6bcc55d7-cfh42" Jul 06 22:48:18.959513 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:48:18.959366 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h8rl\" (UniqueName: \"kubernetes.io/projected/f3552039-a4be-4a33-82f4-423e22b58a6b-kube-api-access-2h8rl\") pod \"e2e-lifecycle-model-09d58787-kserve-f6bcc55d7-cfh42\" (UID: \"f3552039-a4be-4a33-82f4-423e22b58a6b\") " pod="ai-tenant-e2e-mt-09d58787/e2e-lifecycle-model-09d58787-kserve-f6bcc55d7-cfh42" Jul 06 22:48:18.959513 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:48:18.959391 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3552039-a4be-4a33-82f4-423e22b58a6b-home\") pod \"e2e-lifecycle-model-09d58787-kserve-f6bcc55d7-cfh42\" (UID: \"f3552039-a4be-4a33-82f4-423e22b58a6b\") " pod="ai-tenant-e2e-mt-09d58787/e2e-lifecycle-model-09d58787-kserve-f6bcc55d7-cfh42" Jul 06 22:48:18.959513 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:48:18.959421 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3552039-a4be-4a33-82f4-423e22b58a6b-dshm\") pod \"e2e-lifecycle-model-09d58787-kserve-f6bcc55d7-cfh42\" (UID: \"f3552039-a4be-4a33-82f4-423e22b58a6b\") " pod="ai-tenant-e2e-mt-09d58787/e2e-lifecycle-model-09d58787-kserve-f6bcc55d7-cfh42" Jul 06 22:48:18.959513 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:48:18.959438 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3552039-a4be-4a33-82f4-423e22b58a6b-model-cache\") pod \"e2e-lifecycle-model-09d58787-kserve-f6bcc55d7-cfh42\" (UID: \"f3552039-a4be-4a33-82f4-423e22b58a6b\") " pod="ai-tenant-e2e-mt-09d58787/e2e-lifecycle-model-09d58787-kserve-f6bcc55d7-cfh42" Jul 06 22:48:18.959837 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:48:18.959811 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3552039-a4be-4a33-82f4-423e22b58a6b-home\") pod \"e2e-lifecycle-model-09d58787-kserve-f6bcc55d7-cfh42\" (UID: \"f3552039-a4be-4a33-82f4-423e22b58a6b\") " pod="ai-tenant-e2e-mt-09d58787/e2e-lifecycle-model-09d58787-kserve-f6bcc55d7-cfh42" Jul 06 22:48:18.959898 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:48:18.959843 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3552039-a4be-4a33-82f4-423e22b58a6b-tmp-dir\") pod \"e2e-lifecycle-model-09d58787-kserve-f6bcc55d7-cfh42\" (UID: \"f3552039-a4be-4a33-82f4-423e22b58a6b\") " pod="ai-tenant-e2e-mt-09d58787/e2e-lifecycle-model-09d58787-kserve-f6bcc55d7-cfh42" Jul 06 22:48:18.959940 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:48:18.959886 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3552039-a4be-4a33-82f4-423e22b58a6b-model-cache\") pod \"e2e-lifecycle-model-09d58787-kserve-f6bcc55d7-cfh42\" (UID: \"f3552039-a4be-4a33-82f4-423e22b58a6b\") " pod="ai-tenant-e2e-mt-09d58787/e2e-lifecycle-model-09d58787-kserve-f6bcc55d7-cfh42" Jul 06 22:48:18.961804 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:48:18.961779 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3552039-a4be-4a33-82f4-423e22b58a6b-dshm\") pod \"e2e-lifecycle-model-09d58787-kserve-f6bcc55d7-cfh42\" (UID: \"f3552039-a4be-4a33-82f4-423e22b58a6b\") " pod="ai-tenant-e2e-mt-09d58787/e2e-lifecycle-model-09d58787-kserve-f6bcc55d7-cfh42" Jul 06 22:48:18.961933 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:48:18.961915 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3552039-a4be-4a33-82f4-423e22b58a6b-tls-certs\") pod \"e2e-lifecycle-model-09d58787-kserve-f6bcc55d7-cfh42\" (UID: \"f3552039-a4be-4a33-82f4-423e22b58a6b\") " pod="ai-tenant-e2e-mt-09d58787/e2e-lifecycle-model-09d58787-kserve-f6bcc55d7-cfh42" Jul 06 22:48:18.968794 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:48:18.968773 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h8rl\" (UniqueName: \"kubernetes.io/projected/f3552039-a4be-4a33-82f4-423e22b58a6b-kube-api-access-2h8rl\") pod \"e2e-lifecycle-model-09d58787-kserve-f6bcc55d7-cfh42\" (UID: \"f3552039-a4be-4a33-82f4-423e22b58a6b\") " pod="ai-tenant-e2e-mt-09d58787/e2e-lifecycle-model-09d58787-kserve-f6bcc55d7-cfh42" Jul 06 22:48:19.120022 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:48:19.119933 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-09d58787/e2e-lifecycle-model-09d58787-kserve-f6bcc55d7-cfh42" Jul 06 22:48:19.260908 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:48:19.260879 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-09d58787/e2e-lifecycle-model-09d58787-kserve-f6bcc55d7-cfh42"] Jul 06 22:48:19.263562 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:48:19.263520 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3552039_a4be_4a33_82f4_423e22b58a6b.slice/crio-75341a8d229f4bb3a90cb8b94b4ab2a2668dc774e5371cf080e46e0267006fb6 WatchSource:0}: Error finding container 75341a8d229f4bb3a90cb8b94b4ab2a2668dc774e5371cf080e46e0267006fb6: Status 404 returned error can't find the container with id 75341a8d229f4bb3a90cb8b94b4ab2a2668dc774e5371cf080e46e0267006fb6 Jul 06 22:48:19.572764 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:48:19.572724 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-09d58787/e2e-lifecycle-model-09d58787-kserve-f6bcc55d7-cfh42" event={"ID":"f3552039-a4be-4a33-82f4-423e22b58a6b","Type":"ContainerStarted","Data":"612abb01f044db2cd5880d6c7e7b549c4bae8609ec36a2200e8194a6fac8301e"} Jul 06 22:48:19.572953 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:48:19.572775 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-09d58787/e2e-lifecycle-model-09d58787-kserve-f6bcc55d7-cfh42" event={"ID":"f3552039-a4be-4a33-82f4-423e22b58a6b","Type":"ContainerStarted","Data":"75341a8d229f4bb3a90cb8b94b4ab2a2668dc774e5371cf080e46e0267006fb6"} Jul 06 22:48:19.589653 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:48:19.589577 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-09d58787/e2e-lifecycle-model-09d58787-kserve-f6bcc55d7-cfh42" podStartSLOduration=1.589561533 podStartE2EDuration="1.589561533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:48:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:48:19.588833982 +0000 UTC m=+1706.507076832" watchObservedRunningTime="2026-07-06 22:48:19.589561533 +0000 UTC m=+1706.507804389" Jul 06 22:48:29.120309 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:48:29.120273 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-09d58787/e2e-lifecycle-model-09d58787-kserve-f6bcc55d7-cfh42" Jul 06 22:48:29.120309 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:48:29.120317 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-09d58787/e2e-lifecycle-model-09d58787-kserve-f6bcc55d7-cfh42" Jul 06 22:48:29.132905 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:48:29.132871 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-09d58787/e2e-lifecycle-model-09d58787-kserve-f6bcc55d7-cfh42" Jul 06 22:48:29.636552 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:48:29.636518 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-09d58787/e2e-lifecycle-model-09d58787-kserve-f6bcc55d7-cfh42" Jul 06 22:48:37.984901 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:48:37.984868 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:48:38.368620 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:48:38.368530 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-09d58787-5b69699c46-7z7f9"] Jul 06 22:48:38.368780 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:48:38.368760 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-09d58787-5b69699c46-7z7f9" podUID="47c6f238-a633-4434-a836-382770581ac6" containerName="maas-api" containerID="cri-o://ca50d112aa431891e3ca9ac36a168fc56e908a9ea1d6df31134a77581245fad7" gracePeriod=30 Jul 06 22:48:38.393666 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:48:38.393590 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:48:38.607714 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:48:38.607688 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-09d58787-5b69699c46-7z7f9" Jul 06 22:48:38.665392 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:48:38.665355 2571 generic.go:356] "Generic (PLEG): container finished" podID="47c6f238-a633-4434-a836-382770581ac6" containerID="ca50d112aa431891e3ca9ac36a168fc56e908a9ea1d6df31134a77581245fad7" exitCode=0 Jul 06 22:48:38.665602 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:48:38.665417 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-09d58787-5b69699c46-7z7f9" Jul 06 22:48:38.665602 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:48:38.665435 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-09d58787-5b69699c46-7z7f9" event={"ID":"47c6f238-a633-4434-a836-382770581ac6","Type":"ContainerDied","Data":"ca50d112aa431891e3ca9ac36a168fc56e908a9ea1d6df31134a77581245fad7"} Jul 06 22:48:38.665602 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:48:38.665492 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-09d58787-5b69699c46-7z7f9" event={"ID":"47c6f238-a633-4434-a836-382770581ac6","Type":"ContainerDied","Data":"0fee3b51c49c714c408d1fe3b76c3aac1654cda00c0eac19c9c4c9cf47c22c62"} Jul 06 22:48:38.665602 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:48:38.665516 2571 scope.go:117] "RemoveContainer" containerID="ca50d112aa431891e3ca9ac36a168fc56e908a9ea1d6df31134a77581245fad7" Jul 06 22:48:38.674562 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:48:38.674543 2571 scope.go:117] "RemoveContainer" containerID="ca50d112aa431891e3ca9ac36a168fc56e908a9ea1d6df31134a77581245fad7" Jul 06 22:48:38.674862 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:48:38.674844 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca50d112aa431891e3ca9ac36a168fc56e908a9ea1d6df31134a77581245fad7\": container with ID starting with ca50d112aa431891e3ca9ac36a168fc56e908a9ea1d6df31134a77581245fad7 not found: ID does not exist" containerID="ca50d112aa431891e3ca9ac36a168fc56e908a9ea1d6df31134a77581245fad7" Jul 06 22:48:38.674933 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:48:38.674871 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca50d112aa431891e3ca9ac36a168fc56e908a9ea1d6df31134a77581245fad7"} err="failed to get container status \"ca50d112aa431891e3ca9ac36a168fc56e908a9ea1d6df31134a77581245fad7\": rpc error: code = NotFound desc = could not find container \"ca50d112aa431891e3ca9ac36a168fc56e908a9ea1d6df31134a77581245fad7\": container with ID starting with ca50d112aa431891e3ca9ac36a168fc56e908a9ea1d6df31134a77581245fad7 not found: ID does not exist" Jul 06 22:48:38.742816 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:48:38.742781 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47c6f238-a633-4434-a836-382770581ac6-openshift-service-ca\") pod \"47c6f238-a633-4434-a836-382770581ac6\" (UID: \"47c6f238-a633-4434-a836-382770581ac6\") " Jul 06 22:48:38.742993 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:48:38.742865 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47c6f238-a633-4434-a836-382770581ac6-maas-api-tls\") pod \"47c6f238-a633-4434-a836-382770581ac6\" (UID: \"47c6f238-a633-4434-a836-382770581ac6\") " Jul 06 22:48:38.742993 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:48:38.742899 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mvfgv\" (UniqueName: \"kubernetes.io/projected/47c6f238-a633-4434-a836-382770581ac6-kube-api-access-mvfgv\") pod \"47c6f238-a633-4434-a836-382770581ac6\" (UID: \"47c6f238-a633-4434-a836-382770581ac6\") " Jul 06 22:48:38.743196 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:48:38.743168 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47c6f238-a633-4434-a836-382770581ac6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "47c6f238-a633-4434-a836-382770581ac6" (UID: "47c6f238-a633-4434-a836-382770581ac6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:48:38.745115 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:48:38.745083 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47c6f238-a633-4434-a836-382770581ac6-kube-api-access-mvfgv" (OuterVolumeSpecName: "kube-api-access-mvfgv") pod "47c6f238-a633-4434-a836-382770581ac6" (UID: "47c6f238-a633-4434-a836-382770581ac6"). InnerVolumeSpecName "kube-api-access-mvfgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:48:38.745115 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:48:38.745102 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47c6f238-a633-4434-a836-382770581ac6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "47c6f238-a633-4434-a836-382770581ac6" (UID: "47c6f238-a633-4434-a836-382770581ac6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:48:38.844401 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:48:38.844365 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47c6f238-a633-4434-a836-382770581ac6-maas-api-tls\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:48:38.844401 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:48:38.844394 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvfgv\" (UniqueName: \"kubernetes.io/projected/47c6f238-a633-4434-a836-382770581ac6-kube-api-access-mvfgv\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:48:38.844401 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:48:38.844406 2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47c6f238-a633-4434-a836-382770581ac6-openshift-service-ca\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:48:38.988283 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:48:38.988251 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-09d58787-5b69699c46-7z7f9"] Jul 06 22:48:38.990515 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:48:38.990491 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-09d58787-5b69699c46-7z7f9"] Jul 06 22:48:39.635784 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:48:39.635751 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47c6f238-a633-4434-a836-382770581ac6" path="/var/lib/kubelet/pods/47c6f238-a633-4434-a836-382770581ac6/volumes" Jul 06 22:48:44.052689 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:48:44.052656 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:48:45.765082 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:48:45.765034 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:48:45.818941 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:48:45.818899 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:48:49.035589 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:48:49.035549 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-09d58787/e2e-lifecycle-model-09d58787-kserve-f6bcc55d7-cfh42"] Jul 06 22:48:49.078939 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:48:49.078904 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:04.056949 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:49:04.056908 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-mt-09d58787/e2e-lifecycle-model-09d58787-kserve-f6bcc55d7-cfh42" podUID="f3552039-a4be-4a33-82f4-423e22b58a6b" containerName="main" containerID="cri-o://612abb01f044db2cd5880d6c7e7b549c4bae8609ec36a2200e8194a6fac8301e" gracePeriod=45 Jul 06 22:49:04.316992 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:49:04.316920 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-09d58787/e2e-lifecycle-model-09d58787-kserve-f6bcc55d7-cfh42" Jul 06 22:49:04.369997 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:49:04.369959 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3552039-a4be-4a33-82f4-423e22b58a6b-home\") pod \"f3552039-a4be-4a33-82f4-423e22b58a6b\" (UID: \"f3552039-a4be-4a33-82f4-423e22b58a6b\") " Jul 06 22:49:04.369997 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:49:04.370005 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3552039-a4be-4a33-82f4-423e22b58a6b-tmp-dir\") pod \"f3552039-a4be-4a33-82f4-423e22b58a6b\" (UID: \"f3552039-a4be-4a33-82f4-423e22b58a6b\") " Jul 06 22:49:04.370384 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:49:04.370034 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3552039-a4be-4a33-82f4-423e22b58a6b-tls-certs\") pod \"f3552039-a4be-4a33-82f4-423e22b58a6b\" (UID: \"f3552039-a4be-4a33-82f4-423e22b58a6b\") " Jul 06 22:49:04.370384 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:49:04.370058 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3552039-a4be-4a33-82f4-423e22b58a6b-model-cache\") pod \"f3552039-a4be-4a33-82f4-423e22b58a6b\" (UID: \"f3552039-a4be-4a33-82f4-423e22b58a6b\") " Jul 06 22:49:04.370384 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:49:04.370130 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2h8rl\" (UniqueName: \"kubernetes.io/projected/f3552039-a4be-4a33-82f4-423e22b58a6b-kube-api-access-2h8rl\") pod \"f3552039-a4be-4a33-82f4-423e22b58a6b\" (UID: \"f3552039-a4be-4a33-82f4-423e22b58a6b\") " Jul 06 22:49:04.370384 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:49:04.370156 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3552039-a4be-4a33-82f4-423e22b58a6b-dshm\") pod \"f3552039-a4be-4a33-82f4-423e22b58a6b\" (UID: \"f3552039-a4be-4a33-82f4-423e22b58a6b\") " Jul 06 22:49:04.370384 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:49:04.370263 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3552039-a4be-4a33-82f4-423e22b58a6b-home" (OuterVolumeSpecName: "home") pod "f3552039-a4be-4a33-82f4-423e22b58a6b" (UID: "f3552039-a4be-4a33-82f4-423e22b58a6b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:49:04.370384 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:49:04.370341 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3552039-a4be-4a33-82f4-423e22b58a6b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f3552039-a4be-4a33-82f4-423e22b58a6b" (UID: "f3552039-a4be-4a33-82f4-423e22b58a6b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:49:04.370718 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:49:04.370401 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3552039-a4be-4a33-82f4-423e22b58a6b-model-cache" (OuterVolumeSpecName: "model-cache") pod "f3552039-a4be-4a33-82f4-423e22b58a6b" (UID: "f3552039-a4be-4a33-82f4-423e22b58a6b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:49:04.370718 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:49:04.370560 2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3552039-a4be-4a33-82f4-423e22b58a6b-home\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:49:04.370718 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:49:04.370578 2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3552039-a4be-4a33-82f4-423e22b58a6b-tmp-dir\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:49:04.370718 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:49:04.370590 2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3552039-a4be-4a33-82f4-423e22b58a6b-model-cache\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:49:04.372432 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:49:04.372408 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3552039-a4be-4a33-82f4-423e22b58a6b-dshm" (OuterVolumeSpecName: "dshm") pod "f3552039-a4be-4a33-82f4-423e22b58a6b" (UID: "f3552039-a4be-4a33-82f4-423e22b58a6b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:49:04.372684 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:49:04.372654 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3552039-a4be-4a33-82f4-423e22b58a6b-kube-api-access-2h8rl" (OuterVolumeSpecName: "kube-api-access-2h8rl") pod "f3552039-a4be-4a33-82f4-423e22b58a6b" (UID: "f3552039-a4be-4a33-82f4-423e22b58a6b"). InnerVolumeSpecName "kube-api-access-2h8rl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:49:04.372684 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:49:04.372682 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3552039-a4be-4a33-82f4-423e22b58a6b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f3552039-a4be-4a33-82f4-423e22b58a6b" (UID: "f3552039-a4be-4a33-82f4-423e22b58a6b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:49:04.471546 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:49:04.471502 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h8rl\" (UniqueName: \"kubernetes.io/projected/f3552039-a4be-4a33-82f4-423e22b58a6b-kube-api-access-2h8rl\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:49:04.471546 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:49:04.471542 2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3552039-a4be-4a33-82f4-423e22b58a6b-dshm\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:49:04.471546 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:49:04.471556 2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3552039-a4be-4a33-82f4-423e22b58a6b-tls-certs\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:49:04.788088 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:49:04.788049 2571 generic.go:356] "Generic (PLEG): container finished" podID="f3552039-a4be-4a33-82f4-423e22b58a6b" containerID="612abb01f044db2cd5880d6c7e7b549c4bae8609ec36a2200e8194a6fac8301e" exitCode=0 Jul 06 22:49:04.788300 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:49:04.788095 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-09d58787/e2e-lifecycle-model-09d58787-kserve-f6bcc55d7-cfh42" Jul 06 22:49:04.788300 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:49:04.788119 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-09d58787/e2e-lifecycle-model-09d58787-kserve-f6bcc55d7-cfh42" event={"ID":"f3552039-a4be-4a33-82f4-423e22b58a6b","Type":"ContainerDied","Data":"612abb01f044db2cd5880d6c7e7b549c4bae8609ec36a2200e8194a6fac8301e"} Jul 06 22:49:04.788300 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:49:04.788164 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-09d58787/e2e-lifecycle-model-09d58787-kserve-f6bcc55d7-cfh42" event={"ID":"f3552039-a4be-4a33-82f4-423e22b58a6b","Type":"ContainerDied","Data":"75341a8d229f4bb3a90cb8b94b4ab2a2668dc774e5371cf080e46e0267006fb6"} Jul 06 22:49:04.788300 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:49:04.788186 2571 scope.go:117] "RemoveContainer" containerID="612abb01f044db2cd5880d6c7e7b549c4bae8609ec36a2200e8194a6fac8301e" Jul 06 22:49:04.797222 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:49:04.797188 2571 scope.go:117] "RemoveContainer" containerID="612abb01f044db2cd5880d6c7e7b549c4bae8609ec36a2200e8194a6fac8301e" Jul 06 22:49:04.797479 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:49:04.797459 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"612abb01f044db2cd5880d6c7e7b549c4bae8609ec36a2200e8194a6fac8301e\": container with ID starting with 612abb01f044db2cd5880d6c7e7b549c4bae8609ec36a2200e8194a6fac8301e not found: ID does not exist" containerID="612abb01f044db2cd5880d6c7e7b549c4bae8609ec36a2200e8194a6fac8301e" Jul 06 22:49:04.797528 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:49:04.797491 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"612abb01f044db2cd5880d6c7e7b549c4bae8609ec36a2200e8194a6fac8301e"} err="failed to get container status \"612abb01f044db2cd5880d6c7e7b549c4bae8609ec36a2200e8194a6fac8301e\": rpc error: code = NotFound desc = could not find container \"612abb01f044db2cd5880d6c7e7b549c4bae8609ec36a2200e8194a6fac8301e\": container with ID starting with 612abb01f044db2cd5880d6c7e7b549c4bae8609ec36a2200e8194a6fac8301e not found: ID does not exist" Jul 06 22:49:04.810374 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:49:04.810348 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-09d58787/e2e-lifecycle-model-09d58787-kserve-f6bcc55d7-cfh42"] Jul 06 22:49:04.812795 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:49:04.812770 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-09d58787/e2e-lifecycle-model-09d58787-kserve-f6bcc55d7-cfh42"] Jul 06 22:49:05.634154 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:49:05.634115 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3552039-a4be-4a33-82f4-423e22b58a6b" path="/var/lib/kubelet/pods/f3552039-a4be-4a33-82f4-423e22b58a6b/volumes" Jul 06 22:49:26.348540 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:49:26.348502 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:26.707773 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:49:26.707736 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:32.682916 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:49:32.682885 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:53.657436 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:49:53.657400 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-4x6pf_5796a8ea-adcb-4819-bb18-0b85e1ef4681/console-operator/2.log" Jul 06 22:49:53.660908 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:49:53.660880 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qbbrs_bf23ef00-a2d0-499c-8c87-5d6b98510ea0/ovn-acl-logging/0.log" Jul 06 22:49:53.661064 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:49:53.660889 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-4x6pf_5796a8ea-adcb-4819-bb18-0b85e1ef4681/console-operator/2.log" Jul 06 22:49:53.664567 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:49:53.664545 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qbbrs_bf23ef00-a2d0-499c-8c87-5d6b98510ea0/ovn-acl-logging/0.log" Jul 06 22:49:56.229551 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:49:56.229517 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:56.280574 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:49:56.280542 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:56.526629 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:49:56.526540 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:56.762526 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:49:56.762487 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:57.800833 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:49:57.800793 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:57.902357 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:49:57.902316 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:58.000558 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:49:58.000528 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:58.726820 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:49:58.726788 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:58.825865 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:49:58.825830 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:58.923036 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:49:58.923004 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:50:00.327099 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:50:00.327071 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:50:06.766100 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:50:06.766061 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:50:06.866010 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:50:06.865976 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:50:07.423485 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:50:07.423450 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:50:13.432344 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:50:13.432306 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:50:13.490780 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:50:13.490750 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:50:13.569650 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:50:13.569614 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:50:13.668115 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:50:13.668085 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:50:24.135391 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:50:24.135362 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:50:24.467287 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:50:24.467168 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:50:45.424372 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:50:45.424332 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:50:51.487145 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:50:51.487100 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:50:51.548557 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:50:51.548519 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:50:51.853706 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:50:51.853620 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:50:52.211417 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:50:52.211385 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:50:52.261900 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:50:52.261873 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:50:54.856184 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:50:54.856152 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:50:54.925419 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:50:54.925378 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-a-eb8d59-646bfdb49c-p5v9r"] Jul 06 22:50:54.925932 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:50:54.925913 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47c6f238-a633-4434-a836-382770581ac6" containerName="maas-api" Jul 06 22:50:54.925932 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:50:54.925933 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="47c6f238-a633-4434-a836-382770581ac6" containerName="maas-api" Jul 06 22:50:54.926121 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:50:54.925954 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3552039-a4be-4a33-82f4-423e22b58a6b" containerName="main" Jul 06 22:50:54.926121 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:50:54.925963 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3552039-a4be-4a33-82f4-423e22b58a6b" containerName="main" Jul 06 22:50:54.926121 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:50:54.926072 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3552039-a4be-4a33-82f4-423e22b58a6b" containerName="main" Jul 06 22:50:54.926121 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:50:54.926086 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="47c6f238-a633-4434-a836-382770581ac6" containerName="maas-api" Jul 06 22:50:54.927338 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:50:54.927313 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:50:54.929611 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:50:54.929588 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-eb8d59-646bfdb49c-p5v9r" Jul 06 22:50:54.931983 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:50:54.931960 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-a-eb8d59-serving-cert\"" Jul 06 22:50:54.932095 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:50:54.932000 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-98x7w\"" Jul 06 22:50:54.938781 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:50:54.938751 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-eb8d59-646bfdb49c-p5v9r"] Jul 06 22:50:54.987285 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:50:54.987249 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:50:55.045467 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:50:55.045426 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d125a1be-0505-4c5c-9c76-6a7c95b7cc57-openshift-service-ca\") pod \"maas-api-e2e-inf-a-eb8d59-646bfdb49c-p5v9r\" (UID: \"d125a1be-0505-4c5c-9c76-6a7c95b7cc57\") " pod="opendatahub/maas-api-e2e-inf-a-eb8d59-646bfdb49c-p5v9r" Jul 06 22:50:55.045664 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:50:55.045477 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d125a1be-0505-4c5c-9c76-6a7c95b7cc57-maas-api-tls\") pod \"maas-api-e2e-inf-a-eb8d59-646bfdb49c-p5v9r\" (UID: \"d125a1be-0505-4c5c-9c76-6a7c95b7cc57\") " pod="opendatahub/maas-api-e2e-inf-a-eb8d59-646bfdb49c-p5v9r" Jul 06 22:50:55.045664 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:50:55.045548 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szd5n\" (UniqueName: \"kubernetes.io/projected/d125a1be-0505-4c5c-9c76-6a7c95b7cc57-kube-api-access-szd5n\") pod \"maas-api-e2e-inf-a-eb8d59-646bfdb49c-p5v9r\" (UID: \"d125a1be-0505-4c5c-9c76-6a7c95b7cc57\") " pod="opendatahub/maas-api-e2e-inf-a-eb8d59-646bfdb49c-p5v9r" Jul 06 22:50:55.146077 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:50:55.146042 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d125a1be-0505-4c5c-9c76-6a7c95b7cc57-openshift-service-ca\") pod \"maas-api-e2e-inf-a-eb8d59-646bfdb49c-p5v9r\" (UID: \"d125a1be-0505-4c5c-9c76-6a7c95b7cc57\") " pod="opendatahub/maas-api-e2e-inf-a-eb8d59-646bfdb49c-p5v9r" Jul 06 22:50:55.146322 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:50:55.146100 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d125a1be-0505-4c5c-9c76-6a7c95b7cc57-maas-api-tls\") pod \"maas-api-e2e-inf-a-eb8d59-646bfdb49c-p5v9r\" (UID: \"d125a1be-0505-4c5c-9c76-6a7c95b7cc57\") " pod="opendatahub/maas-api-e2e-inf-a-eb8d59-646bfdb49c-p5v9r" Jul 06 22:50:55.146322 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:50:55.146143 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szd5n\" (UniqueName: \"kubernetes.io/projected/d125a1be-0505-4c5c-9c76-6a7c95b7cc57-kube-api-access-szd5n\") pod \"maas-api-e2e-inf-a-eb8d59-646bfdb49c-p5v9r\" (UID: \"d125a1be-0505-4c5c-9c76-6a7c95b7cc57\") " pod="opendatahub/maas-api-e2e-inf-a-eb8d59-646bfdb49c-p5v9r" Jul 06 22:50:55.146743 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:50:55.146713 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d125a1be-0505-4c5c-9c76-6a7c95b7cc57-openshift-service-ca\") pod \"maas-api-e2e-inf-a-eb8d59-646bfdb49c-p5v9r\" (UID: \"d125a1be-0505-4c5c-9c76-6a7c95b7cc57\") " pod="opendatahub/maas-api-e2e-inf-a-eb8d59-646bfdb49c-p5v9r" Jul 06 22:50:55.148636 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:50:55.148610 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d125a1be-0505-4c5c-9c76-6a7c95b7cc57-maas-api-tls\") pod \"maas-api-e2e-inf-a-eb8d59-646bfdb49c-p5v9r\" (UID: \"d125a1be-0505-4c5c-9c76-6a7c95b7cc57\") " pod="opendatahub/maas-api-e2e-inf-a-eb8d59-646bfdb49c-p5v9r" Jul 06 22:50:55.158047 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:50:55.158020 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szd5n\" (UniqueName: \"kubernetes.io/projected/d125a1be-0505-4c5c-9c76-6a7c95b7cc57-kube-api-access-szd5n\") pod \"maas-api-e2e-inf-a-eb8d59-646bfdb49c-p5v9r\" (UID: \"d125a1be-0505-4c5c-9c76-6a7c95b7cc57\") " pod="opendatahub/maas-api-e2e-inf-a-eb8d59-646bfdb49c-p5v9r" Jul 06 22:50:55.194123 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:50:55.194087 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:50:55.245182 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:50:55.245134 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-eb8d59-646bfdb49c-p5v9r" Jul 06 22:50:55.390602 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:50:55.390353 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-eb8d59-646bfdb49c-p5v9r"] Jul 06 22:50:55.393146 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:50:55.393114 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd125a1be_0505_4c5c_9c76_6a7c95b7cc57.slice/crio-aaadcf3149f091382cf80282e9d043890e91ba90543ea319ffd6865f7097de33 WatchSource:0}: Error finding container aaadcf3149f091382cf80282e9d043890e91ba90543ea319ffd6865f7097de33: Status 404 returned error can't find the container with id aaadcf3149f091382cf80282e9d043890e91ba90543ea319ffd6865f7097de33 Jul 06 22:50:55.394911 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:50:55.394890 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 22:50:55.550393 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:50:55.550349 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:50:55.651034 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:50:55.650951 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:50:56.330395 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:50:56.330284 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-eb8d59-646bfdb49c-p5v9r" event={"ID":"d125a1be-0505-4c5c-9c76-6a7c95b7cc57","Type":"ContainerStarted","Data":"78b79ad325d4b3cbbb5d28f1b429a7762e34c39cb20d3bca957fbceee9e56c1e"} Jul 06 22:50:56.330395 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:50:56.330390 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-eb8d59-646bfdb49c-p5v9r" event={"ID":"d125a1be-0505-4c5c-9c76-6a7c95b7cc57","Type":"ContainerStarted","Data":"aaadcf3149f091382cf80282e9d043890e91ba90543ea319ffd6865f7097de33"} Jul 06 22:50:56.330843 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:50:56.330421 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-a-eb8d59-646bfdb49c-p5v9r" Jul 06 22:50:56.348618 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:50:56.348562 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-a-eb8d59-646bfdb49c-p5v9r" podStartSLOduration=1.72514683 podStartE2EDuration="2.348545615s" totalImagesPullingTime="623.398785ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:50:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:50:56.347730196 +0000 UTC m=+1863.265973051" watchObservedRunningTime="2026-07-06 22:50:56.348545615 +0000 UTC m=+1863.266788473" Jul 06 22:50:59.375012 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:50:59.374978 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:50:59.686877 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:50:59.686836 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:00.000717 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:00.000630 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:00.110110 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:00.110072 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:02.342122 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:02.342091 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-a-eb8d59-646bfdb49c-p5v9r" Jul 06 22:51:02.969311 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:02.969280 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:03.068908 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:03.068842 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:03.184561 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:03.184523 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-b-e401d1-777c88c576-lfsmn"] Jul 06 22:51:03.188406 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:03.188380 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-e401d1-777c88c576-lfsmn" Jul 06 22:51:03.190853 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:03.190830 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-b-e401d1-serving-cert\"" Jul 06 22:51:03.197263 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:03.197234 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-e401d1-777c88c576-lfsmn"] Jul 06 22:51:03.232151 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:03.232043 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:03.322103 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:03.322041 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mpxh\" (UniqueName: \"kubernetes.io/projected/e6e45ec3-320d-429e-9b3f-caa137b7df29-kube-api-access-2mpxh\") pod \"maas-api-e2e-inf-b-e401d1-777c88c576-lfsmn\" (UID: \"e6e45ec3-320d-429e-9b3f-caa137b7df29\") " pod="opendatahub/maas-api-e2e-inf-b-e401d1-777c88c576-lfsmn" Jul 06 22:51:03.322345 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:03.322134 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6e45ec3-320d-429e-9b3f-caa137b7df29-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e401d1-777c88c576-lfsmn\" (UID: \"e6e45ec3-320d-429e-9b3f-caa137b7df29\") " pod="opendatahub/maas-api-e2e-inf-b-e401d1-777c88c576-lfsmn" Jul 06 22:51:03.322345 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:03.322174 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6e45ec3-320d-429e-9b3f-caa137b7df29-maas-api-tls\") pod \"maas-api-e2e-inf-b-e401d1-777c88c576-lfsmn\" (UID: \"e6e45ec3-320d-429e-9b3f-caa137b7df29\") " pod="opendatahub/maas-api-e2e-inf-b-e401d1-777c88c576-lfsmn" Jul 06 22:51:03.404686 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:03.404654 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:03.422909 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:03.422870 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mpxh\" (UniqueName: \"kubernetes.io/projected/e6e45ec3-320d-429e-9b3f-caa137b7df29-kube-api-access-2mpxh\") pod \"maas-api-e2e-inf-b-e401d1-777c88c576-lfsmn\" (UID: \"e6e45ec3-320d-429e-9b3f-caa137b7df29\") " pod="opendatahub/maas-api-e2e-inf-b-e401d1-777c88c576-lfsmn" Jul 06 22:51:03.423041 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:03.422968 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6e45ec3-320d-429e-9b3f-caa137b7df29-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e401d1-777c88c576-lfsmn\" (UID: \"e6e45ec3-320d-429e-9b3f-caa137b7df29\") " pod="opendatahub/maas-api-e2e-inf-b-e401d1-777c88c576-lfsmn" Jul 06 22:51:03.423041 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:03.423009 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6e45ec3-320d-429e-9b3f-caa137b7df29-maas-api-tls\") pod \"maas-api-e2e-inf-b-e401d1-777c88c576-lfsmn\" (UID: \"e6e45ec3-320d-429e-9b3f-caa137b7df29\") " pod="opendatahub/maas-api-e2e-inf-b-e401d1-777c88c576-lfsmn" Jul 06 22:51:03.423784 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:03.423756 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6e45ec3-320d-429e-9b3f-caa137b7df29-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e401d1-777c88c576-lfsmn\" (UID: \"e6e45ec3-320d-429e-9b3f-caa137b7df29\") " pod="opendatahub/maas-api-e2e-inf-b-e401d1-777c88c576-lfsmn" Jul 06 22:51:03.425540 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:03.425522 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6e45ec3-320d-429e-9b3f-caa137b7df29-maas-api-tls\") pod \"maas-api-e2e-inf-b-e401d1-777c88c576-lfsmn\" (UID: \"e6e45ec3-320d-429e-9b3f-caa137b7df29\") " pod="opendatahub/maas-api-e2e-inf-b-e401d1-777c88c576-lfsmn" Jul 06 22:51:03.431092 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:03.431068 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mpxh\" (UniqueName: \"kubernetes.io/projected/e6e45ec3-320d-429e-9b3f-caa137b7df29-kube-api-access-2mpxh\") pod \"maas-api-e2e-inf-b-e401d1-777c88c576-lfsmn\" (UID: \"e6e45ec3-320d-429e-9b3f-caa137b7df29\") " pod="opendatahub/maas-api-e2e-inf-b-e401d1-777c88c576-lfsmn" Jul 06 22:51:03.503282 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:03.503140 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-e401d1-777c88c576-lfsmn" Jul 06 22:51:03.646047 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:03.646021 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-e401d1-777c88c576-lfsmn"] Jul 06 22:51:03.646977 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:51:03.646950 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6e45ec3_320d_429e_9b3f_caa137b7df29.slice/crio-8acfc4f55d881c7b4e5c03660e30b65da5f9b487238bfed8885dada1a547c766 WatchSource:0}: Error finding container 8acfc4f55d881c7b4e5c03660e30b65da5f9b487238bfed8885dada1a547c766: Status 404 returned error can't find the container with id 8acfc4f55d881c7b4e5c03660e30b65da5f9b487238bfed8885dada1a547c766 Jul 06 22:51:03.857074 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:03.856988 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:03.956849 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:03.956807 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:04.372063 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:04.372026 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-e401d1-777c88c576-lfsmn" event={"ID":"e6e45ec3-320d-429e-9b3f-caa137b7df29","Type":"ContainerStarted","Data":"03efe569fcb554b2f171f40c47f1da3c37cdc4a518217ecf6dda4196d0324fc8"} Jul 06 22:51:04.372281 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:04.372075 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-e401d1-777c88c576-lfsmn" event={"ID":"e6e45ec3-320d-429e-9b3f-caa137b7df29","Type":"ContainerStarted","Data":"8acfc4f55d881c7b4e5c03660e30b65da5f9b487238bfed8885dada1a547c766"} Jul 06 22:51:04.372281 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:04.372198 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-b-e401d1-777c88c576-lfsmn" Jul 06 22:51:04.390545 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:04.390488 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-b-e401d1-777c88c576-lfsmn" podStartSLOduration=1.084308641 podStartE2EDuration="1.390468261s" totalImagesPullingTime="306.15962ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:51:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:51:04.388822355 +0000 UTC m=+1871.307065213" watchObservedRunningTime="2026-07-06 22:51:04.390468261 +0000 UTC m=+1871.308711117" Jul 06 22:51:07.463292 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:07.463196 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:10.382688 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:10.382658 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-b-e401d1-777c88c576-lfsmn" Jul 06 22:51:12.935277 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:12.935239 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-eb8d59/test-model-eb8d59-kserve-5bb5dcf9bc-g9hk4"] Jul 06 22:51:12.938991 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:12.938969 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-eb8d59/test-model-eb8d59-kserve-5bb5dcf9bc-g9hk4" Jul 06 22:51:12.942469 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:12.942446 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-eb8d59\"/\"kube-root-ca.crt\"" Jul 06 22:51:12.942603 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:12.942446 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-eb8d59\"/\"test-model-eb8d59-kserve-self-signed-certs\"" Jul 06 22:51:12.942603 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:12.942481 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-eb8d59\"/\"openshift-service-ca.crt\"" Jul 06 22:51:12.942603 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:12.942511 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-eb8d59\"/\"default-dockercfg-9qndl\"" Jul 06 22:51:12.949441 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:12.949416 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-eb8d59/test-model-eb8d59-kserve-5bb5dcf9bc-g9hk4"] Jul 06 22:51:12.960912 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:12.960846 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:13.012822 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:13.012785 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d370525e-75f3-4ef1-a89d-4054a5f98663-tmp-dir\") pod \"test-model-eb8d59-kserve-5bb5dcf9bc-g9hk4\" (UID: \"d370525e-75f3-4ef1-a89d-4054a5f98663\") " pod="ai-tenant-e2e-inf-a-eb8d59/test-model-eb8d59-kserve-5bb5dcf9bc-g9hk4" Jul 06 22:51:13.013024 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:13.012851 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q5mk\" (UniqueName: \"kubernetes.io/projected/d370525e-75f3-4ef1-a89d-4054a5f98663-kube-api-access-6q5mk\") pod \"test-model-eb8d59-kserve-5bb5dcf9bc-g9hk4\" (UID: \"d370525e-75f3-4ef1-a89d-4054a5f98663\") " pod="ai-tenant-e2e-inf-a-eb8d59/test-model-eb8d59-kserve-5bb5dcf9bc-g9hk4" Jul 06 22:51:13.013024 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:13.012900 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d370525e-75f3-4ef1-a89d-4054a5f98663-model-cache\") pod \"test-model-eb8d59-kserve-5bb5dcf9bc-g9hk4\" (UID: \"d370525e-75f3-4ef1-a89d-4054a5f98663\") " pod="ai-tenant-e2e-inf-a-eb8d59/test-model-eb8d59-kserve-5bb5dcf9bc-g9hk4" Jul 06 22:51:13.013024 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:13.012958 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d370525e-75f3-4ef1-a89d-4054a5f98663-tls-certs\") pod \"test-model-eb8d59-kserve-5bb5dcf9bc-g9hk4\" (UID: \"d370525e-75f3-4ef1-a89d-4054a5f98663\") " pod="ai-tenant-e2e-inf-a-eb8d59/test-model-eb8d59-kserve-5bb5dcf9bc-g9hk4" Jul 06 22:51:13.013024 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:13.013012 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d370525e-75f3-4ef1-a89d-4054a5f98663-dshm\") pod \"test-model-eb8d59-kserve-5bb5dcf9bc-g9hk4\" (UID: \"d370525e-75f3-4ef1-a89d-4054a5f98663\") " pod="ai-tenant-e2e-inf-a-eb8d59/test-model-eb8d59-kserve-5bb5dcf9bc-g9hk4" Jul 06 22:51:13.013256 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:13.013058 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d370525e-75f3-4ef1-a89d-4054a5f98663-home\") pod \"test-model-eb8d59-kserve-5bb5dcf9bc-g9hk4\" (UID: \"d370525e-75f3-4ef1-a89d-4054a5f98663\") " pod="ai-tenant-e2e-inf-a-eb8d59/test-model-eb8d59-kserve-5bb5dcf9bc-g9hk4" Jul 06 22:51:13.113769 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:13.113731 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d370525e-75f3-4ef1-a89d-4054a5f98663-model-cache\") pod \"test-model-eb8d59-kserve-5bb5dcf9bc-g9hk4\" (UID: \"d370525e-75f3-4ef1-a89d-4054a5f98663\") " pod="ai-tenant-e2e-inf-a-eb8d59/test-model-eb8d59-kserve-5bb5dcf9bc-g9hk4" Jul 06 22:51:13.113769 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:13.113773 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d370525e-75f3-4ef1-a89d-4054a5f98663-tls-certs\") pod \"test-model-eb8d59-kserve-5bb5dcf9bc-g9hk4\" (UID: \"d370525e-75f3-4ef1-a89d-4054a5f98663\") " pod="ai-tenant-e2e-inf-a-eb8d59/test-model-eb8d59-kserve-5bb5dcf9bc-g9hk4" Jul 06 22:51:13.114010 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:13.113805 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d370525e-75f3-4ef1-a89d-4054a5f98663-dshm\") pod \"test-model-eb8d59-kserve-5bb5dcf9bc-g9hk4\" (UID: \"d370525e-75f3-4ef1-a89d-4054a5f98663\") " pod="ai-tenant-e2e-inf-a-eb8d59/test-model-eb8d59-kserve-5bb5dcf9bc-g9hk4" Jul 06 22:51:13.114010 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:13.113830 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d370525e-75f3-4ef1-a89d-4054a5f98663-home\") pod \"test-model-eb8d59-kserve-5bb5dcf9bc-g9hk4\" (UID: \"d370525e-75f3-4ef1-a89d-4054a5f98663\") " pod="ai-tenant-e2e-inf-a-eb8d59/test-model-eb8d59-kserve-5bb5dcf9bc-g9hk4" Jul 06 22:51:13.114010 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:13.113863 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d370525e-75f3-4ef1-a89d-4054a5f98663-tmp-dir\") pod \"test-model-eb8d59-kserve-5bb5dcf9bc-g9hk4\" (UID: \"d370525e-75f3-4ef1-a89d-4054a5f98663\") " pod="ai-tenant-e2e-inf-a-eb8d59/test-model-eb8d59-kserve-5bb5dcf9bc-g9hk4" Jul 06 22:51:13.114010 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:13.113907 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q5mk\" (UniqueName: \"kubernetes.io/projected/d370525e-75f3-4ef1-a89d-4054a5f98663-kube-api-access-6q5mk\") pod \"test-model-eb8d59-kserve-5bb5dcf9bc-g9hk4\" (UID: \"d370525e-75f3-4ef1-a89d-4054a5f98663\") " pod="ai-tenant-e2e-inf-a-eb8d59/test-model-eb8d59-kserve-5bb5dcf9bc-g9hk4" Jul 06 22:51:13.114246 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:13.114194 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d370525e-75f3-4ef1-a89d-4054a5f98663-model-cache\") pod \"test-model-eb8d59-kserve-5bb5dcf9bc-g9hk4\" (UID: \"d370525e-75f3-4ef1-a89d-4054a5f98663\") " pod="ai-tenant-e2e-inf-a-eb8d59/test-model-eb8d59-kserve-5bb5dcf9bc-g9hk4" Jul 06 22:51:13.114341 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:13.114264 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d370525e-75f3-4ef1-a89d-4054a5f98663-home\") pod \"test-model-eb8d59-kserve-5bb5dcf9bc-g9hk4\" (UID: \"d370525e-75f3-4ef1-a89d-4054a5f98663\") " pod="ai-tenant-e2e-inf-a-eb8d59/test-model-eb8d59-kserve-5bb5dcf9bc-g9hk4" Jul 06 22:51:13.114413 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:13.114390 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d370525e-75f3-4ef1-a89d-4054a5f98663-tmp-dir\") pod \"test-model-eb8d59-kserve-5bb5dcf9bc-g9hk4\" (UID: \"d370525e-75f3-4ef1-a89d-4054a5f98663\") " pod="ai-tenant-e2e-inf-a-eb8d59/test-model-eb8d59-kserve-5bb5dcf9bc-g9hk4" Jul 06 22:51:13.116391 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:13.116368 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d370525e-75f3-4ef1-a89d-4054a5f98663-dshm\") pod \"test-model-eb8d59-kserve-5bb5dcf9bc-g9hk4\" (UID: \"d370525e-75f3-4ef1-a89d-4054a5f98663\") " pod="ai-tenant-e2e-inf-a-eb8d59/test-model-eb8d59-kserve-5bb5dcf9bc-g9hk4" Jul 06 22:51:13.116674 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:13.116654 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d370525e-75f3-4ef1-a89d-4054a5f98663-tls-certs\") pod \"test-model-eb8d59-kserve-5bb5dcf9bc-g9hk4\" (UID: \"d370525e-75f3-4ef1-a89d-4054a5f98663\") " pod="ai-tenant-e2e-inf-a-eb8d59/test-model-eb8d59-kserve-5bb5dcf9bc-g9hk4" Jul 06 22:51:13.123731 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:13.123709 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q5mk\" (UniqueName: \"kubernetes.io/projected/d370525e-75f3-4ef1-a89d-4054a5f98663-kube-api-access-6q5mk\") pod \"test-model-eb8d59-kserve-5bb5dcf9bc-g9hk4\" (UID: \"d370525e-75f3-4ef1-a89d-4054a5f98663\") " pod="ai-tenant-e2e-inf-a-eb8d59/test-model-eb8d59-kserve-5bb5dcf9bc-g9hk4" Jul 06 22:51:13.249702 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:13.249613 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-eb8d59/test-model-eb8d59-kserve-5bb5dcf9bc-g9hk4" Jul 06 22:51:13.385563 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:13.385539 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-eb8d59/test-model-eb8d59-kserve-5bb5dcf9bc-g9hk4"] Jul 06 22:51:13.387615 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:51:13.387587 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd370525e_75f3_4ef1_a89d_4054a5f98663.slice/crio-694b873195e057752b280569b0ffdb748658558dd588657cd0510e7d0c4f1e82 WatchSource:0}: Error finding container 694b873195e057752b280569b0ffdb748658558dd588657cd0510e7d0c4f1e82: Status 404 returned error can't find the container with id 694b873195e057752b280569b0ffdb748658558dd588657cd0510e7d0c4f1e82 Jul 06 22:51:13.414669 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:13.414639 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-eb8d59/test-model-eb8d59-kserve-5bb5dcf9bc-g9hk4" event={"ID":"d370525e-75f3-4ef1-a89d-4054a5f98663","Type":"ContainerStarted","Data":"694b873195e057752b280569b0ffdb748658558dd588657cd0510e7d0c4f1e82"} Jul 06 22:51:14.421089 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:14.421052 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-eb8d59/test-model-eb8d59-kserve-5bb5dcf9bc-g9hk4" event={"ID":"d370525e-75f3-4ef1-a89d-4054a5f98663","Type":"ContainerStarted","Data":"aaf17b7180ccf4a0a8b10d73529ddd2f32fbf9f772fbceadd9838e1fdc5a28ff"} Jul 06 22:51:17.399579 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:17.399491 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:17.499057 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:17.499020 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:18.091377 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:18.091345 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:18.222547 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:18.222516 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:23.250075 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:23.250031 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-eb8d59/test-model-eb8d59-kserve-5bb5dcf9bc-g9hk4" Jul 06 22:51:23.250691 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:23.250160 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-eb8d59/test-model-eb8d59-kserve-5bb5dcf9bc-g9hk4" Jul 06 22:51:23.262623 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:23.262597 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-eb8d59/test-model-eb8d59-kserve-5bb5dcf9bc-g9hk4" Jul 06 22:51:23.281989 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:23.281938 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-eb8d59/test-model-eb8d59-kserve-5bb5dcf9bc-g9hk4" podStartSLOduration=11.281924308 podStartE2EDuration="11.281924308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:51:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:51:14.441754957 +0000 UTC m=+1881.359997825" watchObservedRunningTime="2026-07-06 22:51:23.281924308 +0000 UTC m=+1890.200167207" Jul 06 22:51:23.477608 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:23.477573 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-eb8d59/test-model-eb8d59-kserve-5bb5dcf9bc-g9hk4" Jul 06 22:51:25.702836 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:25.702794 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:25.802721 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:25.802684 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:44.114866 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:51:44.114831 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:52:06.023833 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:52:06.023799 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:52:21.533963 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:52:21.533923 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-eb8d59/test-model-eb8d59-kserve-5bb5dcf9bc-g9hk4"] Jul 06 22:52:21.554604 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:52:21.554572 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:52:22.479958 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:52:22.479925 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:52:22.798110 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:52:22.798017 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:52:23.150445 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:52:23.150413 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:52:23.325996 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:52:23.325936 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:52:23.426701 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:52:23.426615 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:52:23.573884 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:52:23.573851 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-eb8d59-646bfdb49c-p5v9r"] Jul 06 22:52:23.574092 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:52:23.574063 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-a-eb8d59-646bfdb49c-p5v9r" podUID="d125a1be-0505-4c5c-9c76-6a7c95b7cc57" containerName="maas-api" containerID="cri-o://78b79ad325d4b3cbbb5d28f1b429a7762e34c39cb20d3bca957fbceee9e56c1e" gracePeriod=30 Jul 06 22:52:23.602516 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:52:23.602485 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:52:24.018561 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:52:24.018533 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-eb8d59-646bfdb49c-p5v9r" Jul 06 22:52:24.171479 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:52:24.171440 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d125a1be-0505-4c5c-9c76-6a7c95b7cc57-openshift-service-ca\") pod \"d125a1be-0505-4c5c-9c76-6a7c95b7cc57\" (UID: \"d125a1be-0505-4c5c-9c76-6a7c95b7cc57\") " Jul 06 22:52:24.171479 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:52:24.171481 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-szd5n\" (UniqueName: \"kubernetes.io/projected/d125a1be-0505-4c5c-9c76-6a7c95b7cc57-kube-api-access-szd5n\") pod \"d125a1be-0505-4c5c-9c76-6a7c95b7cc57\" (UID: \"d125a1be-0505-4c5c-9c76-6a7c95b7cc57\") " Jul 06 22:52:24.171760 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:52:24.171531 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d125a1be-0505-4c5c-9c76-6a7c95b7cc57-maas-api-tls\") pod \"d125a1be-0505-4c5c-9c76-6a7c95b7cc57\" (UID: \"d125a1be-0505-4c5c-9c76-6a7c95b7cc57\") " Jul 06 22:52:24.171946 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:52:24.171917 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d125a1be-0505-4c5c-9c76-6a7c95b7cc57-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d125a1be-0505-4c5c-9c76-6a7c95b7cc57" (UID: "d125a1be-0505-4c5c-9c76-6a7c95b7cc57"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:52:24.173610 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:52:24.173588 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d125a1be-0505-4c5c-9c76-6a7c95b7cc57-kube-api-access-szd5n" (OuterVolumeSpecName: "kube-api-access-szd5n") pod "d125a1be-0505-4c5c-9c76-6a7c95b7cc57" (UID: "d125a1be-0505-4c5c-9c76-6a7c95b7cc57"). InnerVolumeSpecName "kube-api-access-szd5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:52:24.173810 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:52:24.173789 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d125a1be-0505-4c5c-9c76-6a7c95b7cc57-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d125a1be-0505-4c5c-9c76-6a7c95b7cc57" (UID: "d125a1be-0505-4c5c-9c76-6a7c95b7cc57"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:52:24.273032 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:52:24.272993 2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d125a1be-0505-4c5c-9c76-6a7c95b7cc57-openshift-service-ca\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:52:24.273032 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:52:24.273024 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szd5n\" (UniqueName: \"kubernetes.io/projected/d125a1be-0505-4c5c-9c76-6a7c95b7cc57-kube-api-access-szd5n\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:52:24.273032 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:52:24.273034 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d125a1be-0505-4c5c-9c76-6a7c95b7cc57-maas-api-tls\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:52:24.762403 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:52:24.762364 2571 generic.go:356] "Generic (PLEG): container finished" podID="d125a1be-0505-4c5c-9c76-6a7c95b7cc57" containerID="78b79ad325d4b3cbbb5d28f1b429a7762e34c39cb20d3bca957fbceee9e56c1e" exitCode=0 Jul 06 22:52:24.762576 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:52:24.762421 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-eb8d59-646bfdb49c-p5v9r" Jul 06 22:52:24.762576 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:52:24.762434 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-eb8d59-646bfdb49c-p5v9r" event={"ID":"d125a1be-0505-4c5c-9c76-6a7c95b7cc57","Type":"ContainerDied","Data":"78b79ad325d4b3cbbb5d28f1b429a7762e34c39cb20d3bca957fbceee9e56c1e"} Jul 06 22:52:24.762576 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:52:24.762478 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-eb8d59-646bfdb49c-p5v9r" event={"ID":"d125a1be-0505-4c5c-9c76-6a7c95b7cc57","Type":"ContainerDied","Data":"aaadcf3149f091382cf80282e9d043890e91ba90543ea319ffd6865f7097de33"} Jul 06 22:52:24.762576 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:52:24.762501 2571 scope.go:117] "RemoveContainer" containerID="78b79ad325d4b3cbbb5d28f1b429a7762e34c39cb20d3bca957fbceee9e56c1e" Jul 06 22:52:24.772082 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:52:24.772064 2571 scope.go:117] "RemoveContainer" containerID="78b79ad325d4b3cbbb5d28f1b429a7762e34c39cb20d3bca957fbceee9e56c1e" Jul 06 22:52:24.772372 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:52:24.772348 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78b79ad325d4b3cbbb5d28f1b429a7762e34c39cb20d3bca957fbceee9e56c1e\": container with ID starting with 78b79ad325d4b3cbbb5d28f1b429a7762e34c39cb20d3bca957fbceee9e56c1e not found: ID does not exist" containerID="78b79ad325d4b3cbbb5d28f1b429a7762e34c39cb20d3bca957fbceee9e56c1e" Jul 06 22:52:24.772451 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:52:24.772378 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78b79ad325d4b3cbbb5d28f1b429a7762e34c39cb20d3bca957fbceee9e56c1e"} err="failed to get container status \"78b79ad325d4b3cbbb5d28f1b429a7762e34c39cb20d3bca957fbceee9e56c1e\": rpc error: code = NotFound desc = could not find container \"78b79ad325d4b3cbbb5d28f1b429a7762e34c39cb20d3bca957fbceee9e56c1e\": container with ID starting with 78b79ad325d4b3cbbb5d28f1b429a7762e34c39cb20d3bca957fbceee9e56c1e not found: ID does not exist" Jul 06 22:52:24.785089 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:52:24.785052 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-eb8d59-646bfdb49c-p5v9r"] Jul 06 22:52:24.790219 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:52:24.790176 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-eb8d59-646bfdb49c-p5v9r"] Jul 06 22:52:25.636655 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:52:25.636613 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d125a1be-0505-4c5c-9c76-6a7c95b7cc57" path="/var/lib/kubelet/pods/d125a1be-0505-4c5c-9c76-6a7c95b7cc57/volumes" Jul 06 22:52:28.233069 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:52:28.233031 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:52:36.554822 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:52:36.554774 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-eb8d59/test-model-eb8d59-kserve-5bb5dcf9bc-g9hk4" podUID="d370525e-75f3-4ef1-a89d-4054a5f98663" containerName="main" containerID="cri-o://aaf17b7180ccf4a0a8b10d73529ddd2f32fbf9f772fbceadd9838e1fdc5a28ff" gracePeriod=45 Jul 06 22:52:36.808256 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:52:36.808174 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-eb8d59/test-model-eb8d59-kserve-5bb5dcf9bc-g9hk4" Jul 06 22:52:36.827423 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:52:36.827316 2571 generic.go:356] "Generic (PLEG): container finished" podID="d370525e-75f3-4ef1-a89d-4054a5f98663" containerID="aaf17b7180ccf4a0a8b10d73529ddd2f32fbf9f772fbceadd9838e1fdc5a28ff" exitCode=0 Jul 06 22:52:36.827423 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:52:36.827379 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-eb8d59/test-model-eb8d59-kserve-5bb5dcf9bc-g9hk4" Jul 06 22:52:36.827683 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:52:36.827374 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-eb8d59/test-model-eb8d59-kserve-5bb5dcf9bc-g9hk4" event={"ID":"d370525e-75f3-4ef1-a89d-4054a5f98663","Type":"ContainerDied","Data":"aaf17b7180ccf4a0a8b10d73529ddd2f32fbf9f772fbceadd9838e1fdc5a28ff"} Jul 06 22:52:36.827683 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:52:36.827524 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-eb8d59/test-model-eb8d59-kserve-5bb5dcf9bc-g9hk4" event={"ID":"d370525e-75f3-4ef1-a89d-4054a5f98663","Type":"ContainerDied","Data":"694b873195e057752b280569b0ffdb748658558dd588657cd0510e7d0c4f1e82"} Jul 06 22:52:36.827683 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:52:36.827555 2571 scope.go:117] "RemoveContainer" containerID="aaf17b7180ccf4a0a8b10d73529ddd2f32fbf9f772fbceadd9838e1fdc5a28ff" Jul 06 22:52:36.837328 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:52:36.837299 2571 scope.go:117] "RemoveContainer" containerID="aaf17b7180ccf4a0a8b10d73529ddd2f32fbf9f772fbceadd9838e1fdc5a28ff" Jul 06 22:52:36.837706 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:52:36.837681 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaf17b7180ccf4a0a8b10d73529ddd2f32fbf9f772fbceadd9838e1fdc5a28ff\": container with ID starting with aaf17b7180ccf4a0a8b10d73529ddd2f32fbf9f772fbceadd9838e1fdc5a28ff not found: ID does not exist" containerID="aaf17b7180ccf4a0a8b10d73529ddd2f32fbf9f772fbceadd9838e1fdc5a28ff" Jul 06 22:52:36.837812 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:52:36.837719 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaf17b7180ccf4a0a8b10d73529ddd2f32fbf9f772fbceadd9838e1fdc5a28ff"} err="failed to get container status \"aaf17b7180ccf4a0a8b10d73529ddd2f32fbf9f772fbceadd9838e1fdc5a28ff\": rpc error: code = NotFound desc = could not find container \"aaf17b7180ccf4a0a8b10d73529ddd2f32fbf9f772fbceadd9838e1fdc5a28ff\": container with ID starting with aaf17b7180ccf4a0a8b10d73529ddd2f32fbf9f772fbceadd9838e1fdc5a28ff not found: ID does not exist" Jul 06 22:52:36.985823 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:52:36.985782 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d370525e-75f3-4ef1-a89d-4054a5f98663-model-cache\") pod \"d370525e-75f3-4ef1-a89d-4054a5f98663\" (UID: \"d370525e-75f3-4ef1-a89d-4054a5f98663\") " Jul 06 22:52:36.986015 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:52:36.985867 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6q5mk\" (UniqueName: \"kubernetes.io/projected/d370525e-75f3-4ef1-a89d-4054a5f98663-kube-api-access-6q5mk\") pod \"d370525e-75f3-4ef1-a89d-4054a5f98663\" (UID: \"d370525e-75f3-4ef1-a89d-4054a5f98663\") " Jul 06 22:52:36.986015 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:52:36.985910 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d370525e-75f3-4ef1-a89d-4054a5f98663-tmp-dir\") pod \"d370525e-75f3-4ef1-a89d-4054a5f98663\" (UID: \"d370525e-75f3-4ef1-a89d-4054a5f98663\") " Jul 06 22:52:36.986015 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:52:36.985954 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d370525e-75f3-4ef1-a89d-4054a5f98663-tls-certs\") pod \"d370525e-75f3-4ef1-a89d-4054a5f98663\" (UID: \"d370525e-75f3-4ef1-a89d-4054a5f98663\") " Jul 06 22:52:36.986015 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:52:36.985984 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d370525e-75f3-4ef1-a89d-4054a5f98663-home\") pod \"d370525e-75f3-4ef1-a89d-4054a5f98663\" (UID: \"d370525e-75f3-4ef1-a89d-4054a5f98663\") " Jul 06 22:52:36.986186 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:52:36.986029 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d370525e-75f3-4ef1-a89d-4054a5f98663-dshm\") pod \"d370525e-75f3-4ef1-a89d-4054a5f98663\" (UID: \"d370525e-75f3-4ef1-a89d-4054a5f98663\") " Jul 06 22:52:36.986186 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:52:36.986096 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d370525e-75f3-4ef1-a89d-4054a5f98663-model-cache" (OuterVolumeSpecName: "model-cache") pod "d370525e-75f3-4ef1-a89d-4054a5f98663" (UID: "d370525e-75f3-4ef1-a89d-4054a5f98663"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:52:36.986296 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:52:36.986264 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d370525e-75f3-4ef1-a89d-4054a5f98663-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d370525e-75f3-4ef1-a89d-4054a5f98663" (UID: "d370525e-75f3-4ef1-a89d-4054a5f98663"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:52:36.986361 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:52:36.986308 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d370525e-75f3-4ef1-a89d-4054a5f98663-home" (OuterVolumeSpecName: "home") pod "d370525e-75f3-4ef1-a89d-4054a5f98663" (UID: "d370525e-75f3-4ef1-a89d-4054a5f98663"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:52:36.986438 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:52:36.986421 2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d370525e-75f3-4ef1-a89d-4054a5f98663-tmp-dir\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:52:36.986493 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:52:36.986440 2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d370525e-75f3-4ef1-a89d-4054a5f98663-home\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:52:36.986493 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:52:36.986451 2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d370525e-75f3-4ef1-a89d-4054a5f98663-model-cache\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:52:36.988175 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:52:36.988145 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d370525e-75f3-4ef1-a89d-4054a5f98663-dshm" (OuterVolumeSpecName: "dshm") pod "d370525e-75f3-4ef1-a89d-4054a5f98663" (UID: "d370525e-75f3-4ef1-a89d-4054a5f98663"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:52:36.988296 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:52:36.988198 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d370525e-75f3-4ef1-a89d-4054a5f98663-kube-api-access-6q5mk" (OuterVolumeSpecName: "kube-api-access-6q5mk") pod "d370525e-75f3-4ef1-a89d-4054a5f98663" (UID: "d370525e-75f3-4ef1-a89d-4054a5f98663"). InnerVolumeSpecName "kube-api-access-6q5mk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:52:36.988296 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:52:36.988244 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d370525e-75f3-4ef1-a89d-4054a5f98663-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d370525e-75f3-4ef1-a89d-4054a5f98663" (UID: "d370525e-75f3-4ef1-a89d-4054a5f98663"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:52:37.087038 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:52:37.086943 2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d370525e-75f3-4ef1-a89d-4054a5f98663-tls-certs\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:52:37.087038 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:52:37.086978 2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d370525e-75f3-4ef1-a89d-4054a5f98663-dshm\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:52:37.087038 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:52:37.086987 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6q5mk\" (UniqueName: \"kubernetes.io/projected/d370525e-75f3-4ef1-a89d-4054a5f98663-kube-api-access-6q5mk\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:52:37.152821 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:52:37.152786 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-eb8d59/test-model-eb8d59-kserve-5bb5dcf9bc-g9hk4"] Jul 06 22:52:37.157399 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:52:37.157371 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-eb8d59/test-model-eb8d59-kserve-5bb5dcf9bc-g9hk4"] Jul 06 22:52:37.635230 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:52:37.635170 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d370525e-75f3-4ef1-a89d-4054a5f98663" path="/var/lib/kubelet/pods/d370525e-75f3-4ef1-a89d-4054a5f98663/volumes" Jul 06 22:52:53.573839 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:52:53.573741 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:52:54.918009 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:52:54.917980 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:05.429663 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:05.429625 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:05.687517 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:05.687424 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:05.914594 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:05.914555 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:06.092732 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:06.092629 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:06.481707 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:06.481674 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:06.664625 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:06.664599 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:06.773783 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:06.773706 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:06.983006 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:06.982971 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-e401d1-777c88c576-lfsmn"] Jul 06 22:53:06.983233 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:06.983194 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-b-e401d1-777c88c576-lfsmn" podUID="e6e45ec3-320d-429e-9b3f-caa137b7df29" containerName="maas-api" containerID="cri-o://03efe569fcb554b2f171f40c47f1da3c37cdc4a518217ecf6dda4196d0324fc8" gracePeriod=30 Jul 06 22:53:07.008366 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:07.008336 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:07.434640 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:07.434610 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-e401d1-777c88c576-lfsmn" Jul 06 22:53:07.564613 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:07.564516 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2mpxh\" (UniqueName: \"kubernetes.io/projected/e6e45ec3-320d-429e-9b3f-caa137b7df29-kube-api-access-2mpxh\") pod \"e6e45ec3-320d-429e-9b3f-caa137b7df29\" (UID: \"e6e45ec3-320d-429e-9b3f-caa137b7df29\") " Jul 06 22:53:07.564613 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:07.564582 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6e45ec3-320d-429e-9b3f-caa137b7df29-maas-api-tls\") pod \"e6e45ec3-320d-429e-9b3f-caa137b7df29\" (UID: \"e6e45ec3-320d-429e-9b3f-caa137b7df29\") " Jul 06 22:53:07.564613 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:07.564613 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6e45ec3-320d-429e-9b3f-caa137b7df29-openshift-service-ca\") pod \"e6e45ec3-320d-429e-9b3f-caa137b7df29\" (UID: \"e6e45ec3-320d-429e-9b3f-caa137b7df29\") " Jul 06 22:53:07.565140 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:07.565052 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6e45ec3-320d-429e-9b3f-caa137b7df29-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e6e45ec3-320d-429e-9b3f-caa137b7df29" (UID: "e6e45ec3-320d-429e-9b3f-caa137b7df29"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:53:07.566579 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:07.566558 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6e45ec3-320d-429e-9b3f-caa137b7df29-kube-api-access-2mpxh" (OuterVolumeSpecName: "kube-api-access-2mpxh") pod "e6e45ec3-320d-429e-9b3f-caa137b7df29" (UID: "e6e45ec3-320d-429e-9b3f-caa137b7df29"). InnerVolumeSpecName "kube-api-access-2mpxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:53:07.566751 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:07.566732 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6e45ec3-320d-429e-9b3f-caa137b7df29-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e6e45ec3-320d-429e-9b3f-caa137b7df29" (UID: "e6e45ec3-320d-429e-9b3f-caa137b7df29"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:53:07.666097 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:07.666059 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mpxh\" (UniqueName: \"kubernetes.io/projected/e6e45ec3-320d-429e-9b3f-caa137b7df29-kube-api-access-2mpxh\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:53:07.666097 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:07.666089 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6e45ec3-320d-429e-9b3f-caa137b7df29-maas-api-tls\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:53:07.666097 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:07.666098 2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6e45ec3-320d-429e-9b3f-caa137b7df29-openshift-service-ca\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:53:07.979495 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:07.979461 2571 generic.go:356] "Generic (PLEG): container finished" podID="e6e45ec3-320d-429e-9b3f-caa137b7df29" containerID="03efe569fcb554b2f171f40c47f1da3c37cdc4a518217ecf6dda4196d0324fc8" exitCode=0 Jul 06 22:53:07.979709 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:07.979515 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-e401d1-777c88c576-lfsmn" Jul 06 22:53:07.979709 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:07.979549 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-e401d1-777c88c576-lfsmn" event={"ID":"e6e45ec3-320d-429e-9b3f-caa137b7df29","Type":"ContainerDied","Data":"03efe569fcb554b2f171f40c47f1da3c37cdc4a518217ecf6dda4196d0324fc8"} Jul 06 22:53:07.979709 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:07.979613 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-e401d1-777c88c576-lfsmn" event={"ID":"e6e45ec3-320d-429e-9b3f-caa137b7df29","Type":"ContainerDied","Data":"8acfc4f55d881c7b4e5c03660e30b65da5f9b487238bfed8885dada1a547c766"} Jul 06 22:53:07.979709 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:07.979646 2571 scope.go:117] "RemoveContainer" containerID="03efe569fcb554b2f171f40c47f1da3c37cdc4a518217ecf6dda4196d0324fc8" Jul 06 22:53:07.989191 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:07.989006 2571 scope.go:117] "RemoveContainer" containerID="03efe569fcb554b2f171f40c47f1da3c37cdc4a518217ecf6dda4196d0324fc8" Jul 06 22:53:07.989279 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:53:07.989265 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03efe569fcb554b2f171f40c47f1da3c37cdc4a518217ecf6dda4196d0324fc8\": container with ID starting with 03efe569fcb554b2f171f40c47f1da3c37cdc4a518217ecf6dda4196d0324fc8 not found: ID does not exist" containerID="03efe569fcb554b2f171f40c47f1da3c37cdc4a518217ecf6dda4196d0324fc8" Jul 06 22:53:07.989322 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:07.989292 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03efe569fcb554b2f171f40c47f1da3c37cdc4a518217ecf6dda4196d0324fc8"} err="failed to get container status \"03efe569fcb554b2f171f40c47f1da3c37cdc4a518217ecf6dda4196d0324fc8\": rpc error: code = NotFound desc = could not find container \"03efe569fcb554b2f171f40c47f1da3c37cdc4a518217ecf6dda4196d0324fc8\": container with ID starting with 03efe569fcb554b2f171f40c47f1da3c37cdc4a518217ecf6dda4196d0324fc8 not found: ID does not exist" Jul 06 22:53:07.998404 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:07.998378 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-e401d1-777c88c576-lfsmn"] Jul 06 22:53:08.006402 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:08.006376 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-e401d1-777c88c576-lfsmn"] Jul 06 22:53:09.635889 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:09.635854 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6e45ec3-320d-429e-9b3f-caa137b7df29" path="/var/lib/kubelet/pods/e6e45ec3-320d-429e-9b3f-caa137b7df29/volumes" Jul 06 22:53:12.842261 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:12.842229 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:12.896065 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:12.896030 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:13.100553 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:13.100453 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:13.331535 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:13.331502 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:13.678099 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:13.678062 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:13.983960 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:13.983854 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:14.193619 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:14.193585 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:16.673058 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:16.673018 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:16.742705 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:16.742667 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:16.743135 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:16.743115 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-a-8f36f0-86d7757566-4x2xc"] Jul 06 22:53:16.743515 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:16.743499 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d125a1be-0505-4c5c-9c76-6a7c95b7cc57" containerName="maas-api" Jul 06 22:53:16.743624 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:16.743519 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d125a1be-0505-4c5c-9c76-6a7c95b7cc57" containerName="maas-api" Jul 06 22:53:16.743624 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:16.743552 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d370525e-75f3-4ef1-a89d-4054a5f98663" containerName="main" Jul 06 22:53:16.743624 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:16.743562 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d370525e-75f3-4ef1-a89d-4054a5f98663" containerName="main" Jul 06 22:53:16.743624 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:16.743589 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6e45ec3-320d-429e-9b3f-caa137b7df29" containerName="maas-api" Jul 06 22:53:16.743624 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:16.743598 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6e45ec3-320d-429e-9b3f-caa137b7df29" containerName="maas-api" Jul 06 22:53:16.743873 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:16.743701 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="d370525e-75f3-4ef1-a89d-4054a5f98663" containerName="main" Jul 06 22:53:16.743873 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:16.743715 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="d125a1be-0505-4c5c-9c76-6a7c95b7cc57" containerName="maas-api" Jul 06 22:53:16.743873 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:16.743726 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6e45ec3-320d-429e-9b3f-caa137b7df29" containerName="maas-api" Jul 06 22:53:16.748200 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:16.748178 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-8f36f0-86d7757566-4x2xc" Jul 06 22:53:16.750458 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:16.750439 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-98x7w\"" Jul 06 22:53:16.750601 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:16.750438 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-a-8f36f0-serving-cert\"" Jul 06 22:53:16.757724 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:16.757696 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-8f36f0-86d7757566-4x2xc"] Jul 06 22:53:16.803564 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:16.803533 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:16.851242 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:16.851166 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j698t\" (UniqueName: \"kubernetes.io/projected/8feaf161-d6d9-4c61-96e2-cc2608c7b9aa-kube-api-access-j698t\") pod \"maas-api-e2e-api-a-8f36f0-86d7757566-4x2xc\" (UID: \"8feaf161-d6d9-4c61-96e2-cc2608c7b9aa\") " pod="opendatahub/maas-api-e2e-api-a-8f36f0-86d7757566-4x2xc" Jul 06 22:53:16.851242 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:16.851250 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8feaf161-d6d9-4c61-96e2-cc2608c7b9aa-maas-api-tls\") pod \"maas-api-e2e-api-a-8f36f0-86d7757566-4x2xc\" (UID: \"8feaf161-d6d9-4c61-96e2-cc2608c7b9aa\") " pod="opendatahub/maas-api-e2e-api-a-8f36f0-86d7757566-4x2xc" Jul 06 22:53:16.851466 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:16.851292 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8feaf161-d6d9-4c61-96e2-cc2608c7b9aa-openshift-service-ca\") pod \"maas-api-e2e-api-a-8f36f0-86d7757566-4x2xc\" (UID: \"8feaf161-d6d9-4c61-96e2-cc2608c7b9aa\") " pod="opendatahub/maas-api-e2e-api-a-8f36f0-86d7757566-4x2xc" Jul 06 22:53:16.951989 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:16.951897 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j698t\" (UniqueName: \"kubernetes.io/projected/8feaf161-d6d9-4c61-96e2-cc2608c7b9aa-kube-api-access-j698t\") pod \"maas-api-e2e-api-a-8f36f0-86d7757566-4x2xc\" (UID: \"8feaf161-d6d9-4c61-96e2-cc2608c7b9aa\") " pod="opendatahub/maas-api-e2e-api-a-8f36f0-86d7757566-4x2xc" Jul 06 22:53:16.951989 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:16.951939 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8feaf161-d6d9-4c61-96e2-cc2608c7b9aa-maas-api-tls\") pod \"maas-api-e2e-api-a-8f36f0-86d7757566-4x2xc\" (UID: \"8feaf161-d6d9-4c61-96e2-cc2608c7b9aa\") " pod="opendatahub/maas-api-e2e-api-a-8f36f0-86d7757566-4x2xc" Jul 06 22:53:16.952192 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:16.952009 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8feaf161-d6d9-4c61-96e2-cc2608c7b9aa-openshift-service-ca\") pod \"maas-api-e2e-api-a-8f36f0-86d7757566-4x2xc\" (UID: \"8feaf161-d6d9-4c61-96e2-cc2608c7b9aa\") " pod="opendatahub/maas-api-e2e-api-a-8f36f0-86d7757566-4x2xc" Jul 06 22:53:16.952662 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:16.952640 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8feaf161-d6d9-4c61-96e2-cc2608c7b9aa-openshift-service-ca\") pod \"maas-api-e2e-api-a-8f36f0-86d7757566-4x2xc\" (UID: \"8feaf161-d6d9-4c61-96e2-cc2608c7b9aa\") " pod="opendatahub/maas-api-e2e-api-a-8f36f0-86d7757566-4x2xc" Jul 06 22:53:16.953892 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:16.953869 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:16.954732 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:16.954709 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8feaf161-d6d9-4c61-96e2-cc2608c7b9aa-maas-api-tls\") pod \"maas-api-e2e-api-a-8f36f0-86d7757566-4x2xc\" (UID: \"8feaf161-d6d9-4c61-96e2-cc2608c7b9aa\") " pod="opendatahub/maas-api-e2e-api-a-8f36f0-86d7757566-4x2xc" Jul 06 22:53:16.963312 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:16.963289 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j698t\" (UniqueName: \"kubernetes.io/projected/8feaf161-d6d9-4c61-96e2-cc2608c7b9aa-kube-api-access-j698t\") pod \"maas-api-e2e-api-a-8f36f0-86d7757566-4x2xc\" (UID: \"8feaf161-d6d9-4c61-96e2-cc2608c7b9aa\") " pod="opendatahub/maas-api-e2e-api-a-8f36f0-86d7757566-4x2xc" Jul 06 22:53:17.061012 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:17.060978 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-8f36f0-86d7757566-4x2xc" Jul 06 22:53:17.205945 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:17.205911 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-8f36f0-86d7757566-4x2xc"] Jul 06 22:53:17.208249 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:53:17.208199 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8feaf161_d6d9_4c61_96e2_cc2608c7b9aa.slice/crio-95fd413a5355fa13e000025d6a25122b1724a0103a37c710a2202d27580ed98c WatchSource:0}: Error finding container 95fd413a5355fa13e000025d6a25122b1724a0103a37c710a2202d27580ed98c: Status 404 returned error can't find the container with id 95fd413a5355fa13e000025d6a25122b1724a0103a37c710a2202d27580ed98c Jul 06 22:53:17.434946 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:17.434870 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:17.535027 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:17.534934 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:18.030055 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:18.030019 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-8f36f0-86d7757566-4x2xc" event={"ID":"8feaf161-d6d9-4c61-96e2-cc2608c7b9aa","Type":"ContainerStarted","Data":"6356dba9a4b259a09451c835d9f3c8260621e87287c40e9ec6f47744dcd2a335"} Jul 06 22:53:18.030571 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:18.030076 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-8f36f0-86d7757566-4x2xc" event={"ID":"8feaf161-d6d9-4c61-96e2-cc2608c7b9aa","Type":"ContainerStarted","Data":"95fd413a5355fa13e000025d6a25122b1724a0103a37c710a2202d27580ed98c"} Jul 06 22:53:18.030571 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:18.030102 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-a-8f36f0-86d7757566-4x2xc" Jul 06 22:53:18.047657 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:18.047538 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-a-8f36f0-86d7757566-4x2xc" podStartSLOduration=1.4684365609999999 podStartE2EDuration="2.047515111s" totalImagesPullingTime="579.07855ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:53:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:53:18.047116315 +0000 UTC m=+2004.965359196" watchObservedRunningTime="2026-07-06 22:53:18.047515111 +0000 UTC m=+2004.965757970" Jul 06 22:53:21.167691 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:21.167656 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:21.635792 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:21.635740 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:21.963788 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:21.963706 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:22.578631 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:22.578593 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:24.042045 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:24.042017 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-a-8f36f0-86d7757566-4x2xc" Jul 06 22:53:24.718871 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:24.718836 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:24.801090 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:24.801051 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-b-164a1c-98ff65b9b-pr2bl"] Jul 06 22:53:24.804542 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:24.804501 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:24.805589 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:24.805566 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-164a1c-98ff65b9b-pr2bl" Jul 06 22:53:24.807991 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:24.807966 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-b-164a1c-serving-cert\"" Jul 06 22:53:24.814971 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:24.814943 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-164a1c-98ff65b9b-pr2bl"] Jul 06 22:53:24.819462 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:24.819423 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94b9cea6-cf7b-4f2a-9ec9-14bcf7308b94-openshift-service-ca\") pod \"maas-api-e2e-api-b-164a1c-98ff65b9b-pr2bl\" (UID: \"94b9cea6-cf7b-4f2a-9ec9-14bcf7308b94\") " pod="opendatahub/maas-api-e2e-api-b-164a1c-98ff65b9b-pr2bl" Jul 06 22:53:24.819735 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:24.819712 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlc5c\" (UniqueName: \"kubernetes.io/projected/94b9cea6-cf7b-4f2a-9ec9-14bcf7308b94-kube-api-access-zlc5c\") pod \"maas-api-e2e-api-b-164a1c-98ff65b9b-pr2bl\" (UID: \"94b9cea6-cf7b-4f2a-9ec9-14bcf7308b94\") " pod="opendatahub/maas-api-e2e-api-b-164a1c-98ff65b9b-pr2bl" Jul 06 22:53:24.819935 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:24.819917 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94b9cea6-cf7b-4f2a-9ec9-14bcf7308b94-maas-api-tls\") pod \"maas-api-e2e-api-b-164a1c-98ff65b9b-pr2bl\" (UID: \"94b9cea6-cf7b-4f2a-9ec9-14bcf7308b94\") " pod="opendatahub/maas-api-e2e-api-b-164a1c-98ff65b9b-pr2bl" Jul 06 22:53:24.869338 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:24.869297 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:24.920436 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:24.920396 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94b9cea6-cf7b-4f2a-9ec9-14bcf7308b94-openshift-service-ca\") pod \"maas-api-e2e-api-b-164a1c-98ff65b9b-pr2bl\" (UID: \"94b9cea6-cf7b-4f2a-9ec9-14bcf7308b94\") " pod="opendatahub/maas-api-e2e-api-b-164a1c-98ff65b9b-pr2bl" Jul 06 22:53:24.920436 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:24.920434 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlc5c\" (UniqueName: \"kubernetes.io/projected/94b9cea6-cf7b-4f2a-9ec9-14bcf7308b94-kube-api-access-zlc5c\") pod \"maas-api-e2e-api-b-164a1c-98ff65b9b-pr2bl\" (UID: \"94b9cea6-cf7b-4f2a-9ec9-14bcf7308b94\") " pod="opendatahub/maas-api-e2e-api-b-164a1c-98ff65b9b-pr2bl" Jul 06 22:53:24.920692 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:24.920472 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94b9cea6-cf7b-4f2a-9ec9-14bcf7308b94-maas-api-tls\") pod \"maas-api-e2e-api-b-164a1c-98ff65b9b-pr2bl\" (UID: \"94b9cea6-cf7b-4f2a-9ec9-14bcf7308b94\") " pod="opendatahub/maas-api-e2e-api-b-164a1c-98ff65b9b-pr2bl" Jul 06 22:53:24.920692 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:53:24.920597 2571 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-api-b-164a1c-serving-cert: secret "maas-api-e2e-api-b-164a1c-serving-cert" not found Jul 06 22:53:24.920692 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:53:24.920657 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94b9cea6-cf7b-4f2a-9ec9-14bcf7308b94-maas-api-tls podName:94b9cea6-cf7b-4f2a-9ec9-14bcf7308b94 nodeName:}" failed. No retries permitted until 2026-07-06 22:53:25.420638204 +0000 UTC m=+2012.338881049 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/94b9cea6-cf7b-4f2a-9ec9-14bcf7308b94-maas-api-tls") pod "maas-api-e2e-api-b-164a1c-98ff65b9b-pr2bl" (UID: "94b9cea6-cf7b-4f2a-9ec9-14bcf7308b94") : secret "maas-api-e2e-api-b-164a1c-serving-cert" not found Jul 06 22:53:24.921149 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:24.921122 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94b9cea6-cf7b-4f2a-9ec9-14bcf7308b94-openshift-service-ca\") pod \"maas-api-e2e-api-b-164a1c-98ff65b9b-pr2bl\" (UID: \"94b9cea6-cf7b-4f2a-9ec9-14bcf7308b94\") " pod="opendatahub/maas-api-e2e-api-b-164a1c-98ff65b9b-pr2bl" Jul 06 22:53:24.934275 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:24.934244 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlc5c\" (UniqueName: \"kubernetes.io/projected/94b9cea6-cf7b-4f2a-9ec9-14bcf7308b94-kube-api-access-zlc5c\") pod \"maas-api-e2e-api-b-164a1c-98ff65b9b-pr2bl\" (UID: \"94b9cea6-cf7b-4f2a-9ec9-14bcf7308b94\") " pod="opendatahub/maas-api-e2e-api-b-164a1c-98ff65b9b-pr2bl" Jul 06 22:53:24.987628 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:24.987597 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:25.314587 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:25.314547 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:25.414414 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:25.414379 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:25.423707 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:25.423679 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94b9cea6-cf7b-4f2a-9ec9-14bcf7308b94-maas-api-tls\") pod \"maas-api-e2e-api-b-164a1c-98ff65b9b-pr2bl\" (UID: \"94b9cea6-cf7b-4f2a-9ec9-14bcf7308b94\") " pod="opendatahub/maas-api-e2e-api-b-164a1c-98ff65b9b-pr2bl" Jul 06 22:53:25.426218 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:25.426185 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94b9cea6-cf7b-4f2a-9ec9-14bcf7308b94-maas-api-tls\") pod \"maas-api-e2e-api-b-164a1c-98ff65b9b-pr2bl\" (UID: \"94b9cea6-cf7b-4f2a-9ec9-14bcf7308b94\") " pod="opendatahub/maas-api-e2e-api-b-164a1c-98ff65b9b-pr2bl" Jul 06 22:53:25.721170 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:25.721135 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-164a1c-98ff65b9b-pr2bl" Jul 06 22:53:25.862992 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:25.862960 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-164a1c-98ff65b9b-pr2bl"] Jul 06 22:53:25.865449 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:53:25.865421 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94b9cea6_cf7b_4f2a_9ec9_14bcf7308b94.slice/crio-bf4b59e011e27096b6ecccd97985f8c8fbbc3667b3d784e080f466013ac93708 WatchSource:0}: Error finding container bf4b59e011e27096b6ecccd97985f8c8fbbc3667b3d784e080f466013ac93708: Status 404 returned error can't find the container with id bf4b59e011e27096b6ecccd97985f8c8fbbc3667b3d784e080f466013ac93708 Jul 06 22:53:26.072679 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:26.072589 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-164a1c-98ff65b9b-pr2bl" event={"ID":"94b9cea6-cf7b-4f2a-9ec9-14bcf7308b94","Type":"ContainerStarted","Data":"bf4b59e011e27096b6ecccd97985f8c8fbbc3667b3d784e080f466013ac93708"} Jul 06 22:53:27.078535 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:27.078490 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-164a1c-98ff65b9b-pr2bl" event={"ID":"94b9cea6-cf7b-4f2a-9ec9-14bcf7308b94","Type":"ContainerStarted","Data":"60419ddab0dab1b3734fb624c179ba96283e29896c8405a30572edac39fb14cb"} Jul 06 22:53:27.078937 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:27.078570 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-b-164a1c-98ff65b9b-pr2bl" Jul 06 22:53:27.097365 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:27.097319 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-b-164a1c-98ff65b9b-pr2bl" podStartSLOduration=2.721359249 podStartE2EDuration="3.097303063s" totalImagesPullingTime="375.943814ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:53:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:53:27.095928716 +0000 UTC m=+2014.014171573" watchObservedRunningTime="2026-07-06 22:53:27.097303063 +0000 UTC m=+2014.015545918" Jul 06 22:53:29.223254 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:29.223201 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:33.088749 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:33.088718 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-b-164a1c-98ff65b9b-pr2bl" Jul 06 22:53:37.343234 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:37.343167 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:37.721920 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:37.721892 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:37.837813 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:37.837753 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:37.938413 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:37.938378 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:38.104632 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:38.104541 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-8f36f0-86d7757566-4x2xc"] Jul 06 22:53:38.104850 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:38.104826 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-a-8f36f0-86d7757566-4x2xc" podUID="8feaf161-d6d9-4c61-96e2-cc2608c7b9aa" containerName="maas-api" containerID="cri-o://6356dba9a4b259a09451c835d9f3c8260621e87287c40e9ec6f47744dcd2a335" gracePeriod=30 Jul 06 22:53:38.127603 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:38.127568 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:38.359762 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:38.359689 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-8f36f0-86d7757566-4x2xc" Jul 06 22:53:38.439128 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:38.439087 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8feaf161-d6d9-4c61-96e2-cc2608c7b9aa-maas-api-tls\") pod \"8feaf161-d6d9-4c61-96e2-cc2608c7b9aa\" (UID: \"8feaf161-d6d9-4c61-96e2-cc2608c7b9aa\") " Jul 06 22:53:38.439355 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:38.439154 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j698t\" (UniqueName: \"kubernetes.io/projected/8feaf161-d6d9-4c61-96e2-cc2608c7b9aa-kube-api-access-j698t\") pod \"8feaf161-d6d9-4c61-96e2-cc2608c7b9aa\" (UID: \"8feaf161-d6d9-4c61-96e2-cc2608c7b9aa\") " Jul 06 22:53:38.439355 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:38.439277 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8feaf161-d6d9-4c61-96e2-cc2608c7b9aa-openshift-service-ca\") pod \"8feaf161-d6d9-4c61-96e2-cc2608c7b9aa\" (UID: \"8feaf161-d6d9-4c61-96e2-cc2608c7b9aa\") " Jul 06 22:53:38.439691 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:38.439662 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8feaf161-d6d9-4c61-96e2-cc2608c7b9aa-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8feaf161-d6d9-4c61-96e2-cc2608c7b9aa" (UID: "8feaf161-d6d9-4c61-96e2-cc2608c7b9aa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:53:38.441352 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:38.441321 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8feaf161-d6d9-4c61-96e2-cc2608c7b9aa-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8feaf161-d6d9-4c61-96e2-cc2608c7b9aa" (UID: "8feaf161-d6d9-4c61-96e2-cc2608c7b9aa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:53:38.441472 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:38.441451 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8feaf161-d6d9-4c61-96e2-cc2608c7b9aa-kube-api-access-j698t" (OuterVolumeSpecName: "kube-api-access-j698t") pod "8feaf161-d6d9-4c61-96e2-cc2608c7b9aa" (UID: "8feaf161-d6d9-4c61-96e2-cc2608c7b9aa"). InnerVolumeSpecName "kube-api-access-j698t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:53:38.540889 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:38.540853 2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8feaf161-d6d9-4c61-96e2-cc2608c7b9aa-openshift-service-ca\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:53:38.540889 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:38.540886 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8feaf161-d6d9-4c61-96e2-cc2608c7b9aa-maas-api-tls\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:53:38.540889 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:38.540897 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j698t\" (UniqueName: \"kubernetes.io/projected/8feaf161-d6d9-4c61-96e2-cc2608c7b9aa-kube-api-access-j698t\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:53:39.141492 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:39.141453 2571 generic.go:356] "Generic (PLEG): container finished" podID="8feaf161-d6d9-4c61-96e2-cc2608c7b9aa" containerID="6356dba9a4b259a09451c835d9f3c8260621e87287c40e9ec6f47744dcd2a335" exitCode=0 Jul 06 22:53:39.141492 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:39.141500 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-8f36f0-86d7757566-4x2xc" Jul 06 22:53:39.141735 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:39.141542 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-8f36f0-86d7757566-4x2xc" event={"ID":"8feaf161-d6d9-4c61-96e2-cc2608c7b9aa","Type":"ContainerDied","Data":"6356dba9a4b259a09451c835d9f3c8260621e87287c40e9ec6f47744dcd2a335"} Jul 06 22:53:39.141735 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:39.141595 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-8f36f0-86d7757566-4x2xc" event={"ID":"8feaf161-d6d9-4c61-96e2-cc2608c7b9aa","Type":"ContainerDied","Data":"95fd413a5355fa13e000025d6a25122b1724a0103a37c710a2202d27580ed98c"} Jul 06 22:53:39.141735 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:39.141617 2571 scope.go:117] "RemoveContainer" containerID="6356dba9a4b259a09451c835d9f3c8260621e87287c40e9ec6f47744dcd2a335" Jul 06 22:53:39.150733 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:39.150716 2571 scope.go:117] "RemoveContainer" containerID="6356dba9a4b259a09451c835d9f3c8260621e87287c40e9ec6f47744dcd2a335" Jul 06 22:53:39.150990 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:53:39.150971 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6356dba9a4b259a09451c835d9f3c8260621e87287c40e9ec6f47744dcd2a335\": container with ID starting with 6356dba9a4b259a09451c835d9f3c8260621e87287c40e9ec6f47744dcd2a335 not found: ID does not exist" containerID="6356dba9a4b259a09451c835d9f3c8260621e87287c40e9ec6f47744dcd2a335" Jul 06 22:53:39.151047 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:39.150999 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6356dba9a4b259a09451c835d9f3c8260621e87287c40e9ec6f47744dcd2a335"} err="failed to get container status \"6356dba9a4b259a09451c835d9f3c8260621e87287c40e9ec6f47744dcd2a335\": rpc error: code = NotFound desc = could not find container \"6356dba9a4b259a09451c835d9f3c8260621e87287c40e9ec6f47744dcd2a335\": container with ID starting with 6356dba9a4b259a09451c835d9f3c8260621e87287c40e9ec6f47744dcd2a335 not found: ID does not exist" Jul 06 22:53:39.163836 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:39.163808 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-8f36f0-86d7757566-4x2xc"] Jul 06 22:53:39.170572 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:39.170546 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-a-8f36f0-86d7757566-4x2xc"] Jul 06 22:53:39.635395 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:39.635365 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8feaf161-d6d9-4c61-96e2-cc2608c7b9aa" path="/var/lib/kubelet/pods/8feaf161-d6d9-4c61-96e2-cc2608c7b9aa/volumes" Jul 06 22:53:43.824789 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:43.824754 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:43.924687 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:43.924645 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:44.095564 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:44.095471 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:44.325579 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:44.325547 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:44.504220 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:44.504184 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:44.933960 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:44.933927 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:45.130458 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:45.130426 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:45.230748 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:45.230664 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:45.399894 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:45.399863 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-164a1c-98ff65b9b-pr2bl"] Jul 06 22:53:45.400117 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:45.400093 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-b-164a1c-98ff65b9b-pr2bl" podUID="94b9cea6-cf7b-4f2a-9ec9-14bcf7308b94" containerName="maas-api" containerID="cri-o://60419ddab0dab1b3734fb624c179ba96283e29896c8405a30572edac39fb14cb" gracePeriod=30 Jul 06 22:53:45.428827 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:45.428782 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:45.653122 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:45.653089 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-164a1c-98ff65b9b-pr2bl" Jul 06 22:53:45.705484 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:45.705450 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zlc5c\" (UniqueName: \"kubernetes.io/projected/94b9cea6-cf7b-4f2a-9ec9-14bcf7308b94-kube-api-access-zlc5c\") pod \"94b9cea6-cf7b-4f2a-9ec9-14bcf7308b94\" (UID: \"94b9cea6-cf7b-4f2a-9ec9-14bcf7308b94\") " Jul 06 22:53:45.705484 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:45.705483 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94b9cea6-cf7b-4f2a-9ec9-14bcf7308b94-maas-api-tls\") pod \"94b9cea6-cf7b-4f2a-9ec9-14bcf7308b94\" (UID: \"94b9cea6-cf7b-4f2a-9ec9-14bcf7308b94\") " Jul 06 22:53:45.705747 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:45.705567 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94b9cea6-cf7b-4f2a-9ec9-14bcf7308b94-openshift-service-ca\") pod \"94b9cea6-cf7b-4f2a-9ec9-14bcf7308b94\" (UID: \"94b9cea6-cf7b-4f2a-9ec9-14bcf7308b94\") " Jul 06 22:53:45.705970 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:45.705940 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94b9cea6-cf7b-4f2a-9ec9-14bcf7308b94-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "94b9cea6-cf7b-4f2a-9ec9-14bcf7308b94" (UID: "94b9cea6-cf7b-4f2a-9ec9-14bcf7308b94"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:53:45.707629 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:45.707604 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94b9cea6-cf7b-4f2a-9ec9-14bcf7308b94-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "94b9cea6-cf7b-4f2a-9ec9-14bcf7308b94" (UID: "94b9cea6-cf7b-4f2a-9ec9-14bcf7308b94"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:53:45.707629 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:45.707612 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94b9cea6-cf7b-4f2a-9ec9-14bcf7308b94-kube-api-access-zlc5c" (OuterVolumeSpecName: "kube-api-access-zlc5c") pod "94b9cea6-cf7b-4f2a-9ec9-14bcf7308b94" (UID: "94b9cea6-cf7b-4f2a-9ec9-14bcf7308b94"). InnerVolumeSpecName "kube-api-access-zlc5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:53:45.806705 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:45.806615 2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94b9cea6-cf7b-4f2a-9ec9-14bcf7308b94-openshift-service-ca\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:53:45.806705 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:45.806646 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlc5c\" (UniqueName: \"kubernetes.io/projected/94b9cea6-cf7b-4f2a-9ec9-14bcf7308b94-kube-api-access-zlc5c\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:53:45.806705 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:45.806657 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94b9cea6-cf7b-4f2a-9ec9-14bcf7308b94-maas-api-tls\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:53:46.176906 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:46.176870 2571 generic.go:356] "Generic (PLEG): container finished" podID="94b9cea6-cf7b-4f2a-9ec9-14bcf7308b94" containerID="60419ddab0dab1b3734fb624c179ba96283e29896c8405a30572edac39fb14cb" exitCode=0 Jul 06 22:53:46.177367 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:46.176913 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-164a1c-98ff65b9b-pr2bl" Jul 06 22:53:46.177367 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:46.176955 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-164a1c-98ff65b9b-pr2bl" event={"ID":"94b9cea6-cf7b-4f2a-9ec9-14bcf7308b94","Type":"ContainerDied","Data":"60419ddab0dab1b3734fb624c179ba96283e29896c8405a30572edac39fb14cb"} Jul 06 22:53:46.177367 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:46.177003 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-164a1c-98ff65b9b-pr2bl" event={"ID":"94b9cea6-cf7b-4f2a-9ec9-14bcf7308b94","Type":"ContainerDied","Data":"bf4b59e011e27096b6ecccd97985f8c8fbbc3667b3d784e080f466013ac93708"} Jul 06 22:53:46.177367 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:46.177025 2571 scope.go:117] "RemoveContainer" containerID="60419ddab0dab1b3734fb624c179ba96283e29896c8405a30572edac39fb14cb" Jul 06 22:53:46.185680 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:46.185664 2571 scope.go:117] "RemoveContainer" containerID="60419ddab0dab1b3734fb624c179ba96283e29896c8405a30572edac39fb14cb" Jul 06 22:53:46.185964 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:53:46.185945 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60419ddab0dab1b3734fb624c179ba96283e29896c8405a30572edac39fb14cb\": container with ID starting with 60419ddab0dab1b3734fb624c179ba96283e29896c8405a30572edac39fb14cb not found: ID does not exist" containerID="60419ddab0dab1b3734fb624c179ba96283e29896c8405a30572edac39fb14cb" Jul 06 22:53:46.186009 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:46.185973 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60419ddab0dab1b3734fb624c179ba96283e29896c8405a30572edac39fb14cb"} err="failed to get container status \"60419ddab0dab1b3734fb624c179ba96283e29896c8405a30572edac39fb14cb\": rpc error: code = NotFound desc = could not find container \"60419ddab0dab1b3734fb624c179ba96283e29896c8405a30572edac39fb14cb\": container with ID starting with 60419ddab0dab1b3734fb624c179ba96283e29896c8405a30572edac39fb14cb not found: ID does not exist" Jul 06 22:53:46.202837 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:46.202807 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-164a1c-98ff65b9b-pr2bl"] Jul 06 22:53:46.215730 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:46.215699 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-b-164a1c-98ff65b9b-pr2bl"] Jul 06 22:53:47.635471 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:47.635423 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94b9cea6-cf7b-4f2a-9ec9-14bcf7308b94" path="/var/lib/kubelet/pods/94b9cea6-cf7b-4f2a-9ec9-14bcf7308b94/volumes" Jul 06 22:53:51.731850 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:51.731814 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:51.797586 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:51.797546 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:52.013551 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:52.013439 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:52.247227 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:52.247166 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:52.591304 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:52.591271 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:52.945050 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:52.945001 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:53.204275 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:53.204137 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:55.946540 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:55.946502 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:56.031336 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:56.031302 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-a-2ef6a3-66dbb6d8c5-6v2s7"] Jul 06 22:53:56.031868 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:56.031842 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8feaf161-d6d9-4c61-96e2-cc2608c7b9aa" containerName="maas-api" Jul 06 22:53:56.032061 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:56.032039 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="8feaf161-d6d9-4c61-96e2-cc2608c7b9aa" containerName="maas-api" Jul 06 22:53:56.032172 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:56.032076 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94b9cea6-cf7b-4f2a-9ec9-14bcf7308b94" containerName="maas-api" Jul 06 22:53:56.032172 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:56.032086 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="94b9cea6-cf7b-4f2a-9ec9-14bcf7308b94" containerName="maas-api" Jul 06 22:53:56.032363 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:56.032256 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="94b9cea6-cf7b-4f2a-9ec9-14bcf7308b94" containerName="maas-api" Jul 06 22:53:56.032363 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:56.032272 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="8feaf161-d6d9-4c61-96e2-cc2608c7b9aa" containerName="maas-api" Jul 06 22:53:56.034333 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:56.034306 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:56.037118 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:56.037082 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-2ef6a3-66dbb6d8c5-6v2s7" Jul 06 22:53:56.039611 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:56.039590 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-a-2ef6a3-serving-cert\"" Jul 06 22:53:56.039704 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:56.039656 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-98x7w\"" Jul 06 22:53:56.050373 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:56.050353 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-2ef6a3-66dbb6d8c5-6v2s7"] Jul 06 22:53:56.107716 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:56.107676 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:56.197017 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:56.196919 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa9fe430-ff17-4440-9ba6-1c5582cd5344-openshift-service-ca\") pod \"maas-api-e2e-shared-a-2ef6a3-66dbb6d8c5-6v2s7\" (UID: \"aa9fe430-ff17-4440-9ba6-1c5582cd5344\") " pod="opendatahub/maas-api-e2e-shared-a-2ef6a3-66dbb6d8c5-6v2s7" Jul 06 22:53:56.197017 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:56.196989 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa9fe430-ff17-4440-9ba6-1c5582cd5344-maas-api-tls\") pod \"maas-api-e2e-shared-a-2ef6a3-66dbb6d8c5-6v2s7\" (UID: \"aa9fe430-ff17-4440-9ba6-1c5582cd5344\") " pod="opendatahub/maas-api-e2e-shared-a-2ef6a3-66dbb6d8c5-6v2s7" Jul 06 22:53:56.197243 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:56.197043 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m6hs\" (UniqueName: \"kubernetes.io/projected/aa9fe430-ff17-4440-9ba6-1c5582cd5344-kube-api-access-9m6hs\") pod \"maas-api-e2e-shared-a-2ef6a3-66dbb6d8c5-6v2s7\" (UID: \"aa9fe430-ff17-4440-9ba6-1c5582cd5344\") " pod="opendatahub/maas-api-e2e-shared-a-2ef6a3-66dbb6d8c5-6v2s7" Jul 06 22:53:56.198777 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:56.198752 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:56.297828 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:56.297791 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa9fe430-ff17-4440-9ba6-1c5582cd5344-openshift-service-ca\") pod \"maas-api-e2e-shared-a-2ef6a3-66dbb6d8c5-6v2s7\" (UID: \"aa9fe430-ff17-4440-9ba6-1c5582cd5344\") " pod="opendatahub/maas-api-e2e-shared-a-2ef6a3-66dbb6d8c5-6v2s7" Jul 06 22:53:56.298030 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:56.297852 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa9fe430-ff17-4440-9ba6-1c5582cd5344-maas-api-tls\") pod \"maas-api-e2e-shared-a-2ef6a3-66dbb6d8c5-6v2s7\" (UID: \"aa9fe430-ff17-4440-9ba6-1c5582cd5344\") " pod="opendatahub/maas-api-e2e-shared-a-2ef6a3-66dbb6d8c5-6v2s7" Jul 06 22:53:56.298030 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:56.297875 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m6hs\" (UniqueName: \"kubernetes.io/projected/aa9fe430-ff17-4440-9ba6-1c5582cd5344-kube-api-access-9m6hs\") pod \"maas-api-e2e-shared-a-2ef6a3-66dbb6d8c5-6v2s7\" (UID: \"aa9fe430-ff17-4440-9ba6-1c5582cd5344\") " pod="opendatahub/maas-api-e2e-shared-a-2ef6a3-66dbb6d8c5-6v2s7" Jul 06 22:53:56.298545 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:56.298516 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa9fe430-ff17-4440-9ba6-1c5582cd5344-openshift-service-ca\") pod \"maas-api-e2e-shared-a-2ef6a3-66dbb6d8c5-6v2s7\" (UID: \"aa9fe430-ff17-4440-9ba6-1c5582cd5344\") " pod="opendatahub/maas-api-e2e-shared-a-2ef6a3-66dbb6d8c5-6v2s7" Jul 06 22:53:56.300375 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:56.300351 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa9fe430-ff17-4440-9ba6-1c5582cd5344-maas-api-tls\") pod \"maas-api-e2e-shared-a-2ef6a3-66dbb6d8c5-6v2s7\" (UID: \"aa9fe430-ff17-4440-9ba6-1c5582cd5344\") " pod="opendatahub/maas-api-e2e-shared-a-2ef6a3-66dbb6d8c5-6v2s7" Jul 06 22:53:56.306379 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:56.306350 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m6hs\" (UniqueName: \"kubernetes.io/projected/aa9fe430-ff17-4440-9ba6-1c5582cd5344-kube-api-access-9m6hs\") pod \"maas-api-e2e-shared-a-2ef6a3-66dbb6d8c5-6v2s7\" (UID: \"aa9fe430-ff17-4440-9ba6-1c5582cd5344\") " pod="opendatahub/maas-api-e2e-shared-a-2ef6a3-66dbb6d8c5-6v2s7" Jul 06 22:53:56.348388 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:56.348355 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-2ef6a3-66dbb6d8c5-6v2s7" Jul 06 22:53:56.494537 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:56.494504 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-2ef6a3-66dbb6d8c5-6v2s7"] Jul 06 22:53:56.496354 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:53:56.496323 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa9fe430_ff17_4440_9ba6_1c5582cd5344.slice/crio-55c3d457d942f1221197c77880dfa72f2afc2d16a61f836a7971236abf222a48 WatchSource:0}: Error finding container 55c3d457d942f1221197c77880dfa72f2afc2d16a61f836a7971236abf222a48: Status 404 returned error can't find the container with id 55c3d457d942f1221197c77880dfa72f2afc2d16a61f836a7971236abf222a48 Jul 06 22:53:56.704476 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:56.704445 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:56.801311 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:56.801165 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:57.229735 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:57.229695 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-2ef6a3-66dbb6d8c5-6v2s7" event={"ID":"aa9fe430-ff17-4440-9ba6-1c5582cd5344","Type":"ContainerStarted","Data":"4488956c2caf76beed9b08a9890c668f6f184a8bd250138bc5c13e329083bd5f"} Jul 06 22:53:57.230261 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:57.229763 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-2ef6a3-66dbb6d8c5-6v2s7" event={"ID":"aa9fe430-ff17-4440-9ba6-1c5582cd5344","Type":"ContainerStarted","Data":"55c3d457d942f1221197c77880dfa72f2afc2d16a61f836a7971236abf222a48"} Jul 06 22:53:57.230261 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:57.229790 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-a-2ef6a3-66dbb6d8c5-6v2s7" Jul 06 22:53:57.249607 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:53:57.249556 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-a-2ef6a3-66dbb6d8c5-6v2s7" podStartSLOduration=0.939426592 podStartE2EDuration="1.249539173s" totalImagesPullingTime="310.112581ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:53:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:53:57.248701305 +0000 UTC m=+2044.166944164" watchObservedRunningTime="2026-07-06 22:53:57.249539173 +0000 UTC m=+2044.167782029" Jul 06 22:54:00.450430 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:00.450392 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:00.789199 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:00.789102 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:01.131454 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:01.131366 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:01.230976 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:01.230945 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:03.240627 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:03.240598 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-a-2ef6a3-66dbb6d8c5-6v2s7" Jul 06 22:54:04.080728 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:04.080687 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:04.197048 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:04.197014 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:04.218768 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:04.218736 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-b-6c153e-647b666d54-p74jz"] Jul 06 22:54:04.222579 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:04.222555 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-6c153e-647b666d54-p74jz" Jul 06 22:54:04.225250 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:04.225223 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-b-6c153e-serving-cert\"" Jul 06 22:54:04.238547 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:04.238520 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-6c153e-647b666d54-p74jz"] Jul 06 22:54:04.283100 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:04.283068 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:04.369922 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:04.369829 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d43d667-a519-43ed-915e-837b766f397a-maas-api-tls\") pod \"maas-api-e2e-shared-b-6c153e-647b666d54-p74jz\" (UID: \"0d43d667-a519-43ed-915e-837b766f397a\") " pod="opendatahub/maas-api-e2e-shared-b-6c153e-647b666d54-p74jz" Jul 06 22:54:04.369922 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:04.369909 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntw6j\" (UniqueName: \"kubernetes.io/projected/0d43d667-a519-43ed-915e-837b766f397a-kube-api-access-ntw6j\") pod \"maas-api-e2e-shared-b-6c153e-647b666d54-p74jz\" (UID: \"0d43d667-a519-43ed-915e-837b766f397a\") " pod="opendatahub/maas-api-e2e-shared-b-6c153e-647b666d54-p74jz" Jul 06 22:54:04.370116 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:04.369949 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d43d667-a519-43ed-915e-837b766f397a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-6c153e-647b666d54-p74jz\" (UID: \"0d43d667-a519-43ed-915e-837b766f397a\") " pod="opendatahub/maas-api-e2e-shared-b-6c153e-647b666d54-p74jz" Jul 06 22:54:04.471331 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:04.471291 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntw6j\" (UniqueName: \"kubernetes.io/projected/0d43d667-a519-43ed-915e-837b766f397a-kube-api-access-ntw6j\") pod \"maas-api-e2e-shared-b-6c153e-647b666d54-p74jz\" (UID: \"0d43d667-a519-43ed-915e-837b766f397a\") " pod="opendatahub/maas-api-e2e-shared-b-6c153e-647b666d54-p74jz" Jul 06 22:54:04.471331 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:04.471335 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d43d667-a519-43ed-915e-837b766f397a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-6c153e-647b666d54-p74jz\" (UID: \"0d43d667-a519-43ed-915e-837b766f397a\") " pod="opendatahub/maas-api-e2e-shared-b-6c153e-647b666d54-p74jz" Jul 06 22:54:04.471614 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:04.471381 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d43d667-a519-43ed-915e-837b766f397a-maas-api-tls\") pod \"maas-api-e2e-shared-b-6c153e-647b666d54-p74jz\" (UID: \"0d43d667-a519-43ed-915e-837b766f397a\") " pod="opendatahub/maas-api-e2e-shared-b-6c153e-647b666d54-p74jz" Jul 06 22:54:04.472054 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:04.472025 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d43d667-a519-43ed-915e-837b766f397a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-6c153e-647b666d54-p74jz\" (UID: \"0d43d667-a519-43ed-915e-837b766f397a\") " pod="opendatahub/maas-api-e2e-shared-b-6c153e-647b666d54-p74jz" Jul 06 22:54:04.473823 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:04.473793 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d43d667-a519-43ed-915e-837b766f397a-maas-api-tls\") pod \"maas-api-e2e-shared-b-6c153e-647b666d54-p74jz\" (UID: \"0d43d667-a519-43ed-915e-837b766f397a\") " pod="opendatahub/maas-api-e2e-shared-b-6c153e-647b666d54-p74jz" Jul 06 22:54:04.480931 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:04.480907 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntw6j\" (UniqueName: \"kubernetes.io/projected/0d43d667-a519-43ed-915e-837b766f397a-kube-api-access-ntw6j\") pod \"maas-api-e2e-shared-b-6c153e-647b666d54-p74jz\" (UID: \"0d43d667-a519-43ed-915e-837b766f397a\") " pod="opendatahub/maas-api-e2e-shared-b-6c153e-647b666d54-p74jz" Jul 06 22:54:04.501793 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:04.501757 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:04.533808 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:04.533758 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-6c153e-647b666d54-p74jz" Jul 06 22:54:04.675618 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:04.675588 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-6c153e-647b666d54-p74jz"] Jul 06 22:54:04.678256 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:54:04.678222 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d43d667_a519_43ed_915e_837b766f397a.slice/crio-e6b2f041c7f4a81524ca796db5f5d38fc269daf3687b47747a11f312d3cf8282 WatchSource:0}: Error finding container e6b2f041c7f4a81524ca796db5f5d38fc269daf3687b47747a11f312d3cf8282: Status 404 returned error can't find the container with id e6b2f041c7f4a81524ca796db5f5d38fc269daf3687b47747a11f312d3cf8282 Jul 06 22:54:05.027323 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:05.027232 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:05.128969 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:05.128930 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:05.275739 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:05.275706 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-6c153e-647b666d54-p74jz" event={"ID":"0d43d667-a519-43ed-915e-837b766f397a","Type":"ContainerStarted","Data":"e6b2f041c7f4a81524ca796db5f5d38fc269daf3687b47747a11f312d3cf8282"} Jul 06 22:54:06.281917 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:06.281875 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-6c153e-647b666d54-p74jz" event={"ID":"0d43d667-a519-43ed-915e-837b766f397a","Type":"ContainerStarted","Data":"426900e7687381481c95b56a702962d71b4d958aae33a091ce9662693dd2d425"} Jul 06 22:54:06.282325 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:06.282021 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-b-6c153e-647b666d54-p74jz" Jul 06 22:54:06.303954 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:06.303902 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-b-6c153e-647b666d54-p74jz" podStartSLOduration=1.762411873 podStartE2EDuration="2.303883646s" totalImagesPullingTime="541.471773ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:54:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:54:06.301995039 +0000 UTC m=+2053.220237896" watchObservedRunningTime="2026-07-06 22:54:06.303883646 +0000 UTC m=+2053.222126502" Jul 06 22:54:08.546326 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:08.546286 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:12.291933 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:12.291904 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-b-6c153e-647b666d54-p74jz" Jul 06 22:54:17.730030 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:17.729945 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:17.824095 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:17.824016 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:25.949510 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:25.949475 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:26.050000 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:26.049964 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:42.164967 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:42.164929 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-2ef6a3/auth-test-model-2ef6a3-kserve-7ff94b46ff-v9k5c"] Jul 06 22:54:42.168772 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:42.168750 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2ef6a3/auth-test-model-2ef6a3-kserve-7ff94b46ff-v9k5c" Jul 06 22:54:42.171866 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:42.171841 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-2ef6a3\"/\"auth-test-model-2ef6a3-kserve-self-signed-certs\"" Jul 06 22:54:42.172627 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:42.172604 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-2ef6a3\"/\"default-dockercfg-d8nv5\"" Jul 06 22:54:42.172746 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:42.172641 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-2ef6a3\"/\"openshift-service-ca.crt\"" Jul 06 22:54:42.172746 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:42.172652 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-2ef6a3\"/\"kube-root-ca.crt\"" Jul 06 22:54:42.177811 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:42.177786 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-2ef6a3/auth-test-model-2ef6a3-kserve-7ff94b46ff-v9k5c"] Jul 06 22:54:42.204961 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:42.204930 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:42.307467 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:42.307432 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/300fbe96-fadd-49c0-87b6-c7e3497af807-tls-certs\") pod \"auth-test-model-2ef6a3-kserve-7ff94b46ff-v9k5c\" (UID: \"300fbe96-fadd-49c0-87b6-c7e3497af807\") " pod="ai-tenant-e2e-shared-a-2ef6a3/auth-test-model-2ef6a3-kserve-7ff94b46ff-v9k5c" Jul 06 22:54:42.307467 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:42.307475 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/300fbe96-fadd-49c0-87b6-c7e3497af807-tmp-dir\") pod \"auth-test-model-2ef6a3-kserve-7ff94b46ff-v9k5c\" (UID: \"300fbe96-fadd-49c0-87b6-c7e3497af807\") " pod="ai-tenant-e2e-shared-a-2ef6a3/auth-test-model-2ef6a3-kserve-7ff94b46ff-v9k5c" Jul 06 22:54:42.307739 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:42.307553 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/300fbe96-fadd-49c0-87b6-c7e3497af807-model-cache\") pod \"auth-test-model-2ef6a3-kserve-7ff94b46ff-v9k5c\" (UID: \"300fbe96-fadd-49c0-87b6-c7e3497af807\") " pod="ai-tenant-e2e-shared-a-2ef6a3/auth-test-model-2ef6a3-kserve-7ff94b46ff-v9k5c" Jul 06 22:54:42.307739 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:42.307578 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmp9x\" (UniqueName: \"kubernetes.io/projected/300fbe96-fadd-49c0-87b6-c7e3497af807-kube-api-access-lmp9x\") pod \"auth-test-model-2ef6a3-kserve-7ff94b46ff-v9k5c\" (UID: \"300fbe96-fadd-49c0-87b6-c7e3497af807\") " pod="ai-tenant-e2e-shared-a-2ef6a3/auth-test-model-2ef6a3-kserve-7ff94b46ff-v9k5c" Jul 06 22:54:42.307739 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:42.307615 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/300fbe96-fadd-49c0-87b6-c7e3497af807-dshm\") pod \"auth-test-model-2ef6a3-kserve-7ff94b46ff-v9k5c\" (UID: \"300fbe96-fadd-49c0-87b6-c7e3497af807\") " pod="ai-tenant-e2e-shared-a-2ef6a3/auth-test-model-2ef6a3-kserve-7ff94b46ff-v9k5c" Jul 06 22:54:42.307739 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:42.307692 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/300fbe96-fadd-49c0-87b6-c7e3497af807-home\") pod \"auth-test-model-2ef6a3-kserve-7ff94b46ff-v9k5c\" (UID: \"300fbe96-fadd-49c0-87b6-c7e3497af807\") " pod="ai-tenant-e2e-shared-a-2ef6a3/auth-test-model-2ef6a3-kserve-7ff94b46ff-v9k5c" Jul 06 22:54:42.408738 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:42.408698 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/300fbe96-fadd-49c0-87b6-c7e3497af807-tls-certs\") pod \"auth-test-model-2ef6a3-kserve-7ff94b46ff-v9k5c\" (UID: \"300fbe96-fadd-49c0-87b6-c7e3497af807\") " pod="ai-tenant-e2e-shared-a-2ef6a3/auth-test-model-2ef6a3-kserve-7ff94b46ff-v9k5c" Jul 06 22:54:42.408738 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:42.408741 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/300fbe96-fadd-49c0-87b6-c7e3497af807-tmp-dir\") pod \"auth-test-model-2ef6a3-kserve-7ff94b46ff-v9k5c\" (UID: \"300fbe96-fadd-49c0-87b6-c7e3497af807\") " pod="ai-tenant-e2e-shared-a-2ef6a3/auth-test-model-2ef6a3-kserve-7ff94b46ff-v9k5c" Jul 06 22:54:42.409008 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:42.408777 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/300fbe96-fadd-49c0-87b6-c7e3497af807-model-cache\") pod \"auth-test-model-2ef6a3-kserve-7ff94b46ff-v9k5c\" (UID: \"300fbe96-fadd-49c0-87b6-c7e3497af807\") " pod="ai-tenant-e2e-shared-a-2ef6a3/auth-test-model-2ef6a3-kserve-7ff94b46ff-v9k5c" Jul 06 22:54:42.409008 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:42.408796 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmp9x\" (UniqueName: \"kubernetes.io/projected/300fbe96-fadd-49c0-87b6-c7e3497af807-kube-api-access-lmp9x\") pod \"auth-test-model-2ef6a3-kserve-7ff94b46ff-v9k5c\" (UID: \"300fbe96-fadd-49c0-87b6-c7e3497af807\") " pod="ai-tenant-e2e-shared-a-2ef6a3/auth-test-model-2ef6a3-kserve-7ff94b46ff-v9k5c" Jul 06 22:54:42.409008 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:42.408822 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/300fbe96-fadd-49c0-87b6-c7e3497af807-dshm\") pod \"auth-test-model-2ef6a3-kserve-7ff94b46ff-v9k5c\" (UID: \"300fbe96-fadd-49c0-87b6-c7e3497af807\") " pod="ai-tenant-e2e-shared-a-2ef6a3/auth-test-model-2ef6a3-kserve-7ff94b46ff-v9k5c" Jul 06 22:54:42.409008 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:42.408861 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/300fbe96-fadd-49c0-87b6-c7e3497af807-home\") pod \"auth-test-model-2ef6a3-kserve-7ff94b46ff-v9k5c\" (UID: \"300fbe96-fadd-49c0-87b6-c7e3497af807\") " pod="ai-tenant-e2e-shared-a-2ef6a3/auth-test-model-2ef6a3-kserve-7ff94b46ff-v9k5c" Jul 06 22:54:42.409309 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:42.409286 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/300fbe96-fadd-49c0-87b6-c7e3497af807-model-cache\") pod \"auth-test-model-2ef6a3-kserve-7ff94b46ff-v9k5c\" (UID: \"300fbe96-fadd-49c0-87b6-c7e3497af807\") " pod="ai-tenant-e2e-shared-a-2ef6a3/auth-test-model-2ef6a3-kserve-7ff94b46ff-v9k5c" Jul 06 22:54:42.409383 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:42.409334 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/300fbe96-fadd-49c0-87b6-c7e3497af807-home\") pod \"auth-test-model-2ef6a3-kserve-7ff94b46ff-v9k5c\" (UID: \"300fbe96-fadd-49c0-87b6-c7e3497af807\") " pod="ai-tenant-e2e-shared-a-2ef6a3/auth-test-model-2ef6a3-kserve-7ff94b46ff-v9k5c" Jul 06 22:54:42.409383 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:42.409360 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/300fbe96-fadd-49c0-87b6-c7e3497af807-tmp-dir\") pod \"auth-test-model-2ef6a3-kserve-7ff94b46ff-v9k5c\" (UID: \"300fbe96-fadd-49c0-87b6-c7e3497af807\") " pod="ai-tenant-e2e-shared-a-2ef6a3/auth-test-model-2ef6a3-kserve-7ff94b46ff-v9k5c" Jul 06 22:54:42.411091 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:42.411068 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/300fbe96-fadd-49c0-87b6-c7e3497af807-dshm\") pod \"auth-test-model-2ef6a3-kserve-7ff94b46ff-v9k5c\" (UID: \"300fbe96-fadd-49c0-87b6-c7e3497af807\") " pod="ai-tenant-e2e-shared-a-2ef6a3/auth-test-model-2ef6a3-kserve-7ff94b46ff-v9k5c" Jul 06 22:54:42.411354 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:42.411334 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/300fbe96-fadd-49c0-87b6-c7e3497af807-tls-certs\") pod \"auth-test-model-2ef6a3-kserve-7ff94b46ff-v9k5c\" (UID: \"300fbe96-fadd-49c0-87b6-c7e3497af807\") " pod="ai-tenant-e2e-shared-a-2ef6a3/auth-test-model-2ef6a3-kserve-7ff94b46ff-v9k5c" Jul 06 22:54:42.419943 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:42.419880 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmp9x\" (UniqueName: \"kubernetes.io/projected/300fbe96-fadd-49c0-87b6-c7e3497af807-kube-api-access-lmp9x\") pod \"auth-test-model-2ef6a3-kserve-7ff94b46ff-v9k5c\" (UID: \"300fbe96-fadd-49c0-87b6-c7e3497af807\") " pod="ai-tenant-e2e-shared-a-2ef6a3/auth-test-model-2ef6a3-kserve-7ff94b46ff-v9k5c" Jul 06 22:54:42.479370 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:42.479336 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2ef6a3/auth-test-model-2ef6a3-kserve-7ff94b46ff-v9k5c" Jul 06 22:54:42.622772 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:42.622734 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-2ef6a3/auth-test-model-2ef6a3-kserve-7ff94b46ff-v9k5c"] Jul 06 22:54:42.626748 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:54:42.626716 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod300fbe96_fadd_49c0_87b6_c7e3497af807.slice/crio-bd5bc36b0f6c3dde8222059d2ac4fb73758567a14f8d773a43c7bb9a7c335690 WatchSource:0}: Error finding container bd5bc36b0f6c3dde8222059d2ac4fb73758567a14f8d773a43c7bb9a7c335690: Status 404 returned error can't find the container with id bd5bc36b0f6c3dde8222059d2ac4fb73758567a14f8d773a43c7bb9a7c335690 Jul 06 22:54:43.463511 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:43.463473 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2ef6a3/auth-test-model-2ef6a3-kserve-7ff94b46ff-v9k5c" event={"ID":"300fbe96-fadd-49c0-87b6-c7e3497af807","Type":"ContainerStarted","Data":"b8f589ff1fe14bc345c80d2a021dd887f1b9aa3d106f1c9541ecd3f490207903"} Jul 06 22:54:43.463912 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:43.463518 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2ef6a3/auth-test-model-2ef6a3-kserve-7ff94b46ff-v9k5c" event={"ID":"300fbe96-fadd-49c0-87b6-c7e3497af807","Type":"ContainerStarted","Data":"bd5bc36b0f6c3dde8222059d2ac4fb73758567a14f8d773a43c7bb9a7c335690"} Jul 06 22:54:43.499760 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:43.499707 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-2ef6a3/auth-test-model-2ef6a3-kserve-7ff94b46ff-v9k5c" podStartSLOduration=1.4996885629999999 podStartE2EDuration="1.499688563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:54:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:54:43.497706512 +0000 UTC m=+2090.415949371" watchObservedRunningTime="2026-07-06 22:54:43.499688563 +0000 UTC m=+2090.417931419" Jul 06 22:54:52.479728 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:52.479686 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-2ef6a3/auth-test-model-2ef6a3-kserve-7ff94b46ff-v9k5c" Jul 06 22:54:52.480182 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:52.479749 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-2ef6a3/auth-test-model-2ef6a3-kserve-7ff94b46ff-v9k5c" Jul 06 22:54:52.492463 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:52.492434 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-2ef6a3/auth-test-model-2ef6a3-kserve-7ff94b46ff-v9k5c" Jul 06 22:54:52.524265 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:52.524227 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-2ef6a3/auth-test-model-2ef6a3-kserve-7ff94b46ff-v9k5c" Jul 06 22:54:53.688319 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:53.688295 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-4x6pf_5796a8ea-adcb-4819-bb18-0b85e1ef4681/console-operator/2.log" Jul 06 22:54:53.690662 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:53.690639 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-4x6pf_5796a8ea-adcb-4819-bb18-0b85e1ef4681/console-operator/2.log" Jul 06 22:54:53.691824 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:53.691801 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qbbrs_bf23ef00-a2d0-499c-8c87-5d6b98510ea0/ovn-acl-logging/0.log" Jul 06 22:54:53.693973 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:54:53.693951 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qbbrs_bf23ef00-a2d0-499c-8c87-5d6b98510ea0/ovn-acl-logging/0.log" Jul 06 22:55:09.237133 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:55:09.237097 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:43.867319 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:56:43.867287 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:57:16.020968 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:16.020881 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-2ef6a3/auth-test-model-2ef6a3-kserve-7ff94b46ff-v9k5c"] Jul 06 22:57:16.048320 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:16.048278 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:57:16.851799 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:16.851766 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:57:26.443149 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:26.443112 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:57:30.000863 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:30.000827 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:57:30.099628 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:30.099599 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:57:31.041165 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:31.041096 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-2ef6a3/auth-test-model-2ef6a3-kserve-7ff94b46ff-v9k5c" podUID="300fbe96-fadd-49c0-87b6-c7e3497af807" containerName="main" containerID="cri-o://b8f589ff1fe14bc345c80d2a021dd887f1b9aa3d106f1c9541ecd3f490207903" gracePeriod=45 Jul 06 22:57:31.261400 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:31.261362 2571 generic.go:356] "Generic (PLEG): container finished" podID="300fbe96-fadd-49c0-87b6-c7e3497af807" containerID="b8f589ff1fe14bc345c80d2a021dd887f1b9aa3d106f1c9541ecd3f490207903" exitCode=0 Jul 06 22:57:31.261582 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:31.261419 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2ef6a3/auth-test-model-2ef6a3-kserve-7ff94b46ff-v9k5c" event={"ID":"300fbe96-fadd-49c0-87b6-c7e3497af807","Type":"ContainerDied","Data":"b8f589ff1fe14bc345c80d2a021dd887f1b9aa3d106f1c9541ecd3f490207903"} Jul 06 22:57:31.287168 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:31.287141 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2ef6a3/auth-test-model-2ef6a3-kserve-7ff94b46ff-v9k5c" Jul 06 22:57:31.394394 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:31.394362 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lmp9x\" (UniqueName: \"kubernetes.io/projected/300fbe96-fadd-49c0-87b6-c7e3497af807-kube-api-access-lmp9x\") pod \"300fbe96-fadd-49c0-87b6-c7e3497af807\" (UID: \"300fbe96-fadd-49c0-87b6-c7e3497af807\") " Jul 06 22:57:31.394577 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:31.394427 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/300fbe96-fadd-49c0-87b6-c7e3497af807-model-cache\") pod \"300fbe96-fadd-49c0-87b6-c7e3497af807\" (UID: \"300fbe96-fadd-49c0-87b6-c7e3497af807\") " Jul 06 22:57:31.394577 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:31.394547 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/300fbe96-fadd-49c0-87b6-c7e3497af807-tls-certs\") pod \"300fbe96-fadd-49c0-87b6-c7e3497af807\" (UID: \"300fbe96-fadd-49c0-87b6-c7e3497af807\") " Jul 06 22:57:31.394707 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:31.394580 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/300fbe96-fadd-49c0-87b6-c7e3497af807-tmp-dir\") pod \"300fbe96-fadd-49c0-87b6-c7e3497af807\" (UID: \"300fbe96-fadd-49c0-87b6-c7e3497af807\") " Jul 06 22:57:31.394707 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:31.394640 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/300fbe96-fadd-49c0-87b6-c7e3497af807-model-cache" (OuterVolumeSpecName: "model-cache") pod "300fbe96-fadd-49c0-87b6-c7e3497af807" (UID: "300fbe96-fadd-49c0-87b6-c7e3497af807"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:57:31.394707 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:31.394644 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/300fbe96-fadd-49c0-87b6-c7e3497af807-home\") pod \"300fbe96-fadd-49c0-87b6-c7e3497af807\" (UID: \"300fbe96-fadd-49c0-87b6-c7e3497af807\") " Jul 06 22:57:31.394707 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:31.394692 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/300fbe96-fadd-49c0-87b6-c7e3497af807-dshm\") pod \"300fbe96-fadd-49c0-87b6-c7e3497af807\" (UID: \"300fbe96-fadd-49c0-87b6-c7e3497af807\") " Jul 06 22:57:31.394917 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:31.394850 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/300fbe96-fadd-49c0-87b6-c7e3497af807-home" (OuterVolumeSpecName: "home") pod "300fbe96-fadd-49c0-87b6-c7e3497af807" (UID: "300fbe96-fadd-49c0-87b6-c7e3497af807"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:57:31.394917 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:31.394866 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/300fbe96-fadd-49c0-87b6-c7e3497af807-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "300fbe96-fadd-49c0-87b6-c7e3497af807" (UID: "300fbe96-fadd-49c0-87b6-c7e3497af807"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:57:31.395036 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:31.395013 2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/300fbe96-fadd-49c0-87b6-c7e3497af807-model-cache\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:57:31.395036 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:31.395031 2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/300fbe96-fadd-49c0-87b6-c7e3497af807-tmp-dir\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:57:31.395136 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:31.395041 2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/300fbe96-fadd-49c0-87b6-c7e3497af807-home\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:57:31.396718 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:31.396690 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/300fbe96-fadd-49c0-87b6-c7e3497af807-dshm" (OuterVolumeSpecName: "dshm") pod "300fbe96-fadd-49c0-87b6-c7e3497af807" (UID: "300fbe96-fadd-49c0-87b6-c7e3497af807"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:57:31.396826 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:31.396728 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/300fbe96-fadd-49c0-87b6-c7e3497af807-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "300fbe96-fadd-49c0-87b6-c7e3497af807" (UID: "300fbe96-fadd-49c0-87b6-c7e3497af807"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:57:31.396826 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:31.396750 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/300fbe96-fadd-49c0-87b6-c7e3497af807-kube-api-access-lmp9x" (OuterVolumeSpecName: "kube-api-access-lmp9x") pod "300fbe96-fadd-49c0-87b6-c7e3497af807" (UID: "300fbe96-fadd-49c0-87b6-c7e3497af807"). InnerVolumeSpecName "kube-api-access-lmp9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:57:31.496542 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:31.496499 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmp9x\" (UniqueName: \"kubernetes.io/projected/300fbe96-fadd-49c0-87b6-c7e3497af807-kube-api-access-lmp9x\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:57:31.496542 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:31.496533 2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/300fbe96-fadd-49c0-87b6-c7e3497af807-tls-certs\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:57:31.496542 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:31.496543 2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/300fbe96-fadd-49c0-87b6-c7e3497af807-dshm\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:57:32.267533 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:32.267492 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2ef6a3/auth-test-model-2ef6a3-kserve-7ff94b46ff-v9k5c" event={"ID":"300fbe96-fadd-49c0-87b6-c7e3497af807","Type":"ContainerDied","Data":"bd5bc36b0f6c3dde8222059d2ac4fb73758567a14f8d773a43c7bb9a7c335690"} Jul 06 22:57:32.267533 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:32.267508 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2ef6a3/auth-test-model-2ef6a3-kserve-7ff94b46ff-v9k5c" Jul 06 22:57:32.268039 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:32.267543 2571 scope.go:117] "RemoveContainer" containerID="b8f589ff1fe14bc345c80d2a021dd887f1b9aa3d106f1c9541ecd3f490207903" Jul 06 22:57:32.285889 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:32.285862 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-2ef6a3/auth-test-model-2ef6a3-kserve-7ff94b46ff-v9k5c"] Jul 06 22:57:32.292351 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:32.292327 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-2ef6a3/auth-test-model-2ef6a3-kserve-7ff94b46ff-v9k5c"] Jul 06 22:57:33.635789 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:33.635758 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="300fbe96-fadd-49c0-87b6-c7e3497af807" path="/var/lib/kubelet/pods/300fbe96-fadd-49c0-87b6-c7e3497af807/volumes" Jul 06 22:57:48.960118 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:48.960076 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:57:54.132379 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:54.132341 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-6c153e/sub-test-model-6c153e-kserve-5686bd4669-5s2j5"] Jul 06 22:57:54.132752 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:54.132717 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="300fbe96-fadd-49c0-87b6-c7e3497af807" containerName="main" Jul 06 22:57:54.132752 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:54.132728 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="300fbe96-fadd-49c0-87b6-c7e3497af807" containerName="main" Jul 06 22:57:54.132824 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:54.132781 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="300fbe96-fadd-49c0-87b6-c7e3497af807" containerName="main" Jul 06 22:57:54.136810 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:54.136788 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6c153e/sub-test-model-6c153e-kserve-5686bd4669-5s2j5" Jul 06 22:57:54.140096 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:54.140074 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-6c153e\"/\"kube-root-ca.crt\"" Jul 06 22:57:54.140096 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:54.140086 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-6c153e\"/\"default-dockercfg-nsqkg\"" Jul 06 22:57:54.140323 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:54.140074 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-6c153e\"/\"openshift-service-ca.crt\"" Jul 06 22:57:54.140323 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:54.140126 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-6c153e\"/\"sub-test-model-6c153e-kserve-self-signed-certs\"" Jul 06 22:57:54.144319 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:54.144266 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-6c153e/sub-test-model-6c153e-kserve-5686bd4669-5s2j5"] Jul 06 22:57:54.195417 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:54.195377 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/274507cf-cf73-4529-aec7-a455b1c5edab-model-cache\") pod \"sub-test-model-6c153e-kserve-5686bd4669-5s2j5\" (UID: \"274507cf-cf73-4529-aec7-a455b1c5edab\") " pod="ai-tenant-e2e-shared-b-6c153e/sub-test-model-6c153e-kserve-5686bd4669-5s2j5" Jul 06 22:57:54.195417 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:54.195417 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8f6j\" (UniqueName: \"kubernetes.io/projected/274507cf-cf73-4529-aec7-a455b1c5edab-kube-api-access-m8f6j\") pod \"sub-test-model-6c153e-kserve-5686bd4669-5s2j5\" (UID: \"274507cf-cf73-4529-aec7-a455b1c5edab\") " pod="ai-tenant-e2e-shared-b-6c153e/sub-test-model-6c153e-kserve-5686bd4669-5s2j5" Jul 06 22:57:54.195667 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:54.195440 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/274507cf-cf73-4529-aec7-a455b1c5edab-tmp-dir\") pod \"sub-test-model-6c153e-kserve-5686bd4669-5s2j5\" (UID: \"274507cf-cf73-4529-aec7-a455b1c5edab\") " pod="ai-tenant-e2e-shared-b-6c153e/sub-test-model-6c153e-kserve-5686bd4669-5s2j5" Jul 06 22:57:54.195667 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:54.195534 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/274507cf-cf73-4529-aec7-a455b1c5edab-tls-certs\") pod \"sub-test-model-6c153e-kserve-5686bd4669-5s2j5\" (UID: \"274507cf-cf73-4529-aec7-a455b1c5edab\") " pod="ai-tenant-e2e-shared-b-6c153e/sub-test-model-6c153e-kserve-5686bd4669-5s2j5" Jul 06 22:57:54.195667 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:54.195563 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/274507cf-cf73-4529-aec7-a455b1c5edab-dshm\") pod \"sub-test-model-6c153e-kserve-5686bd4669-5s2j5\" (UID: \"274507cf-cf73-4529-aec7-a455b1c5edab\") " pod="ai-tenant-e2e-shared-b-6c153e/sub-test-model-6c153e-kserve-5686bd4669-5s2j5" Jul 06 22:57:54.195667 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:54.195601 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/274507cf-cf73-4529-aec7-a455b1c5edab-home\") pod \"sub-test-model-6c153e-kserve-5686bd4669-5s2j5\" (UID: \"274507cf-cf73-4529-aec7-a455b1c5edab\") " pod="ai-tenant-e2e-shared-b-6c153e/sub-test-model-6c153e-kserve-5686bd4669-5s2j5" Jul 06 22:57:54.218737 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:54.218698 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:57:54.296967 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:54.296876 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/274507cf-cf73-4529-aec7-a455b1c5edab-tls-certs\") pod \"sub-test-model-6c153e-kserve-5686bd4669-5s2j5\" (UID: \"274507cf-cf73-4529-aec7-a455b1c5edab\") " pod="ai-tenant-e2e-shared-b-6c153e/sub-test-model-6c153e-kserve-5686bd4669-5s2j5" Jul 06 22:57:54.296967 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:54.296919 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/274507cf-cf73-4529-aec7-a455b1c5edab-dshm\") pod \"sub-test-model-6c153e-kserve-5686bd4669-5s2j5\" (UID: \"274507cf-cf73-4529-aec7-a455b1c5edab\") " pod="ai-tenant-e2e-shared-b-6c153e/sub-test-model-6c153e-kserve-5686bd4669-5s2j5" Jul 06 22:57:54.296967 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:54.296965 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/274507cf-cf73-4529-aec7-a455b1c5edab-home\") pod \"sub-test-model-6c153e-kserve-5686bd4669-5s2j5\" (UID: \"274507cf-cf73-4529-aec7-a455b1c5edab\") " pod="ai-tenant-e2e-shared-b-6c153e/sub-test-model-6c153e-kserve-5686bd4669-5s2j5" Jul 06 22:57:54.297261 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:54.297066 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/274507cf-cf73-4529-aec7-a455b1c5edab-model-cache\") pod \"sub-test-model-6c153e-kserve-5686bd4669-5s2j5\" (UID: \"274507cf-cf73-4529-aec7-a455b1c5edab\") " pod="ai-tenant-e2e-shared-b-6c153e/sub-test-model-6c153e-kserve-5686bd4669-5s2j5" Jul 06 22:57:54.297261 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:54.297096 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8f6j\" (UniqueName: \"kubernetes.io/projected/274507cf-cf73-4529-aec7-a455b1c5edab-kube-api-access-m8f6j\") pod \"sub-test-model-6c153e-kserve-5686bd4669-5s2j5\" (UID: \"274507cf-cf73-4529-aec7-a455b1c5edab\") " pod="ai-tenant-e2e-shared-b-6c153e/sub-test-model-6c153e-kserve-5686bd4669-5s2j5" Jul 06 22:57:54.297261 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:54.297125 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/274507cf-cf73-4529-aec7-a455b1c5edab-tmp-dir\") pod \"sub-test-model-6c153e-kserve-5686bd4669-5s2j5\" (UID: \"274507cf-cf73-4529-aec7-a455b1c5edab\") " pod="ai-tenant-e2e-shared-b-6c153e/sub-test-model-6c153e-kserve-5686bd4669-5s2j5" Jul 06 22:57:54.297554 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:54.297527 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/274507cf-cf73-4529-aec7-a455b1c5edab-model-cache\") pod \"sub-test-model-6c153e-kserve-5686bd4669-5s2j5\" (UID: \"274507cf-cf73-4529-aec7-a455b1c5edab\") " pod="ai-tenant-e2e-shared-b-6c153e/sub-test-model-6c153e-kserve-5686bd4669-5s2j5" Jul 06 22:57:54.297630 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:54.297540 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/274507cf-cf73-4529-aec7-a455b1c5edab-tmp-dir\") pod \"sub-test-model-6c153e-kserve-5686bd4669-5s2j5\" (UID: \"274507cf-cf73-4529-aec7-a455b1c5edab\") " pod="ai-tenant-e2e-shared-b-6c153e/sub-test-model-6c153e-kserve-5686bd4669-5s2j5" Jul 06 22:57:54.297630 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:54.297598 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/274507cf-cf73-4529-aec7-a455b1c5edab-home\") pod \"sub-test-model-6c153e-kserve-5686bd4669-5s2j5\" (UID: \"274507cf-cf73-4529-aec7-a455b1c5edab\") " pod="ai-tenant-e2e-shared-b-6c153e/sub-test-model-6c153e-kserve-5686bd4669-5s2j5" Jul 06 22:57:54.299445 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:54.299419 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/274507cf-cf73-4529-aec7-a455b1c5edab-dshm\") pod \"sub-test-model-6c153e-kserve-5686bd4669-5s2j5\" (UID: \"274507cf-cf73-4529-aec7-a455b1c5edab\") " pod="ai-tenant-e2e-shared-b-6c153e/sub-test-model-6c153e-kserve-5686bd4669-5s2j5" Jul 06 22:57:54.299587 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:54.299571 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/274507cf-cf73-4529-aec7-a455b1c5edab-tls-certs\") pod \"sub-test-model-6c153e-kserve-5686bd4669-5s2j5\" (UID: \"274507cf-cf73-4529-aec7-a455b1c5edab\") " pod="ai-tenant-e2e-shared-b-6c153e/sub-test-model-6c153e-kserve-5686bd4669-5s2j5" Jul 06 22:57:54.306065 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:54.306038 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8f6j\" (UniqueName: \"kubernetes.io/projected/274507cf-cf73-4529-aec7-a455b1c5edab-kube-api-access-m8f6j\") pod \"sub-test-model-6c153e-kserve-5686bd4669-5s2j5\" (UID: \"274507cf-cf73-4529-aec7-a455b1c5edab\") " pod="ai-tenant-e2e-shared-b-6c153e/sub-test-model-6c153e-kserve-5686bd4669-5s2j5" Jul 06 22:57:54.447133 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:54.447095 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6c153e/sub-test-model-6c153e-kserve-5686bd4669-5s2j5" Jul 06 22:57:54.541809 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:54.541775 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:57:54.589126 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:54.589100 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-6c153e/sub-test-model-6c153e-kserve-5686bd4669-5s2j5"] Jul 06 22:57:54.591400 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:57:54.591367 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod274507cf_cf73_4529_aec7_a455b1c5edab.slice/crio-8df9750c3b17d49f7b87b33fd6c7776143cc7de94038baec2f147b282f9d8051 WatchSource:0}: Error finding container 8df9750c3b17d49f7b87b33fd6c7776143cc7de94038baec2f147b282f9d8051: Status 404 returned error can't find the container with id 8df9750c3b17d49f7b87b33fd6c7776143cc7de94038baec2f147b282f9d8051 Jul 06 22:57:54.641321 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:54.641291 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:57:55.391652 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:55.391615 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6c153e/sub-test-model-6c153e-kserve-5686bd4669-5s2j5" event={"ID":"274507cf-cf73-4529-aec7-a455b1c5edab","Type":"ContainerStarted","Data":"d6867d86a6c281bcf5c0318a91aa901866bcbe2c0db376245fad86cbd9bd2c4c"} Jul 06 22:57:55.391652 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:55.391657 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6c153e/sub-test-model-6c153e-kserve-5686bd4669-5s2j5" event={"ID":"274507cf-cf73-4529-aec7-a455b1c5edab","Type":"ContainerStarted","Data":"8df9750c3b17d49f7b87b33fd6c7776143cc7de94038baec2f147b282f9d8051"} Jul 06 22:57:55.408765 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:57:55.408713 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-6c153e/sub-test-model-6c153e-kserve-5686bd4669-5s2j5" podStartSLOduration=1.408696081 podStartE2EDuration="1.408696081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:57:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:57:55.407298933 +0000 UTC m=+2282.325541791" watchObservedRunningTime="2026-07-06 22:57:55.408696081 +0000 UTC m=+2282.326938937" Jul 06 22:58:04.447618 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:58:04.447581 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-6c153e/sub-test-model-6c153e-kserve-5686bd4669-5s2j5" Jul 06 22:58:04.448007 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:58:04.447653 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-6c153e/sub-test-model-6c153e-kserve-5686bd4669-5s2j5" Jul 06 22:58:04.462298 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:58:04.462271 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-6c153e/sub-test-model-6c153e-kserve-5686bd4669-5s2j5" Jul 06 22:58:05.453258 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:58:05.453224 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-6c153e/sub-test-model-6c153e-kserve-5686bd4669-5s2j5" Jul 06 22:58:51.316456 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:58:51.316371 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:58:52.139950 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:58:52.139914 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-6c153e/sub-test-model-6c153e-kserve-5686bd4669-5s2j5"] Jul 06 22:58:52.489486 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:58:52.489397 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:58:58.064924 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:58:58.064873 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:59:07.161095 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:07.161029 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-6c153e/sub-test-model-6c153e-kserve-5686bd4669-5s2j5" podUID="274507cf-cf73-4529-aec7-a455b1c5edab" containerName="main" containerID="cri-o://d6867d86a6c281bcf5c0318a91aa901866bcbe2c0db376245fad86cbd9bd2c4c" gracePeriod=45 Jul 06 22:59:07.420360 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:07.420292 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6c153e/sub-test-model-6c153e-kserve-5686bd4669-5s2j5" Jul 06 22:59:07.549762 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:07.549720 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/274507cf-cf73-4529-aec7-a455b1c5edab-tmp-dir\") pod \"274507cf-cf73-4529-aec7-a455b1c5edab\" (UID: \"274507cf-cf73-4529-aec7-a455b1c5edab\") " Jul 06 22:59:07.549762 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:07.549760 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8f6j\" (UniqueName: \"kubernetes.io/projected/274507cf-cf73-4529-aec7-a455b1c5edab-kube-api-access-m8f6j\") pod \"274507cf-cf73-4529-aec7-a455b1c5edab\" (UID: \"274507cf-cf73-4529-aec7-a455b1c5edab\") " Jul 06 22:59:07.549982 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:07.549794 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/274507cf-cf73-4529-aec7-a455b1c5edab-tls-certs\") pod \"274507cf-cf73-4529-aec7-a455b1c5edab\" (UID: \"274507cf-cf73-4529-aec7-a455b1c5edab\") " Jul 06 22:59:07.550037 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:07.549982 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/274507cf-cf73-4529-aec7-a455b1c5edab-model-cache\") pod \"274507cf-cf73-4529-aec7-a455b1c5edab\" (UID: \"274507cf-cf73-4529-aec7-a455b1c5edab\") " Jul 06 22:59:07.550093 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:07.550028 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/274507cf-cf73-4529-aec7-a455b1c5edab-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "274507cf-cf73-4529-aec7-a455b1c5edab" (UID: "274507cf-cf73-4529-aec7-a455b1c5edab"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:59:07.550093 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:07.550081 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/274507cf-cf73-4529-aec7-a455b1c5edab-dshm\") pod \"274507cf-cf73-4529-aec7-a455b1c5edab\" (UID: \"274507cf-cf73-4529-aec7-a455b1c5edab\") " Jul 06 22:59:07.550232 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:07.550132 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/274507cf-cf73-4529-aec7-a455b1c5edab-home\") pod \"274507cf-cf73-4529-aec7-a455b1c5edab\" (UID: \"274507cf-cf73-4529-aec7-a455b1c5edab\") " Jul 06 22:59:07.550299 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:07.550226 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/274507cf-cf73-4529-aec7-a455b1c5edab-model-cache" (OuterVolumeSpecName: "model-cache") pod "274507cf-cf73-4529-aec7-a455b1c5edab" (UID: "274507cf-cf73-4529-aec7-a455b1c5edab"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:59:07.550438 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:07.550418 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/274507cf-cf73-4529-aec7-a455b1c5edab-home" (OuterVolumeSpecName: "home") pod "274507cf-cf73-4529-aec7-a455b1c5edab" (UID: "274507cf-cf73-4529-aec7-a455b1c5edab"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:59:07.550501 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:07.550443 2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/274507cf-cf73-4529-aec7-a455b1c5edab-model-cache\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:59:07.550501 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:07.550488 2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/274507cf-cf73-4529-aec7-a455b1c5edab-tmp-dir\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:59:07.552048 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:07.552027 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/274507cf-cf73-4529-aec7-a455b1c5edab-kube-api-access-m8f6j" (OuterVolumeSpecName: "kube-api-access-m8f6j") pod "274507cf-cf73-4529-aec7-a455b1c5edab" (UID: "274507cf-cf73-4529-aec7-a455b1c5edab"). InnerVolumeSpecName "kube-api-access-m8f6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:59:07.552119 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:07.552040 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/274507cf-cf73-4529-aec7-a455b1c5edab-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "274507cf-cf73-4529-aec7-a455b1c5edab" (UID: "274507cf-cf73-4529-aec7-a455b1c5edab"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:59:07.552119 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:07.552048 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/274507cf-cf73-4529-aec7-a455b1c5edab-dshm" (OuterVolumeSpecName: "dshm") pod "274507cf-cf73-4529-aec7-a455b1c5edab" (UID: "274507cf-cf73-4529-aec7-a455b1c5edab"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:59:07.651836 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:07.651800 2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/274507cf-cf73-4529-aec7-a455b1c5edab-home\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:59:07.651836 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:07.651831 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8f6j\" (UniqueName: \"kubernetes.io/projected/274507cf-cf73-4529-aec7-a455b1c5edab-kube-api-access-m8f6j\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:59:07.651836 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:07.651842 2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/274507cf-cf73-4529-aec7-a455b1c5edab-tls-certs\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:59:07.652074 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:07.651851 2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/274507cf-cf73-4529-aec7-a455b1c5edab-dshm\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 22:59:07.745475 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:07.745380 2571 generic.go:356] "Generic (PLEG): container finished" podID="274507cf-cf73-4529-aec7-a455b1c5edab" containerID="d6867d86a6c281bcf5c0318a91aa901866bcbe2c0db376245fad86cbd9bd2c4c" exitCode=0 Jul 06 22:59:07.745475 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:07.745423 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6c153e/sub-test-model-6c153e-kserve-5686bd4669-5s2j5" Jul 06 22:59:07.745475 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:07.745463 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6c153e/sub-test-model-6c153e-kserve-5686bd4669-5s2j5" event={"ID":"274507cf-cf73-4529-aec7-a455b1c5edab","Type":"ContainerDied","Data":"d6867d86a6c281bcf5c0318a91aa901866bcbe2c0db376245fad86cbd9bd2c4c"} Jul 06 22:59:07.745791 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:07.745510 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6c153e/sub-test-model-6c153e-kserve-5686bd4669-5s2j5" event={"ID":"274507cf-cf73-4529-aec7-a455b1c5edab","Type":"ContainerDied","Data":"8df9750c3b17d49f7b87b33fd6c7776143cc7de94038baec2f147b282f9d8051"} Jul 06 22:59:07.745791 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:07.745527 2571 scope.go:117] "RemoveContainer" containerID="d6867d86a6c281bcf5c0318a91aa901866bcbe2c0db376245fad86cbd9bd2c4c" Jul 06 22:59:07.754312 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:07.754292 2571 scope.go:117] "RemoveContainer" containerID="d6867d86a6c281bcf5c0318a91aa901866bcbe2c0db376245fad86cbd9bd2c4c" Jul 06 22:59:07.754611 ip-10-0-137-147 kubenswrapper[2571]: E0706 22:59:07.754590 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6867d86a6c281bcf5c0318a91aa901866bcbe2c0db376245fad86cbd9bd2c4c\": container with ID starting with d6867d86a6c281bcf5c0318a91aa901866bcbe2c0db376245fad86cbd9bd2c4c not found: ID does not exist" containerID="d6867d86a6c281bcf5c0318a91aa901866bcbe2c0db376245fad86cbd9bd2c4c" Jul 06 22:59:07.754686 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:07.754618 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6867d86a6c281bcf5c0318a91aa901866bcbe2c0db376245fad86cbd9bd2c4c"} err="failed to get container status \"d6867d86a6c281bcf5c0318a91aa901866bcbe2c0db376245fad86cbd9bd2c4c\": rpc error: code = NotFound desc = could not find container \"d6867d86a6c281bcf5c0318a91aa901866bcbe2c0db376245fad86cbd9bd2c4c\": container with ID starting with d6867d86a6c281bcf5c0318a91aa901866bcbe2c0db376245fad86cbd9bd2c4c not found: ID does not exist" Jul 06 22:59:07.765381 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:07.765353 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-6c153e/sub-test-model-6c153e-kserve-5686bd4669-5s2j5"] Jul 06 22:59:07.769983 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:07.769961 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-6c153e/sub-test-model-6c153e-kserve-5686bd4669-5s2j5"] Jul 06 22:59:09.639407 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:09.636363 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="274507cf-cf73-4529-aec7-a455b1c5edab" path="/var/lib/kubelet/pods/274507cf-cf73-4529-aec7-a455b1c5edab/volumes" Jul 06 22:59:18.547224 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:18.547172 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:59:18.648167 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:18.648128 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:59:24.524624 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:24.524568 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-6c153e/rate-test-model-6c153e-kserve-577984dd7f-744pm"] Jul 06 22:59:24.525393 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:24.525371 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="274507cf-cf73-4529-aec7-a455b1c5edab" containerName="main" Jul 06 22:59:24.525393 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:24.525394 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="274507cf-cf73-4529-aec7-a455b1c5edab" containerName="main" Jul 06 22:59:24.525548 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:24.525494 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="274507cf-cf73-4529-aec7-a455b1c5edab" containerName="main" Jul 06 22:59:24.529972 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:24.529949 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6c153e/rate-test-model-6c153e-kserve-577984dd7f-744pm" Jul 06 22:59:24.533401 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:24.533378 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-6c153e\"/\"rate-test-model-6c153e-kserve-self-signed-certs\"" Jul 06 22:59:24.533678 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:24.533426 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-6c153e\"/\"openshift-service-ca.crt\"" Jul 06 22:59:24.533678 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:24.533439 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-6c153e\"/\"default-dockercfg-nsqkg\"" Jul 06 22:59:24.533678 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:24.533467 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-6c153e\"/\"kube-root-ca.crt\"" Jul 06 22:59:24.540792 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:24.540765 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-6c153e/rate-test-model-6c153e-kserve-577984dd7f-744pm"] Jul 06 22:59:24.550346 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:24.550323 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:59:24.705569 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:24.705529 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/937eb1db-82f0-45e9-8d0e-e6bddfce47cb-tmp-dir\") pod \"rate-test-model-6c153e-kserve-577984dd7f-744pm\" (UID: \"937eb1db-82f0-45e9-8d0e-e6bddfce47cb\") " pod="ai-tenant-e2e-shared-b-6c153e/rate-test-model-6c153e-kserve-577984dd7f-744pm" Jul 06 22:59:24.705569 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:24.705578 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/937eb1db-82f0-45e9-8d0e-e6bddfce47cb-home\") pod \"rate-test-model-6c153e-kserve-577984dd7f-744pm\" (UID: \"937eb1db-82f0-45e9-8d0e-e6bddfce47cb\") " pod="ai-tenant-e2e-shared-b-6c153e/rate-test-model-6c153e-kserve-577984dd7f-744pm" Jul 06 22:59:24.705808 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:24.705598 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/937eb1db-82f0-45e9-8d0e-e6bddfce47cb-dshm\") pod \"rate-test-model-6c153e-kserve-577984dd7f-744pm\" (UID: \"937eb1db-82f0-45e9-8d0e-e6bddfce47cb\") " pod="ai-tenant-e2e-shared-b-6c153e/rate-test-model-6c153e-kserve-577984dd7f-744pm" Jul 06 22:59:24.705808 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:24.705676 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f66st\" (UniqueName: \"kubernetes.io/projected/937eb1db-82f0-45e9-8d0e-e6bddfce47cb-kube-api-access-f66st\") pod \"rate-test-model-6c153e-kserve-577984dd7f-744pm\" (UID: \"937eb1db-82f0-45e9-8d0e-e6bddfce47cb\") " pod="ai-tenant-e2e-shared-b-6c153e/rate-test-model-6c153e-kserve-577984dd7f-744pm" Jul 06 22:59:24.705808 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:24.705742 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/937eb1db-82f0-45e9-8d0e-e6bddfce47cb-tls-certs\") pod \"rate-test-model-6c153e-kserve-577984dd7f-744pm\" (UID: \"937eb1db-82f0-45e9-8d0e-e6bddfce47cb\") " pod="ai-tenant-e2e-shared-b-6c153e/rate-test-model-6c153e-kserve-577984dd7f-744pm" Jul 06 22:59:24.705808 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:24.705764 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/937eb1db-82f0-45e9-8d0e-e6bddfce47cb-model-cache\") pod \"rate-test-model-6c153e-kserve-577984dd7f-744pm\" (UID: \"937eb1db-82f0-45e9-8d0e-e6bddfce47cb\") " pod="ai-tenant-e2e-shared-b-6c153e/rate-test-model-6c153e-kserve-577984dd7f-744pm" Jul 06 22:59:24.807248 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:24.807124 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/937eb1db-82f0-45e9-8d0e-e6bddfce47cb-tls-certs\") pod \"rate-test-model-6c153e-kserve-577984dd7f-744pm\" (UID: \"937eb1db-82f0-45e9-8d0e-e6bddfce47cb\") " pod="ai-tenant-e2e-shared-b-6c153e/rate-test-model-6c153e-kserve-577984dd7f-744pm" Jul 06 22:59:24.807248 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:24.807176 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/937eb1db-82f0-45e9-8d0e-e6bddfce47cb-model-cache\") pod \"rate-test-model-6c153e-kserve-577984dd7f-744pm\" (UID: \"937eb1db-82f0-45e9-8d0e-e6bddfce47cb\") " pod="ai-tenant-e2e-shared-b-6c153e/rate-test-model-6c153e-kserve-577984dd7f-744pm" Jul 06 22:59:24.807462 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:24.807304 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/937eb1db-82f0-45e9-8d0e-e6bddfce47cb-tmp-dir\") pod \"rate-test-model-6c153e-kserve-577984dd7f-744pm\" (UID: \"937eb1db-82f0-45e9-8d0e-e6bddfce47cb\") " pod="ai-tenant-e2e-shared-b-6c153e/rate-test-model-6c153e-kserve-577984dd7f-744pm" Jul 06 22:59:24.807462 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:24.807351 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/937eb1db-82f0-45e9-8d0e-e6bddfce47cb-home\") pod \"rate-test-model-6c153e-kserve-577984dd7f-744pm\" (UID: \"937eb1db-82f0-45e9-8d0e-e6bddfce47cb\") " pod="ai-tenant-e2e-shared-b-6c153e/rate-test-model-6c153e-kserve-577984dd7f-744pm" Jul 06 22:59:24.807462 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:24.807368 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/937eb1db-82f0-45e9-8d0e-e6bddfce47cb-dshm\") pod \"rate-test-model-6c153e-kserve-577984dd7f-744pm\" (UID: \"937eb1db-82f0-45e9-8d0e-e6bddfce47cb\") " pod="ai-tenant-e2e-shared-b-6c153e/rate-test-model-6c153e-kserve-577984dd7f-744pm" Jul 06 22:59:24.807462 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:24.807414 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f66st\" (UniqueName: \"kubernetes.io/projected/937eb1db-82f0-45e9-8d0e-e6bddfce47cb-kube-api-access-f66st\") pod \"rate-test-model-6c153e-kserve-577984dd7f-744pm\" (UID: \"937eb1db-82f0-45e9-8d0e-e6bddfce47cb\") " pod="ai-tenant-e2e-shared-b-6c153e/rate-test-model-6c153e-kserve-577984dd7f-744pm" Jul 06 22:59:24.807778 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:24.807755 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/937eb1db-82f0-45e9-8d0e-e6bddfce47cb-home\") pod \"rate-test-model-6c153e-kserve-577984dd7f-744pm\" (UID: \"937eb1db-82f0-45e9-8d0e-e6bddfce47cb\") " pod="ai-tenant-e2e-shared-b-6c153e/rate-test-model-6c153e-kserve-577984dd7f-744pm" Jul 06 22:59:24.807778 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:24.807769 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/937eb1db-82f0-45e9-8d0e-e6bddfce47cb-tmp-dir\") pod \"rate-test-model-6c153e-kserve-577984dd7f-744pm\" (UID: \"937eb1db-82f0-45e9-8d0e-e6bddfce47cb\") " pod="ai-tenant-e2e-shared-b-6c153e/rate-test-model-6c153e-kserve-577984dd7f-744pm" Jul 06 22:59:24.807883 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:24.807846 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/937eb1db-82f0-45e9-8d0e-e6bddfce47cb-model-cache\") pod \"rate-test-model-6c153e-kserve-577984dd7f-744pm\" (UID: \"937eb1db-82f0-45e9-8d0e-e6bddfce47cb\") " pod="ai-tenant-e2e-shared-b-6c153e/rate-test-model-6c153e-kserve-577984dd7f-744pm" Jul 06 22:59:24.809621 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:24.809597 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/937eb1db-82f0-45e9-8d0e-e6bddfce47cb-dshm\") pod \"rate-test-model-6c153e-kserve-577984dd7f-744pm\" (UID: \"937eb1db-82f0-45e9-8d0e-e6bddfce47cb\") " pod="ai-tenant-e2e-shared-b-6c153e/rate-test-model-6c153e-kserve-577984dd7f-744pm" Jul 06 22:59:24.809786 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:24.809769 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/937eb1db-82f0-45e9-8d0e-e6bddfce47cb-tls-certs\") pod \"rate-test-model-6c153e-kserve-577984dd7f-744pm\" (UID: \"937eb1db-82f0-45e9-8d0e-e6bddfce47cb\") " pod="ai-tenant-e2e-shared-b-6c153e/rate-test-model-6c153e-kserve-577984dd7f-744pm" Jul 06 22:59:24.816982 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:24.816946 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f66st\" (UniqueName: \"kubernetes.io/projected/937eb1db-82f0-45e9-8d0e-e6bddfce47cb-kube-api-access-f66st\") pod \"rate-test-model-6c153e-kserve-577984dd7f-744pm\" (UID: \"937eb1db-82f0-45e9-8d0e-e6bddfce47cb\") " pod="ai-tenant-e2e-shared-b-6c153e/rate-test-model-6c153e-kserve-577984dd7f-744pm" Jul 06 22:59:24.841775 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:24.841733 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6c153e/rate-test-model-6c153e-kserve-577984dd7f-744pm" Jul 06 22:59:24.978469 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:24.978298 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-6c153e/rate-test-model-6c153e-kserve-577984dd7f-744pm"] Jul 06 22:59:24.980804 ip-10-0-137-147 kubenswrapper[2571]: W0706 22:59:24.980769 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod937eb1db_82f0_45e9_8d0e_e6bddfce47cb.slice/crio-9e8630d7bc765db99f16248a45e3498bdb467e3104e4e1deab8e0995b4fcb4ad WatchSource:0}: Error finding container 9e8630d7bc765db99f16248a45e3498bdb467e3104e4e1deab8e0995b4fcb4ad: Status 404 returned error can't find the container with id 9e8630d7bc765db99f16248a45e3498bdb467e3104e4e1deab8e0995b4fcb4ad Jul 06 22:59:25.830174 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:25.830138 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6c153e/rate-test-model-6c153e-kserve-577984dd7f-744pm" event={"ID":"937eb1db-82f0-45e9-8d0e-e6bddfce47cb","Type":"ContainerStarted","Data":"3a8e58538ec1a149847c33a92516a25751a0b6b39613da54918ab081233ca936"} Jul 06 22:59:25.830614 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:25.830184 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6c153e/rate-test-model-6c153e-kserve-577984dd7f-744pm" event={"ID":"937eb1db-82f0-45e9-8d0e-e6bddfce47cb","Type":"ContainerStarted","Data":"9e8630d7bc765db99f16248a45e3498bdb467e3104e4e1deab8e0995b4fcb4ad"} Jul 06 22:59:25.850930 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:25.850865 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-6c153e/rate-test-model-6c153e-kserve-577984dd7f-744pm" podStartSLOduration=1.8508420810000001 podStartE2EDuration="1.850842081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:59:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:59:25.850165079 +0000 UTC m=+2372.768407936" watchObservedRunningTime="2026-07-06 22:59:25.850842081 +0000 UTC m=+2372.769084938" Jul 06 22:59:26.888734 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:26.888692 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:59:26.988960 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:26.988910 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:59:34.842597 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:34.842549 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-6c153e/rate-test-model-6c153e-kserve-577984dd7f-744pm" Jul 06 22:59:34.843174 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:34.842615 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-6c153e/rate-test-model-6c153e-kserve-577984dd7f-744pm" Jul 06 22:59:34.855310 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:34.855278 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-6c153e/rate-test-model-6c153e-kserve-577984dd7f-744pm" Jul 06 22:59:34.886163 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:34.886132 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-6c153e/rate-test-model-6c153e-kserve-577984dd7f-744pm" Jul 06 22:59:41.310613 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:41.310556 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:59:41.410687 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:41.410651 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:59:53.717502 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:53.717476 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-4x6pf_5796a8ea-adcb-4819-bb18-0b85e1ef4681/console-operator/2.log" Jul 06 22:59:53.720287 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:53.720262 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-4x6pf_5796a8ea-adcb-4819-bb18-0b85e1ef4681/console-operator/2.log" Jul 06 22:59:53.720989 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:53.720970 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qbbrs_bf23ef00-a2d0-499c-8c87-5d6b98510ea0/ovn-acl-logging/0.log" Jul 06 22:59:53.723619 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:53.723599 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qbbrs_bf23ef00-a2d0-499c-8c87-5d6b98510ea0/ovn-acl-logging/0.log" Jul 06 22:59:54.521356 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:54.521321 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:59:54.622298 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:54.622264 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 22:59:58.833771 ip-10-0-137-147 kubenswrapper[2571]: I0706 22:59:58.833730 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 23:00:00.138228 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:00.138175 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-2ef6a3-29722980-cql8k"] Jul 06 23:00:00.141791 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:00.141769 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-2ef6a3-29722980-cql8k" Jul 06 23:00:00.151321 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:00.151290 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-2ef6a3-29722980-cql8k"] Jul 06 23:00:00.219652 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:00.219618 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2grpf\" (UniqueName: \"kubernetes.io/projected/34964a9e-d002-4638-a9d4-5f7c0b7af86e-kube-api-access-2grpf\") pod \"maas-api-key-cleanup-e2e-shared-a-2ef6a3-29722980-cql8k\" (UID: \"34964a9e-d002-4638-a9d4-5f7c0b7af86e\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-2ef6a3-29722980-cql8k" Jul 06 23:00:00.320157 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:00.320113 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2grpf\" (UniqueName: \"kubernetes.io/projected/34964a9e-d002-4638-a9d4-5f7c0b7af86e-kube-api-access-2grpf\") pod \"maas-api-key-cleanup-e2e-shared-a-2ef6a3-29722980-cql8k\" (UID: \"34964a9e-d002-4638-a9d4-5f7c0b7af86e\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-2ef6a3-29722980-cql8k" Jul 06 23:00:00.329149 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:00.329119 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2grpf\" (UniqueName: \"kubernetes.io/projected/34964a9e-d002-4638-a9d4-5f7c0b7af86e-kube-api-access-2grpf\") pod \"maas-api-key-cleanup-e2e-shared-a-2ef6a3-29722980-cql8k\" (UID: \"34964a9e-d002-4638-a9d4-5f7c0b7af86e\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-2ef6a3-29722980-cql8k" Jul 06 23:00:00.453453 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:00.453346 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-2ef6a3-29722980-cql8k" Jul 06 23:00:00.595986 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:00.595960 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-2ef6a3-29722980-cql8k"] Jul 06 23:00:00.598874 ip-10-0-137-147 kubenswrapper[2571]: W0706 23:00:00.598840 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34964a9e_d002_4638_a9d4_5f7c0b7af86e.slice/crio-0311848e8a9b3f14e365ae6608beb9e14788634addd55b69713b1264d7236a7e WatchSource:0}: Error finding container 0311848e8a9b3f14e365ae6608beb9e14788634addd55b69713b1264d7236a7e: Status 404 returned error can't find the container with id 0311848e8a9b3f14e365ae6608beb9e14788634addd55b69713b1264d7236a7e Jul 06 23:00:01.003850 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:01.003818 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-2ef6a3-29722980-cql8k" event={"ID":"34964a9e-d002-4638-a9d4-5f7c0b7af86e","Type":"ContainerStarted","Data":"cf1c423c9267adebea44eb5b6efbbcc7f0616d110f8a8c0929b56327f22dbdb8"} Jul 06 23:00:01.004033 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:01.003873 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-2ef6a3-29722980-cql8k" event={"ID":"34964a9e-d002-4638-a9d4-5f7c0b7af86e","Type":"ContainerStarted","Data":"0311848e8a9b3f14e365ae6608beb9e14788634addd55b69713b1264d7236a7e"} Jul 06 23:00:01.020652 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:01.020590 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-2ef6a3-29722980-cql8k" podStartSLOduration=1.020573131 podStartE2EDuration="1.020573131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:00:01.019862574 +0000 UTC m=+2407.938105432" watchObservedRunningTime="2026-07-06 23:00:01.020573131 +0000 UTC m=+2407.938815988" Jul 06 23:00:02.010551 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:02.010518 2571 generic.go:356] "Generic (PLEG): container finished" podID="34964a9e-d002-4638-a9d4-5f7c0b7af86e" containerID="cf1c423c9267adebea44eb5b6efbbcc7f0616d110f8a8c0929b56327f22dbdb8" exitCode=0 Jul 06 23:00:02.010942 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:02.010586 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-2ef6a3-29722980-cql8k" event={"ID":"34964a9e-d002-4638-a9d4-5f7c0b7af86e","Type":"ContainerDied","Data":"cf1c423c9267adebea44eb5b6efbbcc7f0616d110f8a8c0929b56327f22dbdb8"} Jul 06 23:00:03.150435 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:03.150401 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-2ef6a3-29722980-cql8k" Jul 06 23:00:03.247623 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:03.247587 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2grpf\" (UniqueName: \"kubernetes.io/projected/34964a9e-d002-4638-a9d4-5f7c0b7af86e-kube-api-access-2grpf\") pod \"34964a9e-d002-4638-a9d4-5f7c0b7af86e\" (UID: \"34964a9e-d002-4638-a9d4-5f7c0b7af86e\") " Jul 06 23:00:03.249784 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:03.249756 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34964a9e-d002-4638-a9d4-5f7c0b7af86e-kube-api-access-2grpf" (OuterVolumeSpecName: "kube-api-access-2grpf") pod "34964a9e-d002-4638-a9d4-5f7c0b7af86e" (UID: "34964a9e-d002-4638-a9d4-5f7c0b7af86e"). InnerVolumeSpecName "kube-api-access-2grpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:00:03.349270 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:03.349173 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2grpf\" (UniqueName: \"kubernetes.io/projected/34964a9e-d002-4638-a9d4-5f7c0b7af86e-kube-api-access-2grpf\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 23:00:04.021830 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:04.021749 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-2ef6a3-29722980-cql8k" Jul 06 23:00:04.021830 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:04.021779 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-2ef6a3-29722980-cql8k" event={"ID":"34964a9e-d002-4638-a9d4-5f7c0b7af86e","Type":"ContainerDied","Data":"0311848e8a9b3f14e365ae6608beb9e14788634addd55b69713b1264d7236a7e"} Jul 06 23:00:04.022021 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:04.021845 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0311848e8a9b3f14e365ae6608beb9e14788634addd55b69713b1264d7236a7e" Jul 06 23:00:08.421555 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:08.421497 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 23:00:09.244583 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:09.244542 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-6c153e/rate-test-model-6c153e-kserve-577984dd7f-744pm"] Jul 06 23:00:09.255858 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:09.255826 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 23:00:10.290186 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:10.290152 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 23:00:10.669960 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:10.669918 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 23:00:10.770449 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:10.770407 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 23:00:10.869985 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:10.869953 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 23:00:11.067700 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:11.067611 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-2ef6a3-66dbb6d8c5-6v2s7"] Jul 06 23:00:11.067931 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:11.067902 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-a-2ef6a3-66dbb6d8c5-6v2s7" podUID="aa9fe430-ff17-4440-9ba6-1c5582cd5344" containerName="maas-api" containerID="cri-o://4488956c2caf76beed9b08a9890c668f6f184a8bd250138bc5c13e329083bd5f" gracePeriod=30 Jul 06 23:00:11.097269 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:11.097232 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-2ef6a3-29722980-cql8k"] Jul 06 23:00:11.099714 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:11.099687 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-2ef6a3-29722980-cql8k"] Jul 06 23:00:11.102839 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:11.102817 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 23:00:11.518395 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:11.518368 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-2ef6a3-66dbb6d8c5-6v2s7" Jul 06 23:00:11.626330 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:11.626199 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa9fe430-ff17-4440-9ba6-1c5582cd5344-openshift-service-ca\") pod \"aa9fe430-ff17-4440-9ba6-1c5582cd5344\" (UID: \"aa9fe430-ff17-4440-9ba6-1c5582cd5344\") " Jul 06 23:00:11.626330 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:11.626277 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa9fe430-ff17-4440-9ba6-1c5582cd5344-maas-api-tls\") pod \"aa9fe430-ff17-4440-9ba6-1c5582cd5344\" (UID: \"aa9fe430-ff17-4440-9ba6-1c5582cd5344\") " Jul 06 23:00:11.626330 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:11.626324 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9m6hs\" (UniqueName: \"kubernetes.io/projected/aa9fe430-ff17-4440-9ba6-1c5582cd5344-kube-api-access-9m6hs\") pod \"aa9fe430-ff17-4440-9ba6-1c5582cd5344\" (UID: \"aa9fe430-ff17-4440-9ba6-1c5582cd5344\") " Jul 06 23:00:11.626673 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:11.626629 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa9fe430-ff17-4440-9ba6-1c5582cd5344-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "aa9fe430-ff17-4440-9ba6-1c5582cd5344" (UID: "aa9fe430-ff17-4440-9ba6-1c5582cd5344"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:00:11.628472 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:11.628444 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa9fe430-ff17-4440-9ba6-1c5582cd5344-kube-api-access-9m6hs" (OuterVolumeSpecName: "kube-api-access-9m6hs") pod "aa9fe430-ff17-4440-9ba6-1c5582cd5344" (UID: "aa9fe430-ff17-4440-9ba6-1c5582cd5344"). InnerVolumeSpecName "kube-api-access-9m6hs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:00:11.628599 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:11.628497 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa9fe430-ff17-4440-9ba6-1c5582cd5344-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "aa9fe430-ff17-4440-9ba6-1c5582cd5344" (UID: "aa9fe430-ff17-4440-9ba6-1c5582cd5344"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:00:11.635995 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:11.635963 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34964a9e-d002-4638-a9d4-5f7c0b7af86e" path="/var/lib/kubelet/pods/34964a9e-d002-4638-a9d4-5f7c0b7af86e/volumes" Jul 06 23:00:11.727868 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:11.727828 2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa9fe430-ff17-4440-9ba6-1c5582cd5344-openshift-service-ca\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 23:00:11.727868 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:11.727862 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa9fe430-ff17-4440-9ba6-1c5582cd5344-maas-api-tls\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 23:00:11.727868 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:11.727871 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9m6hs\" (UniqueName: \"kubernetes.io/projected/aa9fe430-ff17-4440-9ba6-1c5582cd5344-kube-api-access-9m6hs\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 23:00:12.061632 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:12.061597 2571 generic.go:356] "Generic (PLEG): container finished" podID="aa9fe430-ff17-4440-9ba6-1c5582cd5344" containerID="4488956c2caf76beed9b08a9890c668f6f184a8bd250138bc5c13e329083bd5f" exitCode=0 Jul 06 23:00:12.061807 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:12.061640 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-2ef6a3-66dbb6d8c5-6v2s7" event={"ID":"aa9fe430-ff17-4440-9ba6-1c5582cd5344","Type":"ContainerDied","Data":"4488956c2caf76beed9b08a9890c668f6f184a8bd250138bc5c13e329083bd5f"} Jul 06 23:00:12.061807 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:12.061648 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-2ef6a3-66dbb6d8c5-6v2s7" Jul 06 23:00:12.061807 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:12.061680 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-2ef6a3-66dbb6d8c5-6v2s7" event={"ID":"aa9fe430-ff17-4440-9ba6-1c5582cd5344","Type":"ContainerDied","Data":"55c3d457d942f1221197c77880dfa72f2afc2d16a61f836a7971236abf222a48"} Jul 06 23:00:12.061807 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:12.061701 2571 scope.go:117] "RemoveContainer" containerID="4488956c2caf76beed9b08a9890c668f6f184a8bd250138bc5c13e329083bd5f" Jul 06 23:00:12.070672 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:12.070505 2571 scope.go:117] "RemoveContainer" containerID="4488956c2caf76beed9b08a9890c668f6f184a8bd250138bc5c13e329083bd5f" Jul 06 23:00:12.070838 ip-10-0-137-147 kubenswrapper[2571]: E0706 23:00:12.070814 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4488956c2caf76beed9b08a9890c668f6f184a8bd250138bc5c13e329083bd5f\": container with ID starting with 4488956c2caf76beed9b08a9890c668f6f184a8bd250138bc5c13e329083bd5f not found: ID does not exist" containerID="4488956c2caf76beed9b08a9890c668f6f184a8bd250138bc5c13e329083bd5f" Jul 06 23:00:12.070883 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:12.070851 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4488956c2caf76beed9b08a9890c668f6f184a8bd250138bc5c13e329083bd5f"} err="failed to get container status \"4488956c2caf76beed9b08a9890c668f6f184a8bd250138bc5c13e329083bd5f\": rpc error: code = NotFound desc = could not find container \"4488956c2caf76beed9b08a9890c668f6f184a8bd250138bc5c13e329083bd5f\": container with ID starting with 4488956c2caf76beed9b08a9890c668f6f184a8bd250138bc5c13e329083bd5f not found: ID does not exist" Jul 06 23:00:12.081034 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:12.081001 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-2ef6a3-66dbb6d8c5-6v2s7"] Jul 06 23:00:12.085773 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:12.085745 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-2ef6a3-66dbb6d8c5-6v2s7"] Jul 06 23:00:13.635941 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:13.635907 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa9fe430-ff17-4440-9ba6-1c5582cd5344" path="/var/lib/kubelet/pods/aa9fe430-ff17-4440-9ba6-1c5582cd5344/volumes" Jul 06 23:00:15.749900 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:15.749819 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 23:00:24.265781 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:24.265733 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-6c153e/rate-test-model-6c153e-kserve-577984dd7f-744pm" podUID="937eb1db-82f0-45e9-8d0e-e6bddfce47cb" containerName="main" containerID="cri-o://3a8e58538ec1a149847c33a92516a25751a0b6b39613da54918ab081233ca936" gracePeriod=45 Jul 06 23:00:24.530845 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:24.530779 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6c153e/rate-test-model-6c153e-kserve-577984dd7f-744pm" Jul 06 23:00:24.546842 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:24.546804 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/937eb1db-82f0-45e9-8d0e-e6bddfce47cb-tls-certs\") pod \"937eb1db-82f0-45e9-8d0e-e6bddfce47cb\" (UID: \"937eb1db-82f0-45e9-8d0e-e6bddfce47cb\") " Jul 06 23:00:24.546991 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:24.546879 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/937eb1db-82f0-45e9-8d0e-e6bddfce47cb-dshm\") pod \"937eb1db-82f0-45e9-8d0e-e6bddfce47cb\" (UID: \"937eb1db-82f0-45e9-8d0e-e6bddfce47cb\") " Jul 06 23:00:24.546991 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:24.546908 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/937eb1db-82f0-45e9-8d0e-e6bddfce47cb-tmp-dir\") pod \"937eb1db-82f0-45e9-8d0e-e6bddfce47cb\" (UID: \"937eb1db-82f0-45e9-8d0e-e6bddfce47cb\") " Jul 06 23:00:24.546991 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:24.546972 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/937eb1db-82f0-45e9-8d0e-e6bddfce47cb-model-cache\") pod \"937eb1db-82f0-45e9-8d0e-e6bddfce47cb\" (UID: \"937eb1db-82f0-45e9-8d0e-e6bddfce47cb\") " Jul 06 23:00:24.547172 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:24.547025 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/937eb1db-82f0-45e9-8d0e-e6bddfce47cb-home\") pod \"937eb1db-82f0-45e9-8d0e-e6bddfce47cb\" (UID: \"937eb1db-82f0-45e9-8d0e-e6bddfce47cb\") " Jul 06 23:00:24.547172 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:24.547079 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f66st\" (UniqueName: \"kubernetes.io/projected/937eb1db-82f0-45e9-8d0e-e6bddfce47cb-kube-api-access-f66st\") pod \"937eb1db-82f0-45e9-8d0e-e6bddfce47cb\" (UID: \"937eb1db-82f0-45e9-8d0e-e6bddfce47cb\") " Jul 06 23:00:24.547308 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:24.547246 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/937eb1db-82f0-45e9-8d0e-e6bddfce47cb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "937eb1db-82f0-45e9-8d0e-e6bddfce47cb" (UID: "937eb1db-82f0-45e9-8d0e-e6bddfce47cb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:00:24.547364 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:24.547315 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/937eb1db-82f0-45e9-8d0e-e6bddfce47cb-model-cache" (OuterVolumeSpecName: "model-cache") pod "937eb1db-82f0-45e9-8d0e-e6bddfce47cb" (UID: "937eb1db-82f0-45e9-8d0e-e6bddfce47cb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:00:24.547423 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:24.547364 2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/937eb1db-82f0-45e9-8d0e-e6bddfce47cb-tmp-dir\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 23:00:24.547423 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:24.547389 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/937eb1db-82f0-45e9-8d0e-e6bddfce47cb-home" (OuterVolumeSpecName: "home") pod "937eb1db-82f0-45e9-8d0e-e6bddfce47cb" (UID: "937eb1db-82f0-45e9-8d0e-e6bddfce47cb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:00:24.549071 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:24.549040 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/937eb1db-82f0-45e9-8d0e-e6bddfce47cb-dshm" (OuterVolumeSpecName: "dshm") pod "937eb1db-82f0-45e9-8d0e-e6bddfce47cb" (UID: "937eb1db-82f0-45e9-8d0e-e6bddfce47cb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:00:24.549071 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:24.549051 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/937eb1db-82f0-45e9-8d0e-e6bddfce47cb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "937eb1db-82f0-45e9-8d0e-e6bddfce47cb" (UID: "937eb1db-82f0-45e9-8d0e-e6bddfce47cb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:00:24.549243 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:24.549123 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/937eb1db-82f0-45e9-8d0e-e6bddfce47cb-kube-api-access-f66st" (OuterVolumeSpecName: "kube-api-access-f66st") pod "937eb1db-82f0-45e9-8d0e-e6bddfce47cb" (UID: "937eb1db-82f0-45e9-8d0e-e6bddfce47cb"). InnerVolumeSpecName "kube-api-access-f66st". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:00:24.648383 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:24.648345 2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/937eb1db-82f0-45e9-8d0e-e6bddfce47cb-model-cache\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 23:00:24.648383 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:24.648377 2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/937eb1db-82f0-45e9-8d0e-e6bddfce47cb-home\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 23:00:24.648383 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:24.648386 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f66st\" (UniqueName: \"kubernetes.io/projected/937eb1db-82f0-45e9-8d0e-e6bddfce47cb-kube-api-access-f66st\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 23:00:24.648383 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:24.648394 2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/937eb1db-82f0-45e9-8d0e-e6bddfce47cb-tls-certs\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 23:00:24.648657 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:24.648402 2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/937eb1db-82f0-45e9-8d0e-e6bddfce47cb-dshm\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 23:00:25.127732 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:25.127696 2571 generic.go:356] "Generic (PLEG): container finished" podID="937eb1db-82f0-45e9-8d0e-e6bddfce47cb" containerID="3a8e58538ec1a149847c33a92516a25751a0b6b39613da54918ab081233ca936" exitCode=0 Jul 06 23:00:25.127732 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:25.127733 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6c153e/rate-test-model-6c153e-kserve-577984dd7f-744pm" Jul 06 23:00:25.127946 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:25.127774 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6c153e/rate-test-model-6c153e-kserve-577984dd7f-744pm" event={"ID":"937eb1db-82f0-45e9-8d0e-e6bddfce47cb","Type":"ContainerDied","Data":"3a8e58538ec1a149847c33a92516a25751a0b6b39613da54918ab081233ca936"} Jul 06 23:00:25.127946 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:25.127814 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6c153e/rate-test-model-6c153e-kserve-577984dd7f-744pm" event={"ID":"937eb1db-82f0-45e9-8d0e-e6bddfce47cb","Type":"ContainerDied","Data":"9e8630d7bc765db99f16248a45e3498bdb467e3104e4e1deab8e0995b4fcb4ad"} Jul 06 23:00:25.127946 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:25.127835 2571 scope.go:117] "RemoveContainer" containerID="3a8e58538ec1a149847c33a92516a25751a0b6b39613da54918ab081233ca936" Jul 06 23:00:25.140629 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:25.140528 2571 scope.go:117] "RemoveContainer" containerID="3a8e58538ec1a149847c33a92516a25751a0b6b39613da54918ab081233ca936" Jul 06 23:00:25.141073 ip-10-0-137-147 kubenswrapper[2571]: E0706 23:00:25.141044 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a8e58538ec1a149847c33a92516a25751a0b6b39613da54918ab081233ca936\": container with ID starting with 3a8e58538ec1a149847c33a92516a25751a0b6b39613da54918ab081233ca936 not found: ID does not exist" containerID="3a8e58538ec1a149847c33a92516a25751a0b6b39613da54918ab081233ca936" Jul 06 23:00:25.141185 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:25.141082 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a8e58538ec1a149847c33a92516a25751a0b6b39613da54918ab081233ca936"} err="failed to get container status \"3a8e58538ec1a149847c33a92516a25751a0b6b39613da54918ab081233ca936\": rpc error: code = NotFound desc = could not find container \"3a8e58538ec1a149847c33a92516a25751a0b6b39613da54918ab081233ca936\": container with ID starting with 3a8e58538ec1a149847c33a92516a25751a0b6b39613da54918ab081233ca936 not found: ID does not exist" Jul 06 23:00:25.152515 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:25.152486 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-6c153e/rate-test-model-6c153e-kserve-577984dd7f-744pm"] Jul 06 23:00:25.157515 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:25.157492 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-6c153e/rate-test-model-6c153e-kserve-577984dd7f-744pm"] Jul 06 23:00:25.635962 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:25.635923 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="937eb1db-82f0-45e9-8d0e-e6bddfce47cb" path="/var/lib/kubelet/pods/937eb1db-82f0-45e9-8d0e-e6bddfce47cb/volumes" Jul 06 23:00:53.018894 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:53.018841 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 23:00:53.304137 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:53.304052 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 23:00:53.518128 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:53.518096 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 23:00:53.699428 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:53.699390 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 23:00:54.098377 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:54.098338 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 23:00:54.213796 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:54.213747 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 23:00:54.315899 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:54.315783 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 23:00:54.438705 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:54.438669 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-6c153e-647b666d54-p74jz"] Jul 06 23:00:54.438963 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:54.438936 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-b-6c153e-647b666d54-p74jz" podUID="0d43d667-a519-43ed-915e-837b766f397a" containerName="maas-api" containerID="cri-o://426900e7687381481c95b56a702962d71b4d958aae33a091ce9662693dd2d425" gracePeriod=30 Jul 06 23:00:54.461038 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:54.461004 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 23:00:54.890620 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:54.890587 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-6c153e-647b666d54-p74jz" Jul 06 23:00:55.026548 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:55.026444 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d43d667-a519-43ed-915e-837b766f397a-openshift-service-ca\") pod \"0d43d667-a519-43ed-915e-837b766f397a\" (UID: \"0d43d667-a519-43ed-915e-837b766f397a\") " Jul 06 23:00:55.026548 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:55.026547 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d43d667-a519-43ed-915e-837b766f397a-maas-api-tls\") pod \"0d43d667-a519-43ed-915e-837b766f397a\" (UID: \"0d43d667-a519-43ed-915e-837b766f397a\") " Jul 06 23:00:55.026778 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:55.026598 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ntw6j\" (UniqueName: \"kubernetes.io/projected/0d43d667-a519-43ed-915e-837b766f397a-kube-api-access-ntw6j\") pod \"0d43d667-a519-43ed-915e-837b766f397a\" (UID: \"0d43d667-a519-43ed-915e-837b766f397a\") " Jul 06 23:00:55.026848 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:55.026812 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d43d667-a519-43ed-915e-837b766f397a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0d43d667-a519-43ed-915e-837b766f397a" (UID: "0d43d667-a519-43ed-915e-837b766f397a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:00:55.028711 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:55.028680 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d43d667-a519-43ed-915e-837b766f397a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0d43d667-a519-43ed-915e-837b766f397a" (UID: "0d43d667-a519-43ed-915e-837b766f397a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:00:55.028834 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:55.028738 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d43d667-a519-43ed-915e-837b766f397a-kube-api-access-ntw6j" (OuterVolumeSpecName: "kube-api-access-ntw6j") pod "0d43d667-a519-43ed-915e-837b766f397a" (UID: "0d43d667-a519-43ed-915e-837b766f397a"). InnerVolumeSpecName "kube-api-access-ntw6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:00:55.127367 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:55.127325 2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d43d667-a519-43ed-915e-837b766f397a-openshift-service-ca\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 23:00:55.127367 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:55.127362 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d43d667-a519-43ed-915e-837b766f397a-maas-api-tls\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 23:00:55.127367 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:55.127371 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntw6j\" (UniqueName: \"kubernetes.io/projected/0d43d667-a519-43ed-915e-837b766f397a-kube-api-access-ntw6j\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 06 23:00:55.270282 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:55.270243 2571 generic.go:356] "Generic (PLEG): container finished" podID="0d43d667-a519-43ed-915e-837b766f397a" containerID="426900e7687381481c95b56a702962d71b4d958aae33a091ce9662693dd2d425" exitCode=0 Jul 06 23:00:55.270495 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:55.270296 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-6c153e-647b666d54-p74jz" Jul 06 23:00:55.270495 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:55.270332 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-6c153e-647b666d54-p74jz" event={"ID":"0d43d667-a519-43ed-915e-837b766f397a","Type":"ContainerDied","Data":"426900e7687381481c95b56a702962d71b4d958aae33a091ce9662693dd2d425"} Jul 06 23:00:55.270495 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:55.270429 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-6c153e-647b666d54-p74jz" event={"ID":"0d43d667-a519-43ed-915e-837b766f397a","Type":"ContainerDied","Data":"e6b2f041c7f4a81524ca796db5f5d38fc269daf3687b47747a11f312d3cf8282"} Jul 06 23:00:55.270495 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:55.270460 2571 scope.go:117] "RemoveContainer" containerID="426900e7687381481c95b56a702962d71b4d958aae33a091ce9662693dd2d425" Jul 06 23:00:55.280509 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:55.280488 2571 scope.go:117] "RemoveContainer" containerID="426900e7687381481c95b56a702962d71b4d958aae33a091ce9662693dd2d425" Jul 06 23:00:55.280760 ip-10-0-137-147 kubenswrapper[2571]: E0706 23:00:55.280742 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"426900e7687381481c95b56a702962d71b4d958aae33a091ce9662693dd2d425\": container with ID starting with 426900e7687381481c95b56a702962d71b4d958aae33a091ce9662693dd2d425 not found: ID does not exist" containerID="426900e7687381481c95b56a702962d71b4d958aae33a091ce9662693dd2d425" Jul 06 23:00:55.280800 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:55.280773 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"426900e7687381481c95b56a702962d71b4d958aae33a091ce9662693dd2d425"} err="failed to get container status \"426900e7687381481c95b56a702962d71b4d958aae33a091ce9662693dd2d425\": rpc error: code = NotFound desc = could not find container \"426900e7687381481c95b56a702962d71b4d958aae33a091ce9662693dd2d425\": container with ID starting with 426900e7687381481c95b56a702962d71b4d958aae33a091ce9662693dd2d425 not found: ID does not exist" Jul 06 23:00:55.294305 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:55.294281 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-6c153e-647b666d54-p74jz"] Jul 06 23:00:55.298609 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:55.298585 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-6c153e-647b666d54-p74jz"] Jul 06 23:00:55.634679 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:00:55.634647 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d43d667-a519-43ed-915e-837b766f397a" path="/var/lib/kubelet/pods/0d43d667-a519-43ed-915e-837b766f397a/volumes" Jul 06 23:01:00.528822 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:00.528785 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 23:01:00.630140 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:00.630108 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 23:01:00.797748 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:00.797662 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 23:01:01.026179 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:01.026142 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 23:01:01.438850 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:01.438813 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d7d6f976b-rgjjr_e9e55f7f-183e-4f3b-975c-188c209a8df7/authorino/0.log" Jul 06 23:01:06.399700 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:06.399669 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-bdbcbfbbd-cz42d_e9108810-4ebe-41de-ad13-f23455bb516d/maas-api/0.log" Jul 06 23:01:06.504903 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:06.504875 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722950-vjkjj_3491aced-8662-4105-b094-476506976c45/cleanup/0.log" Jul 06 23:01:06.848975 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:06.848896 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5b4776d6c4-8zkq6_17688e6e-694d-424c-a7b7-71f9d4cda9fe/manager/0.log" Jul 06 23:01:07.497283 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:07.497256 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c6597d756-dmckl_c180e6ba-1217-4cf6-8e95-bbbf6cc98170/copy-manifests/0.log" Jul 06 23:01:07.512545 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:07.512516 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c6597d756-dmckl_c180e6ba-1217-4cf6-8e95-bbbf6cc98170/manager/0.log" Jul 06 23:01:09.019087 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:09.019057 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d7d6f976b-rgjjr_e9e55f7f-183e-4f3b-975c-188c209a8df7/authorino/0.log" Jul 06 23:01:09.675037 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:09.674997 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-ng78j_5eadc65a-d34e-458a-aa84-5f1a08b89cfc/manager/0.log" Jul 06 23:01:10.544006 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:10.543969 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7588b845d6-brn5g_a00e36ef-1dc2-403d-9cca-4298c2eb5561/kube-auth-proxy/0.log" Jul 06 23:01:11.065059 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:11.065020 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-cmgpv_afb88cc6-f49f-41d0-9634-115396264273/payload-processing/0.log" Jul 06 23:01:11.535074 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:11.535040 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-lcllt_d058f5dd-1fdd-49d3-aad1-0e6908b00b4f/main/0.log" Jul 06 23:01:11.774089 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:11.774062 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-xtcpb_fbe2b1ea-53bf-45a2-8cef-71f2afa45cde/main/0.log" Jul 06 23:01:16.348123 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:16.348084 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 23:01:16.438997 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:16.438954 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" limit=1024 Jul 06 23:01:19.365978 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:19.365945 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-ccgsb_d115b038-4f94-41df-82e3-6d42109603e0/global-pull-secret-syncer/0.log" Jul 06 23:01:19.466342 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:19.466309 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-65zfn_e5f17bc2-eb9a-4dbc-a599-2f6902a646f9/konnectivity-agent/0.log" Jul 06 23:01:19.570589 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:19.570538 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-147.ec2.internal_13c2e0a524b9d1d1320d115d1eeca7b4/haproxy/0.log" Jul 06 23:01:24.996265 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:24.996228 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d7d6f976b-rgjjr_e9e55f7f-183e-4f3b-975c-188c209a8df7/authorino/0.log" Jul 06 23:01:25.314404 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:25.314301 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-ng78j_5eadc65a-d34e-458a-aa84-5f1a08b89cfc/manager/0.log" Jul 06 23:01:27.489984 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:27.489894 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-8fwgn_cf6e4bbe-5fd6-4bec-84de-7bb8e7a68247/kube-state-metrics/0.log" Jul 06 23:01:27.515545 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:27.515513 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-8fwgn_cf6e4bbe-5fd6-4bec-84de-7bb8e7a68247/kube-rbac-proxy-main/0.log" Jul 06 23:01:27.550008 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:27.549977 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-8fwgn_cf6e4bbe-5fd6-4bec-84de-7bb8e7a68247/kube-rbac-proxy-self/0.log" Jul 06 23:01:27.769030 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:27.768947 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m2svj_cb280153-8e0b-4776-8c56-8c47339afb67/node-exporter/0.log" Jul 06 23:01:27.796331 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:27.796305 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m2svj_cb280153-8e0b-4776-8c56-8c47339afb67/kube-rbac-proxy/0.log" Jul 06 23:01:27.823666 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:27.823639 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m2svj_cb280153-8e0b-4776-8c56-8c47339afb67/init-textfile/0.log" Jul 06 23:01:27.961043 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:27.961013 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-7h2fd_b2a1a1e7-c4da-473f-8f3b-0949d9ef885a/kube-rbac-proxy-main/0.log" Jul 06 23:01:27.988025 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:27.987995 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-7h2fd_b2a1a1e7-c4da-473f-8f3b-0949d9ef885a/kube-rbac-proxy-self/0.log" Jul 06 23:01:28.015951 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:28.015921 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-7h2fd_b2a1a1e7-c4da-473f-8f3b-0949d9ef885a/openshift-state-metrics/0.log" Jul 06 23:01:28.065482 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:28.065409 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ce8044dc-7328-4898-a3bf-357ce89db933/prometheus/0.log" Jul 06 23:01:28.093156 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:28.093130 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ce8044dc-7328-4898-a3bf-357ce89db933/config-reloader/0.log" Jul 06 23:01:28.122686 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:28.122656 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ce8044dc-7328-4898-a3bf-357ce89db933/thanos-sidecar/0.log" Jul 06 23:01:28.159902 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:28.159868 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ce8044dc-7328-4898-a3bf-357ce89db933/kube-rbac-proxy-web/0.log" Jul 06 23:01:28.193114 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:28.193086 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ce8044dc-7328-4898-a3bf-357ce89db933/kube-rbac-proxy/0.log" Jul 06 23:01:28.224890 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:28.224866 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ce8044dc-7328-4898-a3bf-357ce89db933/kube-rbac-proxy-thanos/0.log" Jul 06 23:01:28.253536 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:28.253507 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ce8044dc-7328-4898-a3bf-357ce89db933/init-config-reloader/0.log" Jul 06 23:01:29.313166 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:29.313126 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-59pq4/perf-node-gather-daemonset-fsmjl"] Jul 06 23:01:29.313592 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:29.313525 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa9fe430-ff17-4440-9ba6-1c5582cd5344" containerName="maas-api" Jul 06 23:01:29.313592 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:29.313539 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa9fe430-ff17-4440-9ba6-1c5582cd5344" containerName="maas-api" Jul 06 23:01:29.313592 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:29.313554 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d43d667-a519-43ed-915e-837b766f397a" containerName="maas-api" Jul 06 23:01:29.313592 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:29.313559 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d43d667-a519-43ed-915e-837b766f397a" containerName="maas-api" Jul 06 23:01:29.313592 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:29.313569 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="937eb1db-82f0-45e9-8d0e-e6bddfce47cb" containerName="main" Jul 06 23:01:29.313592 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:29.313576 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="937eb1db-82f0-45e9-8d0e-e6bddfce47cb" containerName="main" Jul 06 23:01:29.313592 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:29.313589 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34964a9e-d002-4638-a9d4-5f7c0b7af86e" containerName="cleanup" Jul 06 23:01:29.313592 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:29.313594 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="34964a9e-d002-4638-a9d4-5f7c0b7af86e" containerName="cleanup" Jul 06 23:01:29.313841 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:29.313656 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="34964a9e-d002-4638-a9d4-5f7c0b7af86e" containerName="cleanup" Jul 06 23:01:29.313841 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:29.313666 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d43d667-a519-43ed-915e-837b766f397a" containerName="maas-api" Jul 06 23:01:29.313841 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:29.313673 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa9fe430-ff17-4440-9ba6-1c5582cd5344" containerName="maas-api" Jul 06 23:01:29.313841 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:29.313678 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="937eb1db-82f0-45e9-8d0e-e6bddfce47cb" containerName="main" Jul 06 23:01:29.316986 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:29.316967 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-59pq4/perf-node-gather-daemonset-fsmjl" Jul 06 23:01:29.319518 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:29.319494 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-59pq4\"/\"openshift-service-ca.crt\"" Jul 06 23:01:29.320412 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:29.320393 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-59pq4\"/\"kube-root-ca.crt\"" Jul 06 23:01:29.320540 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:29.320398 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-59pq4\"/\"default-dockercfg-zdbs2\"" Jul 06 23:01:29.326017 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:29.325984 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-59pq4/perf-node-gather-daemonset-fsmjl"] Jul 06 23:01:29.426397 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:29.426362 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/79c3c8d4-d850-4283-9b03-4244cb62dd56-proc\") pod \"perf-node-gather-daemonset-fsmjl\" (UID: \"79c3c8d4-d850-4283-9b03-4244cb62dd56\") " pod="openshift-must-gather-59pq4/perf-node-gather-daemonset-fsmjl" Jul 06 23:01:29.426594 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:29.426414 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g7lq\" (UniqueName: \"kubernetes.io/projected/79c3c8d4-d850-4283-9b03-4244cb62dd56-kube-api-access-9g7lq\") pod \"perf-node-gather-daemonset-fsmjl\" (UID: \"79c3c8d4-d850-4283-9b03-4244cb62dd56\") " pod="openshift-must-gather-59pq4/perf-node-gather-daemonset-fsmjl" Jul 06 23:01:29.426594 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:29.426455 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/79c3c8d4-d850-4283-9b03-4244cb62dd56-lib-modules\") pod \"perf-node-gather-daemonset-fsmjl\" (UID: \"79c3c8d4-d850-4283-9b03-4244cb62dd56\") " pod="openshift-must-gather-59pq4/perf-node-gather-daemonset-fsmjl" Jul 06 23:01:29.426594 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:29.426508 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79c3c8d4-d850-4283-9b03-4244cb62dd56-sys\") pod \"perf-node-gather-daemonset-fsmjl\" (UID: \"79c3c8d4-d850-4283-9b03-4244cb62dd56\") " pod="openshift-must-gather-59pq4/perf-node-gather-daemonset-fsmjl" Jul 06 23:01:29.426594 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:29.426546 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/79c3c8d4-d850-4283-9b03-4244cb62dd56-podres\") pod \"perf-node-gather-daemonset-fsmjl\" (UID: \"79c3c8d4-d850-4283-9b03-4244cb62dd56\") " pod="openshift-must-gather-59pq4/perf-node-gather-daemonset-fsmjl" Jul 06 23:01:29.527657 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:29.527611 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/79c3c8d4-d850-4283-9b03-4244cb62dd56-proc\") pod \"perf-node-gather-daemonset-fsmjl\" (UID: \"79c3c8d4-d850-4283-9b03-4244cb62dd56\") " pod="openshift-must-gather-59pq4/perf-node-gather-daemonset-fsmjl" Jul 06 23:01:29.527835 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:29.527679 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g7lq\" (UniqueName: \"kubernetes.io/projected/79c3c8d4-d850-4283-9b03-4244cb62dd56-kube-api-access-9g7lq\") pod \"perf-node-gather-daemonset-fsmjl\" (UID: \"79c3c8d4-d850-4283-9b03-4244cb62dd56\") " pod="openshift-must-gather-59pq4/perf-node-gather-daemonset-fsmjl" Jul 06 23:01:29.527835 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:29.527741 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/79c3c8d4-d850-4283-9b03-4244cb62dd56-lib-modules\") pod \"perf-node-gather-daemonset-fsmjl\" (UID: \"79c3c8d4-d850-4283-9b03-4244cb62dd56\") " pod="openshift-must-gather-59pq4/perf-node-gather-daemonset-fsmjl" Jul 06 23:01:29.527835 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:29.527749 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/79c3c8d4-d850-4283-9b03-4244cb62dd56-proc\") pod \"perf-node-gather-daemonset-fsmjl\" (UID: \"79c3c8d4-d850-4283-9b03-4244cb62dd56\") " pod="openshift-must-gather-59pq4/perf-node-gather-daemonset-fsmjl" Jul 06 23:01:29.527835 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:29.527778 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79c3c8d4-d850-4283-9b03-4244cb62dd56-sys\") pod \"perf-node-gather-daemonset-fsmjl\" (UID: \"79c3c8d4-d850-4283-9b03-4244cb62dd56\") " pod="openshift-must-gather-59pq4/perf-node-gather-daemonset-fsmjl" Jul 06 23:01:29.527835 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:29.527812 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/79c3c8d4-d850-4283-9b03-4244cb62dd56-podres\") pod \"perf-node-gather-daemonset-fsmjl\" (UID: \"79c3c8d4-d850-4283-9b03-4244cb62dd56\") " pod="openshift-must-gather-59pq4/perf-node-gather-daemonset-fsmjl" Jul 06 23:01:29.527999 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:29.527895 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/79c3c8d4-d850-4283-9b03-4244cb62dd56-lib-modules\") pod \"perf-node-gather-daemonset-fsmjl\" (UID: \"79c3c8d4-d850-4283-9b03-4244cb62dd56\") " pod="openshift-must-gather-59pq4/perf-node-gather-daemonset-fsmjl" Jul 06 23:01:29.527999 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:29.527922 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79c3c8d4-d850-4283-9b03-4244cb62dd56-sys\") pod \"perf-node-gather-daemonset-fsmjl\" (UID: \"79c3c8d4-d850-4283-9b03-4244cb62dd56\") " pod="openshift-must-gather-59pq4/perf-node-gather-daemonset-fsmjl" Jul 06 23:01:29.527999 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:29.527934 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/79c3c8d4-d850-4283-9b03-4244cb62dd56-podres\") pod \"perf-node-gather-daemonset-fsmjl\" (UID: \"79c3c8d4-d850-4283-9b03-4244cb62dd56\") " pod="openshift-must-gather-59pq4/perf-node-gather-daemonset-fsmjl" Jul 06 23:01:29.537598 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:29.537561 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g7lq\" (UniqueName: \"kubernetes.io/projected/79c3c8d4-d850-4283-9b03-4244cb62dd56-kube-api-access-9g7lq\") pod \"perf-node-gather-daemonset-fsmjl\" (UID: \"79c3c8d4-d850-4283-9b03-4244cb62dd56\") " pod="openshift-must-gather-59pq4/perf-node-gather-daemonset-fsmjl" Jul 06 23:01:29.627529 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:29.627439 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-59pq4/perf-node-gather-daemonset-fsmjl" Jul 06 23:01:29.765561 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:29.765530 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-59pq4/perf-node-gather-daemonset-fsmjl"] Jul 06 23:01:29.768247 ip-10-0-137-147 kubenswrapper[2571]: W0706 23:01:29.768195 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod79c3c8d4_d850_4283_9b03_4244cb62dd56.slice/crio-e5c4f1b0a644a1c90dbfe63cf842a0a4e8775ed69f51dff50111e543b4a4d0e4 WatchSource:0}: Error finding container e5c4f1b0a644a1c90dbfe63cf842a0a4e8775ed69f51dff50111e543b4a4d0e4: Status 404 returned error can't find the container with id e5c4f1b0a644a1c90dbfe63cf842a0a4e8775ed69f51dff50111e543b4a4d0e4 Jul 06 23:01:30.439558 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:30.439510 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-59pq4/perf-node-gather-daemonset-fsmjl" event={"ID":"79c3c8d4-d850-4283-9b03-4244cb62dd56","Type":"ContainerStarted","Data":"cb9980dfa758f85115fd2e73af1d4a749faa8aa28509b661ab8b90510f700a01"} Jul 06 23:01:30.440040 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:30.439567 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-59pq4/perf-node-gather-daemonset-fsmjl" event={"ID":"79c3c8d4-d850-4283-9b03-4244cb62dd56","Type":"ContainerStarted","Data":"e5c4f1b0a644a1c90dbfe63cf842a0a4e8775ed69f51dff50111e543b4a4d0e4"} Jul 06 23:01:30.440040 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:30.439599 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-59pq4/perf-node-gather-daemonset-fsmjl" Jul 06 23:01:30.443248 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:30.443201 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-4x6pf_5796a8ea-adcb-4819-bb18-0b85e1ef4681/console-operator/2.log" Jul 06 23:01:30.447085 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:30.447046 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-4x6pf_5796a8ea-adcb-4819-bb18-0b85e1ef4681/console-operator/3.log" Jul 06 23:01:30.458662 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:30.458618 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-59pq4/perf-node-gather-daemonset-fsmjl" podStartSLOduration=1.458601837 podStartE2EDuration="1.458601837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:01:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:01:30.4575868 +0000 UTC m=+2497.375829655" watchObservedRunningTime="2026-07-06 23:01:30.458601837 +0000 UTC m=+2497.376844692" Jul 06 23:01:32.919127 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:32.919101 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xfxdc_13cb55e3-2959-4b92-acff-2d8750c2dc62/dns/0.log" Jul 06 23:01:32.953963 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:32.953934 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xfxdc_13cb55e3-2959-4b92-acff-2d8750c2dc62/kube-rbac-proxy/0.log" Jul 06 23:01:33.072267 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:33.072234 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-l7h9q_e472a07e-8c98-4b7c-9da5-13a2240aaa4c/dns-node-resolver/0.log" Jul 06 23:01:33.744029 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:33.743996 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tj7dx_cddcda7d-c04f-4fca-a269-9c034ad1e2a1/node-ca/0.log" Jul 06 23:01:34.975558 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:34.975528 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7588b845d6-brn5g_a00e36ef-1dc2-403d-9cca-4298c2eb5561/kube-auth-proxy/0.log" Jul 06 23:01:35.276448 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:35.276368 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-cmgpv_afb88cc6-f49f-41d0-9634-115396264273/payload-processing/0.log" Jul 06 23:01:36.094787 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:36.094759 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7hlrt_45c9cd76-e64c-4050-a5d2-88d69415cb36/serve-healthcheck-canary/0.log" Jul 06 23:01:36.455383 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:36.455356 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-59pq4/perf-node-gather-daemonset-fsmjl" Jul 06 23:01:36.662293 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:36.662266 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2ktvb_46b9db84-34e1-414a-a394-2809f8f0fa0b/kube-rbac-proxy/0.log" Jul 06 23:01:36.689372 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:36.689347 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2ktvb_46b9db84-34e1-414a-a394-2809f8f0fa0b/exporter/0.log" Jul 06 23:01:36.719661 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:36.719587 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2ktvb_46b9db84-34e1-414a-a394-2809f8f0fa0b/extractor/0.log" Jul 06 23:01:39.739253 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:39.739217 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-bdbcbfbbd-cz42d_e9108810-4ebe-41de-ad13-f23455bb516d/maas-api/0.log" Jul 06 23:01:39.767915 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:39.767886 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722950-vjkjj_3491aced-8662-4105-b094-476506976c45/cleanup/0.log" Jul 06 23:01:39.956582 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:39.956543 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5b4776d6c4-8zkq6_17688e6e-694d-424c-a7b7-71f9d4cda9fe/manager/0.log" Jul 06 23:01:40.010835 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:40.010754 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7588b845d6-brn5g_a00e36ef-1dc2-403d-9cca-4298c2eb5561/kube-auth-proxy/0.log" Jul 06 23:01:40.211920 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:40.211887 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-cmgpv_afb88cc6-f49f-41d0-9634-115396264273/payload-processing/0.log" Jul 06 23:01:40.350629 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:40.350545 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c6597d756-dmckl_c180e6ba-1217-4cf6-8e95-bbbf6cc98170/manager/0.log" Jul 06 23:01:40.382637 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:40.382610 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c6597d756-dmckl_c180e6ba-1217-4cf6-8e95-bbbf6cc98170/copy-manifests/0.log" Jul 06 23:01:41.677974 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:41.677931 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7588b845d6-brn5g_a00e36ef-1dc2-403d-9cca-4298c2eb5561/kube-auth-proxy/0.log" Jul 06 23:01:41.865462 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:41.865427 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-cmgpv_afb88cc6-f49f-41d0-9634-115396264273/payload-processing/0.log" Jul 06 23:01:41.959839 ip-10-0-137-147 kubenswrapper[2571]: I0706 23:01:41.959739 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-67b4867ff6-mpbh8_278c0f33-7076-4cb1-9801-cc976024cd5e/manager/0.log"