Jul 10 00:13:44.185969 ip-10-0-135-101 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 10 00:13:44.185981 ip-10-0-135-101 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 10 00:13:44.185991 ip-10-0-135-101 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 10 00:13:44.186334 ip-10-0-135-101 systemd[1]: Failed to start Kubernetes Kubelet. Jul 10 00:13:54.316864 ip-10-0-135-101 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 10 00:13:54.316880 ip-10-0-135-101 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 3c0d2f4549bb49abb302b0a10ce09609 -- Jul 10 00:15:33.017335 ip-10-0-135-101 systemd[1]: Starting Kubernetes Kubelet... Jul 10 00:15:33.584396 ip-10-0-135-101 kubenswrapper[2562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 00:15:33.584396 ip-10-0-135-101 kubenswrapper[2562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 10 00:15:33.584396 ip-10-0-135-101 kubenswrapper[2562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 00:15:33.584396 ip-10-0-135-101 kubenswrapper[2562]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 10 00:15:33.584396 ip-10-0-135-101 kubenswrapper[2562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 00:15:33.584396 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.492981 2562 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 10 00:15:33.584396 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497473 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 00:15:33.584396 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497484 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 00:15:33.584396 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497487 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 00:15:33.584396 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497490 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 00:15:33.584396 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497494 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 00:15:33.584396 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497497 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 00:15:33.584396 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497500 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 00:15:33.584396 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497503 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 00:15:33.569819 ip-10-0-135-101 systemd[1]: Started Kubernetes Kubelet. Jul 10 00:15:33.820660 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497505 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 00:15:33.820660 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497509 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 00:15:33.820660 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497513 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 00:15:33.820660 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497516 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 00:15:33.820660 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497519 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 00:15:33.820660 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497521 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 00:15:33.820660 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497524 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 00:15:33.820660 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497528 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 00:15:33.820660 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497531 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 00:15:33.820660 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497533 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 00:15:33.820660 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497537 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 00:15:33.820660 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497539 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 00:15:33.820660 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497541 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 00:15:33.820660 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497548 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 00:15:33.820660 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497551 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 00:15:33.820660 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497555 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 00:15:33.820660 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497558 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 00:15:33.820660 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497560 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 00:15:33.820660 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497563 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 00:15:33.834257 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497566 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 00:15:33.834257 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497568 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 00:15:33.834257 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497571 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 00:15:33.834257 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497574 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 00:15:33.834257 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497576 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 00:15:33.834257 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497579 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 00:15:33.834257 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497581 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 10 00:15:33.834257 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497584 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 00:15:33.834257 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497586 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 00:15:33.834257 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497589 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 00:15:33.834257 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497591 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 00:15:33.834257 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497594 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 00:15:33.834257 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497596 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 00:15:33.834257 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497599 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 00:15:33.834257 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497601 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 00:15:33.834257 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497604 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 00:15:33.834257 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497607 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 00:15:33.834257 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497609 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 00:15:33.834257 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497612 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 00:15:33.834257 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497614 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 00:15:33.835312 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497617 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 00:15:33.835312 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497620 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 00:15:33.835312 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497622 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 00:15:33.835312 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497624 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 00:15:33.835312 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497627 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 00:15:33.835312 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497634 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 00:15:33.835312 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497637 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 00:15:33.835312 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497639 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 00:15:33.835312 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497642 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 00:15:33.835312 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497644 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 00:15:33.835312 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497647 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 00:15:33.835312 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497649 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 00:15:33.835312 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497651 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 00:15:33.835312 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497654 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 00:15:33.835312 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497657 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 00:15:33.835312 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497659 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 00:15:33.835312 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497662 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 00:15:33.835312 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497664 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 00:15:33.835312 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497667 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 00:15:33.835312 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497670 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 00:15:33.836999 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497672 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 00:15:33.836999 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497675 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 00:15:33.836999 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497677 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 00:15:33.836999 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497680 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 00:15:33.836999 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497683 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 00:15:33.836999 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497685 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 00:15:33.836999 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497688 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 00:15:33.836999 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497690 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 00:15:33.836999 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497693 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 00:15:33.836999 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497695 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 00:15:33.836999 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497698 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 00:15:33.836999 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497702 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 00:15:33.836999 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497705 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 00:15:33.836999 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497708 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 00:15:33.836999 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497711 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 00:15:33.836999 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497714 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 00:15:33.836999 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497716 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 00:15:33.836999 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497719 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 00:15:33.836999 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497721 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 00:15:33.836999 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497724 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 00:15:33.838685 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497727 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 00:15:33.838685 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497729 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 00:15:33.838685 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497734 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 00:15:33.838685 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497736 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 00:15:33.838685 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497739 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 00:15:33.838685 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497744 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 00:15:33.838685 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497747 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 00:15:33.838685 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497749 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 00:15:33.838685 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497753 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 00:15:33.838685 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497755 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 00:15:33.838685 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497758 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 00:15:33.838685 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.497760 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 00:15:33.838685 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498436 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 00:15:33.838685 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498442 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 00:15:33.838685 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498445 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 00:15:33.838685 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498448 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 00:15:33.838685 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498450 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 00:15:33.838685 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498456 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 00:15:33.838685 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498459 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 00:15:33.839573 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498462 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 00:15:33.839573 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498464 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 00:15:33.839573 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498467 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 00:15:33.839573 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498470 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 00:15:33.839573 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498489 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 00:15:33.839573 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498495 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 00:15:33.839573 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498498 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 00:15:33.839573 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498500 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 00:15:33.839573 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498517 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 00:15:33.839573 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498521 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 00:15:33.839573 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498525 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 00:15:33.839573 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498528 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 00:15:33.839573 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498531 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 00:15:33.839573 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498534 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 00:15:33.839573 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498536 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 00:15:33.839573 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498541 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 00:15:33.839573 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498543 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 00:15:33.839573 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498546 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 00:15:33.839573 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498549 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 00:15:33.840468 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498551 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 00:15:33.840468 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498553 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 00:15:33.840468 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498556 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 00:15:33.840468 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498558 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 00:15:33.840468 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498561 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 00:15:33.840468 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498564 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 00:15:33.840468 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498567 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 00:15:33.840468 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498569 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 00:15:33.840468 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498571 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 00:15:33.840468 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498574 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 00:15:33.840468 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498577 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 00:15:33.840468 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498580 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 00:15:33.840468 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498583 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 00:15:33.840468 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498585 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 00:15:33.840468 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498587 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 00:15:33.840468 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498590 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 00:15:33.840468 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498594 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 00:15:33.840468 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498597 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 00:15:33.840468 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498600 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 00:15:33.840468 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498603 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 00:15:33.841528 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498606 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 00:15:33.841528 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498609 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 00:15:33.841528 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498611 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 00:15:33.841528 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498613 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 00:15:33.841528 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498617 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 00:15:33.841528 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498620 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 00:15:33.841528 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498623 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 00:15:33.841528 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498625 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 00:15:33.841528 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498629 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 00:15:33.841528 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498631 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 00:15:33.841528 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498634 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 00:15:33.841528 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498636 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 00:15:33.841528 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498639 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 00:15:33.841528 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498641 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 00:15:33.841528 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498643 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 00:15:33.841528 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498646 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 00:15:33.841528 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498648 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 00:15:33.841528 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498651 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 00:15:33.841528 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498654 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 00:15:33.842388 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498656 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 00:15:33.842388 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498658 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 00:15:33.842388 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498661 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 00:15:33.842388 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498663 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 00:15:33.842388 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498666 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 00:15:33.842388 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498668 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 00:15:33.842388 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498671 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 00:15:33.842388 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498673 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 00:15:33.842388 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498676 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 00:15:33.842388 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498678 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 00:15:33.842388 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498680 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 00:15:33.842388 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498683 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 00:15:33.842388 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498685 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 00:15:33.842388 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498689 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 00:15:33.842388 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498691 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 00:15:33.842388 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498694 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 00:15:33.842388 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498696 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 00:15:33.842388 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498699 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 00:15:33.842388 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498701 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 00:15:33.842388 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498704 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 00:15:33.842993 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498706 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 10 00:15:33.842993 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498710 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 00:15:33.842993 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498712 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 00:15:33.842993 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498715 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 00:15:33.842993 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498717 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 00:15:33.842993 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498720 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 00:15:33.842993 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498722 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 00:15:33.842993 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498725 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 00:15:33.842993 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498727 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 00:15:33.842993 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498729 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 00:15:33.842993 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498732 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 00:15:33.842993 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498734 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 00:15:33.842993 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498737 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 00:15:33.842993 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.498739 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 00:15:33.842993 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499113 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 00:15:33.842993 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499119 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 00:15:33.842993 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499122 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 00:15:33.842993 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499126 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 00:15:33.842993 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499129 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 00:15:33.843620 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499132 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 00:15:33.843620 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499135 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 00:15:33.843620 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499138 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 00:15:33.843620 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499140 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 00:15:33.843620 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499143 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 00:15:33.843620 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499145 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 00:15:33.843620 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499150 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 00:15:33.843620 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499153 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 00:15:33.843620 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499155 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 00:15:33.843620 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499158 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 00:15:33.843620 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499160 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 00:15:33.843620 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499162 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 00:15:33.843620 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499165 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 00:15:33.843620 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499167 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 00:15:33.843620 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499172 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 00:15:33.843620 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499174 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 00:15:33.843620 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499177 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 00:15:33.843620 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499179 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 00:15:33.843620 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499182 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 00:15:33.843620 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499184 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 00:15:33.844448 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499187 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 00:15:33.844448 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499189 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 00:15:33.844448 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499193 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 00:15:33.844448 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499197 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 00:15:33.844448 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499200 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 00:15:33.844448 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499203 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 00:15:33.844448 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499205 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 00:15:33.844448 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499208 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 00:15:33.844448 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499211 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 00:15:33.844448 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499214 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 00:15:33.844448 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499216 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 00:15:33.844448 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499219 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 00:15:33.844448 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499222 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 00:15:33.844448 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499224 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 00:15:33.844448 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499226 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 00:15:33.844448 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499229 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 00:15:33.844448 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499231 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 00:15:33.844448 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499234 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 00:15:33.844448 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499237 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 00:15:33.844448 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499240 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 00:15:33.845063 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499242 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 00:15:33.845063 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499245 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 00:15:33.845063 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499247 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 00:15:33.845063 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499249 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 00:15:33.845063 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499252 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 00:15:33.845063 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499254 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 00:15:33.845063 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499258 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 00:15:33.845063 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499260 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 00:15:33.845063 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499263 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 00:15:33.845063 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499265 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 00:15:33.845063 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499267 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 00:15:33.845063 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499270 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 00:15:33.845063 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499272 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 00:15:33.845063 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499274 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 00:15:33.845063 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499277 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 00:15:33.845063 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499279 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 00:15:33.845063 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499282 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 00:15:33.845063 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499284 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 00:15:33.845063 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499287 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 00:15:33.845063 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499289 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 00:15:33.845843 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499292 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 00:15:33.845843 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499294 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 00:15:33.845843 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499296 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 00:15:33.845843 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499299 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 00:15:33.845843 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499301 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 00:15:33.845843 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499305 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 00:15:33.845843 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499308 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 00:15:33.845843 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499311 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 00:15:33.845843 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499314 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 00:15:33.845843 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499317 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 00:15:33.845843 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499320 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 00:15:33.845843 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499323 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 00:15:33.845843 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499326 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 00:15:33.845843 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499330 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 00:15:33.845843 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499333 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 00:15:33.845843 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499336 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 00:15:33.845843 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499339 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 00:15:33.845843 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499341 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 00:15:33.845843 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499345 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 00:15:33.846399 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499348 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 00:15:33.846399 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499350 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 00:15:33.846399 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499353 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 00:15:33.846399 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499355 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 10 00:15:33.846399 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499358 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 00:15:33.846399 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499360 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 00:15:33.846399 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499363 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 00:15:33.846399 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499366 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 00:15:33.846399 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499368 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 00:15:33.846399 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499371 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 00:15:33.846399 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499373 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 00:15:33.846399 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499375 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 00:15:33.846399 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499378 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 00:15:33.846399 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499380 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 00:15:33.846399 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.499383 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 00:15:33.846399 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499442 2562 flags.go:64] FLAG: --address="0.0.0.0" Jul 10 00:15:33.846399 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499453 2562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 10 00:15:33.846399 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499458 2562 flags.go:64] FLAG: --anonymous-auth="true" Jul 10 00:15:33.846399 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499462 2562 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 10 00:15:33.846399 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499467 2562 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 10 00:15:33.846399 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499470 2562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 10 00:15:33.846973 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499474 2562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 10 00:15:33.846973 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499478 2562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 10 00:15:33.846973 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499482 2562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 10 00:15:33.846973 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499484 2562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 10 00:15:33.846973 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499488 2562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 10 00:15:33.846973 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499491 2562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 10 00:15:33.846973 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499494 2562 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 10 00:15:33.846973 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499497 2562 flags.go:64] FLAG: --cgroup-root="" Jul 10 00:15:33.846973 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499500 2562 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 10 00:15:33.846973 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499503 2562 flags.go:64] FLAG: --client-ca-file="" Jul 10 00:15:33.846973 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499507 2562 flags.go:64] FLAG: --cloud-provider="external" Jul 10 00:15:33.846973 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499509 2562 flags.go:64] FLAG: --cluster-dns="[]" Jul 10 00:15:33.846973 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499513 2562 flags.go:64] FLAG: --cluster-domain="" Jul 10 00:15:33.846973 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499516 2562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 10 00:15:33.846973 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499518 2562 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 10 00:15:33.846973 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499522 2562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 10 00:15:33.846973 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499526 2562 flags.go:64] FLAG: --container-log-max-files="5" Jul 10 00:15:33.846973 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499529 2562 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 10 00:15:33.846973 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499532 2562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 10 00:15:33.846973 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499535 2562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 10 00:15:33.846973 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499538 2562 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 10 00:15:33.846973 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499541 2562 flags.go:64] FLAG: --contention-profiling="false" Jul 10 00:15:33.846973 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499544 2562 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 10 00:15:33.846973 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499547 2562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 10 00:15:33.847579 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499550 2562 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 10 00:15:33.847579 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499553 2562 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 10 00:15:33.847579 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499556 2562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 10 00:15:33.847579 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499559 2562 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 10 00:15:33.847579 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499562 2562 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 10 00:15:33.847579 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499565 2562 flags.go:64] FLAG: --enable-load-reader="false" Jul 10 00:15:33.847579 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499567 2562 flags.go:64] FLAG: --enable-server="true" Jul 10 00:15:33.847579 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499571 2562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 10 00:15:33.847579 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499575 2562 flags.go:64] FLAG: --event-burst="100" Jul 10 00:15:33.847579 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499578 2562 flags.go:64] FLAG: --event-qps="50" Jul 10 00:15:33.847579 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499583 2562 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 10 00:15:33.847579 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499586 2562 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 10 00:15:33.847579 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499588 2562 flags.go:64] FLAG: --eviction-hard="" Jul 10 00:15:33.847579 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499592 2562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 10 00:15:33.847579 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499595 2562 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 10 00:15:33.847579 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499597 2562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 10 00:15:33.847579 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499600 2562 flags.go:64] FLAG: --eviction-soft="" Jul 10 00:15:33.847579 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499603 2562 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 10 00:15:33.847579 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499607 2562 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 10 00:15:33.847579 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499610 2562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 10 00:15:33.847579 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499613 2562 flags.go:64] FLAG: --experimental-mounter-path="" Jul 10 00:15:33.847579 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499616 2562 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 10 00:15:33.847579 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499619 2562 flags.go:64] FLAG: --fail-swap-on="true" Jul 10 00:15:33.847579 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499621 2562 flags.go:64] FLAG: --feature-gates="" Jul 10 00:15:33.847579 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499625 2562 flags.go:64] FLAG: --file-check-frequency="20s" Jul 10 00:15:33.848368 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499628 2562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 10 00:15:33.848368 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499631 2562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 10 00:15:33.848368 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499634 2562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 10 00:15:33.848368 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499637 2562 flags.go:64] FLAG: --healthz-port="10248" Jul 10 00:15:33.848368 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499640 2562 flags.go:64] FLAG: --help="false" Jul 10 00:15:33.848368 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499642 2562 flags.go:64] FLAG: --hostname-override="ip-10-0-135-101.ec2.internal" Jul 10 00:15:33.848368 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499645 2562 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 10 00:15:33.848368 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499648 2562 flags.go:64] FLAG: --http-check-frequency="20s" Jul 10 00:15:33.848368 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499651 2562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 10 00:15:33.848368 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499654 2562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 10 00:15:33.848368 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499657 2562 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 10 00:15:33.848368 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499660 2562 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 10 00:15:33.848368 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499662 2562 flags.go:64] FLAG: --image-service-endpoint="" Jul 10 00:15:33.848368 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499665 2562 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 10 00:15:33.848368 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499668 2562 flags.go:64] FLAG: --kube-api-burst="100" Jul 10 00:15:33.848368 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499674 2562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 10 00:15:33.848368 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499678 2562 flags.go:64] FLAG: --kube-api-qps="50" Jul 10 00:15:33.848368 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499682 2562 flags.go:64] FLAG: --kube-reserved="" Jul 10 00:15:33.848368 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499684 2562 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 10 00:15:33.848368 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499687 2562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 10 00:15:33.848368 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499690 2562 flags.go:64] FLAG: --kubelet-cgroups="" Jul 10 00:15:33.848368 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499693 2562 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 10 00:15:33.848368 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499695 2562 flags.go:64] FLAG: --lock-file="" Jul 10 00:15:33.848368 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499698 2562 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 10 00:15:33.848988 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499701 2562 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 10 00:15:33.848988 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499705 2562 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 10 00:15:33.848988 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499710 2562 flags.go:64] FLAG: --log-json-split-stream="false" Jul 10 00:15:33.848988 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499713 2562 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 10 00:15:33.848988 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499715 2562 flags.go:64] FLAG: --log-text-split-stream="false" Jul 10 00:15:33.848988 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499718 2562 flags.go:64] FLAG: --logging-format="text" Jul 10 00:15:33.848988 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499721 2562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 10 00:15:33.848988 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499724 2562 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 10 00:15:33.848988 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499727 2562 flags.go:64] FLAG: --manifest-url="" Jul 10 00:15:33.848988 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499729 2562 flags.go:64] FLAG: --manifest-url-header="" Jul 10 00:15:33.848988 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499733 2562 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 10 00:15:33.848988 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499736 2562 flags.go:64] FLAG: --max-open-files="1000000" Jul 10 00:15:33.848988 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499740 2562 flags.go:64] FLAG: --max-pods="110" Jul 10 00:15:33.848988 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499743 2562 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 10 00:15:33.848988 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499745 2562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 10 00:15:33.848988 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499748 2562 flags.go:64] FLAG: --memory-manager-policy="None" Jul 10 00:15:33.848988 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499751 2562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 10 00:15:33.848988 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499754 2562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 10 00:15:33.848988 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499757 2562 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 10 00:15:33.848988 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499759 2562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 10 00:15:33.848988 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499765 2562 flags.go:64] FLAG: --node-status-max-images="50" Jul 10 00:15:33.848988 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499768 2562 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 10 00:15:33.848988 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499771 2562 flags.go:64] FLAG: --oom-score-adj="-999" Jul 10 00:15:33.848988 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499774 2562 flags.go:64] FLAG: --pod-cidr="" Jul 10 00:15:33.849660 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499783 2562 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 10 00:15:33.849660 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499789 2562 flags.go:64] FLAG: --pod-manifest-path="" Jul 10 00:15:33.849660 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499792 2562 flags.go:64] FLAG: --pod-max-pids="-1" Jul 10 00:15:33.849660 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499795 2562 flags.go:64] FLAG: --pods-per-core="0" Jul 10 00:15:33.849660 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499798 2562 flags.go:64] FLAG: --port="10250" Jul 10 00:15:33.849660 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499801 2562 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 10 00:15:33.849660 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499804 2562 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e1f14c9a8eda6d9c" Jul 10 00:15:33.849660 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499807 2562 flags.go:64] FLAG: --qos-reserved="" Jul 10 00:15:33.849660 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499810 2562 flags.go:64] FLAG: --read-only-port="10255" Jul 10 00:15:33.849660 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499813 2562 flags.go:64] FLAG: --register-node="true" Jul 10 00:15:33.849660 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499817 2562 flags.go:64] FLAG: --register-with-taints="" Jul 10 00:15:33.849660 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499820 2562 flags.go:64] FLAG: --registry-burst="10" Jul 10 00:15:33.849660 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499823 2562 flags.go:64] FLAG: --registry-qps="5" Jul 10 00:15:33.849660 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499826 2562 flags.go:64] FLAG: --reserved-cpus="" Jul 10 00:15:33.849660 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499829 2562 flags.go:64] FLAG: --reserved-memory="" Jul 10 00:15:33.849660 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499832 2562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 10 00:15:33.849660 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499835 2562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 10 00:15:33.849660 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499838 2562 flags.go:64] FLAG: --rotate-certificates="false" Jul 10 00:15:33.849660 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499841 2562 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 10 00:15:33.849660 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499844 2562 flags.go:64] FLAG: --runonce="false" Jul 10 00:15:33.849660 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499846 2562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 10 00:15:33.849660 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499849 2562 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 10 00:15:33.849660 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499852 2562 flags.go:64] FLAG: --seccomp-default="false" Jul 10 00:15:33.849660 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499855 2562 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 10 00:15:33.849660 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499857 2562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 10 00:15:33.850624 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499860 2562 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 10 00:15:33.850624 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499863 2562 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 10 00:15:33.850624 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499866 2562 flags.go:64] FLAG: --storage-driver-password="root" Jul 10 00:15:33.850624 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499869 2562 flags.go:64] FLAG: --storage-driver-secure="false" Jul 10 00:15:33.850624 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499872 2562 flags.go:64] FLAG: --storage-driver-table="stats" Jul 10 00:15:33.850624 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499874 2562 flags.go:64] FLAG: --storage-driver-user="root" Jul 10 00:15:33.850624 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499877 2562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 10 00:15:33.850624 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499880 2562 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 10 00:15:33.850624 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499890 2562 flags.go:64] FLAG: --system-cgroups="" Jul 10 00:15:33.850624 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499893 2562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 10 00:15:33.850624 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499898 2562 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 10 00:15:33.850624 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499900 2562 flags.go:64] FLAG: --tls-cert-file="" Jul 10 00:15:33.850624 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499903 2562 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 10 00:15:33.850624 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499907 2562 flags.go:64] FLAG: --tls-min-version="" Jul 10 00:15:33.850624 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499909 2562 flags.go:64] FLAG: --tls-private-key-file="" Jul 10 00:15:33.850624 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499912 2562 flags.go:64] FLAG: --topology-manager-policy="none" Jul 10 00:15:33.850624 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499915 2562 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 10 00:15:33.850624 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499918 2562 flags.go:64] FLAG: --topology-manager-scope="container" Jul 10 00:15:33.850624 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499921 2562 flags.go:64] FLAG: --v="2" Jul 10 00:15:33.850624 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499925 2562 flags.go:64] FLAG: --version="false" Jul 10 00:15:33.850624 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499929 2562 flags.go:64] FLAG: --vmodule="" Jul 10 00:15:33.850624 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499932 2562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 10 00:15:33.850624 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.499935 2562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 10 00:15:33.850624 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500048 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 00:15:33.851325 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500052 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 00:15:33.851325 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500055 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 00:15:33.851325 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500057 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 00:15:33.851325 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500059 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 00:15:33.851325 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500062 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 00:15:33.851325 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500065 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 00:15:33.851325 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500067 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 00:15:33.851325 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500070 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 00:15:33.851325 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500085 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 00:15:33.851325 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500088 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 00:15:33.851325 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500091 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 00:15:33.851325 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500094 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 00:15:33.851325 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500096 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 00:15:33.851325 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500099 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 00:15:33.851325 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500101 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 00:15:33.851325 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500104 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 00:15:33.851325 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500108 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 00:15:33.851325 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500111 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 00:15:33.851325 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500114 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 10 00:15:33.851325 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500116 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 00:15:34.007907 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500119 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 00:15:34.007907 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500122 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 00:15:34.007907 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500124 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 00:15:34.007907 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500127 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 00:15:34.007907 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500129 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 00:15:34.007907 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500134 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 00:15:34.007907 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500136 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 00:15:34.007907 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500140 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 00:15:34.007907 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500143 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 00:15:34.007907 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500146 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 00:15:34.007907 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500149 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 00:15:34.007907 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500151 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 00:15:34.007907 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500154 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 00:15:34.007907 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500157 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 00:15:34.007907 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500159 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 00:15:34.007907 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500162 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 00:15:34.007907 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500166 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 00:15:34.007907 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500169 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 00:15:34.007907 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500172 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 00:15:34.008724 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500175 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 00:15:34.008724 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500178 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 00:15:34.008724 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500180 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 00:15:34.008724 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500183 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 00:15:34.008724 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500185 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 00:15:34.008724 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500188 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 00:15:34.008724 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500190 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 00:15:34.008724 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500192 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 00:15:34.008724 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500195 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 00:15:34.008724 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500199 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 00:15:34.008724 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500201 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 00:15:34.008724 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500205 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 00:15:34.008724 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500207 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 00:15:34.008724 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500210 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 00:15:34.008724 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500212 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 00:15:34.008724 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500215 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 00:15:34.008724 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500228 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 00:15:34.008724 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500231 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 00:15:34.008724 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500235 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 00:15:34.008724 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500238 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 00:15:34.009356 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500240 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 00:15:34.009356 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500243 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 00:15:34.009356 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500245 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 00:15:34.009356 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500248 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 00:15:34.009356 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500250 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 00:15:34.009356 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500253 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 00:15:34.009356 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500274 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 00:15:34.009356 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500277 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 00:15:34.009356 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500280 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 00:15:34.009356 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500282 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 00:15:34.009356 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500285 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 00:15:34.009356 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500288 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 00:15:34.009356 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500290 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 00:15:34.009356 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500293 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 00:15:34.009356 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500295 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 00:15:34.009356 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500298 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 00:15:34.009356 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500302 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 00:15:34.009356 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500305 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 00:15:34.009356 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500308 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 00:15:34.009356 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500311 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 00:15:34.009948 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500314 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 00:15:34.009948 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500318 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 00:15:34.009948 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500321 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 00:15:34.009948 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500324 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 00:15:34.009948 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500327 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 00:15:34.009948 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500330 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 00:15:34.009948 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500333 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 00:15:34.009948 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500336 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 00:15:34.009948 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500338 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 00:15:34.009948 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500341 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 00:15:34.009948 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500345 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 00:15:34.009948 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500348 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 00:15:34.009948 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500350 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 00:15:34.009948 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500353 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 00:15:34.009948 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500355 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 00:15:34.009948 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500358 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 00:15:34.009948 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500360 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 00:15:34.009948 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500363 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 00:15:34.009948 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.500366 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 00:15:34.010461 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.500961 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 00:15:34.010461 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.510681 2562 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 10 00:15:34.010461 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.510697 2562 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 10 00:15:34.010461 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.511605 2562 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 10 00:15:34.010461 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.511613 2562 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 10 00:15:34.010461 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511666 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 00:15:34.010461 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511672 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 00:15:34.010461 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511676 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 00:15:34.010461 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511679 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 00:15:34.010461 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511681 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 00:15:34.010461 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511684 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 00:15:34.010461 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511687 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 00:15:34.010461 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511690 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 00:15:34.010461 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511692 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 00:15:34.010461 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511695 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 00:15:34.010915 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511697 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 00:15:34.010915 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511700 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 00:15:34.010915 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511702 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 00:15:34.010915 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511705 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 00:15:34.010915 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511707 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 00:15:34.010915 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511710 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 00:15:34.010915 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511715 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 00:15:34.010915 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511719 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 00:15:34.010915 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511722 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 00:15:34.010915 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511726 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 00:15:34.010915 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511730 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 00:15:34.010915 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511733 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 00:15:34.010915 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511736 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 00:15:34.010915 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511739 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 00:15:34.010915 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511742 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 00:15:34.010915 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511744 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 00:15:34.010915 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511747 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 00:15:34.010915 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511750 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 00:15:34.010915 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511752 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 00:15:34.011435 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511756 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 00:15:34.011435 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511759 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 00:15:34.011435 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511762 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 00:15:34.011435 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511764 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 00:15:34.011435 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511767 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 00:15:34.011435 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511770 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 00:15:34.011435 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511772 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 00:15:34.011435 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511775 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 00:15:34.011435 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511777 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 00:15:34.011435 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511780 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 00:15:34.011435 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511782 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 00:15:34.011435 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511785 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 00:15:34.011435 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511788 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 00:15:34.011435 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511790 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 00:15:34.011435 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511793 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 00:15:34.011435 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511795 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 00:15:34.011435 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511798 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 00:15:34.011435 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511800 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 00:15:34.011435 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511804 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 00:15:34.011435 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511807 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 00:15:34.011968 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511812 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 00:15:34.011968 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511815 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 10 00:15:34.011968 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511818 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 00:15:34.011968 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511821 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 00:15:34.011968 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511824 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 00:15:34.011968 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511826 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 00:15:34.011968 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511828 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 00:15:34.011968 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511831 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 00:15:34.011968 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511834 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 00:15:34.011968 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511836 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 00:15:34.011968 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511839 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 00:15:34.011968 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511841 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 00:15:34.011968 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511844 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 00:15:34.011968 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511847 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 00:15:34.011968 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511852 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 00:15:34.011968 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511854 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 00:15:34.011968 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511857 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 00:15:34.011968 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511860 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 00:15:34.011968 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511862 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 00:15:34.011968 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511865 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 00:15:34.012527 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511867 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 00:15:34.012527 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511870 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 00:15:34.012527 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511872 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 00:15:34.012527 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511875 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 00:15:34.012527 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511877 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 00:15:34.012527 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511880 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 00:15:34.012527 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511882 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 00:15:34.012527 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511884 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 00:15:34.012527 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511887 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 00:15:34.012527 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511889 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 00:15:34.012527 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511892 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 00:15:34.012527 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511895 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 00:15:34.012527 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511899 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 00:15:34.012527 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511901 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 00:15:34.012527 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511904 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 00:15:34.012527 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511906 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 00:15:34.012527 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511909 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 00:15:34.012527 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511911 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 00:15:34.012527 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511914 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 00:15:34.012527 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511917 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 00:15:34.013096 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511919 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 00:15:34.013096 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511922 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 00:15:34.013096 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511924 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 00:15:34.013096 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511927 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 00:15:34.013096 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511930 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 00:15:34.013096 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511932 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 00:15:34.013096 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511937 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 00:15:34.013096 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511939 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 00:15:34.013096 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511942 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 00:15:34.013096 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.511944 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 00:15:34.013096 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.511949 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 00:15:34.013096 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512090 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 00:15:34.013096 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512096 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 00:15:34.013096 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512100 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 00:15:34.013496 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512103 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 00:15:34.013496 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512106 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 00:15:34.013496 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512108 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 00:15:34.013496 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512111 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 00:15:34.013496 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512115 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 00:15:34.013496 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512117 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 00:15:34.013496 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512120 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 00:15:34.013496 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512123 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 00:15:34.013496 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512126 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 00:15:34.013496 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512129 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 00:15:34.013496 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512133 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 00:15:34.013496 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512136 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 00:15:34.013496 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512138 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 00:15:34.013496 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512141 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 00:15:34.013496 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512143 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 00:15:34.013496 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512146 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 00:15:34.013496 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512148 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 00:15:34.013496 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512151 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 00:15:34.013496 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512153 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 00:15:34.014003 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512156 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 00:15:34.014003 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512159 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 00:15:34.014003 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512161 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 00:15:34.014003 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512163 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 00:15:34.014003 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512166 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 00:15:34.014003 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512168 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 00:15:34.014003 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512173 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 00:15:34.014003 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512176 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 00:15:34.014003 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512179 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 00:15:34.014003 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512182 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 00:15:34.014003 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512184 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 00:15:34.014003 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512187 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 00:15:34.014003 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512189 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 00:15:34.014003 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512192 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 10 00:15:34.014003 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512194 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 00:15:34.014003 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512197 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 00:15:34.014003 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512199 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 00:15:34.014003 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512202 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 00:15:34.014003 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512204 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 00:15:34.014003 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512206 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 00:15:34.014523 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512209 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 00:15:34.014523 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512211 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 00:15:34.014523 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512214 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 00:15:34.014523 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512229 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 00:15:34.014523 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512232 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 00:15:34.014523 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512235 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 00:15:34.014523 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512238 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 00:15:34.014523 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512241 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 00:15:34.014523 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512245 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 00:15:34.014523 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512248 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 00:15:34.014523 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512250 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 00:15:34.014523 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512253 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 00:15:34.014523 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512274 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 00:15:34.014523 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512276 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 00:15:34.014523 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512279 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 00:15:34.014523 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512282 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 00:15:34.014523 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512285 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 00:15:34.014523 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512287 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 00:15:34.014523 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512291 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 00:15:34.014523 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512297 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 00:15:34.015061 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512300 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 00:15:34.015061 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512303 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 00:15:34.015061 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512306 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 00:15:34.015061 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512308 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 00:15:34.015061 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512311 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 00:15:34.015061 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512314 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 00:15:34.015061 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512316 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 00:15:34.015061 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512321 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 00:15:34.015061 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512324 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 00:15:34.015061 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512327 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 00:15:34.015061 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512330 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 00:15:34.015061 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512333 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 00:15:34.015061 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512336 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 00:15:34.015061 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512338 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 00:15:34.015061 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512341 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 00:15:34.015061 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512345 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 00:15:34.015061 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512348 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 00:15:34.015061 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512351 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 00:15:34.015061 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512353 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 00:15:34.015061 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512356 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 00:15:34.015593 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512358 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 00:15:34.015593 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512361 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 00:15:34.015593 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512363 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 00:15:34.015593 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512366 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 00:15:34.015593 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512368 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 00:15:34.015593 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512371 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 00:15:34.015593 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512373 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 00:15:34.015593 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512376 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 00:15:34.015593 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512378 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 00:15:34.015593 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512381 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 00:15:34.015593 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512384 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 00:15:34.015593 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512387 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 00:15:34.015593 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512390 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 00:15:34.015593 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512393 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 00:15:34.015593 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512396 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 00:15:34.015593 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512399 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 00:15:34.015593 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:33.512402 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 00:15:34.016067 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.512407 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 00:15:34.016067 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.512508 2562 server.go:962] "Client rotation is on, will bootstrap in background" Jul 10 00:15:34.016067 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.515305 2562 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 10 00:15:34.016067 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.516771 2562 server.go:1019] "Starting client certificate rotation" Jul 10 00:15:34.016067 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.517801 2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 00:15:34.016067 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.517830 2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 00:15:34.016067 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.519641 2562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 00:15:34.016067 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.525278 2562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 00:15:34.016067 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.537692 2562 log.go:25] "Validated CRI v1 runtime API" Jul 10 00:15:34.016067 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.539884 2562 log.go:25] "Validated CRI v1 image API" Jul 10 00:15:34.016067 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.540657 2562 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 10 00:15:34.016067 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.543990 2562 fs.go:135] Filesystem UUIDs: map[62c52635-78a1-43e8-b543-505f1afc4342:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 ad03beb5-1378-4524-b89d-02d70cecc109:/dev/nvme0n1p4] Jul 10 00:15:34.016449 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.544004 2562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 10 00:15:34.016550 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.548954 2562 manager.go:216] Machine: {Timestamp:2026-07-10 00:15:33.547659844 +0000 UTC m=+0.365965328 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3198986 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec298b32f617c6f1ba848b9a427320ef SystemUUID:ec298b32-f617-c6f1-ba84-8b9a427320ef BootID:3c0d2f45-49bb-49ab-b302-b0a10ce09609 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:17:95:49:8e:cd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:17:95:49:8e:cd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a6:38:1c:24:d7:f4 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 10 00:15:34.016550 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.549097 2562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 10 00:15:34.016550 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.549173 2562 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 10 00:15:34.016550 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.550736 2562 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 10 00:15:34.016550 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.550778 2562 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-101.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 10 00:15:34.016550 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.551695 2562 topology_manager.go:138] "Creating topology manager with none policy" Jul 10 00:15:34.016550 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.551706 2562 container_manager_linux.go:309] "Creating device plugin manager" Jul 10 00:15:34.016550 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.551718 2562 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 00:15:34.016825 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.552695 2562 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 00:15:34.016825 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.553988 2562 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 10 00:15:34.016825 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.557162 2562 state_mem.go:36] "Initialized new in-memory state store" Jul 10 00:15:34.016825 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.557277 2562 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 10 00:15:34.016825 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.557305 2562 kubelet.go:486] "Attempting to sync node with API server" Jul 10 00:15:34.016825 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.557315 2562 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 10 00:15:34.016825 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.557329 2562 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 10 00:15:34.016825 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.557337 2562 kubelet.go:398] "Adding apiserver pod source" Jul 10 00:15:34.016825 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.557346 2562 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 10 00:15:34.016825 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.557687 2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 00:15:34.016825 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.557710 2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 00:15:34.016825 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.558896 2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 00:15:34.016825 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.559862 2562 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 10 00:15:34.016825 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.561812 2562 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 10 00:15:34.016825 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.561838 2562 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 10 00:15:34.016825 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.561981 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 10 00:15:34.016825 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.561993 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 10 00:15:34.016825 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.561999 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 10 00:15:34.016825 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.562005 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 10 00:15:34.016825 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.562013 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 10 00:15:34.016825 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.562020 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 10 00:15:34.016825 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.562674 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 10 00:15:34.016825 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.562681 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 10 00:15:34.016825 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.562688 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 10 00:15:34.016825 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.562693 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 10 00:15:34.016825 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.562701 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 10 00:15:34.016825 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.562709 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 10 00:15:34.016825 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.563550 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 10 00:15:34.016825 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.563558 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 10 00:15:34.016825 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.569051 2562 server.go:1268] "Started kubelet" Jul 10 00:15:34.016825 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.569258 2562 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 10 00:15:34.016825 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.569386 2562 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 10 00:15:34.016825 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.569622 2562 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 10 00:15:34.016825 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.571178 2562 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 10 00:15:34.016825 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.572784 2562 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-101.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 10 00:15:34.016825 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:33.572808 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-101.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 10 00:15:34.016825 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:33.572902 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 10 00:15:34.016825 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.573109 2562 server.go:310] "Adding debug handlers to kubelet server" Jul 10 00:15:34.016825 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:33.575997 2562 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 10 00:15:34.017872 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.576943 2562 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 10 00:15:34.017872 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.576953 2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 10 00:15:34.017872 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:33.577013 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-101.ec2.internal\" not found" Jul 10 00:15:34.017872 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.577568 2562 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 10 00:15:34.017872 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.577603 2562 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 10 00:15:34.017872 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.577619 2562 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 10 00:15:34.017872 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.577727 2562 reconstruct.go:97] "Volume reconstruction finished" Jul 10 00:15:34.017872 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.577736 2562 reconciler.go:29] "Reconciler: start to sync state" Jul 10 00:15:34.017872 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.577782 2562 factory.go:55] Registering systemd factory Jul 10 00:15:34.017872 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.577817 2562 factory.go:223] Registration of the systemd container factory successfully Jul 10 00:15:34.017872 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.577989 2562 factory.go:153] Registering CRI-O factory Jul 10 00:15:34.017872 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.578002 2562 factory.go:223] Registration of the crio container factory successfully Jul 10 00:15:34.017872 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.578047 2562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 10 00:15:34.017872 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.578063 2562 factory.go:103] Registering Raw factory Jul 10 00:15:34.017872 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.578101 2562 manager.go:1196] Started watching for new ooms in manager Jul 10 00:15:34.017872 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:33.577298 2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-101.ec2.internal.18c0c56f23a72f30 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-101.ec2.internal,UID:ip-10-0-135-101.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-101.ec2.internal,},FirstTimestamp:2026-07-10 00:15:33.568917296 +0000 UTC m=+0.387222787,LastTimestamp:2026-07-10 00:15:33.568917296 +0000 UTC m=+0.387222787,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-101.ec2.internal,}" Jul 10 00:15:34.017872 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.578436 2562 manager.go:349] Starting recovery of all containers Jul 10 00:15:34.017872 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:33.588350 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 10 00:15:34.017872 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:33.588366 2562 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-101.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 10 00:15:34.017872 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.589746 2562 manager.go:354] Recovery completed Jul 10 00:15:34.017872 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.594049 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 00:15:34.017872 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.596000 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-101.ec2.internal" event="NodeHasSufficientMemory" Jul 10 00:15:34.017872 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.596022 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-101.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 00:15:34.017872 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.596031 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-101.ec2.internal" event="NodeHasSufficientPID" Jul 10 00:15:34.017872 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.596931 2562 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 10 00:15:34.017872 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.596939 2562 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 10 00:15:34.017872 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.596953 2562 state_mem.go:36] "Initialized new in-memory state store" Jul 10 00:15:34.017872 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.596975 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 00:15:34.018913 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:33.598214 2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-101.ec2.internal.18c0c56f25449aba default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-101.ec2.internal,UID:ip-10-0-135-101.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-101.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-101.ec2.internal,},FirstTimestamp:2026-07-10 00:15:33.596011194 +0000 UTC m=+0.414316679,LastTimestamp:2026-07-10 00:15:33.596011194 +0000 UTC m=+0.414316679,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-101.ec2.internal,}" Jul 10 00:15:34.018913 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.598265 2562 policy_none.go:49] "None policy: Start" Jul 10 00:15:34.018913 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.598280 2562 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 10 00:15:34.018913 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.598293 2562 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 10 00:15:34.018913 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.599290 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-101.ec2.internal" event="NodeHasSufficientMemory" Jul 10 00:15:34.018913 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.599309 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-101.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 00:15:34.018913 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.599325 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-101.ec2.internal" event="NodeHasSufficientPID" Jul 10 00:15:34.018913 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.599411 2562 policy_none.go:47] "Start" Jul 10 00:15:34.018913 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.604190 2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-j8t2s" Jul 10 00:15:34.018913 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:33.607698 2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-101.ec2.internal.18c0c56f2544d7bb default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-101.ec2.internal,UID:ip-10-0-135-101.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-101.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-101.ec2.internal,},FirstTimestamp:2026-07-10 00:15:33.596026811 +0000 UTC m=+0.414332295,LastTimestamp:2026-07-10 00:15:33.596026811 +0000 UTC m=+0.414332295,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-101.ec2.internal,}" Jul 10 00:15:34.018913 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.609398 2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-j8t2s" Jul 10 00:15:34.018913 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:33.678086 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-101.ec2.internal\" not found" Jul 10 00:15:34.018913 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.685685 2562 manager.go:337] "Starting Device Plugin manager" Jul 10 00:15:34.018913 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:33.685724 2562 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 10 00:15:34.018913 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.685736 2562 server.go:87] "Starting device plugin registration server" Jul 10 00:15:34.018913 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.685968 2562 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 10 00:15:34.018913 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.685982 2562 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 10 00:15:34.018913 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.686071 2562 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 10 00:15:34.018913 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.686154 2562 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 10 00:15:34.018913 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.686162 2562 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 10 00:15:34.018913 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:33.686619 2562 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 10 00:15:34.018913 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:33.686661 2562 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-101.ec2.internal\" not found" Jul 10 00:15:34.018913 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.727676 2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 10 00:15:34.018913 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.728851 2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 10 00:15:34.018913 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.728883 2562 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 10 00:15:34.018913 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.728903 2562 kubelet.go:2448] "Starting kubelet main sync loop" Jul 10 00:15:34.018913 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:33.728935 2562 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 10 00:15:34.166720 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.731697 2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 00:15:34.166720 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.786070 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 00:15:34.166720 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.786945 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-101.ec2.internal" event="NodeHasSufficientMemory" Jul 10 00:15:34.166720 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.786969 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-101.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 00:15:34.166720 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.786979 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-101.ec2.internal" event="NodeHasSufficientPID" Jul 10 00:15:34.166720 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.787000 2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-101.ec2.internal" Jul 10 00:15:34.166720 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:33.794959 2562 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-135-101.ec2.internal\" not found" node="ip-10-0-135-101.ec2.internal" Jul 10 00:15:34.166720 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.797386 2562 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-101.ec2.internal" Jul 10 00:15:34.166720 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:33.797405 2562 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-101.ec2.internal\": node \"ip-10-0-135-101.ec2.internal\" not found" Jul 10 00:15:34.166720 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:33.824320 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-101.ec2.internal\" not found" Jul 10 00:15:34.166720 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.829362 2562 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-101.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-101.ec2.internal"] Jul 10 00:15:34.166720 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.829422 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 00:15:34.166720 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.831250 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-101.ec2.internal" event="NodeHasSufficientMemory" Jul 10 00:15:34.166720 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.831276 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-101.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 00:15:34.166720 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.831291 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-101.ec2.internal" event="NodeHasSufficientPID" Jul 10 00:15:34.166720 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.833396 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 00:15:34.166720 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.833531 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-101.ec2.internal" Jul 10 00:15:34.166720 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.833562 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 00:15:34.166720 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.834168 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-101.ec2.internal" event="NodeHasSufficientMemory" Jul 10 00:15:34.166720 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.834189 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-101.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 00:15:34.166720 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.834168 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-101.ec2.internal" event="NodeHasSufficientMemory" Jul 10 00:15:34.166720 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.834224 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-101.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 00:15:34.166720 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.834240 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-101.ec2.internal" event="NodeHasSufficientPID" Jul 10 00:15:34.166720 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.834199 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-101.ec2.internal" event="NodeHasSufficientPID" Jul 10 00:15:34.166720 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.835712 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-101.ec2.internal" Jul 10 00:15:34.166720 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.835741 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 00:15:34.166720 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.845666 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-101.ec2.internal" event="NodeHasSufficientMemory" Jul 10 00:15:34.166720 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.845695 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-101.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 00:15:34.166720 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.845709 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-101.ec2.internal" event="NodeHasSufficientPID" Jul 10 00:15:34.166720 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:33.863656 2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-101.ec2.internal\" not found" node="ip-10-0-135-101.ec2.internal" Jul 10 00:15:34.166720 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:33.869779 2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-101.ec2.internal\" not found" node="ip-10-0-135-101.ec2.internal" Jul 10 00:15:34.169428 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.880959 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/65368ef6b324db716ad2e852e6f60436-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-101.ec2.internal\" (UID: \"65368ef6b324db716ad2e852e6f60436\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-101.ec2.internal" Jul 10 00:15:34.169428 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:33.924956 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-101.ec2.internal\" not found" Jul 10 00:15:34.169428 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.981753 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/65368ef6b324db716ad2e852e6f60436-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-101.ec2.internal\" (UID: \"65368ef6b324db716ad2e852e6f60436\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-101.ec2.internal" Jul 10 00:15:34.169428 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.981769 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/65368ef6b324db716ad2e852e6f60436-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-101.ec2.internal\" (UID: \"65368ef6b324db716ad2e852e6f60436\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-101.ec2.internal" Jul 10 00:15:34.169428 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.981798 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/65368ef6b324db716ad2e852e6f60436-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-101.ec2.internal\" (UID: \"65368ef6b324db716ad2e852e6f60436\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-101.ec2.internal" Jul 10 00:15:34.169428 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:33.981840 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/29d40385cd1620022aeccb7b81ab99c1-config\") pod \"kube-apiserver-proxy-ip-10-0-135-101.ec2.internal\" (UID: \"29d40385cd1620022aeccb7b81ab99c1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-101.ec2.internal" Jul 10 00:15:34.169428 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:34.025942 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-101.ec2.internal\" not found" Jul 10 00:15:34.169428 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:34.082493 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/65368ef6b324db716ad2e852e6f60436-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-101.ec2.internal\" (UID: \"65368ef6b324db716ad2e852e6f60436\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-101.ec2.internal" Jul 10 00:15:34.169428 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:34.082517 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/29d40385cd1620022aeccb7b81ab99c1-config\") pod \"kube-apiserver-proxy-ip-10-0-135-101.ec2.internal\" (UID: \"29d40385cd1620022aeccb7b81ab99c1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-101.ec2.internal" Jul 10 00:15:34.169428 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:34.082561 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/29d40385cd1620022aeccb7b81ab99c1-config\") pod \"kube-apiserver-proxy-ip-10-0-135-101.ec2.internal\" (UID: \"29d40385cd1620022aeccb7b81ab99c1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-101.ec2.internal" Jul 10 00:15:34.169428 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:34.082588 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/65368ef6b324db716ad2e852e6f60436-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-101.ec2.internal\" (UID: \"65368ef6b324db716ad2e852e6f60436\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-101.ec2.internal" Jul 10 00:15:34.169428 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:34.126611 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-101.ec2.internal\" not found" Jul 10 00:15:34.169428 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:34.166415 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-101.ec2.internal" Jul 10 00:15:34.327289 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:34.173221 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-101.ec2.internal" Jul 10 00:15:34.327289 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:34.227244 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-101.ec2.internal\" not found" Jul 10 00:15:34.328336 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:34.327861 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-101.ec2.internal\" not found" Jul 10 00:15:34.469012 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:34.428328 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-101.ec2.internal\" not found" Jul 10 00:15:34.630960 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:34.487812 2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 00:15:34.630960 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:34.516875 2562 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 10 00:15:34.630960 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:34.517024 2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 10 00:15:34.630960 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:34.517054 2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 10 00:15:34.630960 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:34.529019 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-101.ec2.internal\" not found" Jul 10 00:15:34.630960 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:34.613571 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-09 00:10:34 +0000 UTC" deadline="2027-12-03 01:03:23.088412093 +0000 UTC" Jul 10 00:15:34.630960 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:34.613605 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12264h47m48.474811149s" Jul 10 00:15:34.630960 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:34.630111 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-101.ec2.internal\" not found" Jul 10 00:15:34.736329 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:34.647498 2562 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 00:15:34.736329 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:34.663368 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 00:15:34.736329 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:34.667310 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-101.ec2.internal" event="NodeHasSufficientMemory" Jul 10 00:15:34.736329 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:34.667332 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-101.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 00:15:34.736329 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:34.667343 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-101.ec2.internal" event="NodeHasSufficientPID" Jul 10 00:15:34.736329 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:34.730355 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-101.ec2.internal\" not found" Jul 10 00:15:34.874113 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:34.799991 2562 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 00:15:34.874113 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:34.834956 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65368ef6b324db716ad2e852e6f60436.slice/crio-dbe19180d285d317eb6a53072920d6bf0f0a5d37f8b99b2dffdf256bb5d301ae WatchSource:0}: Error finding container dbe19180d285d317eb6a53072920d6bf0f0a5d37f8b99b2dffdf256bb5d301ae: Status 404 returned error can't find the container with id dbe19180d285d317eb6a53072920d6bf0f0a5d37f8b99b2dffdf256bb5d301ae Jul 10 00:15:34.874113 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:34.845984 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 00:15:35.080682 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:34.878223 2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-101.ec2.internal" Jul 10 00:15:35.080682 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:34.891752 2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 00:15:35.080682 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:34.892852 2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-101.ec2.internal" Jul 10 00:15:35.080682 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:34.900502 2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 00:15:35.699868 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.558645 2562 apiserver.go:52] "Watching apiserver" Jul 10 00:15:35.699868 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.564417 2562 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 10 00:15:35.699868 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.565737 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-5zlgk","openshift-multus/network-metrics-daemon-x8v89","openshift-network-diagnostics/network-check-target-qbbd4","kube-system/kube-apiserver-proxy-ip-10-0-135-101.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6xw4","openshift-image-registry/node-ca-gjk2r","openshift-multus/multus-additional-cni-plugins-5kkqc","openshift-network-operator/iptables-alerter-f55lr","openshift-ovn-kubernetes/ovnkube-node-h5g9w","kube-system/global-pull-secret-syncer-nkcmr","kube-system/konnectivity-agent-shx2p","openshift-cluster-node-tuning-operator/tuned-n5xgr","openshift-dns/node-resolver-2mpf9","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-101.ec2.internal"] Jul 10 00:15:35.699868 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.578094 2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 10 00:15:35.699868 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.578347 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5zlgk" Jul 10 00:15:35.699868 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.579737 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x8v89" Jul 10 00:15:35.699868 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:35.579856 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x8v89" podUID="62933c8a-818d-4580-a6c9-6d26dbdfd50d" Jul 10 00:15:35.699868 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.580640 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 10 00:15:35.699868 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.580747 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-wf5ff\"" Jul 10 00:15:35.699868 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.580969 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 10 00:15:35.699868 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.581052 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 10 00:15:35.699868 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.581057 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 10 00:15:35.699868 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.581357 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2mpf9" Jul 10 00:15:35.699868 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.582634 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-n5xgr" Jul 10 00:15:35.699868 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.583066 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 10 00:15:35.699868 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.583312 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tp8qp\"" Jul 10 00:15:35.699868 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.583404 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 10 00:15:35.699868 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.583852 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-f55lr" Jul 10 00:15:35.699868 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.584675 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 10 00:15:35.699868 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.584755 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 10 00:15:35.699868 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.584770 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pk5p7\"" Jul 10 00:15:35.699868 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.585134 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gjk2r" Jul 10 00:15:35.699868 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.585642 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 10 00:15:35.699868 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.585907 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 10 00:15:35.699868 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.586263 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qsvl8\"" Jul 10 00:15:35.699868 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.586320 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 10 00:15:35.699868 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.586714 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5kkqc" Jul 10 00:15:35.699868 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.586788 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 10 00:15:35.701962 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.587078 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-qs4pw\"" Jul 10 00:15:35.701962 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.587184 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 10 00:15:35.701962 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.587796 2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 00:15:35.701962 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.588057 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6xw4" Jul 10 00:15:35.701962 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.589298 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 10 00:15:35.701962 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.589692 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 10 00:15:35.701962 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.595929 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0318fc8c-a177-4995-a1b6-80ccf2cf11cd-multus-socket-dir-parent\") pod \"multus-5zlgk\" (UID: \"0318fc8c-a177-4995-a1b6-80ccf2cf11cd\") " pod="openshift-multus/multus-5zlgk" Jul 10 00:15:35.701962 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.595977 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0318fc8c-a177-4995-a1b6-80ccf2cf11cd-host-run-k8s-cni-cncf-io\") pod \"multus-5zlgk\" (UID: \"0318fc8c-a177-4995-a1b6-80ccf2cf11cd\") " pod="openshift-multus/multus-5zlgk" Jul 10 00:15:35.701962 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.596014 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0318fc8c-a177-4995-a1b6-80ccf2cf11cd-host-var-lib-cni-bin\") pod \"multus-5zlgk\" (UID: \"0318fc8c-a177-4995-a1b6-80ccf2cf11cd\") " pod="openshift-multus/multus-5zlgk" Jul 10 00:15:35.701962 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.596049 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0318fc8c-a177-4995-a1b6-80ccf2cf11cd-hostroot\") pod \"multus-5zlgk\" (UID: \"0318fc8c-a177-4995-a1b6-80ccf2cf11cd\") " pod="openshift-multus/multus-5zlgk" Jul 10 00:15:35.701962 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.596081 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f78vw\" (UniqueName: \"kubernetes.io/projected/62933c8a-818d-4580-a6c9-6d26dbdfd50d-kube-api-access-f78vw\") pod \"network-metrics-daemon-x8v89\" (UID: \"62933c8a-818d-4580-a6c9-6d26dbdfd50d\") " pod="openshift-multus/network-metrics-daemon-x8v89" Jul 10 00:15:35.701962 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.596173 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0318fc8c-a177-4995-a1b6-80ccf2cf11cd-cnibin\") pod \"multus-5zlgk\" (UID: \"0318fc8c-a177-4995-a1b6-80ccf2cf11cd\") " pod="openshift-multus/multus-5zlgk" Jul 10 00:15:35.701962 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.596209 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0318fc8c-a177-4995-a1b6-80ccf2cf11cd-multus-conf-dir\") pod \"multus-5zlgk\" (UID: \"0318fc8c-a177-4995-a1b6-80ccf2cf11cd\") " pod="openshift-multus/multus-5zlgk" Jul 10 00:15:35.701962 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.596235 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 10 00:15:35.701962 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.596241 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0318fc8c-a177-4995-a1b6-80ccf2cf11cd-multus-daemon-config\") pod \"multus-5zlgk\" (UID: \"0318fc8c-a177-4995-a1b6-80ccf2cf11cd\") " pod="openshift-multus/multus-5zlgk" Jul 10 00:15:35.701962 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.596346 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9mfc\" (UniqueName: \"kubernetes.io/projected/0318fc8c-a177-4995-a1b6-80ccf2cf11cd-kube-api-access-p9mfc\") pod \"multus-5zlgk\" (UID: \"0318fc8c-a177-4995-a1b6-80ccf2cf11cd\") " pod="openshift-multus/multus-5zlgk" Jul 10 00:15:35.701962 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.596395 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0318fc8c-a177-4995-a1b6-80ccf2cf11cd-os-release\") pod \"multus-5zlgk\" (UID: \"0318fc8c-a177-4995-a1b6-80ccf2cf11cd\") " pod="openshift-multus/multus-5zlgk" Jul 10 00:15:35.701962 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.596426 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0318fc8c-a177-4995-a1b6-80ccf2cf11cd-cni-binary-copy\") pod \"multus-5zlgk\" (UID: \"0318fc8c-a177-4995-a1b6-80ccf2cf11cd\") " pod="openshift-multus/multus-5zlgk" Jul 10 00:15:35.702748 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.596456 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0318fc8c-a177-4995-a1b6-80ccf2cf11cd-host-var-lib-kubelet\") pod \"multus-5zlgk\" (UID: \"0318fc8c-a177-4995-a1b6-80ccf2cf11cd\") " pod="openshift-multus/multus-5zlgk" Jul 10 00:15:35.702748 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.596480 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-t5l8d\"" Jul 10 00:15:35.702748 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.596490 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0318fc8c-a177-4995-a1b6-80ccf2cf11cd-host-run-netns\") pod \"multus-5zlgk\" (UID: \"0318fc8c-a177-4995-a1b6-80ccf2cf11cd\") " pod="openshift-multus/multus-5zlgk" Jul 10 00:15:35.702748 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.596521 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0318fc8c-a177-4995-a1b6-80ccf2cf11cd-host-var-lib-cni-multus\") pod \"multus-5zlgk\" (UID: \"0318fc8c-a177-4995-a1b6-80ccf2cf11cd\") " pod="openshift-multus/multus-5zlgk" Jul 10 00:15:35.702748 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.596548 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0318fc8c-a177-4995-a1b6-80ccf2cf11cd-host-run-multus-certs\") pod \"multus-5zlgk\" (UID: \"0318fc8c-a177-4995-a1b6-80ccf2cf11cd\") " pod="openshift-multus/multus-5zlgk" Jul 10 00:15:35.702748 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.596580 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0318fc8c-a177-4995-a1b6-80ccf2cf11cd-etc-kubernetes\") pod \"multus-5zlgk\" (UID: \"0318fc8c-a177-4995-a1b6-80ccf2cf11cd\") " pod="openshift-multus/multus-5zlgk" Jul 10 00:15:35.702748 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.596694 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62933c8a-818d-4580-a6c9-6d26dbdfd50d-metrics-certs\") pod \"network-metrics-daemon-x8v89\" (UID: \"62933c8a-818d-4580-a6c9-6d26dbdfd50d\") " pod="openshift-multus/network-metrics-daemon-x8v89" Jul 10 00:15:35.702748 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.596730 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0318fc8c-a177-4995-a1b6-80ccf2cf11cd-system-cni-dir\") pod \"multus-5zlgk\" (UID: \"0318fc8c-a177-4995-a1b6-80ccf2cf11cd\") " pod="openshift-multus/multus-5zlgk" Jul 10 00:15:35.702748 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.596763 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0318fc8c-a177-4995-a1b6-80ccf2cf11cd-multus-cni-dir\") pod \"multus-5zlgk\" (UID: \"0318fc8c-a177-4995-a1b6-80ccf2cf11cd\") " pod="openshift-multus/multus-5zlgk" Jul 10 00:15:35.702748 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.597042 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-79x2w\"" Jul 10 00:15:35.702748 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.597214 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 10 00:15:35.702748 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.598113 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:15:35.702748 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.599151 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 10 00:15:35.702748 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.599211 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 10 00:15:35.702748 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.600298 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9lc9q\"" Jul 10 00:15:35.702748 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.600720 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 10 00:15:35.702748 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.600808 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 10 00:15:35.702748 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.600723 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 10 00:15:35.702748 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.601008 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nkcmr" Jul 10 00:15:35.702748 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.601042 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 10 00:15:35.704073 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:35.601078 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nkcmr" podUID="451130fe-547b-40c5-a7dc-412d57fb5db5" Jul 10 00:15:35.704073 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.601120 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 10 00:15:35.704073 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.602217 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-shx2p" Jul 10 00:15:35.704073 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.602713 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 10 00:15:35.704073 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.603828 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qbbd4" Jul 10 00:15:35.704073 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:35.603911 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qbbd4" podUID="6fe3e675-7e07-49d2-a664-a47d434dfd3b" Jul 10 00:15:35.704073 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.604276 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 10 00:15:35.704073 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.604532 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-c29g4\"" Jul 10 00:15:35.704073 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.604644 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 10 00:15:35.704073 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:35.606342 2562 log.go:32] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = creating pod sandbox with name \"k8s_kube-apiserver-proxy-ip-10-0-135-101.ec2.internal_kube-system_29d40385cd1620022aeccb7b81ab99c1_0\": unable to pull image or OCI artifact: pull image err: copying system image from manifest list: parsing image configuration: fetching blob: received unexpected HTTP status: 503 Service Unavailable; artifact err: provided artifact is a container image" Jul 10 00:15:35.704073 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:35.606418 2562 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = creating pod sandbox with name \"k8s_kube-apiserver-proxy-ip-10-0-135-101.ec2.internal_kube-system_29d40385cd1620022aeccb7b81ab99c1_0\": unable to pull image or OCI artifact: pull image err: copying system image from manifest list: parsing image configuration: fetching blob: received unexpected HTTP status: 503 Service Unavailable; artifact err: provided artifact is a container image" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-101.ec2.internal" Jul 10 00:15:35.704073 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:35.606437 2562 kuberuntime_manager.go:1353] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = creating pod sandbox with name \"k8s_kube-apiserver-proxy-ip-10-0-135-101.ec2.internal_kube-system_29d40385cd1620022aeccb7b81ab99c1_0\": unable to pull image or OCI artifact: pull image err: copying system image from manifest list: parsing image configuration: fetching blob: received unexpected HTTP status: 503 Service Unavailable; artifact err: provided artifact is a container image" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-101.ec2.internal" Jul 10 00:15:35.704073 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:35.606524 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"kube-apiserver-proxy-ip-10-0-135-101.ec2.internal_kube-system(29d40385cd1620022aeccb7b81ab99c1)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"kube-apiserver-proxy-ip-10-0-135-101.ec2.internal_kube-system(29d40385cd1620022aeccb7b81ab99c1)\\\": rpc error: code = Unknown desc = creating pod sandbox with name \\\"k8s_kube-apiserver-proxy-ip-10-0-135-101.ec2.internal_kube-system_29d40385cd1620022aeccb7b81ab99c1_0\\\": unable to pull image or OCI artifact: pull image err: copying system image from manifest list: parsing image configuration: fetching blob: received unexpected HTTP status: 503 Service Unavailable; artifact err: provided artifact is a container image\"" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-101.ec2.internal" podUID="29d40385cd1620022aeccb7b81ab99c1" Jul 10 00:15:35.704073 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.616829 2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-v4xnk" Jul 10 00:15:35.704073 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.626524 2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-v4xnk" Jul 10 00:15:35.704073 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.679563 2562 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 10 00:15:35.704816 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.696940 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/269e1974-6855-40f9-9f47-3dc20f729981-device-dir\") pod \"aws-ebs-csi-driver-node-g6xw4\" (UID: \"269e1974-6855-40f9-9f47-3dc20f729981\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6xw4" Jul 10 00:15:35.704816 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.696977 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/08be9fc2-0ae2-45bc-96d1-5306a46e1357-host\") pod \"tuned-n5xgr\" (UID: \"08be9fc2-0ae2-45bc-96d1-5306a46e1357\") " pod="openshift-cluster-node-tuning-operator/tuned-n5xgr" Jul 10 00:15:35.704816 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.697001 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0318fc8c-a177-4995-a1b6-80ccf2cf11cd-host-var-lib-cni-multus\") pod \"multus-5zlgk\" (UID: \"0318fc8c-a177-4995-a1b6-80ccf2cf11cd\") " pod="openshift-multus/multus-5zlgk" Jul 10 00:15:35.704816 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.697017 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0da6eed2-eb77-4e83-bb07-a6163eaca53a-hosts-file\") pod \"node-resolver-2mpf9\" (UID: \"0da6eed2-eb77-4e83-bb07-a6163eaca53a\") " pod="openshift-dns/node-resolver-2mpf9" Jul 10 00:15:35.704816 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.697032 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32-etc-openvswitch\") pod \"ovnkube-node-h5g9w\" (UID: \"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:15:35.704816 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.697078 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32-log-socket\") pod \"ovnkube-node-h5g9w\" (UID: \"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:15:35.704816 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.697108 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0318fc8c-a177-4995-a1b6-80ccf2cf11cd-host-var-lib-cni-multus\") pod \"multus-5zlgk\" (UID: \"0318fc8c-a177-4995-a1b6-80ccf2cf11cd\") " pod="openshift-multus/multus-5zlgk" Jul 10 00:15:35.704816 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.697123 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32-host-cni-bin\") pod \"ovnkube-node-h5g9w\" (UID: \"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:15:35.704816 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.697142 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32-ovn-node-metrics-cert\") pod \"ovnkube-node-h5g9w\" (UID: \"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:15:35.704816 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.697180 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wswmt\" (UniqueName: \"kubernetes.io/projected/0da6eed2-eb77-4e83-bb07-a6163eaca53a-kube-api-access-wswmt\") pod \"node-resolver-2mpf9\" (UID: \"0da6eed2-eb77-4e83-bb07-a6163eaca53a\") " pod="openshift-dns/node-resolver-2mpf9" Jul 10 00:15:35.704816 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.697219 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62933c8a-818d-4580-a6c9-6d26dbdfd50d-metrics-certs\") pod \"network-metrics-daemon-x8v89\" (UID: \"62933c8a-818d-4580-a6c9-6d26dbdfd50d\") " pod="openshift-multus/network-metrics-daemon-x8v89" Jul 10 00:15:35.704816 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.697258 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0318fc8c-a177-4995-a1b6-80ccf2cf11cd-system-cni-dir\") pod \"multus-5zlgk\" (UID: \"0318fc8c-a177-4995-a1b6-80ccf2cf11cd\") " pod="openshift-multus/multus-5zlgk" Jul 10 00:15:35.704816 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:35.697355 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:15:35.704816 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.697362 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0318fc8c-a177-4995-a1b6-80ccf2cf11cd-multus-cni-dir\") pod \"multus-5zlgk\" (UID: \"0318fc8c-a177-4995-a1b6-80ccf2cf11cd\") " pod="openshift-multus/multus-5zlgk" Jul 10 00:15:35.704816 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.697408 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0318fc8c-a177-4995-a1b6-80ccf2cf11cd-system-cni-dir\") pod \"multus-5zlgk\" (UID: \"0318fc8c-a177-4995-a1b6-80ccf2cf11cd\") " pod="openshift-multus/multus-5zlgk" Jul 10 00:15:35.705497 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.697410 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32-host-run-netns\") pod \"ovnkube-node-h5g9w\" (UID: \"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:15:35.705497 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:35.697564 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62933c8a-818d-4580-a6c9-6d26dbdfd50d-metrics-certs podName:62933c8a-818d-4580-a6c9-6d26dbdfd50d nodeName:}" failed. No retries permitted until 2026-07-10 00:15:36.197492315 +0000 UTC m=+3.015797813 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/62933c8a-818d-4580-a6c9-6d26dbdfd50d-metrics-certs") pod "network-metrics-daemon-x8v89" (UID: "62933c8a-818d-4580-a6c9-6d26dbdfd50d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:15:35.705497 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.697580 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0318fc8c-a177-4995-a1b6-80ccf2cf11cd-multus-cni-dir\") pod \"multus-5zlgk\" (UID: \"0318fc8c-a177-4995-a1b6-80ccf2cf11cd\") " pod="openshift-multus/multus-5zlgk" Jul 10 00:15:35.705497 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.697592 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h5g9w\" (UID: \"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:15:35.705497 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.697627 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/08be9fc2-0ae2-45bc-96d1-5306a46e1357-etc-sysconfig\") pod \"tuned-n5xgr\" (UID: \"08be9fc2-0ae2-45bc-96d1-5306a46e1357\") " pod="openshift-cluster-node-tuning-operator/tuned-n5xgr" Jul 10 00:15:35.705497 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.697653 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm2j9\" (UniqueName: \"kubernetes.io/projected/08be9fc2-0ae2-45bc-96d1-5306a46e1357-kube-api-access-pm2j9\") pod \"tuned-n5xgr\" (UID: \"08be9fc2-0ae2-45bc-96d1-5306a46e1357\") " pod="openshift-cluster-node-tuning-operator/tuned-n5xgr" Jul 10 00:15:35.705497 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.697677 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a88d645c-45e4-4592-b451-04eea24d3afb-os-release\") pod \"multus-additional-cni-plugins-5kkqc\" (UID: \"a88d645c-45e4-4592-b451-04eea24d3afb\") " pod="openshift-multus/multus-additional-cni-plugins-5kkqc" Jul 10 00:15:35.705497 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.697707 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a88d645c-45e4-4592-b451-04eea24d3afb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5kkqc\" (UID: \"a88d645c-45e4-4592-b451-04eea24d3afb\") " pod="openshift-multus/multus-additional-cni-plugins-5kkqc" Jul 10 00:15:35.705497 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.697735 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/269e1974-6855-40f9-9f47-3dc20f729981-registration-dir\") pod \"aws-ebs-csi-driver-node-g6xw4\" (UID: \"269e1974-6855-40f9-9f47-3dc20f729981\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6xw4" Jul 10 00:15:35.705497 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.697758 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32-host-run-ovn-kubernetes\") pod \"ovnkube-node-h5g9w\" (UID: \"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:15:35.705497 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.697780 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08be9fc2-0ae2-45bc-96d1-5306a46e1357-etc-kubernetes\") pod \"tuned-n5xgr\" (UID: \"08be9fc2-0ae2-45bc-96d1-5306a46e1357\") " pod="openshift-cluster-node-tuning-operator/tuned-n5xgr" Jul 10 00:15:35.705497 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.697799 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/86b0dedd-e7fd-44d5-9663-ed7335d94063-iptables-alerter-script\") pod \"iptables-alerter-f55lr\" (UID: \"86b0dedd-e7fd-44d5-9663-ed7335d94063\") " pod="openshift-network-operator/iptables-alerter-f55lr" Jul 10 00:15:35.705497 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.697840 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/451130fe-547b-40c5-a7dc-412d57fb5db5-kubelet-config\") pod \"global-pull-secret-syncer-nkcmr\" (UID: \"451130fe-547b-40c5-a7dc-412d57fb5db5\") " pod="kube-system/global-pull-secret-syncer-nkcmr" Jul 10 00:15:35.706167 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.697858 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0318fc8c-a177-4995-a1b6-80ccf2cf11cd-os-release\") pod \"multus-5zlgk\" (UID: \"0318fc8c-a177-4995-a1b6-80ccf2cf11cd\") " pod="openshift-multus/multus-5zlgk" Jul 10 00:15:35.706167 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.697883 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/269e1974-6855-40f9-9f47-3dc20f729981-sys-fs\") pod \"aws-ebs-csi-driver-node-g6xw4\" (UID: \"269e1974-6855-40f9-9f47-3dc20f729981\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6xw4" Jul 10 00:15:35.706167 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.697909 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32-run-openvswitch\") pod \"ovnkube-node-h5g9w\" (UID: \"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:15:35.706167 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.697932 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/03b7fa42-b493-40ce-a0d1-ce267379f3df-serviceca\") pod \"node-ca-gjk2r\" (UID: \"03b7fa42-b493-40ce-a0d1-ce267379f3df\") " pod="openshift-image-registry/node-ca-gjk2r" Jul 10 00:15:35.706167 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.697959 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32-node-log\") pod \"ovnkube-node-h5g9w\" (UID: \"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:15:35.706167 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.697984 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32-env-overrides\") pod \"ovnkube-node-h5g9w\" (UID: \"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:15:35.706167 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.698008 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phkgc\" (UniqueName: \"kubernetes.io/projected/6fe3e675-7e07-49d2-a664-a47d434dfd3b-kube-api-access-phkgc\") pod \"network-check-target-qbbd4\" (UID: \"6fe3e675-7e07-49d2-a664-a47d434dfd3b\") " pod="openshift-network-diagnostics/network-check-target-qbbd4" Jul 10 00:15:35.706167 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.698007 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0318fc8c-a177-4995-a1b6-80ccf2cf11cd-os-release\") pod \"multus-5zlgk\" (UID: \"0318fc8c-a177-4995-a1b6-80ccf2cf11cd\") " pod="openshift-multus/multus-5zlgk" Jul 10 00:15:35.706167 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.698034 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0318fc8c-a177-4995-a1b6-80ccf2cf11cd-multus-socket-dir-parent\") pod \"multus-5zlgk\" (UID: \"0318fc8c-a177-4995-a1b6-80ccf2cf11cd\") " pod="openshift-multus/multus-5zlgk" Jul 10 00:15:35.706167 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.698060 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0318fc8c-a177-4995-a1b6-80ccf2cf11cd-host-var-lib-cni-bin\") pod \"multus-5zlgk\" (UID: \"0318fc8c-a177-4995-a1b6-80ccf2cf11cd\") " pod="openshift-multus/multus-5zlgk" Jul 10 00:15:35.706167 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.698087 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0318fc8c-a177-4995-a1b6-80ccf2cf11cd-hostroot\") pod \"multus-5zlgk\" (UID: \"0318fc8c-a177-4995-a1b6-80ccf2cf11cd\") " pod="openshift-multus/multus-5zlgk" Jul 10 00:15:35.706167 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.698110 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08be9fc2-0ae2-45bc-96d1-5306a46e1357-tmp\") pod \"tuned-n5xgr\" (UID: \"08be9fc2-0ae2-45bc-96d1-5306a46e1357\") " pod="openshift-cluster-node-tuning-operator/tuned-n5xgr" Jul 10 00:15:35.706167 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.698127 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0318fc8c-a177-4995-a1b6-80ccf2cf11cd-multus-socket-dir-parent\") pod \"multus-5zlgk\" (UID: \"0318fc8c-a177-4995-a1b6-80ccf2cf11cd\") " pod="openshift-multus/multus-5zlgk" Jul 10 00:15:35.706167 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.698140 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0318fc8c-a177-4995-a1b6-80ccf2cf11cd-host-var-lib-cni-bin\") pod \"multus-5zlgk\" (UID: \"0318fc8c-a177-4995-a1b6-80ccf2cf11cd\") " pod="openshift-multus/multus-5zlgk" Jul 10 00:15:35.707001 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.698133 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32-systemd-units\") pod \"ovnkube-node-h5g9w\" (UID: \"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:15:35.707001 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.698181 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0318fc8c-a177-4995-a1b6-80ccf2cf11cd-hostroot\") pod \"multus-5zlgk\" (UID: \"0318fc8c-a177-4995-a1b6-80ccf2cf11cd\") " pod="openshift-multus/multus-5zlgk" Jul 10 00:15:35.707001 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.698191 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32-host-slash\") pod \"ovnkube-node-h5g9w\" (UID: \"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:15:35.707001 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.698284 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32-run-systemd\") pod \"ovnkube-node-h5g9w\" (UID: \"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:15:35.707001 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.698309 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32-run-ovn\") pod \"ovnkube-node-h5g9w\" (UID: \"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:15:35.707001 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.698675 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0318fc8c-a177-4995-a1b6-80ccf2cf11cd-cnibin\") pod \"multus-5zlgk\" (UID: \"0318fc8c-a177-4995-a1b6-80ccf2cf11cd\") " pod="openshift-multus/multus-5zlgk" Jul 10 00:15:35.707001 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.698757 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0318fc8c-a177-4995-a1b6-80ccf2cf11cd-cnibin\") pod \"multus-5zlgk\" (UID: \"0318fc8c-a177-4995-a1b6-80ccf2cf11cd\") " pod="openshift-multus/multus-5zlgk" Jul 10 00:15:35.707001 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.699451 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dfc778a3-c60f-4f5e-8b6f-3e613228a7f9-konnectivity-ca\") pod \"konnectivity-agent-shx2p\" (UID: \"dfc778a3-c60f-4f5e-8b6f-3e613228a7f9\") " pod="kube-system/konnectivity-agent-shx2p" Jul 10 00:15:35.707001 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.699913 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a88d645c-45e4-4592-b451-04eea24d3afb-cni-binary-copy\") pod \"multus-additional-cni-plugins-5kkqc\" (UID: \"a88d645c-45e4-4592-b451-04eea24d3afb\") " pod="openshift-multus/multus-additional-cni-plugins-5kkqc" Jul 10 00:15:35.707001 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.700372 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0318fc8c-a177-4995-a1b6-80ccf2cf11cd-cni-binary-copy\") pod \"multus-5zlgk\" (UID: \"0318fc8c-a177-4995-a1b6-80ccf2cf11cd\") " pod="openshift-multus/multus-5zlgk" Jul 10 00:15:35.707001 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.700610 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0318fc8c-a177-4995-a1b6-80ccf2cf11cd-host-var-lib-kubelet\") pod \"multus-5zlgk\" (UID: \"0318fc8c-a177-4995-a1b6-80ccf2cf11cd\") " pod="openshift-multus/multus-5zlgk" Jul 10 00:15:35.707001 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.700616 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0318fc8c-a177-4995-a1b6-80ccf2cf11cd-cni-binary-copy\") pod \"multus-5zlgk\" (UID: \"0318fc8c-a177-4995-a1b6-80ccf2cf11cd\") " pod="openshift-multus/multus-5zlgk" Jul 10 00:15:35.707001 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.700628 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dfc778a3-c60f-4f5e-8b6f-3e613228a7f9-agent-certs\") pod \"konnectivity-agent-shx2p\" (UID: \"dfc778a3-c60f-4f5e-8b6f-3e613228a7f9\") " pod="kube-system/konnectivity-agent-shx2p" Jul 10 00:15:35.707001 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.700650 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a88d645c-45e4-4592-b451-04eea24d3afb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5kkqc\" (UID: \"a88d645c-45e4-4592-b451-04eea24d3afb\") " pod="openshift-multus/multus-additional-cni-plugins-5kkqc" Jul 10 00:15:35.707763 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.700667 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/269e1974-6855-40f9-9f47-3dc20f729981-socket-dir\") pod \"aws-ebs-csi-driver-node-g6xw4\" (UID: \"269e1974-6855-40f9-9f47-3dc20f729981\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6xw4" Jul 10 00:15:35.707763 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.700681 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/08be9fc2-0ae2-45bc-96d1-5306a46e1357-run\") pod \"tuned-n5xgr\" (UID: \"08be9fc2-0ae2-45bc-96d1-5306a46e1357\") " pod="openshift-cluster-node-tuning-operator/tuned-n5xgr" Jul 10 00:15:35.707763 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.700687 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0318fc8c-a177-4995-a1b6-80ccf2cf11cd-host-var-lib-kubelet\") pod \"multus-5zlgk\" (UID: \"0318fc8c-a177-4995-a1b6-80ccf2cf11cd\") " pod="openshift-multus/multus-5zlgk" Jul 10 00:15:35.707763 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.700703 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/08be9fc2-0ae2-45bc-96d1-5306a46e1357-var-lib-kubelet\") pod \"tuned-n5xgr\" (UID: \"08be9fc2-0ae2-45bc-96d1-5306a46e1357\") " pod="openshift-cluster-node-tuning-operator/tuned-n5xgr" Jul 10 00:15:35.707763 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.700761 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/451130fe-547b-40c5-a7dc-412d57fb5db5-original-pull-secret\") pod \"global-pull-secret-syncer-nkcmr\" (UID: \"451130fe-547b-40c5-a7dc-412d57fb5db5\") " pod="kube-system/global-pull-secret-syncer-nkcmr" Jul 10 00:15:35.707763 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.700810 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0318fc8c-a177-4995-a1b6-80ccf2cf11cd-host-run-netns\") pod \"multus-5zlgk\" (UID: \"0318fc8c-a177-4995-a1b6-80ccf2cf11cd\") " pod="openshift-multus/multus-5zlgk" Jul 10 00:15:35.707763 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.700841 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0318fc8c-a177-4995-a1b6-80ccf2cf11cd-host-run-multus-certs\") pod \"multus-5zlgk\" (UID: \"0318fc8c-a177-4995-a1b6-80ccf2cf11cd\") " pod="openshift-multus/multus-5zlgk" Jul 10 00:15:35.707763 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.700853 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0318fc8c-a177-4995-a1b6-80ccf2cf11cd-host-run-netns\") pod \"multus-5zlgk\" (UID: \"0318fc8c-a177-4995-a1b6-80ccf2cf11cd\") " pod="openshift-multus/multus-5zlgk" Jul 10 00:15:35.707763 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.700868 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32-var-lib-openvswitch\") pod \"ovnkube-node-h5g9w\" (UID: \"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:15:35.707763 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.700898 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32-host-cni-netd\") pod \"ovnkube-node-h5g9w\" (UID: \"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:15:35.707763 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.700932 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/08be9fc2-0ae2-45bc-96d1-5306a46e1357-lib-modules\") pod \"tuned-n5xgr\" (UID: \"08be9fc2-0ae2-45bc-96d1-5306a46e1357\") " pod="openshift-cluster-node-tuning-operator/tuned-n5xgr" Jul 10 00:15:35.707763 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.700950 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h769c\" (UniqueName: \"kubernetes.io/projected/86b0dedd-e7fd-44d5-9663-ed7335d94063-kube-api-access-h769c\") pod \"iptables-alerter-f55lr\" (UID: \"86b0dedd-e7fd-44d5-9663-ed7335d94063\") " pod="openshift-network-operator/iptables-alerter-f55lr" Jul 10 00:15:35.707763 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.700966 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/03b7fa42-b493-40ce-a0d1-ce267379f3df-host\") pod \"node-ca-gjk2r\" (UID: \"03b7fa42-b493-40ce-a0d1-ce267379f3df\") " pod="openshift-image-registry/node-ca-gjk2r" Jul 10 00:15:35.707763 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.700897 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0318fc8c-a177-4995-a1b6-80ccf2cf11cd-host-run-multus-certs\") pod \"multus-5zlgk\" (UID: \"0318fc8c-a177-4995-a1b6-80ccf2cf11cd\") " pod="openshift-multus/multus-5zlgk" Jul 10 00:15:35.708709 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.700985 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a88d645c-45e4-4592-b451-04eea24d3afb-system-cni-dir\") pod \"multus-additional-cni-plugins-5kkqc\" (UID: \"a88d645c-45e4-4592-b451-04eea24d3afb\") " pod="openshift-multus/multus-additional-cni-plugins-5kkqc" Jul 10 00:15:35.708709 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.701010 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbhs7\" (UniqueName: \"kubernetes.io/projected/f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32-kube-api-access-wbhs7\") pod \"ovnkube-node-h5g9w\" (UID: \"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:15:35.708709 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.701036 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/08be9fc2-0ae2-45bc-96d1-5306a46e1357-etc-modprobe-d\") pod \"tuned-n5xgr\" (UID: \"08be9fc2-0ae2-45bc-96d1-5306a46e1357\") " pod="openshift-cluster-node-tuning-operator/tuned-n5xgr" Jul 10 00:15:35.708709 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.701052 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/08be9fc2-0ae2-45bc-96d1-5306a46e1357-etc-sysctl-d\") pod \"tuned-n5xgr\" (UID: \"08be9fc2-0ae2-45bc-96d1-5306a46e1357\") " pod="openshift-cluster-node-tuning-operator/tuned-n5xgr" Jul 10 00:15:35.708709 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.701083 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/08be9fc2-0ae2-45bc-96d1-5306a46e1357-etc-tuned\") pod \"tuned-n5xgr\" (UID: \"08be9fc2-0ae2-45bc-96d1-5306a46e1357\") " pod="openshift-cluster-node-tuning-operator/tuned-n5xgr" Jul 10 00:15:35.708709 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.701119 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f78vw\" (UniqueName: \"kubernetes.io/projected/62933c8a-818d-4580-a6c9-6d26dbdfd50d-kube-api-access-f78vw\") pod \"network-metrics-daemon-x8v89\" (UID: \"62933c8a-818d-4580-a6c9-6d26dbdfd50d\") " pod="openshift-multus/network-metrics-daemon-x8v89" Jul 10 00:15:35.708709 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.701135 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0da6eed2-eb77-4e83-bb07-a6163eaca53a-tmp-dir\") pod \"node-resolver-2mpf9\" (UID: \"0da6eed2-eb77-4e83-bb07-a6163eaca53a\") " pod="openshift-dns/node-resolver-2mpf9" Jul 10 00:15:35.708709 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.701154 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a88d645c-45e4-4592-b451-04eea24d3afb-cnibin\") pod \"multus-additional-cni-plugins-5kkqc\" (UID: \"a88d645c-45e4-4592-b451-04eea24d3afb\") " pod="openshift-multus/multus-additional-cni-plugins-5kkqc" Jul 10 00:15:35.708709 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.701262 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a88d645c-45e4-4592-b451-04eea24d3afb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5kkqc\" (UID: \"a88d645c-45e4-4592-b451-04eea24d3afb\") " pod="openshift-multus/multus-additional-cni-plugins-5kkqc" Jul 10 00:15:35.708709 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.701282 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-249ls\" (UniqueName: \"kubernetes.io/projected/269e1974-6855-40f9-9f47-3dc20f729981-kube-api-access-249ls\") pod \"aws-ebs-csi-driver-node-g6xw4\" (UID: \"269e1974-6855-40f9-9f47-3dc20f729981\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6xw4" Jul 10 00:15:35.708709 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.701299 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lntcb\" (UniqueName: \"kubernetes.io/projected/a88d645c-45e4-4592-b451-04eea24d3afb-kube-api-access-lntcb\") pod \"multus-additional-cni-plugins-5kkqc\" (UID: \"a88d645c-45e4-4592-b451-04eea24d3afb\") " pod="openshift-multus/multus-additional-cni-plugins-5kkqc" Jul 10 00:15:35.708709 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.701317 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32-ovnkube-config\") pod \"ovnkube-node-h5g9w\" (UID: \"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:15:35.708709 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.701353 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32-ovnkube-script-lib\") pod \"ovnkube-node-h5g9w\" (UID: \"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:15:35.709395 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.701393 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0318fc8c-a177-4995-a1b6-80ccf2cf11cd-etc-kubernetes\") pod \"multus-5zlgk\" (UID: \"0318fc8c-a177-4995-a1b6-80ccf2cf11cd\") " pod="openshift-multus/multus-5zlgk" Jul 10 00:15:35.709395 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.701430 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l62wx\" (UniqueName: \"kubernetes.io/projected/03b7fa42-b493-40ce-a0d1-ce267379f3df-kube-api-access-l62wx\") pod \"node-ca-gjk2r\" (UID: \"03b7fa42-b493-40ce-a0d1-ce267379f3df\") " pod="openshift-image-registry/node-ca-gjk2r" Jul 10 00:15:35.709395 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.701446 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0318fc8c-a177-4995-a1b6-80ccf2cf11cd-etc-kubernetes\") pod \"multus-5zlgk\" (UID: \"0318fc8c-a177-4995-a1b6-80ccf2cf11cd\") " pod="openshift-multus/multus-5zlgk" Jul 10 00:15:35.709395 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.701453 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32-host-kubelet\") pod \"ovnkube-node-h5g9w\" (UID: \"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:15:35.709395 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.701477 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08be9fc2-0ae2-45bc-96d1-5306a46e1357-sys\") pod \"tuned-n5xgr\" (UID: \"08be9fc2-0ae2-45bc-96d1-5306a46e1357\") " pod="openshift-cluster-node-tuning-operator/tuned-n5xgr" Jul 10 00:15:35.709395 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.701498 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/451130fe-547b-40c5-a7dc-412d57fb5db5-dbus\") pod \"global-pull-secret-syncer-nkcmr\" (UID: \"451130fe-547b-40c5-a7dc-412d57fb5db5\") " pod="kube-system/global-pull-secret-syncer-nkcmr" Jul 10 00:15:35.709395 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.701530 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0318fc8c-a177-4995-a1b6-80ccf2cf11cd-host-run-k8s-cni-cncf-io\") pod \"multus-5zlgk\" (UID: \"0318fc8c-a177-4995-a1b6-80ccf2cf11cd\") " pod="openshift-multus/multus-5zlgk" Jul 10 00:15:35.709395 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.701561 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0318fc8c-a177-4995-a1b6-80ccf2cf11cd-host-run-k8s-cni-cncf-io\") pod \"multus-5zlgk\" (UID: \"0318fc8c-a177-4995-a1b6-80ccf2cf11cd\") " pod="openshift-multus/multus-5zlgk" Jul 10 00:15:35.709395 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.701586 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0318fc8c-a177-4995-a1b6-80ccf2cf11cd-multus-daemon-config\") pod \"multus-5zlgk\" (UID: \"0318fc8c-a177-4995-a1b6-80ccf2cf11cd\") " pod="openshift-multus/multus-5zlgk" Jul 10 00:15:35.709395 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.701634 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0318fc8c-a177-4995-a1b6-80ccf2cf11cd-multus-conf-dir\") pod \"multus-5zlgk\" (UID: \"0318fc8c-a177-4995-a1b6-80ccf2cf11cd\") " pod="openshift-multus/multus-5zlgk" Jul 10 00:15:35.709395 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.701667 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9mfc\" (UniqueName: \"kubernetes.io/projected/0318fc8c-a177-4995-a1b6-80ccf2cf11cd-kube-api-access-p9mfc\") pod \"multus-5zlgk\" (UID: \"0318fc8c-a177-4995-a1b6-80ccf2cf11cd\") " pod="openshift-multus/multus-5zlgk" Jul 10 00:15:35.709395 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.701691 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/269e1974-6855-40f9-9f47-3dc20f729981-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g6xw4\" (UID: \"269e1974-6855-40f9-9f47-3dc20f729981\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6xw4" Jul 10 00:15:35.709395 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.701717 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0318fc8c-a177-4995-a1b6-80ccf2cf11cd-multus-conf-dir\") pod \"multus-5zlgk\" (UID: \"0318fc8c-a177-4995-a1b6-80ccf2cf11cd\") " pod="openshift-multus/multus-5zlgk" Jul 10 00:15:35.709395 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.701751 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/269e1974-6855-40f9-9f47-3dc20f729981-etc-selinux\") pod \"aws-ebs-csi-driver-node-g6xw4\" (UID: \"269e1974-6855-40f9-9f47-3dc20f729981\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6xw4" Jul 10 00:15:35.710353 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.701774 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/08be9fc2-0ae2-45bc-96d1-5306a46e1357-etc-sysctl-conf\") pod \"tuned-n5xgr\" (UID: \"08be9fc2-0ae2-45bc-96d1-5306a46e1357\") " pod="openshift-cluster-node-tuning-operator/tuned-n5xgr" Jul 10 00:15:35.710353 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.701887 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/08be9fc2-0ae2-45bc-96d1-5306a46e1357-etc-systemd\") pod \"tuned-n5xgr\" (UID: \"08be9fc2-0ae2-45bc-96d1-5306a46e1357\") " pod="openshift-cluster-node-tuning-operator/tuned-n5xgr" Jul 10 00:15:35.710353 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.701911 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/86b0dedd-e7fd-44d5-9663-ed7335d94063-host-slash\") pod \"iptables-alerter-f55lr\" (UID: \"86b0dedd-e7fd-44d5-9663-ed7335d94063\") " pod="openshift-network-operator/iptables-alerter-f55lr" Jul 10 00:15:35.710353 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.701978 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0318fc8c-a177-4995-a1b6-80ccf2cf11cd-multus-daemon-config\") pod \"multus-5zlgk\" (UID: \"0318fc8c-a177-4995-a1b6-80ccf2cf11cd\") " pod="openshift-multus/multus-5zlgk" Jul 10 00:15:35.710353 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.709170 2562 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 10 00:15:35.817671 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.713561 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f78vw\" (UniqueName: \"kubernetes.io/projected/62933c8a-818d-4580-a6c9-6d26dbdfd50d-kube-api-access-f78vw\") pod \"network-metrics-daemon-x8v89\" (UID: \"62933c8a-818d-4580-a6c9-6d26dbdfd50d\") " pod="openshift-multus/network-metrics-daemon-x8v89" Jul 10 00:15:35.817671 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.713615 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9mfc\" (UniqueName: \"kubernetes.io/projected/0318fc8c-a177-4995-a1b6-80ccf2cf11cd-kube-api-access-p9mfc\") pod \"multus-5zlgk\" (UID: \"0318fc8c-a177-4995-a1b6-80ccf2cf11cd\") " pod="openshift-multus/multus-5zlgk" Jul 10 00:15:35.817671 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.746619 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-101.ec2.internal" Jul 10 00:15:35.817671 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.746868 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-101.ec2.internal" Jul 10 00:15:35.817671 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.747524 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-101.ec2.internal" event={"ID":"65368ef6b324db716ad2e852e6f60436","Type":"ContainerStarted","Data":"dbe19180d285d317eb6a53072920d6bf0f0a5d37f8b99b2dffdf256bb5d301ae"} Jul 10 00:15:35.817671 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:35.765163 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29d40385cd1620022aeccb7b81ab99c1.slice/crio-0231fd010c0babd432a6b64849f607a7dd609822e026a4ee5e14962a78323d77 WatchSource:0}: Error finding container 0231fd010c0babd432a6b64849f607a7dd609822e026a4ee5e14962a78323d77: Status 404 returned error can't find the container with id 0231fd010c0babd432a6b64849f607a7dd609822e026a4ee5e14962a78323d77 Jul 10 00:15:35.817671 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.765721 2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 00:15:35.817671 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.802454 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32-systemd-units\") pod \"ovnkube-node-h5g9w\" (UID: \"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:15:35.817671 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.802482 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32-host-slash\") pod \"ovnkube-node-h5g9w\" (UID: \"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:15:35.817671 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.802499 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32-run-systemd\") pod \"ovnkube-node-h5g9w\" (UID: \"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:15:35.817671 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.802515 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32-run-ovn\") pod \"ovnkube-node-h5g9w\" (UID: \"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:15:35.817671 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.802543 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dfc778a3-c60f-4f5e-8b6f-3e613228a7f9-konnectivity-ca\") pod \"konnectivity-agent-shx2p\" (UID: \"dfc778a3-c60f-4f5e-8b6f-3e613228a7f9\") " pod="kube-system/konnectivity-agent-shx2p" Jul 10 00:15:35.817671 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.802570 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a88d645c-45e4-4592-b451-04eea24d3afb-cni-binary-copy\") pod \"multus-additional-cni-plugins-5kkqc\" (UID: \"a88d645c-45e4-4592-b451-04eea24d3afb\") " pod="openshift-multus/multus-additional-cni-plugins-5kkqc" Jul 10 00:15:35.817671 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.802581 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32-host-slash\") pod \"ovnkube-node-h5g9w\" (UID: \"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:15:35.817671 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.802595 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dfc778a3-c60f-4f5e-8b6f-3e613228a7f9-agent-certs\") pod \"konnectivity-agent-shx2p\" (UID: \"dfc778a3-c60f-4f5e-8b6f-3e613228a7f9\") " pod="kube-system/konnectivity-agent-shx2p" Jul 10 00:15:35.817671 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.802599 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32-run-systemd\") pod \"ovnkube-node-h5g9w\" (UID: \"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:15:35.819053 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.802624 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a88d645c-45e4-4592-b451-04eea24d3afb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5kkqc\" (UID: \"a88d645c-45e4-4592-b451-04eea24d3afb\") " pod="openshift-multus/multus-additional-cni-plugins-5kkqc" Jul 10 00:15:35.819053 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.802637 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32-run-ovn\") pod \"ovnkube-node-h5g9w\" (UID: \"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:15:35.819053 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.802651 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/269e1974-6855-40f9-9f47-3dc20f729981-socket-dir\") pod \"aws-ebs-csi-driver-node-g6xw4\" (UID: \"269e1974-6855-40f9-9f47-3dc20f729981\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6xw4" Jul 10 00:15:35.819053 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.802680 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/08be9fc2-0ae2-45bc-96d1-5306a46e1357-run\") pod \"tuned-n5xgr\" (UID: \"08be9fc2-0ae2-45bc-96d1-5306a46e1357\") " pod="openshift-cluster-node-tuning-operator/tuned-n5xgr" Jul 10 00:15:35.819053 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.802588 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32-systemd-units\") pod \"ovnkube-node-h5g9w\" (UID: \"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:15:35.819053 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.802708 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/08be9fc2-0ae2-45bc-96d1-5306a46e1357-var-lib-kubelet\") pod \"tuned-n5xgr\" (UID: \"08be9fc2-0ae2-45bc-96d1-5306a46e1357\") " pod="openshift-cluster-node-tuning-operator/tuned-n5xgr" Jul 10 00:15:35.819053 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.802733 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/451130fe-547b-40c5-a7dc-412d57fb5db5-original-pull-secret\") pod \"global-pull-secret-syncer-nkcmr\" (UID: \"451130fe-547b-40c5-a7dc-412d57fb5db5\") " pod="kube-system/global-pull-secret-syncer-nkcmr" Jul 10 00:15:35.819053 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.802759 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32-var-lib-openvswitch\") pod \"ovnkube-node-h5g9w\" (UID: \"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:15:35.819053 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.802766 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/08be9fc2-0ae2-45bc-96d1-5306a46e1357-run\") pod \"tuned-n5xgr\" (UID: \"08be9fc2-0ae2-45bc-96d1-5306a46e1357\") " pod="openshift-cluster-node-tuning-operator/tuned-n5xgr" Jul 10 00:15:35.819053 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.802787 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32-host-cni-netd\") pod \"ovnkube-node-h5g9w\" (UID: \"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:15:35.819053 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.802811 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/08be9fc2-0ae2-45bc-96d1-5306a46e1357-lib-modules\") pod \"tuned-n5xgr\" (UID: \"08be9fc2-0ae2-45bc-96d1-5306a46e1357\") " pod="openshift-cluster-node-tuning-operator/tuned-n5xgr" Jul 10 00:15:35.819053 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.802825 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32-var-lib-openvswitch\") pod \"ovnkube-node-h5g9w\" (UID: \"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:15:35.819053 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.802788 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/269e1974-6855-40f9-9f47-3dc20f729981-socket-dir\") pod \"aws-ebs-csi-driver-node-g6xw4\" (UID: \"269e1974-6855-40f9-9f47-3dc20f729981\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6xw4" Jul 10 00:15:35.819053 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.802853 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h769c\" (UniqueName: \"kubernetes.io/projected/86b0dedd-e7fd-44d5-9663-ed7335d94063-kube-api-access-h769c\") pod \"iptables-alerter-f55lr\" (UID: \"86b0dedd-e7fd-44d5-9663-ed7335d94063\") " pod="openshift-network-operator/iptables-alerter-f55lr" Jul 10 00:15:35.819975 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.802863 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/08be9fc2-0ae2-45bc-96d1-5306a46e1357-var-lib-kubelet\") pod \"tuned-n5xgr\" (UID: \"08be9fc2-0ae2-45bc-96d1-5306a46e1357\") " pod="openshift-cluster-node-tuning-operator/tuned-n5xgr" Jul 10 00:15:35.819975 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.802880 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/03b7fa42-b493-40ce-a0d1-ce267379f3df-host\") pod \"node-ca-gjk2r\" (UID: \"03b7fa42-b493-40ce-a0d1-ce267379f3df\") " pod="openshift-image-registry/node-ca-gjk2r" Jul 10 00:15:35.819975 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.802910 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a88d645c-45e4-4592-b451-04eea24d3afb-system-cni-dir\") pod \"multus-additional-cni-plugins-5kkqc\" (UID: \"a88d645c-45e4-4592-b451-04eea24d3afb\") " pod="openshift-multus/multus-additional-cni-plugins-5kkqc" Jul 10 00:15:35.819975 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.802936 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbhs7\" (UniqueName: \"kubernetes.io/projected/f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32-kube-api-access-wbhs7\") pod \"ovnkube-node-h5g9w\" (UID: \"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:15:35.819975 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:35.802945 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 00:15:35.819975 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.802961 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/08be9fc2-0ae2-45bc-96d1-5306a46e1357-etc-modprobe-d\") pod \"tuned-n5xgr\" (UID: \"08be9fc2-0ae2-45bc-96d1-5306a46e1357\") " pod="openshift-cluster-node-tuning-operator/tuned-n5xgr" Jul 10 00:15:35.819975 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.802986 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/08be9fc2-0ae2-45bc-96d1-5306a46e1357-etc-sysctl-d\") pod \"tuned-n5xgr\" (UID: \"08be9fc2-0ae2-45bc-96d1-5306a46e1357\") " pod="openshift-cluster-node-tuning-operator/tuned-n5xgr" Jul 10 00:15:35.819975 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:35.803014 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/451130fe-547b-40c5-a7dc-412d57fb5db5-original-pull-secret podName:451130fe-547b-40c5-a7dc-412d57fb5db5 nodeName:}" failed. No retries permitted until 2026-07-10 00:15:36.302991756 +0000 UTC m=+3.121297247 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/451130fe-547b-40c5-a7dc-412d57fb5db5-original-pull-secret") pod "global-pull-secret-syncer-nkcmr" (UID: "451130fe-547b-40c5-a7dc-412d57fb5db5") : object "kube-system"/"original-pull-secret" not registered Jul 10 00:15:35.819975 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.803038 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/08be9fc2-0ae2-45bc-96d1-5306a46e1357-etc-tuned\") pod \"tuned-n5xgr\" (UID: \"08be9fc2-0ae2-45bc-96d1-5306a46e1357\") " pod="openshift-cluster-node-tuning-operator/tuned-n5xgr" Jul 10 00:15:35.819975 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.803088 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0da6eed2-eb77-4e83-bb07-a6163eaca53a-tmp-dir\") pod \"node-resolver-2mpf9\" (UID: \"0da6eed2-eb77-4e83-bb07-a6163eaca53a\") " pod="openshift-dns/node-resolver-2mpf9" Jul 10 00:15:35.819975 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.803100 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/08be9fc2-0ae2-45bc-96d1-5306a46e1357-etc-sysctl-d\") pod \"tuned-n5xgr\" (UID: \"08be9fc2-0ae2-45bc-96d1-5306a46e1357\") " pod="openshift-cluster-node-tuning-operator/tuned-n5xgr" Jul 10 00:15:35.819975 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.803120 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a88d645c-45e4-4592-b451-04eea24d3afb-cnibin\") pod \"multus-additional-cni-plugins-5kkqc\" (UID: \"a88d645c-45e4-4592-b451-04eea24d3afb\") " pod="openshift-multus/multus-additional-cni-plugins-5kkqc" Jul 10 00:15:35.819975 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.803166 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a88d645c-45e4-4592-b451-04eea24d3afb-system-cni-dir\") pod \"multus-additional-cni-plugins-5kkqc\" (UID: \"a88d645c-45e4-4592-b451-04eea24d3afb\") " pod="openshift-multus/multus-additional-cni-plugins-5kkqc" Jul 10 00:15:35.819975 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.803165 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a88d645c-45e4-4592-b451-04eea24d3afb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5kkqc\" (UID: \"a88d645c-45e4-4592-b451-04eea24d3afb\") " pod="openshift-multus/multus-additional-cni-plugins-5kkqc" Jul 10 00:15:35.821832 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.803207 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-249ls\" (UniqueName: \"kubernetes.io/projected/269e1974-6855-40f9-9f47-3dc20f729981-kube-api-access-249ls\") pod \"aws-ebs-csi-driver-node-g6xw4\" (UID: \"269e1974-6855-40f9-9f47-3dc20f729981\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6xw4" Jul 10 00:15:35.821832 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.803234 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lntcb\" (UniqueName: \"kubernetes.io/projected/a88d645c-45e4-4592-b451-04eea24d3afb-kube-api-access-lntcb\") pod \"multus-additional-cni-plugins-5kkqc\" (UID: \"a88d645c-45e4-4592-b451-04eea24d3afb\") " pod="openshift-multus/multus-additional-cni-plugins-5kkqc" Jul 10 00:15:35.821832 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.803262 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32-ovnkube-config\") pod \"ovnkube-node-h5g9w\" (UID: \"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:15:35.821832 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.803266 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a88d645c-45e4-4592-b451-04eea24d3afb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5kkqc\" (UID: \"a88d645c-45e4-4592-b451-04eea24d3afb\") " pod="openshift-multus/multus-additional-cni-plugins-5kkqc" Jul 10 00:15:35.821832 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.803295 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/03b7fa42-b493-40ce-a0d1-ce267379f3df-host\") pod \"node-ca-gjk2r\" (UID: \"03b7fa42-b493-40ce-a0d1-ce267379f3df\") " pod="openshift-image-registry/node-ca-gjk2r" Jul 10 00:15:35.821832 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.803360 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/08be9fc2-0ae2-45bc-96d1-5306a46e1357-etc-modprobe-d\") pod \"tuned-n5xgr\" (UID: \"08be9fc2-0ae2-45bc-96d1-5306a46e1357\") " pod="openshift-cluster-node-tuning-operator/tuned-n5xgr" Jul 10 00:15:35.821832 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.803267 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dfc778a3-c60f-4f5e-8b6f-3e613228a7f9-konnectivity-ca\") pod \"konnectivity-agent-shx2p\" (UID: \"dfc778a3-c60f-4f5e-8b6f-3e613228a7f9\") " pod="kube-system/konnectivity-agent-shx2p" Jul 10 00:15:35.821832 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.803564 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a88d645c-45e4-4592-b451-04eea24d3afb-cnibin\") pod \"multus-additional-cni-plugins-5kkqc\" (UID: \"a88d645c-45e4-4592-b451-04eea24d3afb\") " pod="openshift-multus/multus-additional-cni-plugins-5kkqc" Jul 10 00:15:35.821832 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.802881 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32-host-cni-netd\") pod \"ovnkube-node-h5g9w\" (UID: \"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:15:35.821832 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.802991 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/08be9fc2-0ae2-45bc-96d1-5306a46e1357-lib-modules\") pod \"tuned-n5xgr\" (UID: \"08be9fc2-0ae2-45bc-96d1-5306a46e1357\") " pod="openshift-cluster-node-tuning-operator/tuned-n5xgr" Jul 10 00:15:35.821832 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.803653 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a88d645c-45e4-4592-b451-04eea24d3afb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5kkqc\" (UID: \"a88d645c-45e4-4592-b451-04eea24d3afb\") " pod="openshift-multus/multus-additional-cni-plugins-5kkqc" Jul 10 00:15:35.821832 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.803691 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32-ovnkube-script-lib\") pod \"ovnkube-node-h5g9w\" (UID: \"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:15:35.821832 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.803734 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l62wx\" (UniqueName: \"kubernetes.io/projected/03b7fa42-b493-40ce-a0d1-ce267379f3df-kube-api-access-l62wx\") pod \"node-ca-gjk2r\" (UID: \"03b7fa42-b493-40ce-a0d1-ce267379f3df\") " pod="openshift-image-registry/node-ca-gjk2r" Jul 10 00:15:35.821832 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.803761 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32-host-kubelet\") pod \"ovnkube-node-h5g9w\" (UID: \"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:15:35.822530 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.803783 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0da6eed2-eb77-4e83-bb07-a6163eaca53a-tmp-dir\") pod \"node-resolver-2mpf9\" (UID: \"0da6eed2-eb77-4e83-bb07-a6163eaca53a\") " pod="openshift-dns/node-resolver-2mpf9" Jul 10 00:15:35.822530 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.803786 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08be9fc2-0ae2-45bc-96d1-5306a46e1357-sys\") pod \"tuned-n5xgr\" (UID: \"08be9fc2-0ae2-45bc-96d1-5306a46e1357\") " pod="openshift-cluster-node-tuning-operator/tuned-n5xgr" Jul 10 00:15:35.822530 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.803839 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08be9fc2-0ae2-45bc-96d1-5306a46e1357-sys\") pod \"tuned-n5xgr\" (UID: \"08be9fc2-0ae2-45bc-96d1-5306a46e1357\") " pod="openshift-cluster-node-tuning-operator/tuned-n5xgr" Jul 10 00:15:35.822530 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.803834 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/451130fe-547b-40c5-a7dc-412d57fb5db5-dbus\") pod \"global-pull-secret-syncer-nkcmr\" (UID: \"451130fe-547b-40c5-a7dc-412d57fb5db5\") " pod="kube-system/global-pull-secret-syncer-nkcmr" Jul 10 00:15:35.822530 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.803889 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/269e1974-6855-40f9-9f47-3dc20f729981-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g6xw4\" (UID: \"269e1974-6855-40f9-9f47-3dc20f729981\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6xw4" Jul 10 00:15:35.822530 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.803905 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32-host-kubelet\") pod \"ovnkube-node-h5g9w\" (UID: \"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:15:35.822530 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.803914 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/269e1974-6855-40f9-9f47-3dc20f729981-etc-selinux\") pod \"aws-ebs-csi-driver-node-g6xw4\" (UID: \"269e1974-6855-40f9-9f47-3dc20f729981\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6xw4" Jul 10 00:15:35.822530 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.803970 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/269e1974-6855-40f9-9f47-3dc20f729981-etc-selinux\") pod \"aws-ebs-csi-driver-node-g6xw4\" (UID: \"269e1974-6855-40f9-9f47-3dc20f729981\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6xw4" Jul 10 00:15:35.822530 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.804036 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/451130fe-547b-40c5-a7dc-412d57fb5db5-dbus\") pod \"global-pull-secret-syncer-nkcmr\" (UID: \"451130fe-547b-40c5-a7dc-412d57fb5db5\") " pod="kube-system/global-pull-secret-syncer-nkcmr" Jul 10 00:15:35.822530 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.804080 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/269e1974-6855-40f9-9f47-3dc20f729981-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g6xw4\" (UID: \"269e1974-6855-40f9-9f47-3dc20f729981\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6xw4" Jul 10 00:15:35.822530 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.804105 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/08be9fc2-0ae2-45bc-96d1-5306a46e1357-etc-sysctl-conf\") pod \"tuned-n5xgr\" (UID: \"08be9fc2-0ae2-45bc-96d1-5306a46e1357\") " pod="openshift-cluster-node-tuning-operator/tuned-n5xgr" Jul 10 00:15:35.822530 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.804131 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/08be9fc2-0ae2-45bc-96d1-5306a46e1357-etc-systemd\") pod \"tuned-n5xgr\" (UID: \"08be9fc2-0ae2-45bc-96d1-5306a46e1357\") " pod="openshift-cluster-node-tuning-operator/tuned-n5xgr" Jul 10 00:15:35.822530 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.804177 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/86b0dedd-e7fd-44d5-9663-ed7335d94063-host-slash\") pod \"iptables-alerter-f55lr\" (UID: \"86b0dedd-e7fd-44d5-9663-ed7335d94063\") " pod="openshift-network-operator/iptables-alerter-f55lr" Jul 10 00:15:35.822530 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.804205 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/269e1974-6855-40f9-9f47-3dc20f729981-device-dir\") pod \"aws-ebs-csi-driver-node-g6xw4\" (UID: \"269e1974-6855-40f9-9f47-3dc20f729981\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6xw4" Jul 10 00:15:35.822530 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.804213 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/08be9fc2-0ae2-45bc-96d1-5306a46e1357-etc-systemd\") pod \"tuned-n5xgr\" (UID: \"08be9fc2-0ae2-45bc-96d1-5306a46e1357\") " pod="openshift-cluster-node-tuning-operator/tuned-n5xgr" Jul 10 00:15:35.824673 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.804230 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/08be9fc2-0ae2-45bc-96d1-5306a46e1357-host\") pod \"tuned-n5xgr\" (UID: \"08be9fc2-0ae2-45bc-96d1-5306a46e1357\") " pod="openshift-cluster-node-tuning-operator/tuned-n5xgr" Jul 10 00:15:35.824673 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.804251 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/08be9fc2-0ae2-45bc-96d1-5306a46e1357-etc-sysctl-conf\") pod \"tuned-n5xgr\" (UID: \"08be9fc2-0ae2-45bc-96d1-5306a46e1357\") " pod="openshift-cluster-node-tuning-operator/tuned-n5xgr" Jul 10 00:15:35.824673 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.804260 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0da6eed2-eb77-4e83-bb07-a6163eaca53a-hosts-file\") pod \"node-resolver-2mpf9\" (UID: \"0da6eed2-eb77-4e83-bb07-a6163eaca53a\") " pod="openshift-dns/node-resolver-2mpf9" Jul 10 00:15:35.824673 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.804261 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/86b0dedd-e7fd-44d5-9663-ed7335d94063-host-slash\") pod \"iptables-alerter-f55lr\" (UID: \"86b0dedd-e7fd-44d5-9663-ed7335d94063\") " pod="openshift-network-operator/iptables-alerter-f55lr" Jul 10 00:15:35.824673 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.804303 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32-etc-openvswitch\") pod \"ovnkube-node-h5g9w\" (UID: \"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:15:35.824673 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.804307 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0da6eed2-eb77-4e83-bb07-a6163eaca53a-hosts-file\") pod \"node-resolver-2mpf9\" (UID: \"0da6eed2-eb77-4e83-bb07-a6163eaca53a\") " pod="openshift-dns/node-resolver-2mpf9" Jul 10 00:15:35.824673 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.804317 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/269e1974-6855-40f9-9f47-3dc20f729981-device-dir\") pod \"aws-ebs-csi-driver-node-g6xw4\" (UID: \"269e1974-6855-40f9-9f47-3dc20f729981\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6xw4" Jul 10 00:15:35.824673 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.804332 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32-log-socket\") pod \"ovnkube-node-h5g9w\" (UID: \"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:15:35.824673 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.804353 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/08be9fc2-0ae2-45bc-96d1-5306a46e1357-host\") pod \"tuned-n5xgr\" (UID: \"08be9fc2-0ae2-45bc-96d1-5306a46e1357\") " pod="openshift-cluster-node-tuning-operator/tuned-n5xgr" Jul 10 00:15:35.824673 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.804358 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32-host-cni-bin\") pod \"ovnkube-node-h5g9w\" (UID: \"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:15:35.824673 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.804391 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32-ovn-node-metrics-cert\") pod \"ovnkube-node-h5g9w\" (UID: \"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:15:35.824673 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.804361 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32-etc-openvswitch\") pod \"ovnkube-node-h5g9w\" (UID: \"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:15:35.824673 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.804395 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32-log-socket\") pod \"ovnkube-node-h5g9w\" (UID: \"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:15:35.824673 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.804398 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32-host-cni-bin\") pod \"ovnkube-node-h5g9w\" (UID: \"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:15:35.824673 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.804429 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wswmt\" (UniqueName: \"kubernetes.io/projected/0da6eed2-eb77-4e83-bb07-a6163eaca53a-kube-api-access-wswmt\") pod \"node-resolver-2mpf9\" (UID: \"0da6eed2-eb77-4e83-bb07-a6163eaca53a\") " pod="openshift-dns/node-resolver-2mpf9" Jul 10 00:15:35.825398 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.804474 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32-host-run-netns\") pod \"ovnkube-node-h5g9w\" (UID: \"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:15:35.825398 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.804481 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32-ovnkube-config\") pod \"ovnkube-node-h5g9w\" (UID: \"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:15:35.825398 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.804503 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h5g9w\" (UID: \"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:15:35.825398 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.804528 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32-ovnkube-script-lib\") pod \"ovnkube-node-h5g9w\" (UID: \"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:15:35.825398 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.804545 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32-host-run-netns\") pod \"ovnkube-node-h5g9w\" (UID: \"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:15:35.825398 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.804531 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/08be9fc2-0ae2-45bc-96d1-5306a46e1357-etc-sysconfig\") pod \"tuned-n5xgr\" (UID: \"08be9fc2-0ae2-45bc-96d1-5306a46e1357\") " pod="openshift-cluster-node-tuning-operator/tuned-n5xgr" Jul 10 00:15:35.825398 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.804573 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/08be9fc2-0ae2-45bc-96d1-5306a46e1357-etc-sysconfig\") pod \"tuned-n5xgr\" (UID: \"08be9fc2-0ae2-45bc-96d1-5306a46e1357\") " pod="openshift-cluster-node-tuning-operator/tuned-n5xgr" Jul 10 00:15:35.825398 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.804584 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h5g9w\" (UID: \"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:15:35.825398 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.804591 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm2j9\" (UniqueName: \"kubernetes.io/projected/08be9fc2-0ae2-45bc-96d1-5306a46e1357-kube-api-access-pm2j9\") pod \"tuned-n5xgr\" (UID: \"08be9fc2-0ae2-45bc-96d1-5306a46e1357\") " pod="openshift-cluster-node-tuning-operator/tuned-n5xgr" Jul 10 00:15:35.825398 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.804632 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a88d645c-45e4-4592-b451-04eea24d3afb-os-release\") pod \"multus-additional-cni-plugins-5kkqc\" (UID: \"a88d645c-45e4-4592-b451-04eea24d3afb\") " pod="openshift-multus/multus-additional-cni-plugins-5kkqc" Jul 10 00:15:35.825398 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.804659 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a88d645c-45e4-4592-b451-04eea24d3afb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5kkqc\" (UID: \"a88d645c-45e4-4592-b451-04eea24d3afb\") " pod="openshift-multus/multus-additional-cni-plugins-5kkqc" Jul 10 00:15:35.825398 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.804711 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/269e1974-6855-40f9-9f47-3dc20f729981-registration-dir\") pod \"aws-ebs-csi-driver-node-g6xw4\" (UID: \"269e1974-6855-40f9-9f47-3dc20f729981\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6xw4" Jul 10 00:15:35.825398 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.804724 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a88d645c-45e4-4592-b451-04eea24d3afb-os-release\") pod \"multus-additional-cni-plugins-5kkqc\" (UID: \"a88d645c-45e4-4592-b451-04eea24d3afb\") " pod="openshift-multus/multus-additional-cni-plugins-5kkqc" Jul 10 00:15:35.825398 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.804740 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32-host-run-ovn-kubernetes\") pod \"ovnkube-node-h5g9w\" (UID: \"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:15:35.828470 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.804769 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08be9fc2-0ae2-45bc-96d1-5306a46e1357-etc-kubernetes\") pod \"tuned-n5xgr\" (UID: \"08be9fc2-0ae2-45bc-96d1-5306a46e1357\") " pod="openshift-cluster-node-tuning-operator/tuned-n5xgr" Jul 10 00:15:35.828470 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.804803 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/86b0dedd-e7fd-44d5-9663-ed7335d94063-iptables-alerter-script\") pod \"iptables-alerter-f55lr\" (UID: \"86b0dedd-e7fd-44d5-9663-ed7335d94063\") " pod="openshift-network-operator/iptables-alerter-f55lr" Jul 10 00:15:35.828470 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.804813 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a88d645c-45e4-4592-b451-04eea24d3afb-cni-binary-copy\") pod \"multus-additional-cni-plugins-5kkqc\" (UID: \"a88d645c-45e4-4592-b451-04eea24d3afb\") " pod="openshift-multus/multus-additional-cni-plugins-5kkqc" Jul 10 00:15:35.828470 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.804829 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/451130fe-547b-40c5-a7dc-412d57fb5db5-kubelet-config\") pod \"global-pull-secret-syncer-nkcmr\" (UID: \"451130fe-547b-40c5-a7dc-412d57fb5db5\") " pod="kube-system/global-pull-secret-syncer-nkcmr" Jul 10 00:15:35.828470 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.804843 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/269e1974-6855-40f9-9f47-3dc20f729981-registration-dir\") pod \"aws-ebs-csi-driver-node-g6xw4\" (UID: \"269e1974-6855-40f9-9f47-3dc20f729981\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6xw4" Jul 10 00:15:35.828470 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.804864 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/269e1974-6855-40f9-9f47-3dc20f729981-sys-fs\") pod \"aws-ebs-csi-driver-node-g6xw4\" (UID: \"269e1974-6855-40f9-9f47-3dc20f729981\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6xw4" Jul 10 00:15:35.828470 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.804871 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a88d645c-45e4-4592-b451-04eea24d3afb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5kkqc\" (UID: \"a88d645c-45e4-4592-b451-04eea24d3afb\") " pod="openshift-multus/multus-additional-cni-plugins-5kkqc" Jul 10 00:15:35.828470 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.804879 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08be9fc2-0ae2-45bc-96d1-5306a46e1357-etc-kubernetes\") pod \"tuned-n5xgr\" (UID: \"08be9fc2-0ae2-45bc-96d1-5306a46e1357\") " pod="openshift-cluster-node-tuning-operator/tuned-n5xgr" Jul 10 00:15:35.828470 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.804892 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32-run-openvswitch\") pod \"ovnkube-node-h5g9w\" (UID: \"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:15:35.828470 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.804918 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/03b7fa42-b493-40ce-a0d1-ce267379f3df-serviceca\") pod \"node-ca-gjk2r\" (UID: \"03b7fa42-b493-40ce-a0d1-ce267379f3df\") " pod="openshift-image-registry/node-ca-gjk2r" Jul 10 00:15:35.828470 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.804929 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/451130fe-547b-40c5-a7dc-412d57fb5db5-kubelet-config\") pod \"global-pull-secret-syncer-nkcmr\" (UID: \"451130fe-547b-40c5-a7dc-412d57fb5db5\") " pod="kube-system/global-pull-secret-syncer-nkcmr" Jul 10 00:15:35.828470 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.804944 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32-node-log\") pod \"ovnkube-node-h5g9w\" (UID: \"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:15:35.828470 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.804961 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/269e1974-6855-40f9-9f47-3dc20f729981-sys-fs\") pod \"aws-ebs-csi-driver-node-g6xw4\" (UID: \"269e1974-6855-40f9-9f47-3dc20f729981\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6xw4" Jul 10 00:15:35.828470 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.804971 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32-env-overrides\") pod \"ovnkube-node-h5g9w\" (UID: \"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:15:35.829191 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.805002 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32-run-openvswitch\") pod \"ovnkube-node-h5g9w\" (UID: \"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:15:35.829191 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.805019 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phkgc\" (UniqueName: \"kubernetes.io/projected/6fe3e675-7e07-49d2-a664-a47d434dfd3b-kube-api-access-phkgc\") pod \"network-check-target-qbbd4\" (UID: \"6fe3e675-7e07-49d2-a664-a47d434dfd3b\") " pod="openshift-network-diagnostics/network-check-target-qbbd4" Jul 10 00:15:35.829191 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.805051 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08be9fc2-0ae2-45bc-96d1-5306a46e1357-tmp\") pod \"tuned-n5xgr\" (UID: \"08be9fc2-0ae2-45bc-96d1-5306a46e1357\") " pod="openshift-cluster-node-tuning-operator/tuned-n5xgr" Jul 10 00:15:35.829191 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.805434 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/03b7fa42-b493-40ce-a0d1-ce267379f3df-serviceca\") pod \"node-ca-gjk2r\" (UID: \"03b7fa42-b493-40ce-a0d1-ce267379f3df\") " pod="openshift-image-registry/node-ca-gjk2r" Jul 10 00:15:35.829191 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.804919 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32-host-run-ovn-kubernetes\") pod \"ovnkube-node-h5g9w\" (UID: \"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:15:35.829191 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.805458 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32-env-overrides\") pod \"ovnkube-node-h5g9w\" (UID: \"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:15:35.829191 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.805492 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/86b0dedd-e7fd-44d5-9663-ed7335d94063-iptables-alerter-script\") pod \"iptables-alerter-f55lr\" (UID: \"86b0dedd-e7fd-44d5-9663-ed7335d94063\") " pod="openshift-network-operator/iptables-alerter-f55lr" Jul 10 00:15:35.829191 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.805499 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32-node-log\") pod \"ovnkube-node-h5g9w\" (UID: \"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:15:35.829191 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.805823 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/08be9fc2-0ae2-45bc-96d1-5306a46e1357-etc-tuned\") pod \"tuned-n5xgr\" (UID: \"08be9fc2-0ae2-45bc-96d1-5306a46e1357\") " pod="openshift-cluster-node-tuning-operator/tuned-n5xgr" Jul 10 00:15:35.829191 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.806208 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dfc778a3-c60f-4f5e-8b6f-3e613228a7f9-agent-certs\") pod \"konnectivity-agent-shx2p\" (UID: \"dfc778a3-c60f-4f5e-8b6f-3e613228a7f9\") " pod="kube-system/konnectivity-agent-shx2p" Jul 10 00:15:35.829191 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.806875 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32-ovn-node-metrics-cert\") pod \"ovnkube-node-h5g9w\" (UID: \"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:15:35.829191 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.807235 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08be9fc2-0ae2-45bc-96d1-5306a46e1357-tmp\") pod \"tuned-n5xgr\" (UID: \"08be9fc2-0ae2-45bc-96d1-5306a46e1357\") " pod="openshift-cluster-node-tuning-operator/tuned-n5xgr" Jul 10 00:15:35.829191 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.814530 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbhs7\" (UniqueName: \"kubernetes.io/projected/f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32-kube-api-access-wbhs7\") pod \"ovnkube-node-h5g9w\" (UID: \"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:15:35.829191 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.818990 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-249ls\" (UniqueName: \"kubernetes.io/projected/269e1974-6855-40f9-9f47-3dc20f729981-kube-api-access-249ls\") pod \"aws-ebs-csi-driver-node-g6xw4\" (UID: \"269e1974-6855-40f9-9f47-3dc20f729981\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6xw4" Jul 10 00:15:35.831507 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.821121 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lntcb\" (UniqueName: \"kubernetes.io/projected/a88d645c-45e4-4592-b451-04eea24d3afb-kube-api-access-lntcb\") pod \"multus-additional-cni-plugins-5kkqc\" (UID: \"a88d645c-45e4-4592-b451-04eea24d3afb\") " pod="openshift-multus/multus-additional-cni-plugins-5kkqc" Jul 10 00:15:35.831507 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.823218 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h769c\" (UniqueName: \"kubernetes.io/projected/86b0dedd-e7fd-44d5-9663-ed7335d94063-kube-api-access-h769c\") pod \"iptables-alerter-f55lr\" (UID: \"86b0dedd-e7fd-44d5-9663-ed7335d94063\") " pod="openshift-network-operator/iptables-alerter-f55lr" Jul 10 00:15:35.831507 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.827588 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm2j9\" (UniqueName: \"kubernetes.io/projected/08be9fc2-0ae2-45bc-96d1-5306a46e1357-kube-api-access-pm2j9\") pod \"tuned-n5xgr\" (UID: \"08be9fc2-0ae2-45bc-96d1-5306a46e1357\") " pod="openshift-cluster-node-tuning-operator/tuned-n5xgr" Jul 10 00:15:35.831507 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:35.827664 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 00:15:35.831507 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:35.827683 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 00:15:35.831507 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:35.827697 2562 projected.go:196] Error preparing data for projected volume kube-api-access-phkgc for pod openshift-network-diagnostics/network-check-target-qbbd4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:15:35.831507 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:35.827752 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6fe3e675-7e07-49d2-a664-a47d434dfd3b-kube-api-access-phkgc podName:6fe3e675-7e07-49d2-a664-a47d434dfd3b nodeName:}" failed. No retries permitted until 2026-07-10 00:15:36.327735355 +0000 UTC m=+3.146040845 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-phkgc" (UniqueName: "kubernetes.io/projected/6fe3e675-7e07-49d2-a664-a47d434dfd3b-kube-api-access-phkgc") pod "network-check-target-qbbd4" (UID: "6fe3e675-7e07-49d2-a664-a47d434dfd3b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:15:35.831507 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.830136 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l62wx\" (UniqueName: \"kubernetes.io/projected/03b7fa42-b493-40ce-a0d1-ce267379f3df-kube-api-access-l62wx\") pod \"node-ca-gjk2r\" (UID: \"03b7fa42-b493-40ce-a0d1-ce267379f3df\") " pod="openshift-image-registry/node-ca-gjk2r" Jul 10 00:15:35.834928 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.832250 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wswmt\" (UniqueName: \"kubernetes.io/projected/0da6eed2-eb77-4e83-bb07-a6163eaca53a-kube-api-access-wswmt\") pod \"node-resolver-2mpf9\" (UID: \"0da6eed2-eb77-4e83-bb07-a6163eaca53a\") " pod="openshift-dns/node-resolver-2mpf9" Jul 10 00:15:35.956176 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.901157 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5zlgk" Jul 10 00:15:35.956176 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.907724 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2mpf9" Jul 10 00:15:35.956176 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.916739 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-n5xgr" Jul 10 00:15:35.956176 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.926571 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-f55lr" Jul 10 00:15:35.956176 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.929493 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gjk2r" Jul 10 00:15:35.956176 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.936545 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5kkqc" Jul 10 00:15:35.956176 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:35.936790 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86b0dedd_e7fd_44d5_9663_ed7335d94063.slice/crio-c792a8dab6250aa392bf0eab3182cb5fd47a3703447456b9283a26fb5c8c51b1 WatchSource:0}: Error finding container c792a8dab6250aa392bf0eab3182cb5fd47a3703447456b9283a26fb5c8c51b1: Status 404 returned error can't find the container with id c792a8dab6250aa392bf0eab3182cb5fd47a3703447456b9283a26fb5c8c51b1 Jul 10 00:15:35.956176 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:35.938901 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03b7fa42_b493_40ce_a0d1_ce267379f3df.slice/crio-cf4e2326671ac65387016d721472b1300213ef3c4f4b367ebc6cf4a92f3b44b5 WatchSource:0}: Error finding container cf4e2326671ac65387016d721472b1300213ef3c4f4b367ebc6cf4a92f3b44b5: Status 404 returned error can't find the container with id cf4e2326671ac65387016d721472b1300213ef3c4f4b367ebc6cf4a92f3b44b5 Jul 10 00:15:35.956176 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.944639 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6xw4" Jul 10 00:15:35.956176 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:35.949091 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda88d645c_45e4_4592_b451_04eea24d3afb.slice/crio-9e2d02b21660cc80d19b1508a06cb556982867e3237ba321551dff464a8b4060 WatchSource:0}: Error finding container 9e2d02b21660cc80d19b1508a06cb556982867e3237ba321551dff464a8b4060: Status 404 returned error can't find the container with id 9e2d02b21660cc80d19b1508a06cb556982867e3237ba321551dff464a8b4060 Jul 10 00:15:35.956176 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.950122 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:15:35.961413 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:35.959425 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0a68b3c_8fb7_4b9e_b1c2_fb1ad811fd32.slice/crio-580dbf82de806d77b61a0908da8b624a9749063205860f6c8fcf523aba303620 WatchSource:0}: Error finding container 580dbf82de806d77b61a0908da8b624a9749063205860f6c8fcf523aba303620: Status 404 returned error can't find the container with id 580dbf82de806d77b61a0908da8b624a9749063205860f6c8fcf523aba303620 Jul 10 00:15:35.965221 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:35.964874 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-shx2p" Jul 10 00:15:36.115615 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:15:35.973086 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfc778a3_c60f_4f5e_8b6f_3e613228a7f9.slice/crio-008341e334ece3bd626ee29707cdf139faf33c1bfc0f28eafc864481932eb75e WatchSource:0}: Error finding container 008341e334ece3bd626ee29707cdf139faf33c1bfc0f28eafc864481932eb75e: Status 404 returned error can't find the container with id 008341e334ece3bd626ee29707cdf139faf33c1bfc0f28eafc864481932eb75e Jul 10 00:15:36.229831 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:36.207578 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62933c8a-818d-4580-a6c9-6d26dbdfd50d-metrics-certs\") pod \"network-metrics-daemon-x8v89\" (UID: \"62933c8a-818d-4580-a6c9-6d26dbdfd50d\") " pod="openshift-multus/network-metrics-daemon-x8v89" Jul 10 00:15:36.229831 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:36.207734 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:15:36.229831 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:36.207805 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62933c8a-818d-4580-a6c9-6d26dbdfd50d-metrics-certs podName:62933c8a-818d-4580-a6c9-6d26dbdfd50d nodeName:}" failed. No retries permitted until 2026-07-10 00:15:37.207783214 +0000 UTC m=+4.026088703 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/62933c8a-818d-4580-a6c9-6d26dbdfd50d-metrics-certs") pod "network-metrics-daemon-x8v89" (UID: "62933c8a-818d-4580-a6c9-6d26dbdfd50d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:15:36.335351 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:36.249506 2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 00:15:36.335351 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:36.308539 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/451130fe-547b-40c5-a7dc-412d57fb5db5-original-pull-secret\") pod \"global-pull-secret-syncer-nkcmr\" (UID: \"451130fe-547b-40c5-a7dc-412d57fb5db5\") " pod="kube-system/global-pull-secret-syncer-nkcmr" Jul 10 00:15:36.335351 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:36.308693 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 00:15:36.335351 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:36.308761 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/451130fe-547b-40c5-a7dc-412d57fb5db5-original-pull-secret podName:451130fe-547b-40c5-a7dc-412d57fb5db5 nodeName:}" failed. No retries permitted until 2026-07-10 00:15:37.308740432 +0000 UTC m=+4.127045933 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/451130fe-547b-40c5-a7dc-412d57fb5db5-original-pull-secret") pod "global-pull-secret-syncer-nkcmr" (UID: "451130fe-547b-40c5-a7dc-412d57fb5db5") : object "kube-system"/"original-pull-secret" not registered Jul 10 00:15:36.409520 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:36.409467 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phkgc\" (UniqueName: \"kubernetes.io/projected/6fe3e675-7e07-49d2-a664-a47d434dfd3b-kube-api-access-phkgc\") pod \"network-check-target-qbbd4\" (UID: \"6fe3e675-7e07-49d2-a664-a47d434dfd3b\") " pod="openshift-network-diagnostics/network-check-target-qbbd4" Jul 10 00:15:36.409694 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:36.409661 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 00:15:36.409694 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:36.409688 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 00:15:36.409797 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:36.409703 2562 projected.go:196] Error preparing data for projected volume kube-api-access-phkgc for pod openshift-network-diagnostics/network-check-target-qbbd4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:15:36.409797 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:36.409775 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6fe3e675-7e07-49d2-a664-a47d434dfd3b-kube-api-access-phkgc podName:6fe3e675-7e07-49d2-a664-a47d434dfd3b nodeName:}" failed. No retries permitted until 2026-07-10 00:15:37.409751469 +0000 UTC m=+4.228056959 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-phkgc" (UniqueName: "kubernetes.io/projected/6fe3e675-7e07-49d2-a664-a47d434dfd3b-kube-api-access-phkgc") pod "network-check-target-qbbd4" (UID: "6fe3e675-7e07-49d2-a664-a47d434dfd3b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:15:36.629037 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:36.628889 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 00:10:36 +0000 UTC" deadline="2028-04-24 12:51:47.983920686 +0000 UTC" Jul 10 00:15:36.629037 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:36.628956 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15708h36m11.354969297s" Jul 10 00:15:36.730279 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:36.729785 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qbbd4" Jul 10 00:15:36.730279 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:36.729920 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qbbd4" podUID="6fe3e675-7e07-49d2-a664-a47d434dfd3b" Jul 10 00:15:36.751132 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:36.751065 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5zlgk" event={"ID":"0318fc8c-a177-4995-a1b6-80ccf2cf11cd","Type":"ContainerStarted","Data":"5ffaf9505d180866f31b07748195e56857c8d5327a80dbe58d2d61b10d29b6d6"} Jul 10 00:15:36.752821 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:36.752792 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6xw4" event={"ID":"269e1974-6855-40f9-9f47-3dc20f729981","Type":"ContainerStarted","Data":"821937cdf42b34be77377820bb3f66cd0f969bdf8a14488e0a1e2b607445e789"} Jul 10 00:15:36.754480 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:36.754451 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gjk2r" event={"ID":"03b7fa42-b493-40ce-a0d1-ce267379f3df","Type":"ContainerStarted","Data":"cf4e2326671ac65387016d721472b1300213ef3c4f4b367ebc6cf4a92f3b44b5"} Jul 10 00:15:36.756247 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:36.756218 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-f55lr" event={"ID":"86b0dedd-e7fd-44d5-9663-ed7335d94063","Type":"ContainerStarted","Data":"c792a8dab6250aa392bf0eab3182cb5fd47a3703447456b9283a26fb5c8c51b1"} Jul 10 00:15:36.758639 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:36.758614 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-n5xgr" event={"ID":"08be9fc2-0ae2-45bc-96d1-5306a46e1357","Type":"ContainerStarted","Data":"9bca638d6c8f431f8a7d646f842757cf028c83e2ce6381e09e7fc4aca936e1fb"} Jul 10 00:15:36.760335 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:36.760309 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2mpf9" event={"ID":"0da6eed2-eb77-4e83-bb07-a6163eaca53a","Type":"ContainerStarted","Data":"7215e3b7bbb83d7f8af9569490b60b122edfb4e4739e6f083c3cc18557dcb95a"} Jul 10 00:15:36.762306 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:36.762277 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-101.ec2.internal" event={"ID":"29d40385cd1620022aeccb7b81ab99c1","Type":"ContainerStarted","Data":"0231fd010c0babd432a6b64849f607a7dd609822e026a4ee5e14962a78323d77"} Jul 10 00:15:36.763922 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:36.763895 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-shx2p" event={"ID":"dfc778a3-c60f-4f5e-8b6f-3e613228a7f9","Type":"ContainerStarted","Data":"008341e334ece3bd626ee29707cdf139faf33c1bfc0f28eafc864481932eb75e"} Jul 10 00:15:36.765422 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:36.765399 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" event={"ID":"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32","Type":"ContainerStarted","Data":"580dbf82de806d77b61a0908da8b624a9749063205860f6c8fcf523aba303620"} Jul 10 00:15:36.768491 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:36.768469 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5kkqc" event={"ID":"a88d645c-45e4-4592-b451-04eea24d3afb","Type":"ContainerStarted","Data":"9e2d02b21660cc80d19b1508a06cb556982867e3237ba321551dff464a8b4060"} Jul 10 00:15:37.216601 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:37.216567 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62933c8a-818d-4580-a6c9-6d26dbdfd50d-metrics-certs\") pod \"network-metrics-daemon-x8v89\" (UID: \"62933c8a-818d-4580-a6c9-6d26dbdfd50d\") " pod="openshift-multus/network-metrics-daemon-x8v89" Jul 10 00:15:37.216791 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:37.216774 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:15:37.216850 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:37.216843 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62933c8a-818d-4580-a6c9-6d26dbdfd50d-metrics-certs podName:62933c8a-818d-4580-a6c9-6d26dbdfd50d nodeName:}" failed. No retries permitted until 2026-07-10 00:15:39.216823136 +0000 UTC m=+6.035128615 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/62933c8a-818d-4580-a6c9-6d26dbdfd50d-metrics-certs") pod "network-metrics-daemon-x8v89" (UID: "62933c8a-818d-4580-a6c9-6d26dbdfd50d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:15:37.317723 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:37.317687 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/451130fe-547b-40c5-a7dc-412d57fb5db5-original-pull-secret\") pod \"global-pull-secret-syncer-nkcmr\" (UID: \"451130fe-547b-40c5-a7dc-412d57fb5db5\") " pod="kube-system/global-pull-secret-syncer-nkcmr" Jul 10 00:15:37.319709 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:37.317900 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 00:15:37.319709 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:37.317969 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/451130fe-547b-40c5-a7dc-412d57fb5db5-original-pull-secret podName:451130fe-547b-40c5-a7dc-412d57fb5db5 nodeName:}" failed. No retries permitted until 2026-07-10 00:15:39.317949246 +0000 UTC m=+6.136254725 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/451130fe-547b-40c5-a7dc-412d57fb5db5-original-pull-secret") pod "global-pull-secret-syncer-nkcmr" (UID: "451130fe-547b-40c5-a7dc-412d57fb5db5") : object "kube-system"/"original-pull-secret" not registered Jul 10 00:15:37.418518 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:37.418482 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phkgc\" (UniqueName: \"kubernetes.io/projected/6fe3e675-7e07-49d2-a664-a47d434dfd3b-kube-api-access-phkgc\") pod \"network-check-target-qbbd4\" (UID: \"6fe3e675-7e07-49d2-a664-a47d434dfd3b\") " pod="openshift-network-diagnostics/network-check-target-qbbd4" Jul 10 00:15:37.418776 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:37.418756 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 00:15:37.418843 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:37.418788 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 00:15:37.418843 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:37.418803 2562 projected.go:196] Error preparing data for projected volume kube-api-access-phkgc for pod openshift-network-diagnostics/network-check-target-qbbd4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:15:37.418964 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:37.418896 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6fe3e675-7e07-49d2-a664-a47d434dfd3b-kube-api-access-phkgc podName:6fe3e675-7e07-49d2-a664-a47d434dfd3b nodeName:}" failed. No retries permitted until 2026-07-10 00:15:39.418851053 +0000 UTC m=+6.237156531 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-phkgc" (UniqueName: "kubernetes.io/projected/6fe3e675-7e07-49d2-a664-a47d434dfd3b-kube-api-access-phkgc") pod "network-check-target-qbbd4" (UID: "6fe3e675-7e07-49d2-a664-a47d434dfd3b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:15:37.630628 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:37.629863 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 00:10:36 +0000 UTC" deadline="2028-01-18 05:21:57.511612413 +0000 UTC" Jul 10 00:15:37.630628 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:37.629904 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13373h6m19.881711986s" Jul 10 00:15:37.732707 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:37.732547 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x8v89" Jul 10 00:15:37.733312 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:37.732741 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x8v89" podUID="62933c8a-818d-4580-a6c9-6d26dbdfd50d" Jul 10 00:15:37.733312 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:37.732827 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nkcmr" Jul 10 00:15:37.733312 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:37.732908 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nkcmr" podUID="451130fe-547b-40c5-a7dc-412d57fb5db5" Jul 10 00:15:37.790953 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:37.790909 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-101.ec2.internal" event={"ID":"65368ef6b324db716ad2e852e6f60436","Type":"ContainerStarted","Data":"3b496dbabeb111c03ddfbf7a7c8b6354adfe35d0c86590803851622a009cc6fd"} Jul 10 00:15:38.730859 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:38.730348 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qbbd4" Jul 10 00:15:38.730859 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:38.730481 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qbbd4" podUID="6fe3e675-7e07-49d2-a664-a47d434dfd3b" Jul 10 00:15:39.235481 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:39.235433 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62933c8a-818d-4580-a6c9-6d26dbdfd50d-metrics-certs\") pod \"network-metrics-daemon-x8v89\" (UID: \"62933c8a-818d-4580-a6c9-6d26dbdfd50d\") " pod="openshift-multus/network-metrics-daemon-x8v89" Jul 10 00:15:39.235986 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:39.235756 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:15:39.235986 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:39.235883 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62933c8a-818d-4580-a6c9-6d26dbdfd50d-metrics-certs podName:62933c8a-818d-4580-a6c9-6d26dbdfd50d nodeName:}" failed. No retries permitted until 2026-07-10 00:15:43.235862785 +0000 UTC m=+10.054168261 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/62933c8a-818d-4580-a6c9-6d26dbdfd50d-metrics-certs") pod "network-metrics-daemon-x8v89" (UID: "62933c8a-818d-4580-a6c9-6d26dbdfd50d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:15:39.336534 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:39.336490 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/451130fe-547b-40c5-a7dc-412d57fb5db5-original-pull-secret\") pod \"global-pull-secret-syncer-nkcmr\" (UID: \"451130fe-547b-40c5-a7dc-412d57fb5db5\") " pod="kube-system/global-pull-secret-syncer-nkcmr" Jul 10 00:15:39.336715 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:39.336682 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 00:15:39.336798 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:39.336748 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/451130fe-547b-40c5-a7dc-412d57fb5db5-original-pull-secret podName:451130fe-547b-40c5-a7dc-412d57fb5db5 nodeName:}" failed. No retries permitted until 2026-07-10 00:15:43.33672969 +0000 UTC m=+10.155035181 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/451130fe-547b-40c5-a7dc-412d57fb5db5-original-pull-secret") pod "global-pull-secret-syncer-nkcmr" (UID: "451130fe-547b-40c5-a7dc-412d57fb5db5") : object "kube-system"/"original-pull-secret" not registered Jul 10 00:15:39.436955 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:39.436910 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phkgc\" (UniqueName: \"kubernetes.io/projected/6fe3e675-7e07-49d2-a664-a47d434dfd3b-kube-api-access-phkgc\") pod \"network-check-target-qbbd4\" (UID: \"6fe3e675-7e07-49d2-a664-a47d434dfd3b\") " pod="openshift-network-diagnostics/network-check-target-qbbd4" Jul 10 00:15:39.437217 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:39.437133 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 00:15:39.437217 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:39.437156 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 00:15:39.437217 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:39.437170 2562 projected.go:196] Error preparing data for projected volume kube-api-access-phkgc for pod openshift-network-diagnostics/network-check-target-qbbd4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:15:39.437389 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:39.437248 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6fe3e675-7e07-49d2-a664-a47d434dfd3b-kube-api-access-phkgc podName:6fe3e675-7e07-49d2-a664-a47d434dfd3b nodeName:}" failed. No retries permitted until 2026-07-10 00:15:43.437228113 +0000 UTC m=+10.255533592 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-phkgc" (UniqueName: "kubernetes.io/projected/6fe3e675-7e07-49d2-a664-a47d434dfd3b-kube-api-access-phkgc") pod "network-check-target-qbbd4" (UID: "6fe3e675-7e07-49d2-a664-a47d434dfd3b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:15:39.730401 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:39.730282 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x8v89" Jul 10 00:15:39.730401 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:39.730285 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nkcmr" Jul 10 00:15:39.730617 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:39.730462 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x8v89" podUID="62933c8a-818d-4580-a6c9-6d26dbdfd50d" Jul 10 00:15:39.730617 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:39.730584 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nkcmr" podUID="451130fe-547b-40c5-a7dc-412d57fb5db5" Jul 10 00:15:39.811167 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:39.811125 2562 generic.go:356] "Generic (PLEG): container finished" podID="65368ef6b324db716ad2e852e6f60436" containerID="3b496dbabeb111c03ddfbf7a7c8b6354adfe35d0c86590803851622a009cc6fd" exitCode=0 Jul 10 00:15:39.811356 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:39.811196 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-101.ec2.internal" event={"ID":"65368ef6b324db716ad2e852e6f60436","Type":"ContainerDied","Data":"3b496dbabeb111c03ddfbf7a7c8b6354adfe35d0c86590803851622a009cc6fd"} Jul 10 00:15:40.730215 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:40.730179 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qbbd4" Jul 10 00:15:40.730655 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:40.730327 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qbbd4" podUID="6fe3e675-7e07-49d2-a664-a47d434dfd3b" Jul 10 00:15:41.731201 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:41.731167 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x8v89" Jul 10 00:15:41.731702 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:41.731321 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x8v89" podUID="62933c8a-818d-4580-a6c9-6d26dbdfd50d" Jul 10 00:15:41.731702 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:41.731687 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nkcmr" Jul 10 00:15:41.731844 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:41.731808 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nkcmr" podUID="451130fe-547b-40c5-a7dc-412d57fb5db5" Jul 10 00:15:42.729521 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:42.729487 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qbbd4" Jul 10 00:15:42.729718 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:42.729600 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qbbd4" podUID="6fe3e675-7e07-49d2-a664-a47d434dfd3b" Jul 10 00:15:43.272234 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:43.272200 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62933c8a-818d-4580-a6c9-6d26dbdfd50d-metrics-certs\") pod \"network-metrics-daemon-x8v89\" (UID: \"62933c8a-818d-4580-a6c9-6d26dbdfd50d\") " pod="openshift-multus/network-metrics-daemon-x8v89" Jul 10 00:15:43.272673 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:43.272320 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:15:43.272673 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:43.272378 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62933c8a-818d-4580-a6c9-6d26dbdfd50d-metrics-certs podName:62933c8a-818d-4580-a6c9-6d26dbdfd50d nodeName:}" failed. No retries permitted until 2026-07-10 00:15:51.272359054 +0000 UTC m=+18.090664529 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/62933c8a-818d-4580-a6c9-6d26dbdfd50d-metrics-certs") pod "network-metrics-daemon-x8v89" (UID: "62933c8a-818d-4580-a6c9-6d26dbdfd50d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:15:43.372955 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:43.372917 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/451130fe-547b-40c5-a7dc-412d57fb5db5-original-pull-secret\") pod \"global-pull-secret-syncer-nkcmr\" (UID: \"451130fe-547b-40c5-a7dc-412d57fb5db5\") " pod="kube-system/global-pull-secret-syncer-nkcmr" Jul 10 00:15:43.373145 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:43.373084 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 00:15:43.373195 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:43.373172 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/451130fe-547b-40c5-a7dc-412d57fb5db5-original-pull-secret podName:451130fe-547b-40c5-a7dc-412d57fb5db5 nodeName:}" failed. No retries permitted until 2026-07-10 00:15:51.373149793 +0000 UTC m=+18.191455269 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/451130fe-547b-40c5-a7dc-412d57fb5db5-original-pull-secret") pod "global-pull-secret-syncer-nkcmr" (UID: "451130fe-547b-40c5-a7dc-412d57fb5db5") : object "kube-system"/"original-pull-secret" not registered Jul 10 00:15:43.473550 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:43.473514 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phkgc\" (UniqueName: \"kubernetes.io/projected/6fe3e675-7e07-49d2-a664-a47d434dfd3b-kube-api-access-phkgc\") pod \"network-check-target-qbbd4\" (UID: \"6fe3e675-7e07-49d2-a664-a47d434dfd3b\") " pod="openshift-network-diagnostics/network-check-target-qbbd4" Jul 10 00:15:43.473796 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:43.473776 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 00:15:43.473873 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:43.473803 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 00:15:43.473873 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:43.473814 2562 projected.go:196] Error preparing data for projected volume kube-api-access-phkgc for pod openshift-network-diagnostics/network-check-target-qbbd4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:15:43.473873 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:43.473868 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6fe3e675-7e07-49d2-a664-a47d434dfd3b-kube-api-access-phkgc podName:6fe3e675-7e07-49d2-a664-a47d434dfd3b nodeName:}" failed. No retries permitted until 2026-07-10 00:15:51.473848953 +0000 UTC m=+18.292154427 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-phkgc" (UniqueName: "kubernetes.io/projected/6fe3e675-7e07-49d2-a664-a47d434dfd3b-kube-api-access-phkgc") pod "network-check-target-qbbd4" (UID: "6fe3e675-7e07-49d2-a664-a47d434dfd3b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:15:43.730684 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:43.730595 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nkcmr" Jul 10 00:15:43.730845 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:43.730706 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nkcmr" podUID="451130fe-547b-40c5-a7dc-412d57fb5db5" Jul 10 00:15:43.730845 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:43.730761 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x8v89" Jul 10 00:15:43.730957 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:43.730887 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x8v89" podUID="62933c8a-818d-4580-a6c9-6d26dbdfd50d" Jul 10 00:15:44.729322 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:44.729287 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qbbd4" Jul 10 00:15:44.729809 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:44.729415 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qbbd4" podUID="6fe3e675-7e07-49d2-a664-a47d434dfd3b" Jul 10 00:15:45.730201 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:45.730163 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x8v89" Jul 10 00:15:45.730626 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:45.730329 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x8v89" podUID="62933c8a-818d-4580-a6c9-6d26dbdfd50d" Jul 10 00:15:45.730626 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:45.730381 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nkcmr" Jul 10 00:15:45.730626 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:45.730491 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nkcmr" podUID="451130fe-547b-40c5-a7dc-412d57fb5db5" Jul 10 00:15:46.729419 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:46.729386 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qbbd4" Jul 10 00:15:46.729594 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:46.729487 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qbbd4" podUID="6fe3e675-7e07-49d2-a664-a47d434dfd3b" Jul 10 00:15:47.730861 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:47.729908 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nkcmr" Jul 10 00:15:47.730861 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:47.730032 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nkcmr" podUID="451130fe-547b-40c5-a7dc-412d57fb5db5" Jul 10 00:15:47.730861 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:47.730444 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x8v89" Jul 10 00:15:47.730861 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:47.730545 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x8v89" podUID="62933c8a-818d-4580-a6c9-6d26dbdfd50d" Jul 10 00:15:47.838964 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:47.838877 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-101.ec2.internal" event={"ID":"29d40385cd1620022aeccb7b81ab99c1","Type":"ContainerStarted","Data":"b46155373fede585bc58b6d824b02ab6c1f447bec0324192ffb66bfeb8c34b0c"} Jul 10 00:15:47.849656 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:47.849629 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-101.ec2.internal" event={"ID":"65368ef6b324db716ad2e852e6f60436","Type":"ContainerStarted","Data":"96b23a355ebc07afb39c69793a00eb24e7164420245e0b1042c2240408208b1b"} Jul 10 00:15:47.855580 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:47.855557 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-shx2p" event={"ID":"dfc778a3-c60f-4f5e-8b6f-3e613228a7f9","Type":"ContainerStarted","Data":"7a4dea91bb9532122a001a225b72305ee8099eb7161a49d35110b9fe1675fc52"} Jul 10 00:15:47.868125 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:47.865982 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-101.ec2.internal" podStartSLOduration=12.865962608 podStartE2EDuration="12.865962608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 00:15:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:15:47.865524285 +0000 UTC m=+14.683829781" watchObservedRunningTime="2026-07-10 00:15:47.865962608 +0000 UTC m=+14.684268104" Jul 10 00:15:47.868125 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:47.866439 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-101.ec2.internal" podStartSLOduration=12.866430001 podStartE2EDuration="12.866430001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 00:15:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:15:47.851974284 +0000 UTC m=+14.670279780" watchObservedRunningTime="2026-07-10 00:15:47.866430001 +0000 UTC m=+14.684735495" Jul 10 00:15:47.880322 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:47.879523 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-shx2p" podStartSLOduration=1.230512729 podStartE2EDuration="12.879504956s" totalImagesPullingTime="11.648992227s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:15:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:15:47.879428475 +0000 UTC m=+14.697733969" watchObservedRunningTime="2026-07-10 00:15:47.879504956 +0000 UTC m=+14.697810451" Jul 10 00:15:48.729828 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:48.729640 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qbbd4" Jul 10 00:15:48.729979 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:48.729924 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qbbd4" podUID="6fe3e675-7e07-49d2-a664-a47d434dfd3b" Jul 10 00:15:48.859991 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:48.859951 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-n5xgr" event={"ID":"08be9fc2-0ae2-45bc-96d1-5306a46e1357","Type":"ContainerStarted","Data":"6952db2a4dc2a8aeb338defcbaa1b72552f5e2e25ce81b1a28d7ed2fe1c4e18b"} Jul 10 00:15:48.861706 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:48.861674 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2mpf9" event={"ID":"0da6eed2-eb77-4e83-bb07-a6163eaca53a","Type":"ContainerStarted","Data":"aec5cc43ad2faaf696c4b1a1ca4e7eedf9c22c3091eb4b167ecd3c8b408e39c2"} Jul 10 00:15:48.865787 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:48.863477 2562 generic.go:356] "Generic (PLEG): container finished" podID="a88d645c-45e4-4592-b451-04eea24d3afb" containerID="dc4b4f1c22002c46d17005ff37fd62b2e5c261e0e1e5e2bf32257d83abcf0cc7" exitCode=0 Jul 10 00:15:48.865787 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:48.863625 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5kkqc" event={"ID":"a88d645c-45e4-4592-b451-04eea24d3afb","Type":"ContainerDied","Data":"dc4b4f1c22002c46d17005ff37fd62b2e5c261e0e1e5e2bf32257d83abcf0cc7"} Jul 10 00:15:48.867656 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:48.867631 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6xw4" event={"ID":"269e1974-6855-40f9-9f47-3dc20f729981","Type":"ContainerStarted","Data":"423b99c6efe911772f3beda07248083320f885ad4e5f6f2883d945134654d63f"} Jul 10 00:15:48.869624 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:48.869596 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gjk2r" event={"ID":"03b7fa42-b493-40ce-a0d1-ce267379f3df","Type":"ContainerStarted","Data":"e63c4467bfc99befe52ba2a0e3f9d797b4789e2203c5c5b826f6ac2b47710b26"} Jul 10 00:15:48.876279 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:48.876241 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-n5xgr" podStartSLOduration=3.034341283 podStartE2EDuration="14.876227423s" totalImagesPullingTime="11.84188614s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 00:15:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:15:48.87584462 +0000 UTC m=+15.694150117" watchObservedRunningTime="2026-07-10 00:15:48.876227423 +0000 UTC m=+15.694532916" Jul 10 00:15:48.888581 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:48.888529 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-2mpf9" podStartSLOduration=3.185524338 podStartE2EDuration="14.888513906s" totalImagesPullingTime="11.702989568s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 00:15:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:15:48.88810946 +0000 UTC m=+15.706414949" watchObservedRunningTime="2026-07-10 00:15:48.888513906 +0000 UTC m=+15.706819400" Jul 10 00:15:48.934113 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:48.933998 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-gjk2r" podStartSLOduration=3.249534718 podStartE2EDuration="14.93397437s" totalImagesPullingTime="11.684439652s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 00:15:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:15:48.910469473 +0000 UTC m=+15.728774968" watchObservedRunningTime="2026-07-10 00:15:48.93397437 +0000 UTC m=+15.752279867" Jul 10 00:15:49.733220 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:49.733189 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nkcmr" Jul 10 00:15:49.733220 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:49.733220 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x8v89" Jul 10 00:15:49.733497 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:49.733296 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nkcmr" podUID="451130fe-547b-40c5-a7dc-412d57fb5db5" Jul 10 00:15:49.733497 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:49.733442 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x8v89" podUID="62933c8a-818d-4580-a6c9-6d26dbdfd50d" Jul 10 00:15:49.873744 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:49.873707 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-f55lr" event={"ID":"86b0dedd-e7fd-44d5-9663-ed7335d94063","Type":"ContainerStarted","Data":"993255b52bdcdb30343a4eaa40a8e733150dd7b920cfe0bbc913d42dea509595"} Jul 10 00:15:49.886291 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:49.886243 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-f55lr" podStartSLOduration=4.200507997 podStartE2EDuration="15.886227627s" totalImagesPullingTime="11.68571963s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 00:15:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:15:49.885858572 +0000 UTC m=+16.704164066" watchObservedRunningTime="2026-07-10 00:15:49.886227627 +0000 UTC m=+16.704533121" Jul 10 00:15:50.729701 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:50.729664 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qbbd4" Jul 10 00:15:50.729891 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:50.729798 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qbbd4" podUID="6fe3e675-7e07-49d2-a664-a47d434dfd3b" Jul 10 00:15:51.333900 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:51.333865 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62933c8a-818d-4580-a6c9-6d26dbdfd50d-metrics-certs\") pod \"network-metrics-daemon-x8v89\" (UID: \"62933c8a-818d-4580-a6c9-6d26dbdfd50d\") " pod="openshift-multus/network-metrics-daemon-x8v89" Jul 10 00:15:51.334432 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:51.333976 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:15:51.334432 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:51.334036 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62933c8a-818d-4580-a6c9-6d26dbdfd50d-metrics-certs podName:62933c8a-818d-4580-a6c9-6d26dbdfd50d nodeName:}" failed. No retries permitted until 2026-07-10 00:16:07.334020961 +0000 UTC m=+34.152326435 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/62933c8a-818d-4580-a6c9-6d26dbdfd50d-metrics-certs") pod "network-metrics-daemon-x8v89" (UID: "62933c8a-818d-4580-a6c9-6d26dbdfd50d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:15:51.434739 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:51.434699 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/451130fe-547b-40c5-a7dc-412d57fb5db5-original-pull-secret\") pod \"global-pull-secret-syncer-nkcmr\" (UID: \"451130fe-547b-40c5-a7dc-412d57fb5db5\") " pod="kube-system/global-pull-secret-syncer-nkcmr" Jul 10 00:15:51.434916 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:51.434873 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 00:15:51.434972 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:51.434942 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/451130fe-547b-40c5-a7dc-412d57fb5db5-original-pull-secret podName:451130fe-547b-40c5-a7dc-412d57fb5db5 nodeName:}" failed. No retries permitted until 2026-07-10 00:16:07.434926341 +0000 UTC m=+34.253231831 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/451130fe-547b-40c5-a7dc-412d57fb5db5-original-pull-secret") pod "global-pull-secret-syncer-nkcmr" (UID: "451130fe-547b-40c5-a7dc-412d57fb5db5") : object "kube-system"/"original-pull-secret" not registered Jul 10 00:15:51.535826 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:51.535792 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phkgc\" (UniqueName: \"kubernetes.io/projected/6fe3e675-7e07-49d2-a664-a47d434dfd3b-kube-api-access-phkgc\") pod \"network-check-target-qbbd4\" (UID: \"6fe3e675-7e07-49d2-a664-a47d434dfd3b\") " pod="openshift-network-diagnostics/network-check-target-qbbd4" Jul 10 00:15:51.536007 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:51.535987 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 00:15:51.536047 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:51.536013 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 00:15:51.536047 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:51.536023 2562 projected.go:196] Error preparing data for projected volume kube-api-access-phkgc for pod openshift-network-diagnostics/network-check-target-qbbd4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:15:51.536138 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:51.536103 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6fe3e675-7e07-49d2-a664-a47d434dfd3b-kube-api-access-phkgc podName:6fe3e675-7e07-49d2-a664-a47d434dfd3b nodeName:}" failed. No retries permitted until 2026-07-10 00:16:07.536070755 +0000 UTC m=+34.354376244 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-phkgc" (UniqueName: "kubernetes.io/projected/6fe3e675-7e07-49d2-a664-a47d434dfd3b-kube-api-access-phkgc") pod "network-check-target-qbbd4" (UID: "6fe3e675-7e07-49d2-a664-a47d434dfd3b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:15:51.729488 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:51.729366 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x8v89" Jul 10 00:15:51.729644 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:51.729544 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x8v89" podUID="62933c8a-818d-4580-a6c9-6d26dbdfd50d" Jul 10 00:15:51.729701 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:51.729636 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nkcmr" Jul 10 00:15:51.729805 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:51.729745 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nkcmr" podUID="451130fe-547b-40c5-a7dc-412d57fb5db5" Jul 10 00:15:51.801203 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:51.801168 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-shx2p" Jul 10 00:15:51.801795 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:51.801776 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-shx2p" Jul 10 00:15:52.730108 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:52.730065 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qbbd4" Jul 10 00:15:52.730557 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:52.730209 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qbbd4" podUID="6fe3e675-7e07-49d2-a664-a47d434dfd3b" Jul 10 00:15:52.825123 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:52.824876 2562 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 10 00:15:52.881567 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:52.881530 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5zlgk" event={"ID":"0318fc8c-a177-4995-a1b6-80ccf2cf11cd","Type":"ContainerStarted","Data":"dce00af93ce69ee74c4d29f1074ebdec79093013e7d08e46bd41ce6d7d0b5e68"} Jul 10 00:15:52.883290 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:52.883269 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6xw4" event={"ID":"269e1974-6855-40f9-9f47-3dc20f729981","Type":"ContainerStarted","Data":"9f8322f6ff3ca12d086920492c50b520f0cbbfccccefb49ae48abad96b20d67c"} Jul 10 00:15:52.898497 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:52.898442 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-5zlgk" podStartSLOduration=2.065479142 podStartE2EDuration="18.89842289s" totalImagesPullingTime="16.832943748s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 00:15:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:15:52.897540982 +0000 UTC m=+19.715846476" watchObservedRunningTime="2026-07-10 00:15:52.89842289 +0000 UTC m=+19.716728385" Jul 10 00:15:53.697973 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:53.697886 2562 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-10T00:15:52.825054258Z","UUID":"f2595504-7514-4fca-a697-cd8bf19c6e94","Handler":null,"Name":"","Endpoint":""} Jul 10 00:15:53.699278 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:53.699258 2562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 10 00:15:53.699389 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:53.699291 2562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 10 00:15:53.730649 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:53.730625 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x8v89" Jul 10 00:15:53.730976 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:53.730705 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x8v89" podUID="62933c8a-818d-4580-a6c9-6d26dbdfd50d" Jul 10 00:15:53.730976 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:53.730760 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nkcmr" Jul 10 00:15:53.730976 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:53.730866 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nkcmr" podUID="451130fe-547b-40c5-a7dc-412d57fb5db5" Jul 10 00:15:54.729852 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:54.729819 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qbbd4" Jul 10 00:15:54.730035 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:54.729947 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qbbd4" podUID="6fe3e675-7e07-49d2-a664-a47d434dfd3b" Jul 10 00:15:55.729460 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:55.729161 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x8v89" Jul 10 00:15:55.729460 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:55.729305 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x8v89" podUID="62933c8a-818d-4580-a6c9-6d26dbdfd50d" Jul 10 00:15:55.729460 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:55.729366 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nkcmr" Jul 10 00:15:55.730156 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:55.729482 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nkcmr" podUID="451130fe-547b-40c5-a7dc-412d57fb5db5" Jul 10 00:15:56.729469 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:56.729432 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qbbd4" Jul 10 00:15:56.729654 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:56.729567 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qbbd4" podUID="6fe3e675-7e07-49d2-a664-a47d434dfd3b" Jul 10 00:15:57.729897 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:57.729857 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x8v89" Jul 10 00:15:57.730444 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:57.729866 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nkcmr" Jul 10 00:15:57.730444 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:57.729987 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x8v89" podUID="62933c8a-818d-4580-a6c9-6d26dbdfd50d" Jul 10 00:15:57.730444 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:57.730107 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nkcmr" podUID="451130fe-547b-40c5-a7dc-412d57fb5db5" Jul 10 00:15:57.756464 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:57.756437 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-shx2p" Jul 10 00:15:57.756630 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:57.756541 2562 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 10 00:15:57.757155 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:57.757122 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-shx2p" Jul 10 00:15:58.729237 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:58.729210 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qbbd4" Jul 10 00:15:58.729406 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:58.729305 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qbbd4" podUID="6fe3e675-7e07-49d2-a664-a47d434dfd3b" Jul 10 00:15:59.729735 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:59.729700 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x8v89" Jul 10 00:15:59.730227 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:15:59.729711 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nkcmr" Jul 10 00:15:59.730227 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:59.729835 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x8v89" podUID="62933c8a-818d-4580-a6c9-6d26dbdfd50d" Jul 10 00:15:59.730227 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:15:59.729909 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nkcmr" podUID="451130fe-547b-40c5-a7dc-412d57fb5db5" Jul 10 00:16:00.729827 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:00.729806 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qbbd4" Jul 10 00:16:00.730188 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:16:00.729900 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qbbd4" podUID="6fe3e675-7e07-49d2-a664-a47d434dfd3b" Jul 10 00:16:00.902041 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:00.902010 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h5g9w_f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32/ovn-acl-logging/0.log" Jul 10 00:16:00.902350 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:00.902330 2562 generic.go:356] "Generic (PLEG): container finished" podID="f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32" containerID="1318ef4a0a61e25ed20d14ebb51dc8e893d606cab3da2babbf54029bdcabe1ed" exitCode=1 Jul 10 00:16:00.902420 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:00.902404 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" event={"ID":"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32","Type":"ContainerStarted","Data":"7f92cd84332b5027fb3238fd497cf6c36fb9084e5f476564c5bd4542c560c51a"} Jul 10 00:16:00.902476 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:00.902464 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" event={"ID":"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32","Type":"ContainerDied","Data":"1318ef4a0a61e25ed20d14ebb51dc8e893d606cab3da2babbf54029bdcabe1ed"} Jul 10 00:16:00.902520 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:00.902506 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" event={"ID":"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32","Type":"ContainerStarted","Data":"76c658ce1c80cb1d3d7b03c22c57cd2e626008456388be95bcb8c7b8dc90eb85"} Jul 10 00:16:00.904113 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:00.904094 2562 generic.go:356] "Generic (PLEG): container finished" podID="a88d645c-45e4-4592-b451-04eea24d3afb" containerID="1a62d89345ea9d6924df16a9ff8cc7ed3e7c9dd61d20a560b1b2ba8c1d73fed5" exitCode=0 Jul 10 00:16:00.904186 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:00.904159 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5kkqc" event={"ID":"a88d645c-45e4-4592-b451-04eea24d3afb","Type":"ContainerDied","Data":"1a62d89345ea9d6924df16a9ff8cc7ed3e7c9dd61d20a560b1b2ba8c1d73fed5"} Jul 10 00:16:00.906224 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:00.906204 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6xw4" event={"ID":"269e1974-6855-40f9-9f47-3dc20f729981","Type":"ContainerStarted","Data":"94378722a45e11e3f1018ea5e31288fff9ecd66cc44eed9d48cca35eea77c1d5"} Jul 10 00:16:01.730111 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:01.729914 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x8v89" Jul 10 00:16:01.730695 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:01.729965 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nkcmr" Jul 10 00:16:01.730695 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:16:01.730183 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x8v89" podUID="62933c8a-818d-4580-a6c9-6d26dbdfd50d" Jul 10 00:16:01.730695 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:16:01.730251 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nkcmr" podUID="451130fe-547b-40c5-a7dc-412d57fb5db5" Jul 10 00:16:01.911017 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:01.910992 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h5g9w_f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32/ovn-acl-logging/0.log" Jul 10 00:16:01.911419 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:01.911395 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" event={"ID":"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32","Type":"ContainerStarted","Data":"07b2754a4abd2f9ad65fab9dda6c5b48472fd32738b9d14a8ab692a542ca3202"} Jul 10 00:16:01.911481 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:01.911458 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" event={"ID":"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32","Type":"ContainerStarted","Data":"5d4e9ab504be77e3ffb189beb874bb858e3befdef64cae5ecf3c0f6982acbf3c"} Jul 10 00:16:01.911515 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:01.911502 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" event={"ID":"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32","Type":"ContainerStarted","Data":"a14a123830067aba1e3542cfdd12a94632dc5986aad8d6d7a6feca6bbfffbae8"} Jul 10 00:16:02.730072 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:02.730030 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qbbd4" Jul 10 00:16:02.730236 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:16:02.730149 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qbbd4" podUID="6fe3e675-7e07-49d2-a664-a47d434dfd3b" Jul 10 00:16:03.729833 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:03.729796 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x8v89" Jul 10 00:16:03.729994 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:16:03.729897 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x8v89" podUID="62933c8a-818d-4580-a6c9-6d26dbdfd50d" Jul 10 00:16:03.729994 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:03.729974 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nkcmr" Jul 10 00:16:03.730155 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:16:03.730072 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nkcmr" podUID="451130fe-547b-40c5-a7dc-412d57fb5db5" Jul 10 00:16:03.917336 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:03.917310 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h5g9w_f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32/ovn-acl-logging/0.log" Jul 10 00:16:03.917867 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:03.917648 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" event={"ID":"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32","Type":"ContainerStarted","Data":"e9d4f25b539397362c83b0bb389c2cd2c4454acc3458905d0fe97be3c1e2c90c"} Jul 10 00:16:03.919357 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:03.919330 2562 generic.go:356] "Generic (PLEG): container finished" podID="a88d645c-45e4-4592-b451-04eea24d3afb" containerID="f269c4c8f020ebd18082c92462cb55f3d49e47058fec1935bf300b96e78b02fc" exitCode=0 Jul 10 00:16:03.919468 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:03.919365 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5kkqc" event={"ID":"a88d645c-45e4-4592-b451-04eea24d3afb","Type":"ContainerDied","Data":"f269c4c8f020ebd18082c92462cb55f3d49e47058fec1935bf300b96e78b02fc"} Jul 10 00:16:03.942005 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:03.941964 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6xw4" podStartSLOduration=4.878802114 podStartE2EDuration="28.941951574s" totalImagesPullingTime="24.06314946s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 00:15:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:16:00.937835206 +0000 UTC m=+27.756140721" watchObservedRunningTime="2026-07-10 00:16:03.941951574 +0000 UTC m=+30.760257069" Jul 10 00:16:04.729243 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:04.729214 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qbbd4" Jul 10 00:16:04.729404 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:16:04.729309 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qbbd4" podUID="6fe3e675-7e07-49d2-a664-a47d434dfd3b" Jul 10 00:16:05.729459 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:05.729296 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x8v89" Jul 10 00:16:05.729834 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:05.729362 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nkcmr" Jul 10 00:16:05.729834 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:16:05.729576 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x8v89" podUID="62933c8a-818d-4580-a6c9-6d26dbdfd50d" Jul 10 00:16:05.729834 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:16:05.729605 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nkcmr" podUID="451130fe-547b-40c5-a7dc-412d57fb5db5" Jul 10 00:16:05.926829 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:05.926761 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h5g9w_f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32/ovn-acl-logging/0.log" Jul 10 00:16:05.927155 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:05.927117 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" event={"ID":"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32","Type":"ContainerStarted","Data":"ed8e4f4b31f7e8bef85aac662ea94516c2f635a317c44b72819f28c9b46ee122"} Jul 10 00:16:05.927491 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:05.927461 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:16:05.927565 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:05.927556 2562 scope.go:117] "RemoveContainer" containerID="1318ef4a0a61e25ed20d14ebb51dc8e893d606cab3da2babbf54029bdcabe1ed" Jul 10 00:16:05.929641 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:05.929619 2562 generic.go:356] "Generic (PLEG): container finished" podID="a88d645c-45e4-4592-b451-04eea24d3afb" containerID="777c93e62968123ce50928799c143fa086799d1d4fffc78c1e2d5d0694a2d59d" exitCode=0 Jul 10 00:16:05.929748 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:05.929657 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5kkqc" event={"ID":"a88d645c-45e4-4592-b451-04eea24d3afb","Type":"ContainerDied","Data":"777c93e62968123ce50928799c143fa086799d1d4fffc78c1e2d5d0694a2d59d"} Jul 10 00:16:05.942863 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:05.942847 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:16:06.729939 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:06.729904 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qbbd4" Jul 10 00:16:06.730432 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:16:06.730043 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qbbd4" podUID="6fe3e675-7e07-49d2-a664-a47d434dfd3b" Jul 10 00:16:06.936451 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:06.936426 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h5g9w_f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32/ovn-acl-logging/0.log" Jul 10 00:16:06.936866 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:06.936822 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" event={"ID":"f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32","Type":"ContainerStarted","Data":"0a0186176c9563c00ebc76e3ae9397e4c4b0f073cbe82c3e2728cef23ce42e9e"} Jul 10 00:16:06.937348 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:06.937278 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:16:06.937348 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:06.937351 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:16:06.954183 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:06.954160 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:16:06.965266 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:06.965218 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" podStartSLOduration=7.41964751 podStartE2EDuration="31.965200065s" totalImagesPullingTime="24.545552555s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 00:15:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:16:06.964155335 +0000 UTC m=+33.782460830" watchObservedRunningTime="2026-07-10 00:16:06.965200065 +0000 UTC m=+33.783505561" Jul 10 00:16:07.358182 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:07.358156 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62933c8a-818d-4580-a6c9-6d26dbdfd50d-metrics-certs\") pod \"network-metrics-daemon-x8v89\" (UID: \"62933c8a-818d-4580-a6c9-6d26dbdfd50d\") " pod="openshift-multus/network-metrics-daemon-x8v89" Jul 10 00:16:07.358357 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:16:07.358305 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:16:07.358394 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:16:07.358362 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62933c8a-818d-4580-a6c9-6d26dbdfd50d-metrics-certs podName:62933c8a-818d-4580-a6c9-6d26dbdfd50d nodeName:}" failed. No retries permitted until 2026-07-10 00:16:39.358347714 +0000 UTC m=+66.176653191 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/62933c8a-818d-4580-a6c9-6d26dbdfd50d-metrics-certs") pod "network-metrics-daemon-x8v89" (UID: "62933c8a-818d-4580-a6c9-6d26dbdfd50d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:16:07.372328 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:07.372298 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nkcmr"] Jul 10 00:16:07.372479 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:07.372426 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nkcmr" Jul 10 00:16:07.372559 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:16:07.372523 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nkcmr" podUID="451130fe-547b-40c5-a7dc-412d57fb5db5" Jul 10 00:16:07.375712 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:07.375609 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qbbd4"] Jul 10 00:16:07.375712 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:07.375705 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qbbd4" Jul 10 00:16:07.376474 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:16:07.375799 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qbbd4" podUID="6fe3e675-7e07-49d2-a664-a47d434dfd3b" Jul 10 00:16:07.376474 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:07.376455 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-x8v89"] Jul 10 00:16:07.376583 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:07.376560 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x8v89" Jul 10 00:16:07.376681 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:16:07.376655 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x8v89" podUID="62933c8a-818d-4580-a6c9-6d26dbdfd50d" Jul 10 00:16:07.458531 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:07.458499 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/451130fe-547b-40c5-a7dc-412d57fb5db5-original-pull-secret\") pod \"global-pull-secret-syncer-nkcmr\" (UID: \"451130fe-547b-40c5-a7dc-412d57fb5db5\") " pod="kube-system/global-pull-secret-syncer-nkcmr" Jul 10 00:16:07.458697 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:16:07.458680 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 00:16:07.458752 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:16:07.458743 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/451130fe-547b-40c5-a7dc-412d57fb5db5-original-pull-secret podName:451130fe-547b-40c5-a7dc-412d57fb5db5 nodeName:}" failed. No retries permitted until 2026-07-10 00:16:39.458728994 +0000 UTC m=+66.277034469 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/451130fe-547b-40c5-a7dc-412d57fb5db5-original-pull-secret") pod "global-pull-secret-syncer-nkcmr" (UID: "451130fe-547b-40c5-a7dc-412d57fb5db5") : object "kube-system"/"original-pull-secret" not registered Jul 10 00:16:07.559268 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:07.559237 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phkgc\" (UniqueName: \"kubernetes.io/projected/6fe3e675-7e07-49d2-a664-a47d434dfd3b-kube-api-access-phkgc\") pod \"network-check-target-qbbd4\" (UID: \"6fe3e675-7e07-49d2-a664-a47d434dfd3b\") " pod="openshift-network-diagnostics/network-check-target-qbbd4" Jul 10 00:16:07.559422 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:16:07.559407 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 00:16:07.559461 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:16:07.559431 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 00:16:07.559461 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:16:07.559443 2562 projected.go:196] Error preparing data for projected volume kube-api-access-phkgc for pod openshift-network-diagnostics/network-check-target-qbbd4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:16:07.559542 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:16:07.559497 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6fe3e675-7e07-49d2-a664-a47d434dfd3b-kube-api-access-phkgc podName:6fe3e675-7e07-49d2-a664-a47d434dfd3b nodeName:}" failed. No retries permitted until 2026-07-10 00:16:39.559480751 +0000 UTC m=+66.377786227 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-phkgc" (UniqueName: "kubernetes.io/projected/6fe3e675-7e07-49d2-a664-a47d434dfd3b-kube-api-access-phkgc") pod "network-check-target-qbbd4" (UID: "6fe3e675-7e07-49d2-a664-a47d434dfd3b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:16:08.729644 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:08.729610 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nkcmr" Jul 10 00:16:08.730121 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:16:08.729737 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nkcmr" podUID="451130fe-547b-40c5-a7dc-412d57fb5db5" Jul 10 00:16:08.730121 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:08.729843 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x8v89" Jul 10 00:16:08.730121 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:16:08.729964 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x8v89" podUID="62933c8a-818d-4580-a6c9-6d26dbdfd50d" Jul 10 00:16:09.729860 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:09.729755 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qbbd4" Jul 10 00:16:09.730314 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:16:09.729884 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qbbd4" podUID="6fe3e675-7e07-49d2-a664-a47d434dfd3b" Jul 10 00:16:10.729777 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:10.729583 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nkcmr" Jul 10 00:16:10.729873 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:10.729584 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x8v89" Jul 10 00:16:10.729873 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:16:10.729837 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nkcmr" podUID="451130fe-547b-40c5-a7dc-412d57fb5db5" Jul 10 00:16:10.730331 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:16:10.729897 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x8v89" podUID="62933c8a-818d-4580-a6c9-6d26dbdfd50d" Jul 10 00:16:10.949192 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:10.949162 2562 generic.go:356] "Generic (PLEG): container finished" podID="a88d645c-45e4-4592-b451-04eea24d3afb" containerID="b46273d9a1b8cc29be3d0d6bf643e11eb190254d84b075f74461d91840cf2e3a" exitCode=0 Jul 10 00:16:10.949319 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:10.949209 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5kkqc" event={"ID":"a88d645c-45e4-4592-b451-04eea24d3afb","Type":"ContainerDied","Data":"b46273d9a1b8cc29be3d0d6bf643e11eb190254d84b075f74461d91840cf2e3a"} Jul 10 00:16:11.729645 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:11.729612 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qbbd4" Jul 10 00:16:11.729827 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:16:11.729729 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qbbd4" podUID="6fe3e675-7e07-49d2-a664-a47d434dfd3b" Jul 10 00:16:11.954186 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:11.954151 2562 generic.go:356] "Generic (PLEG): container finished" podID="a88d645c-45e4-4592-b451-04eea24d3afb" containerID="934466c65f36366c8aa36840dbcc1b58cbc32d8ba594fd7efb737598e802526d" exitCode=0 Jul 10 00:16:11.954512 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:11.954199 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5kkqc" event={"ID":"a88d645c-45e4-4592-b451-04eea24d3afb","Type":"ContainerDied","Data":"934466c65f36366c8aa36840dbcc1b58cbc32d8ba594fd7efb737598e802526d"} Jul 10 00:16:12.730072 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:12.730032 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nkcmr" Jul 10 00:16:12.730291 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:16:12.730162 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nkcmr" podUID="451130fe-547b-40c5-a7dc-412d57fb5db5" Jul 10 00:16:12.730291 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:12.730216 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x8v89" Jul 10 00:16:12.730376 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:16:12.730309 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x8v89" podUID="62933c8a-818d-4580-a6c9-6d26dbdfd50d" Jul 10 00:16:12.960024 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:12.959993 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5kkqc" event={"ID":"a88d645c-45e4-4592-b451-04eea24d3afb","Type":"ContainerStarted","Data":"b61fe2c433555fa70692c4949d1094d5a69ec601276620fe87cd1b99087e1998"} Jul 10 00:16:12.993773 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:12.993680 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-5kkqc" podStartSLOduration=6.468286615 podStartE2EDuration="37.993664724s" totalImagesPullingTime="31.525378109s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 00:15:35 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:16:12.978888271 +0000 UTC m=+39.797193778" watchObservedRunningTime="2026-07-10 00:16:12.993664724 +0000 UTC m=+39.811970220" Jul 10 00:16:13.476784 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.476749 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-101.ec2.internal" event="NodeReady" Jul 10 00:16:13.476942 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.476883 2562 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 10 00:16:13.510341 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.510308 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5d58f56c87-wn9cc"] Jul 10 00:16:13.535181 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.535155 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9mt8s"] Jul 10 00:16:13.535330 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.535281 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d58f56c87-wn9cc" Jul 10 00:16:13.537464 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.537230 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 10 00:16:13.537577 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.537243 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 10 00:16:13.537577 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.537384 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 10 00:16:13.537577 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.537444 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-s69wz\"" Jul 10 00:16:13.543020 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.543001 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 10 00:16:13.550971 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.550952 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-tjh96"] Jul 10 00:16:13.550971 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.550961 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9mt8s" Jul 10 00:16:13.553095 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.553058 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 10 00:16:13.553287 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.553272 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 10 00:16:13.553367 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.553300 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-crl8x\"" Jul 10 00:16:13.572995 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.572976 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d58f56c87-wn9cc"] Jul 10 00:16:13.573103 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.573005 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9mt8s"] Jul 10 00:16:13.573103 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.573025 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tjh96"] Jul 10 00:16:13.573103 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.573098 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tjh96" Jul 10 00:16:13.575099 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.575061 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 10 00:16:13.575195 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.575102 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 10 00:16:13.575195 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.575129 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 10 00:16:13.575195 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.575129 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vlbng\"" Jul 10 00:16:13.705549 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.705520 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/55dcffca-f335-4b01-a6b8-4b2356d2fe57-cert\") pod \"ingress-canary-tjh96\" (UID: \"55dcffca-f335-4b01-a6b8-4b2356d2fe57\") " pod="openshift-ingress-canary/ingress-canary-tjh96" Jul 10 00:16:13.705549 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.705552 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f4cp\" (UniqueName: \"kubernetes.io/projected/b56f3715-4283-4226-8188-d1b3869826db-kube-api-access-9f4cp\") pod \"dns-default-9mt8s\" (UID: \"b56f3715-4283-4226-8188-d1b3869826db\") " pod="openshift-dns/dns-default-9mt8s" Jul 10 00:16:13.705744 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.705570 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b56f3715-4283-4226-8188-d1b3869826db-config-volume\") pod \"dns-default-9mt8s\" (UID: \"b56f3715-4283-4226-8188-d1b3869826db\") " pod="openshift-dns/dns-default-9mt8s" Jul 10 00:16:13.705744 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.705589 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bbbf0b29-8280-450c-99e6-5965971c89da-bound-sa-token\") pod \"image-registry-5d58f56c87-wn9cc\" (UID: \"bbbf0b29-8280-450c-99e6-5965971c89da\") " pod="openshift-image-registry/image-registry-5d58f56c87-wn9cc" Jul 10 00:16:13.705744 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.705701 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bbbf0b29-8280-450c-99e6-5965971c89da-installation-pull-secrets\") pod \"image-registry-5d58f56c87-wn9cc\" (UID: \"bbbf0b29-8280-450c-99e6-5965971c89da\") " pod="openshift-image-registry/image-registry-5d58f56c87-wn9cc" Jul 10 00:16:13.705744 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.705725 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bbbf0b29-8280-450c-99e6-5965971c89da-trusted-ca\") pod \"image-registry-5d58f56c87-wn9cc\" (UID: \"bbbf0b29-8280-450c-99e6-5965971c89da\") " pod="openshift-image-registry/image-registry-5d58f56c87-wn9cc" Jul 10 00:16:13.705886 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.705762 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b56f3715-4283-4226-8188-d1b3869826db-metrics-tls\") pod \"dns-default-9mt8s\" (UID: \"b56f3715-4283-4226-8188-d1b3869826db\") " pod="openshift-dns/dns-default-9mt8s" Jul 10 00:16:13.705886 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.705785 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bbbf0b29-8280-450c-99e6-5965971c89da-ca-trust-extracted\") pod \"image-registry-5d58f56c87-wn9cc\" (UID: \"bbbf0b29-8280-450c-99e6-5965971c89da\") " pod="openshift-image-registry/image-registry-5d58f56c87-wn9cc" Jul 10 00:16:13.705886 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.705807 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcdpg\" (UniqueName: \"kubernetes.io/projected/bbbf0b29-8280-450c-99e6-5965971c89da-kube-api-access-tcdpg\") pod \"image-registry-5d58f56c87-wn9cc\" (UID: \"bbbf0b29-8280-450c-99e6-5965971c89da\") " pod="openshift-image-registry/image-registry-5d58f56c87-wn9cc" Jul 10 00:16:13.705886 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.705826 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b56f3715-4283-4226-8188-d1b3869826db-tmp-dir\") pod \"dns-default-9mt8s\" (UID: \"b56f3715-4283-4226-8188-d1b3869826db\") " pod="openshift-dns/dns-default-9mt8s" Jul 10 00:16:13.705886 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.705841 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rttkr\" (UniqueName: \"kubernetes.io/projected/55dcffca-f335-4b01-a6b8-4b2356d2fe57-kube-api-access-rttkr\") pod \"ingress-canary-tjh96\" (UID: \"55dcffca-f335-4b01-a6b8-4b2356d2fe57\") " pod="openshift-ingress-canary/ingress-canary-tjh96" Jul 10 00:16:13.705886 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.705857 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bbbf0b29-8280-450c-99e6-5965971c89da-registry-tls\") pod \"image-registry-5d58f56c87-wn9cc\" (UID: \"bbbf0b29-8280-450c-99e6-5965971c89da\") " pod="openshift-image-registry/image-registry-5d58f56c87-wn9cc" Jul 10 00:16:13.705886 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.705872 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bbbf0b29-8280-450c-99e6-5965971c89da-registry-certificates\") pod \"image-registry-5d58f56c87-wn9cc\" (UID: \"bbbf0b29-8280-450c-99e6-5965971c89da\") " pod="openshift-image-registry/image-registry-5d58f56c87-wn9cc" Jul 10 00:16:13.706178 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.705891 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bbbf0b29-8280-450c-99e6-5965971c89da-image-registry-private-configuration\") pod \"image-registry-5d58f56c87-wn9cc\" (UID: \"bbbf0b29-8280-450c-99e6-5965971c89da\") " pod="openshift-image-registry/image-registry-5d58f56c87-wn9cc" Jul 10 00:16:13.730878 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.730784 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qbbd4" Jul 10 00:16:13.733003 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.732986 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 10 00:16:13.733148 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.733008 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 10 00:16:13.733148 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.733008 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qmtcf\"" Jul 10 00:16:13.806185 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.806164 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bbbf0b29-8280-450c-99e6-5965971c89da-installation-pull-secrets\") pod \"image-registry-5d58f56c87-wn9cc\" (UID: \"bbbf0b29-8280-450c-99e6-5965971c89da\") " pod="openshift-image-registry/image-registry-5d58f56c87-wn9cc" Jul 10 00:16:13.806303 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.806191 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bbbf0b29-8280-450c-99e6-5965971c89da-trusted-ca\") pod \"image-registry-5d58f56c87-wn9cc\" (UID: \"bbbf0b29-8280-450c-99e6-5965971c89da\") " pod="openshift-image-registry/image-registry-5d58f56c87-wn9cc" Jul 10 00:16:13.806303 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.806220 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b56f3715-4283-4226-8188-d1b3869826db-metrics-tls\") pod \"dns-default-9mt8s\" (UID: \"b56f3715-4283-4226-8188-d1b3869826db\") " pod="openshift-dns/dns-default-9mt8s" Jul 10 00:16:13.806303 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.806236 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bbbf0b29-8280-450c-99e6-5965971c89da-ca-trust-extracted\") pod \"image-registry-5d58f56c87-wn9cc\" (UID: \"bbbf0b29-8280-450c-99e6-5965971c89da\") " pod="openshift-image-registry/image-registry-5d58f56c87-wn9cc" Jul 10 00:16:13.806303 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.806254 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcdpg\" (UniqueName: \"kubernetes.io/projected/bbbf0b29-8280-450c-99e6-5965971c89da-kube-api-access-tcdpg\") pod \"image-registry-5d58f56c87-wn9cc\" (UID: \"bbbf0b29-8280-450c-99e6-5965971c89da\") " pod="openshift-image-registry/image-registry-5d58f56c87-wn9cc" Jul 10 00:16:13.806303 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.806274 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b56f3715-4283-4226-8188-d1b3869826db-tmp-dir\") pod \"dns-default-9mt8s\" (UID: \"b56f3715-4283-4226-8188-d1b3869826db\") " pod="openshift-dns/dns-default-9mt8s" Jul 10 00:16:13.806303 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.806296 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rttkr\" (UniqueName: \"kubernetes.io/projected/55dcffca-f335-4b01-a6b8-4b2356d2fe57-kube-api-access-rttkr\") pod \"ingress-canary-tjh96\" (UID: \"55dcffca-f335-4b01-a6b8-4b2356d2fe57\") " pod="openshift-ingress-canary/ingress-canary-tjh96" Jul 10 00:16:13.806570 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.806317 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bbbf0b29-8280-450c-99e6-5965971c89da-registry-tls\") pod \"image-registry-5d58f56c87-wn9cc\" (UID: \"bbbf0b29-8280-450c-99e6-5965971c89da\") " pod="openshift-image-registry/image-registry-5d58f56c87-wn9cc" Jul 10 00:16:13.806570 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.806342 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bbbf0b29-8280-450c-99e6-5965971c89da-registry-certificates\") pod \"image-registry-5d58f56c87-wn9cc\" (UID: \"bbbf0b29-8280-450c-99e6-5965971c89da\") " pod="openshift-image-registry/image-registry-5d58f56c87-wn9cc" Jul 10 00:16:13.806570 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.806368 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bbbf0b29-8280-450c-99e6-5965971c89da-image-registry-private-configuration\") pod \"image-registry-5d58f56c87-wn9cc\" (UID: \"bbbf0b29-8280-450c-99e6-5965971c89da\") " pod="openshift-image-registry/image-registry-5d58f56c87-wn9cc" Jul 10 00:16:13.806570 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.806393 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/55dcffca-f335-4b01-a6b8-4b2356d2fe57-cert\") pod \"ingress-canary-tjh96\" (UID: \"55dcffca-f335-4b01-a6b8-4b2356d2fe57\") " pod="openshift-ingress-canary/ingress-canary-tjh96" Jul 10 00:16:13.806570 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:16:13.806414 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 00:16:13.806570 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.806422 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f4cp\" (UniqueName: \"kubernetes.io/projected/b56f3715-4283-4226-8188-d1b3869826db-kube-api-access-9f4cp\") pod \"dns-default-9mt8s\" (UID: \"b56f3715-4283-4226-8188-d1b3869826db\") " pod="openshift-dns/dns-default-9mt8s" Jul 10 00:16:13.806570 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.806449 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b56f3715-4283-4226-8188-d1b3869826db-config-volume\") pod \"dns-default-9mt8s\" (UID: \"b56f3715-4283-4226-8188-d1b3869826db\") " pod="openshift-dns/dns-default-9mt8s" Jul 10 00:16:13.806570 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:16:13.806476 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b56f3715-4283-4226-8188-d1b3869826db-metrics-tls podName:b56f3715-4283-4226-8188-d1b3869826db nodeName:}" failed. No retries permitted until 2026-07-10 00:16:14.306454436 +0000 UTC m=+41.124759910 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b56f3715-4283-4226-8188-d1b3869826db-metrics-tls") pod "dns-default-9mt8s" (UID: "b56f3715-4283-4226-8188-d1b3869826db") : secret "dns-default-metrics-tls" not found Jul 10 00:16:13.806570 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:16:13.806415 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 00:16:13.806964 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.806941 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b56f3715-4283-4226-8188-d1b3869826db-tmp-dir\") pod \"dns-default-9mt8s\" (UID: \"b56f3715-4283-4226-8188-d1b3869826db\") " pod="openshift-dns/dns-default-9mt8s" Jul 10 00:16:13.807133 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.807114 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bbbf0b29-8280-450c-99e6-5965971c89da-ca-trust-extracted\") pod \"image-registry-5d58f56c87-wn9cc\" (UID: \"bbbf0b29-8280-450c-99e6-5965971c89da\") " pod="openshift-image-registry/image-registry-5d58f56c87-wn9cc" Jul 10 00:16:13.807208 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:16:13.807169 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d58f56c87-wn9cc: secret "image-registry-tls" not found Jul 10 00:16:13.807267 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:16:13.807230 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bbbf0b29-8280-450c-99e6-5965971c89da-registry-tls podName:bbbf0b29-8280-450c-99e6-5965971c89da nodeName:}" failed. No retries permitted until 2026-07-10 00:16:14.307204566 +0000 UTC m=+41.125510044 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bbbf0b29-8280-450c-99e6-5965971c89da-registry-tls") pod "image-registry-5d58f56c87-wn9cc" (UID: "bbbf0b29-8280-450c-99e6-5965971c89da") : secret "image-registry-tls" not found Jul 10 00:16:13.807331 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.807268 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bbbf0b29-8280-450c-99e6-5965971c89da-bound-sa-token\") pod \"image-registry-5d58f56c87-wn9cc\" (UID: \"bbbf0b29-8280-450c-99e6-5965971c89da\") " pod="openshift-image-registry/image-registry-5d58f56c87-wn9cc" Jul 10 00:16:13.807485 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.807448 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bbbf0b29-8280-450c-99e6-5965971c89da-registry-certificates\") pod \"image-registry-5d58f56c87-wn9cc\" (UID: \"bbbf0b29-8280-450c-99e6-5965971c89da\") " pod="openshift-image-registry/image-registry-5d58f56c87-wn9cc" Jul 10 00:16:13.807594 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.807572 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bbbf0b29-8280-450c-99e6-5965971c89da-trusted-ca\") pod \"image-registry-5d58f56c87-wn9cc\" (UID: \"bbbf0b29-8280-450c-99e6-5965971c89da\") " pod="openshift-image-registry/image-registry-5d58f56c87-wn9cc" Jul 10 00:16:13.807920 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.807852 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b56f3715-4283-4226-8188-d1b3869826db-config-volume\") pod \"dns-default-9mt8s\" (UID: \"b56f3715-4283-4226-8188-d1b3869826db\") " pod="openshift-dns/dns-default-9mt8s" Jul 10 00:16:13.812102 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.812049 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bbbf0b29-8280-450c-99e6-5965971c89da-installation-pull-secrets\") pod \"image-registry-5d58f56c87-wn9cc\" (UID: \"bbbf0b29-8280-450c-99e6-5965971c89da\") " pod="openshift-image-registry/image-registry-5d58f56c87-wn9cc" Jul 10 00:16:13.812484 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.812461 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bbbf0b29-8280-450c-99e6-5965971c89da-image-registry-private-configuration\") pod \"image-registry-5d58f56c87-wn9cc\" (UID: \"bbbf0b29-8280-450c-99e6-5965971c89da\") " pod="openshift-image-registry/image-registry-5d58f56c87-wn9cc" Jul 10 00:16:13.812627 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.812604 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/55dcffca-f335-4b01-a6b8-4b2356d2fe57-cert\") pod \"ingress-canary-tjh96\" (UID: \"55dcffca-f335-4b01-a6b8-4b2356d2fe57\") " pod="openshift-ingress-canary/ingress-canary-tjh96" Jul 10 00:16:13.814519 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.814499 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rttkr\" (UniqueName: \"kubernetes.io/projected/55dcffca-f335-4b01-a6b8-4b2356d2fe57-kube-api-access-rttkr\") pod \"ingress-canary-tjh96\" (UID: \"55dcffca-f335-4b01-a6b8-4b2356d2fe57\") " pod="openshift-ingress-canary/ingress-canary-tjh96" Jul 10 00:16:13.817421 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.817367 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f4cp\" (UniqueName: \"kubernetes.io/projected/b56f3715-4283-4226-8188-d1b3869826db-kube-api-access-9f4cp\") pod \"dns-default-9mt8s\" (UID: \"b56f3715-4283-4226-8188-d1b3869826db\") " pod="openshift-dns/dns-default-9mt8s" Jul 10 00:16:13.817759 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.817741 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bbbf0b29-8280-450c-99e6-5965971c89da-bound-sa-token\") pod \"image-registry-5d58f56c87-wn9cc\" (UID: \"bbbf0b29-8280-450c-99e6-5965971c89da\") " pod="openshift-image-registry/image-registry-5d58f56c87-wn9cc" Jul 10 00:16:13.817934 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.817906 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcdpg\" (UniqueName: \"kubernetes.io/projected/bbbf0b29-8280-450c-99e6-5965971c89da-kube-api-access-tcdpg\") pod \"image-registry-5d58f56c87-wn9cc\" (UID: \"bbbf0b29-8280-450c-99e6-5965971c89da\") " pod="openshift-image-registry/image-registry-5d58f56c87-wn9cc" Jul 10 00:16:13.880985 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:13.880962 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tjh96" Jul 10 00:16:14.073243 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:14.073216 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tjh96"] Jul 10 00:16:14.077337 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:16:14.077308 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55dcffca_f335_4b01_a6b8_4b2356d2fe57.slice/crio-05f717fc1a9f95198947a1cae5cbe8d4c572d52b442bd17290b6cbfec9044d3e WatchSource:0}: Error finding container 05f717fc1a9f95198947a1cae5cbe8d4c572d52b442bd17290b6cbfec9044d3e: Status 404 returned error can't find the container with id 05f717fc1a9f95198947a1cae5cbe8d4c572d52b442bd17290b6cbfec9044d3e Jul 10 00:16:14.310851 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:14.310821 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b56f3715-4283-4226-8188-d1b3869826db-metrics-tls\") pod \"dns-default-9mt8s\" (UID: \"b56f3715-4283-4226-8188-d1b3869826db\") " pod="openshift-dns/dns-default-9mt8s" Jul 10 00:16:14.311020 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:14.310859 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bbbf0b29-8280-450c-99e6-5965971c89da-registry-tls\") pod \"image-registry-5d58f56c87-wn9cc\" (UID: \"bbbf0b29-8280-450c-99e6-5965971c89da\") " pod="openshift-image-registry/image-registry-5d58f56c87-wn9cc" Jul 10 00:16:14.311020 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:16:14.310979 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 00:16:14.311020 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:16:14.310982 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 00:16:14.311216 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:16:14.311054 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b56f3715-4283-4226-8188-d1b3869826db-metrics-tls podName:b56f3715-4283-4226-8188-d1b3869826db nodeName:}" failed. No retries permitted until 2026-07-10 00:16:15.311032154 +0000 UTC m=+42.129337644 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b56f3715-4283-4226-8188-d1b3869826db-metrics-tls") pod "dns-default-9mt8s" (UID: "b56f3715-4283-4226-8188-d1b3869826db") : secret "dns-default-metrics-tls" not found Jul 10 00:16:14.311216 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:16:14.310990 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d58f56c87-wn9cc: secret "image-registry-tls" not found Jul 10 00:16:14.311216 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:16:14.311139 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bbbf0b29-8280-450c-99e6-5965971c89da-registry-tls podName:bbbf0b29-8280-450c-99e6-5965971c89da nodeName:}" failed. No retries permitted until 2026-07-10 00:16:15.311127745 +0000 UTC m=+42.129433219 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bbbf0b29-8280-450c-99e6-5965971c89da-registry-tls") pod "image-registry-5d58f56c87-wn9cc" (UID: "bbbf0b29-8280-450c-99e6-5965971c89da") : secret "image-registry-tls" not found Jul 10 00:16:14.729448 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:14.729361 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nkcmr" Jul 10 00:16:14.729616 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:14.729370 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x8v89" Jul 10 00:16:14.731610 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:14.731585 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 00:16:14.732347 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:14.732333 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 00:16:14.732422 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:14.732383 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qn6qf\"" Jul 10 00:16:14.965977 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:14.965940 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tjh96" event={"ID":"55dcffca-f335-4b01-a6b8-4b2356d2fe57","Type":"ContainerStarted","Data":"05f717fc1a9f95198947a1cae5cbe8d4c572d52b442bd17290b6cbfec9044d3e"} Jul 10 00:16:15.319057 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:15.319024 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b56f3715-4283-4226-8188-d1b3869826db-metrics-tls\") pod \"dns-default-9mt8s\" (UID: \"b56f3715-4283-4226-8188-d1b3869826db\") " pod="openshift-dns/dns-default-9mt8s" Jul 10 00:16:15.319471 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:15.319067 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bbbf0b29-8280-450c-99e6-5965971c89da-registry-tls\") pod \"image-registry-5d58f56c87-wn9cc\" (UID: \"bbbf0b29-8280-450c-99e6-5965971c89da\") " pod="openshift-image-registry/image-registry-5d58f56c87-wn9cc" Jul 10 00:16:15.319471 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:16:15.319199 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 00:16:15.319471 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:16:15.319204 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 00:16:15.319471 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:16:15.319264 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b56f3715-4283-4226-8188-d1b3869826db-metrics-tls podName:b56f3715-4283-4226-8188-d1b3869826db nodeName:}" failed. No retries permitted until 2026-07-10 00:16:17.319249719 +0000 UTC m=+44.137555192 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b56f3715-4283-4226-8188-d1b3869826db-metrics-tls") pod "dns-default-9mt8s" (UID: "b56f3715-4283-4226-8188-d1b3869826db") : secret "dns-default-metrics-tls" not found Jul 10 00:16:15.319471 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:16:15.319212 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d58f56c87-wn9cc: secret "image-registry-tls" not found Jul 10 00:16:15.319471 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:16:15.319358 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bbbf0b29-8280-450c-99e6-5965971c89da-registry-tls podName:bbbf0b29-8280-450c-99e6-5965971c89da nodeName:}" failed. No retries permitted until 2026-07-10 00:16:17.319340414 +0000 UTC m=+44.137645907 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bbbf0b29-8280-450c-99e6-5965971c89da-registry-tls") pod "image-registry-5d58f56c87-wn9cc" (UID: "bbbf0b29-8280-450c-99e6-5965971c89da") : secret "image-registry-tls" not found Jul 10 00:16:17.333991 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:17.333795 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b56f3715-4283-4226-8188-d1b3869826db-metrics-tls\") pod \"dns-default-9mt8s\" (UID: \"b56f3715-4283-4226-8188-d1b3869826db\") " pod="openshift-dns/dns-default-9mt8s" Jul 10 00:16:17.334433 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:16:17.333951 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 00:16:17.334433 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:17.334018 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bbbf0b29-8280-450c-99e6-5965971c89da-registry-tls\") pod \"image-registry-5d58f56c87-wn9cc\" (UID: \"bbbf0b29-8280-450c-99e6-5965971c89da\") " pod="openshift-image-registry/image-registry-5d58f56c87-wn9cc" Jul 10 00:16:17.334433 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:16:17.334117 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b56f3715-4283-4226-8188-d1b3869826db-metrics-tls podName:b56f3715-4283-4226-8188-d1b3869826db nodeName:}" failed. No retries permitted until 2026-07-10 00:16:21.33409006 +0000 UTC m=+48.152395546 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b56f3715-4283-4226-8188-d1b3869826db-metrics-tls") pod "dns-default-9mt8s" (UID: "b56f3715-4283-4226-8188-d1b3869826db") : secret "dns-default-metrics-tls" not found Jul 10 00:16:17.334433 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:16:17.334189 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 00:16:17.334433 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:16:17.334206 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d58f56c87-wn9cc: secret "image-registry-tls" not found Jul 10 00:16:17.334433 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:16:17.334251 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bbbf0b29-8280-450c-99e6-5965971c89da-registry-tls podName:bbbf0b29-8280-450c-99e6-5965971c89da nodeName:}" failed. No retries permitted until 2026-07-10 00:16:21.334239194 +0000 UTC m=+48.152544670 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bbbf0b29-8280-450c-99e6-5965971c89da-registry-tls") pod "image-registry-5d58f56c87-wn9cc" (UID: "bbbf0b29-8280-450c-99e6-5965971c89da") : secret "image-registry-tls" not found Jul 10 00:16:18.975038 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:18.975005 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tjh96" event={"ID":"55dcffca-f335-4b01-a6b8-4b2356d2fe57","Type":"ContainerStarted","Data":"2b6720dc69430a5c9ee6f7a2a0b5e2535198d049fdcad69d511beb8d485d8977"} Jul 10 00:16:18.992276 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:18.992225 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-tjh96" podStartSLOduration=1.907965201 podStartE2EDuration="5.992210946s" totalImagesPullingTime="4.084245745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:16:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:16:18.990950561 +0000 UTC m=+45.809256057" watchObservedRunningTime="2026-07-10 00:16:18.992210946 +0000 UTC m=+45.810516440" Jul 10 00:16:21.367816 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:21.367775 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b56f3715-4283-4226-8188-d1b3869826db-metrics-tls\") pod \"dns-default-9mt8s\" (UID: \"b56f3715-4283-4226-8188-d1b3869826db\") " pod="openshift-dns/dns-default-9mt8s" Jul 10 00:16:21.367816 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:21.367818 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bbbf0b29-8280-450c-99e6-5965971c89da-registry-tls\") pod \"image-registry-5d58f56c87-wn9cc\" (UID: \"bbbf0b29-8280-450c-99e6-5965971c89da\") " pod="openshift-image-registry/image-registry-5d58f56c87-wn9cc" Jul 10 00:16:21.368328 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:16:21.367930 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 00:16:21.368328 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:16:21.367941 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d58f56c87-wn9cc: secret "image-registry-tls" not found Jul 10 00:16:21.368328 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:16:21.367954 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 00:16:21.368328 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:16:21.367990 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bbbf0b29-8280-450c-99e6-5965971c89da-registry-tls podName:bbbf0b29-8280-450c-99e6-5965971c89da nodeName:}" failed. No retries permitted until 2026-07-10 00:16:29.367975917 +0000 UTC m=+56.186281391 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bbbf0b29-8280-450c-99e6-5965971c89da-registry-tls") pod "image-registry-5d58f56c87-wn9cc" (UID: "bbbf0b29-8280-450c-99e6-5965971c89da") : secret "image-registry-tls" not found Jul 10 00:16:21.368328 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:16:21.368026 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b56f3715-4283-4226-8188-d1b3869826db-metrics-tls podName:b56f3715-4283-4226-8188-d1b3869826db nodeName:}" failed. No retries permitted until 2026-07-10 00:16:29.368011051 +0000 UTC m=+56.186316523 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b56f3715-4283-4226-8188-d1b3869826db-metrics-tls") pod "dns-default-9mt8s" (UID: "b56f3715-4283-4226-8188-d1b3869826db") : secret "dns-default-metrics-tls" not found Jul 10 00:16:29.426323 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:29.426280 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bbbf0b29-8280-450c-99e6-5965971c89da-registry-tls\") pod \"image-registry-5d58f56c87-wn9cc\" (UID: \"bbbf0b29-8280-450c-99e6-5965971c89da\") " pod="openshift-image-registry/image-registry-5d58f56c87-wn9cc" Jul 10 00:16:29.441220 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:29.426375 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b56f3715-4283-4226-8188-d1b3869826db-metrics-tls\") pod \"dns-default-9mt8s\" (UID: \"b56f3715-4283-4226-8188-d1b3869826db\") " pod="openshift-dns/dns-default-9mt8s" Jul 10 00:16:29.441220 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:16:29.426430 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 00:16:29.441220 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:16:29.426454 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d58f56c87-wn9cc: secret "image-registry-tls" not found Jul 10 00:16:29.441220 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:16:29.426461 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 00:16:29.441220 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:16:29.426509 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bbbf0b29-8280-450c-99e6-5965971c89da-registry-tls podName:bbbf0b29-8280-450c-99e6-5965971c89da nodeName:}" failed. No retries permitted until 2026-07-10 00:16:45.426494595 +0000 UTC m=+72.244800069 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bbbf0b29-8280-450c-99e6-5965971c89da-registry-tls") pod "image-registry-5d58f56c87-wn9cc" (UID: "bbbf0b29-8280-450c-99e6-5965971c89da") : secret "image-registry-tls" not found Jul 10 00:16:29.441220 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:16:29.426525 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b56f3715-4283-4226-8188-d1b3869826db-metrics-tls podName:b56f3715-4283-4226-8188-d1b3869826db nodeName:}" failed. No retries permitted until 2026-07-10 00:16:45.426518647 +0000 UTC m=+72.244824119 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b56f3715-4283-4226-8188-d1b3869826db-metrics-tls") pod "dns-default-9mt8s" (UID: "b56f3715-4283-4226-8188-d1b3869826db") : secret "dns-default-metrics-tls" not found Jul 10 00:16:38.952392 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:38.952362 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h5g9w" Jul 10 00:16:39.394960 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:39.394919 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62933c8a-818d-4580-a6c9-6d26dbdfd50d-metrics-certs\") pod \"network-metrics-daemon-x8v89\" (UID: \"62933c8a-818d-4580-a6c9-6d26dbdfd50d\") " pod="openshift-multus/network-metrics-daemon-x8v89" Jul 10 00:16:39.397419 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:39.397399 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 00:16:39.406059 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:16:39.406035 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 10 00:16:39.406153 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:16:39.406142 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62933c8a-818d-4580-a6c9-6d26dbdfd50d-metrics-certs podName:62933c8a-818d-4580-a6c9-6d26dbdfd50d nodeName:}" failed. No retries permitted until 2026-07-10 00:17:43.406124398 +0000 UTC m=+130.224429888 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/62933c8a-818d-4580-a6c9-6d26dbdfd50d-metrics-certs") pod "network-metrics-daemon-x8v89" (UID: "62933c8a-818d-4580-a6c9-6d26dbdfd50d") : secret "metrics-daemon-secret" not found Jul 10 00:16:39.495809 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:39.495769 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/451130fe-547b-40c5-a7dc-412d57fb5db5-original-pull-secret\") pod \"global-pull-secret-syncer-nkcmr\" (UID: \"451130fe-547b-40c5-a7dc-412d57fb5db5\") " pod="kube-system/global-pull-secret-syncer-nkcmr" Jul 10 00:16:39.498347 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:39.498329 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 00:16:39.509458 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:39.509424 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/451130fe-547b-40c5-a7dc-412d57fb5db5-original-pull-secret\") pod \"global-pull-secret-syncer-nkcmr\" (UID: \"451130fe-547b-40c5-a7dc-412d57fb5db5\") " pod="kube-system/global-pull-secret-syncer-nkcmr" Jul 10 00:16:39.596272 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:39.596234 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phkgc\" (UniqueName: \"kubernetes.io/projected/6fe3e675-7e07-49d2-a664-a47d434dfd3b-kube-api-access-phkgc\") pod \"network-check-target-qbbd4\" (UID: \"6fe3e675-7e07-49d2-a664-a47d434dfd3b\") " pod="openshift-network-diagnostics/network-check-target-qbbd4" Jul 10 00:16:39.598797 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:39.598779 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 10 00:16:39.609002 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:39.608978 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 10 00:16:39.620104 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:39.620056 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phkgc\" (UniqueName: \"kubernetes.io/projected/6fe3e675-7e07-49d2-a664-a47d434dfd3b-kube-api-access-phkgc\") pod \"network-check-target-qbbd4\" (UID: \"6fe3e675-7e07-49d2-a664-a47d434dfd3b\") " pod="openshift-network-diagnostics/network-check-target-qbbd4" Jul 10 00:16:39.640005 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:39.639970 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nkcmr" Jul 10 00:16:39.763865 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:39.763834 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nkcmr"] Jul 10 00:16:39.766723 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:16:39.766688 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod451130fe_547b_40c5_a7dc_412d57fb5db5.slice/crio-37f3cdf74be50da29a57fe0a6cbc3a5c3013df03bec0b626fa12e9409364b36e WatchSource:0}: Error finding container 37f3cdf74be50da29a57fe0a6cbc3a5c3013df03bec0b626fa12e9409364b36e: Status 404 returned error can't find the container with id 37f3cdf74be50da29a57fe0a6cbc3a5c3013df03bec0b626fa12e9409364b36e Jul 10 00:16:39.842367 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:39.842337 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qmtcf\"" Jul 10 00:16:39.850928 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:39.850911 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qbbd4" Jul 10 00:16:39.972106 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:39.971932 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qbbd4"] Jul 10 00:16:39.974450 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:16:39.974421 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fe3e675_7e07_49d2_a664_a47d434dfd3b.slice/crio-27ed128d1a10ebe256596eacfa718c74573f4fedf559dbb7156a555432618eb2 WatchSource:0}: Error finding container 27ed128d1a10ebe256596eacfa718c74573f4fedf559dbb7156a555432618eb2: Status 404 returned error can't find the container with id 27ed128d1a10ebe256596eacfa718c74573f4fedf559dbb7156a555432618eb2 Jul 10 00:16:40.023669 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:40.023630 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qbbd4" event={"ID":"6fe3e675-7e07-49d2-a664-a47d434dfd3b","Type":"ContainerStarted","Data":"27ed128d1a10ebe256596eacfa718c74573f4fedf559dbb7156a555432618eb2"} Jul 10 00:16:40.024847 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:40.024822 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nkcmr" event={"ID":"451130fe-547b-40c5-a7dc-412d57fb5db5","Type":"ContainerStarted","Data":"37f3cdf74be50da29a57fe0a6cbc3a5c3013df03bec0b626fa12e9409364b36e"} Jul 10 00:16:45.441863 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:45.441821 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b56f3715-4283-4226-8188-d1b3869826db-metrics-tls\") pod \"dns-default-9mt8s\" (UID: \"b56f3715-4283-4226-8188-d1b3869826db\") " pod="openshift-dns/dns-default-9mt8s" Jul 10 00:16:45.442337 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:45.441873 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bbbf0b29-8280-450c-99e6-5965971c89da-registry-tls\") pod \"image-registry-5d58f56c87-wn9cc\" (UID: \"bbbf0b29-8280-450c-99e6-5965971c89da\") " pod="openshift-image-registry/image-registry-5d58f56c87-wn9cc" Jul 10 00:16:45.442337 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:16:45.442007 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 00:16:45.442337 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:16:45.442031 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d58f56c87-wn9cc: secret "image-registry-tls" not found Jul 10 00:16:45.442337 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:16:45.442010 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 00:16:45.442337 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:16:45.442110 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bbbf0b29-8280-450c-99e6-5965971c89da-registry-tls podName:bbbf0b29-8280-450c-99e6-5965971c89da nodeName:}" failed. No retries permitted until 2026-07-10 00:17:17.442094831 +0000 UTC m=+104.260400322 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bbbf0b29-8280-450c-99e6-5965971c89da-registry-tls") pod "image-registry-5d58f56c87-wn9cc" (UID: "bbbf0b29-8280-450c-99e6-5965971c89da") : secret "image-registry-tls" not found Jul 10 00:16:45.442337 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:16:45.442144 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b56f3715-4283-4226-8188-d1b3869826db-metrics-tls podName:b56f3715-4283-4226-8188-d1b3869826db nodeName:}" failed. No retries permitted until 2026-07-10 00:17:17.442129719 +0000 UTC m=+104.260435191 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b56f3715-4283-4226-8188-d1b3869826db-metrics-tls") pod "dns-default-9mt8s" (UID: "b56f3715-4283-4226-8188-d1b3869826db") : secret "dns-default-metrics-tls" not found Jul 10 00:16:46.043756 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:46.043659 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qbbd4" event={"ID":"6fe3e675-7e07-49d2-a664-a47d434dfd3b","Type":"ContainerStarted","Data":"ac49e9960a0e05a53e8abfa535d69b3a19ceac64a3198ee4c7b9616f6758f024"} Jul 10 00:16:46.043756 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:46.043749 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qbbd4" Jul 10 00:16:46.045127 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:46.045096 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nkcmr" event={"ID":"451130fe-547b-40c5-a7dc-412d57fb5db5","Type":"ContainerStarted","Data":"c8f1140d37fb6459937ed89c18f6839165bcacedda2528ab4587b0cbfb91f58d"} Jul 10 00:16:46.061558 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:46.058694 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qbbd4" podStartSLOduration=65.397560977 podStartE2EDuration="1m11.058647501s" totalImagesPullingTime="5.661086524s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:15:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:16:46.057503878 +0000 UTC m=+72.875809371" watchObservedRunningTime="2026-07-10 00:16:46.058647501 +0000 UTC m=+72.876952998" Jul 10 00:16:46.071297 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:16:46.071252 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-nkcmr" podStartSLOduration=65.201983396 podStartE2EDuration="1m11.071237016s" totalImagesPullingTime="5.86925362s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 00:15:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:16:46.070356814 +0000 UTC m=+72.888662323" watchObservedRunningTime="2026-07-10 00:16:46.071237016 +0000 UTC m=+72.889542511" Jul 10 00:17:17.049854 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:17:17.049815 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qbbd4" Jul 10 00:17:17.466392 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:17:17.466357 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b56f3715-4283-4226-8188-d1b3869826db-metrics-tls\") pod \"dns-default-9mt8s\" (UID: \"b56f3715-4283-4226-8188-d1b3869826db\") " pod="openshift-dns/dns-default-9mt8s" Jul 10 00:17:17.466392 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:17:17.466402 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bbbf0b29-8280-450c-99e6-5965971c89da-registry-tls\") pod \"image-registry-5d58f56c87-wn9cc\" (UID: \"bbbf0b29-8280-450c-99e6-5965971c89da\") " pod="openshift-image-registry/image-registry-5d58f56c87-wn9cc" Jul 10 00:17:17.466620 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:17:17.466509 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 00:17:17.466620 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:17:17.466520 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 00:17:17.466620 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:17:17.466534 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d58f56c87-wn9cc: secret "image-registry-tls" not found Jul 10 00:17:17.466620 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:17:17.466585 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bbbf0b29-8280-450c-99e6-5965971c89da-registry-tls podName:bbbf0b29-8280-450c-99e6-5965971c89da nodeName:}" failed. No retries permitted until 2026-07-10 00:18:21.466568347 +0000 UTC m=+168.284873824 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bbbf0b29-8280-450c-99e6-5965971c89da-registry-tls") pod "image-registry-5d58f56c87-wn9cc" (UID: "bbbf0b29-8280-450c-99e6-5965971c89da") : secret "image-registry-tls" not found Jul 10 00:17:17.466620 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:17:17.466598 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b56f3715-4283-4226-8188-d1b3869826db-metrics-tls podName:b56f3715-4283-4226-8188-d1b3869826db nodeName:}" failed. No retries permitted until 2026-07-10 00:18:21.466591886 +0000 UTC m=+168.284897358 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b56f3715-4283-4226-8188-d1b3869826db-metrics-tls") pod "dns-default-9mt8s" (UID: "b56f3715-4283-4226-8188-d1b3869826db") : secret "dns-default-metrics-tls" not found Jul 10 00:17:20.597506 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:17:20.597472 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ccc9546c7-575np"] Jul 10 00:17:20.632586 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:17:20.632556 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ccc9546c7-575np"] Jul 10 00:17:20.632751 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:17:20.632654 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ccc9546c7-575np" Jul 10 00:17:20.635450 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:17:20.635424 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 10 00:17:20.635597 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:17:20.635432 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 10 00:17:20.635597 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:17:20.635461 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 10 00:17:20.635597 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:17:20.635463 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 10 00:17:20.787712 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:17:20.787674 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/91e87db0-a4a5-4156-81c7-127610fa96d3-klusterlet-config\") pod \"klusterlet-addon-workmgr-5ccc9546c7-575np\" (UID: \"91e87db0-a4a5-4156-81c7-127610fa96d3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ccc9546c7-575np" Jul 10 00:17:20.787891 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:17:20.787748 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91e87db0-a4a5-4156-81c7-127610fa96d3-tmp\") pod \"klusterlet-addon-workmgr-5ccc9546c7-575np\" (UID: \"91e87db0-a4a5-4156-81c7-127610fa96d3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ccc9546c7-575np" Jul 10 00:17:20.787891 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:17:20.787836 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7fp7\" (UniqueName: \"kubernetes.io/projected/91e87db0-a4a5-4156-81c7-127610fa96d3-kube-api-access-n7fp7\") pod \"klusterlet-addon-workmgr-5ccc9546c7-575np\" (UID: \"91e87db0-a4a5-4156-81c7-127610fa96d3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ccc9546c7-575np" Jul 10 00:17:20.889030 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:17:20.888960 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91e87db0-a4a5-4156-81c7-127610fa96d3-tmp\") pod \"klusterlet-addon-workmgr-5ccc9546c7-575np\" (UID: \"91e87db0-a4a5-4156-81c7-127610fa96d3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ccc9546c7-575np" Jul 10 00:17:20.889030 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:17:20.889023 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7fp7\" (UniqueName: \"kubernetes.io/projected/91e87db0-a4a5-4156-81c7-127610fa96d3-kube-api-access-n7fp7\") pod \"klusterlet-addon-workmgr-5ccc9546c7-575np\" (UID: \"91e87db0-a4a5-4156-81c7-127610fa96d3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ccc9546c7-575np" Jul 10 00:17:20.889187 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:17:20.889062 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/91e87db0-a4a5-4156-81c7-127610fa96d3-klusterlet-config\") pod \"klusterlet-addon-workmgr-5ccc9546c7-575np\" (UID: \"91e87db0-a4a5-4156-81c7-127610fa96d3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ccc9546c7-575np" Jul 10 00:17:20.889406 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:17:20.889382 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91e87db0-a4a5-4156-81c7-127610fa96d3-tmp\") pod \"klusterlet-addon-workmgr-5ccc9546c7-575np\" (UID: \"91e87db0-a4a5-4156-81c7-127610fa96d3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ccc9546c7-575np" Jul 10 00:17:20.891527 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:17:20.891510 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/91e87db0-a4a5-4156-81c7-127610fa96d3-klusterlet-config\") pod \"klusterlet-addon-workmgr-5ccc9546c7-575np\" (UID: \"91e87db0-a4a5-4156-81c7-127610fa96d3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ccc9546c7-575np" Jul 10 00:17:20.896933 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:17:20.896906 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7fp7\" (UniqueName: \"kubernetes.io/projected/91e87db0-a4a5-4156-81c7-127610fa96d3-kube-api-access-n7fp7\") pod \"klusterlet-addon-workmgr-5ccc9546c7-575np\" (UID: \"91e87db0-a4a5-4156-81c7-127610fa96d3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ccc9546c7-575np" Jul 10 00:17:20.941752 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:17:20.941716 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ccc9546c7-575np" Jul 10 00:17:21.083123 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:17:21.083065 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ccc9546c7-575np"] Jul 10 00:17:21.085809 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:17:21.085784 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e87db0_a4a5_4156_81c7_127610fa96d3.slice/crio-97ddb6484906a2410fe2a083230d6395ae3d66539e0a45fffc288492faebd774 WatchSource:0}: Error finding container 97ddb6484906a2410fe2a083230d6395ae3d66539e0a45fffc288492faebd774: Status 404 returned error can't find the container with id 97ddb6484906a2410fe2a083230d6395ae3d66539e0a45fffc288492faebd774 Jul 10 00:17:21.128735 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:17:21.128703 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ccc9546c7-575np" event={"ID":"91e87db0-a4a5-4156-81c7-127610fa96d3","Type":"ContainerStarted","Data":"97ddb6484906a2410fe2a083230d6395ae3d66539e0a45fffc288492faebd774"} Jul 10 00:17:26.144885 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:17:26.144849 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ccc9546c7-575np" event={"ID":"91e87db0-a4a5-4156-81c7-127610fa96d3","Type":"ContainerStarted","Data":"1bda70aebfe47a0fde709782141fb70c31a3c3af7ba90fb18f44f99ac403524a"} Jul 10 00:17:26.145362 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:17:26.145203 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ccc9546c7-575np" Jul 10 00:17:26.146506 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:17:26.146485 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ccc9546c7-575np" Jul 10 00:17:26.159397 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:17:26.159345 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ccc9546c7-575np" podStartSLOduration=1.192097283 podStartE2EDuration="6.159328712s" totalImagesPullingTime="4.967231429s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:17:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:17:26.158562579 +0000 UTC m=+112.976868075" watchObservedRunningTime="2026-07-10 00:17:26.159328712 +0000 UTC m=+112.977634218" Jul 10 00:17:43.459317 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:17:43.459272 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62933c8a-818d-4580-a6c9-6d26dbdfd50d-metrics-certs\") pod \"network-metrics-daemon-x8v89\" (UID: \"62933c8a-818d-4580-a6c9-6d26dbdfd50d\") " pod="openshift-multus/network-metrics-daemon-x8v89" Jul 10 00:17:43.459817 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:17:43.459429 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 10 00:17:43.459817 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:17:43.459496 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62933c8a-818d-4580-a6c9-6d26dbdfd50d-metrics-certs podName:62933c8a-818d-4580-a6c9-6d26dbdfd50d nodeName:}" failed. No retries permitted until 2026-07-10 00:19:45.459479135 +0000 UTC m=+252.277784608 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/62933c8a-818d-4580-a6c9-6d26dbdfd50d-metrics-certs") pod "network-metrics-daemon-x8v89" (UID: "62933c8a-818d-4580-a6c9-6d26dbdfd50d") : secret "metrics-daemon-secret" not found Jul 10 00:18:16.547051 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:18:16.547009 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-5d58f56c87-wn9cc" podUID="bbbf0b29-8280-450c-99e6-5965971c89da" Jul 10 00:18:16.559291 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:18:16.559270 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-9mt8s" podUID="b56f3715-4283-4226-8188-d1b3869826db" Jul 10 00:18:17.269517 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:18:17.269445 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d58f56c87-wn9cc" Jul 10 00:18:17.269646 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:18:17.269445 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9mt8s" Jul 10 00:18:17.744116 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:18:17.744065 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-x8v89" podUID="62933c8a-818d-4580-a6c9-6d26dbdfd50d" Jul 10 00:18:21.527588 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:18:21.527554 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b56f3715-4283-4226-8188-d1b3869826db-metrics-tls\") pod \"dns-default-9mt8s\" (UID: \"b56f3715-4283-4226-8188-d1b3869826db\") " pod="openshift-dns/dns-default-9mt8s" Jul 10 00:18:21.527588 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:18:21.527592 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bbbf0b29-8280-450c-99e6-5965971c89da-registry-tls\") pod \"image-registry-5d58f56c87-wn9cc\" (UID: \"bbbf0b29-8280-450c-99e6-5965971c89da\") " pod="openshift-image-registry/image-registry-5d58f56c87-wn9cc" Jul 10 00:18:21.528119 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:18:21.527701 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 00:18:21.528119 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:18:21.527709 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 00:18:21.528119 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:18:21.527794 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b56f3715-4283-4226-8188-d1b3869826db-metrics-tls podName:b56f3715-4283-4226-8188-d1b3869826db nodeName:}" failed. No retries permitted until 2026-07-10 00:20:23.527774862 +0000 UTC m=+290.346080335 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b56f3715-4283-4226-8188-d1b3869826db-metrics-tls") pod "dns-default-9mt8s" (UID: "b56f3715-4283-4226-8188-d1b3869826db") : secret "dns-default-metrics-tls" not found Jul 10 00:18:21.528119 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:18:21.527712 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d58f56c87-wn9cc: secret "image-registry-tls" not found Jul 10 00:18:21.528119 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:18:21.527846 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bbbf0b29-8280-450c-99e6-5965971c89da-registry-tls podName:bbbf0b29-8280-450c-99e6-5965971c89da nodeName:}" failed. No retries permitted until 2026-07-10 00:20:23.527833094 +0000 UTC m=+290.346138566 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bbbf0b29-8280-450c-99e6-5965971c89da-registry-tls") pod "image-registry-5d58f56c87-wn9cc" (UID: "bbbf0b29-8280-450c-99e6-5965971c89da") : secret "image-registry-tls" not found Jul 10 00:18:28.729940 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:18:28.729856 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x8v89" Jul 10 00:19:26.146287 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:26.146240 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ccc9546c7-575np" podUID="91e87db0-a4a5-4156-81c7-127610fa96d3" containerName="acm-agent" probeResult="failure" output="Get \"http://10.132.0.9:8000/readyz\": dial tcp 10.132.0.9:8000: connect: connection refused" Jul 10 00:19:26.435092 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:26.434985 2562 generic.go:356] "Generic (PLEG): container finished" podID="91e87db0-a4a5-4156-81c7-127610fa96d3" containerID="1bda70aebfe47a0fde709782141fb70c31a3c3af7ba90fb18f44f99ac403524a" exitCode=1 Jul 10 00:19:26.435092 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:26.435059 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ccc9546c7-575np" event={"ID":"91e87db0-a4a5-4156-81c7-127610fa96d3","Type":"ContainerDied","Data":"1bda70aebfe47a0fde709782141fb70c31a3c3af7ba90fb18f44f99ac403524a"} Jul 10 00:19:26.435402 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:26.435386 2562 scope.go:117] "RemoveContainer" containerID="1bda70aebfe47a0fde709782141fb70c31a3c3af7ba90fb18f44f99ac403524a" Jul 10 00:19:27.439642 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:27.439601 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ccc9546c7-575np" event={"ID":"91e87db0-a4a5-4156-81c7-127610fa96d3","Type":"ContainerStarted","Data":"53f8a9c2e7d41f73df311471f3e09cff5cf005353e68cadba29412efcff8b3ae"} Jul 10 00:19:27.440030 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:27.439891 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ccc9546c7-575np" Jul 10 00:19:27.440477 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:27.440460 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ccc9546c7-575np" Jul 10 00:19:32.557496 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.557459 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7866497494-kjhg9"] Jul 10 00:19:32.559592 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.559571 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-kjhg9" Jul 10 00:19:32.562176 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.562155 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 10 00:19:32.562293 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.562157 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 10 00:19:32.562442 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.562426 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 10 00:19:32.563050 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.563034 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 10 00:19:32.563163 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.563146 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-k96vz\"" Jul 10 00:19:32.569346 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.569327 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 10 00:19:32.573194 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.573177 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-kjhg9"] Jul 10 00:19:32.627274 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.627241 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj8p6\" (UniqueName: \"kubernetes.io/projected/4651a90d-ef18-4974-9efd-a77fff3c9dfd-kube-api-access-nj8p6\") pod \"console-operator-7866497494-kjhg9\" (UID: \"4651a90d-ef18-4974-9efd-a77fff3c9dfd\") " pod="openshift-console-operator/console-operator-7866497494-kjhg9" Jul 10 00:19:32.627274 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.627277 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4651a90d-ef18-4974-9efd-a77fff3c9dfd-trusted-ca\") pod \"console-operator-7866497494-kjhg9\" (UID: \"4651a90d-ef18-4974-9efd-a77fff3c9dfd\") " pod="openshift-console-operator/console-operator-7866497494-kjhg9" Jul 10 00:19:32.627444 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.627296 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4651a90d-ef18-4974-9efd-a77fff3c9dfd-config\") pod \"console-operator-7866497494-kjhg9\" (UID: \"4651a90d-ef18-4974-9efd-a77fff3c9dfd\") " pod="openshift-console-operator/console-operator-7866497494-kjhg9" Jul 10 00:19:32.627444 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.627315 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4651a90d-ef18-4974-9efd-a77fff3c9dfd-serving-cert\") pod \"console-operator-7866497494-kjhg9\" (UID: \"4651a90d-ef18-4974-9efd-a77fff3c9dfd\") " pod="openshift-console-operator/console-operator-7866497494-kjhg9" Jul 10 00:19:32.660230 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.660194 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-kkbl9"] Jul 10 00:19:32.662124 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.662104 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-kkbl9" Jul 10 00:19:32.664348 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.664331 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-p2c7b\"" Jul 10 00:19:32.673418 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.673396 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-kkbl9"] Jul 10 00:19:32.727844 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.727816 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj8p6\" (UniqueName: \"kubernetes.io/projected/4651a90d-ef18-4974-9efd-a77fff3c9dfd-kube-api-access-nj8p6\") pod \"console-operator-7866497494-kjhg9\" (UID: \"4651a90d-ef18-4974-9efd-a77fff3c9dfd\") " pod="openshift-console-operator/console-operator-7866497494-kjhg9" Jul 10 00:19:32.727844 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.727846 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4651a90d-ef18-4974-9efd-a77fff3c9dfd-trusted-ca\") pod \"console-operator-7866497494-kjhg9\" (UID: \"4651a90d-ef18-4974-9efd-a77fff3c9dfd\") " pod="openshift-console-operator/console-operator-7866497494-kjhg9" Jul 10 00:19:32.727997 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.727867 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkhck\" (UniqueName: \"kubernetes.io/projected/fb6b765c-8264-4e4d-81d3-dc17e7568325-kube-api-access-gkhck\") pod \"network-check-source-869dcfcdcd-kkbl9\" (UID: \"fb6b765c-8264-4e4d-81d3-dc17e7568325\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-kkbl9" Jul 10 00:19:32.727997 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.727887 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4651a90d-ef18-4974-9efd-a77fff3c9dfd-config\") pod \"console-operator-7866497494-kjhg9\" (UID: \"4651a90d-ef18-4974-9efd-a77fff3c9dfd\") " pod="openshift-console-operator/console-operator-7866497494-kjhg9" Jul 10 00:19:32.727997 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.727917 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4651a90d-ef18-4974-9efd-a77fff3c9dfd-serving-cert\") pod \"console-operator-7866497494-kjhg9\" (UID: \"4651a90d-ef18-4974-9efd-a77fff3c9dfd\") " pod="openshift-console-operator/console-operator-7866497494-kjhg9" Jul 10 00:19:32.729607 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.729584 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4651a90d-ef18-4974-9efd-a77fff3c9dfd-config\") pod \"console-operator-7866497494-kjhg9\" (UID: \"4651a90d-ef18-4974-9efd-a77fff3c9dfd\") " pod="openshift-console-operator/console-operator-7866497494-kjhg9" Jul 10 00:19:32.729737 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.729720 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4651a90d-ef18-4974-9efd-a77fff3c9dfd-trusted-ca\") pod \"console-operator-7866497494-kjhg9\" (UID: \"4651a90d-ef18-4974-9efd-a77fff3c9dfd\") " pod="openshift-console-operator/console-operator-7866497494-kjhg9" Jul 10 00:19:32.730411 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.730396 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4651a90d-ef18-4974-9efd-a77fff3c9dfd-serving-cert\") pod \"console-operator-7866497494-kjhg9\" (UID: \"4651a90d-ef18-4974-9efd-a77fff3c9dfd\") " pod="openshift-console-operator/console-operator-7866497494-kjhg9" Jul 10 00:19:32.734610 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.734588 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj8p6\" (UniqueName: \"kubernetes.io/projected/4651a90d-ef18-4974-9efd-a77fff3c9dfd-kube-api-access-nj8p6\") pod \"console-operator-7866497494-kjhg9\" (UID: \"4651a90d-ef18-4974-9efd-a77fff3c9dfd\") " pod="openshift-console-operator/console-operator-7866497494-kjhg9" Jul 10 00:19:32.754790 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.754764 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-dt7gx"] Jul 10 00:19:32.756806 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.756779 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-dt7gx" Jul 10 00:19:32.758261 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.758241 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-76d8bdc488-dw9ns"] Jul 10 00:19:32.758931 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.758917 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 10 00:19:32.758981 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.758917 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 10 00:19:32.759428 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.759402 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 10 00:19:32.759536 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.759489 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-f4bdj\"" Jul 10 00:19:32.759536 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.759501 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 10 00:19:32.760003 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.759987 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-76d8bdc488-dw9ns" Jul 10 00:19:32.762034 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.762013 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-qgwtt\"" Jul 10 00:19:32.762171 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.762066 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 10 00:19:32.762243 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.762192 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 10 00:19:32.762347 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.762292 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 10 00:19:32.762485 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.762470 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 10 00:19:32.762754 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.762730 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 10 00:19:32.762834 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.762740 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 10 00:19:32.765749 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.765728 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-dt7gx"] Jul 10 00:19:32.774307 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.774286 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-76d8bdc488-dw9ns"] Jul 10 00:19:32.828861 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.828784 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7946a2a9-a12b-4302-9f33-93692ecf91ff-config\") pod \"service-ca-operator-7695fc6c8c-dt7gx\" (UID: \"7946a2a9-a12b-4302-9f33-93692ecf91ff\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-dt7gx" Jul 10 00:19:32.828861 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.828829 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3d7b4fce-5123-477b-800c-bec0da7fd372-stats-auth\") pod \"router-default-76d8bdc488-dw9ns\" (UID: \"3d7b4fce-5123-477b-800c-bec0da7fd372\") " pod="openshift-ingress/router-default-76d8bdc488-dw9ns" Jul 10 00:19:32.828861 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.828852 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkhck\" (UniqueName: \"kubernetes.io/projected/fb6b765c-8264-4e4d-81d3-dc17e7568325-kube-api-access-gkhck\") pod \"network-check-source-869dcfcdcd-kkbl9\" (UID: \"fb6b765c-8264-4e4d-81d3-dc17e7568325\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-kkbl9" Jul 10 00:19:32.829100 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.828915 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3d7b4fce-5123-477b-800c-bec0da7fd372-default-certificate\") pod \"router-default-76d8bdc488-dw9ns\" (UID: \"3d7b4fce-5123-477b-800c-bec0da7fd372\") " pod="openshift-ingress/router-default-76d8bdc488-dw9ns" Jul 10 00:19:32.829100 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.828961 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx6z7\" (UniqueName: \"kubernetes.io/projected/7946a2a9-a12b-4302-9f33-93692ecf91ff-kube-api-access-jx6z7\") pod \"service-ca-operator-7695fc6c8c-dt7gx\" (UID: \"7946a2a9-a12b-4302-9f33-93692ecf91ff\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-dt7gx" Jul 10 00:19:32.829100 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.828990 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7946a2a9-a12b-4302-9f33-93692ecf91ff-serving-cert\") pod \"service-ca-operator-7695fc6c8c-dt7gx\" (UID: \"7946a2a9-a12b-4302-9f33-93692ecf91ff\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-dt7gx" Jul 10 00:19:32.829100 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.829013 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d7b4fce-5123-477b-800c-bec0da7fd372-service-ca-bundle\") pod \"router-default-76d8bdc488-dw9ns\" (UID: \"3d7b4fce-5123-477b-800c-bec0da7fd372\") " pod="openshift-ingress/router-default-76d8bdc488-dw9ns" Jul 10 00:19:32.829100 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.829057 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d7b4fce-5123-477b-800c-bec0da7fd372-metrics-certs\") pod \"router-default-76d8bdc488-dw9ns\" (UID: \"3d7b4fce-5123-477b-800c-bec0da7fd372\") " pod="openshift-ingress/router-default-76d8bdc488-dw9ns" Jul 10 00:19:32.829251 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.829112 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnncs\" (UniqueName: \"kubernetes.io/projected/3d7b4fce-5123-477b-800c-bec0da7fd372-kube-api-access-vnncs\") pod \"router-default-76d8bdc488-dw9ns\" (UID: \"3d7b4fce-5123-477b-800c-bec0da7fd372\") " pod="openshift-ingress/router-default-76d8bdc488-dw9ns" Jul 10 00:19:32.836319 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.836291 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkhck\" (UniqueName: \"kubernetes.io/projected/fb6b765c-8264-4e4d-81d3-dc17e7568325-kube-api-access-gkhck\") pod \"network-check-source-869dcfcdcd-kkbl9\" (UID: \"fb6b765c-8264-4e4d-81d3-dc17e7568325\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-kkbl9" Jul 10 00:19:32.868124 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.868093 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-kjhg9" Jul 10 00:19:32.930023 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.929533 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3d7b4fce-5123-477b-800c-bec0da7fd372-stats-auth\") pod \"router-default-76d8bdc488-dw9ns\" (UID: \"3d7b4fce-5123-477b-800c-bec0da7fd372\") " pod="openshift-ingress/router-default-76d8bdc488-dw9ns" Jul 10 00:19:32.930023 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.929592 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3d7b4fce-5123-477b-800c-bec0da7fd372-default-certificate\") pod \"router-default-76d8bdc488-dw9ns\" (UID: \"3d7b4fce-5123-477b-800c-bec0da7fd372\") " pod="openshift-ingress/router-default-76d8bdc488-dw9ns" Jul 10 00:19:32.930023 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.929642 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx6z7\" (UniqueName: \"kubernetes.io/projected/7946a2a9-a12b-4302-9f33-93692ecf91ff-kube-api-access-jx6z7\") pod \"service-ca-operator-7695fc6c8c-dt7gx\" (UID: \"7946a2a9-a12b-4302-9f33-93692ecf91ff\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-dt7gx" Jul 10 00:19:32.930023 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.929675 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7946a2a9-a12b-4302-9f33-93692ecf91ff-serving-cert\") pod \"service-ca-operator-7695fc6c8c-dt7gx\" (UID: \"7946a2a9-a12b-4302-9f33-93692ecf91ff\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-dt7gx" Jul 10 00:19:32.930023 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.929709 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d7b4fce-5123-477b-800c-bec0da7fd372-service-ca-bundle\") pod \"router-default-76d8bdc488-dw9ns\" (UID: \"3d7b4fce-5123-477b-800c-bec0da7fd372\") " pod="openshift-ingress/router-default-76d8bdc488-dw9ns" Jul 10 00:19:32.930023 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.929742 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d7b4fce-5123-477b-800c-bec0da7fd372-metrics-certs\") pod \"router-default-76d8bdc488-dw9ns\" (UID: \"3d7b4fce-5123-477b-800c-bec0da7fd372\") " pod="openshift-ingress/router-default-76d8bdc488-dw9ns" Jul 10 00:19:32.930023 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.929781 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnncs\" (UniqueName: \"kubernetes.io/projected/3d7b4fce-5123-477b-800c-bec0da7fd372-kube-api-access-vnncs\") pod \"router-default-76d8bdc488-dw9ns\" (UID: \"3d7b4fce-5123-477b-800c-bec0da7fd372\") " pod="openshift-ingress/router-default-76d8bdc488-dw9ns" Jul 10 00:19:32.930023 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.929827 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7946a2a9-a12b-4302-9f33-93692ecf91ff-config\") pod \"service-ca-operator-7695fc6c8c-dt7gx\" (UID: \"7946a2a9-a12b-4302-9f33-93692ecf91ff\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-dt7gx" Jul 10 00:19:32.930556 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:19:32.930475 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3d7b4fce-5123-477b-800c-bec0da7fd372-service-ca-bundle podName:3d7b4fce-5123-477b-800c-bec0da7fd372 nodeName:}" failed. No retries permitted until 2026-07-10 00:19:33.430455297 +0000 UTC m=+240.248760795 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3d7b4fce-5123-477b-800c-bec0da7fd372-service-ca-bundle") pod "router-default-76d8bdc488-dw9ns" (UID: "3d7b4fce-5123-477b-800c-bec0da7fd372") : configmap references non-existent config key: service-ca.crt Jul 10 00:19:32.930848 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.930639 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7946a2a9-a12b-4302-9f33-93692ecf91ff-config\") pod \"service-ca-operator-7695fc6c8c-dt7gx\" (UID: \"7946a2a9-a12b-4302-9f33-93692ecf91ff\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-dt7gx" Jul 10 00:19:32.930848 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:19:32.930755 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 10 00:19:32.930848 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:19:32.930819 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d7b4fce-5123-477b-800c-bec0da7fd372-metrics-certs podName:3d7b4fce-5123-477b-800c-bec0da7fd372 nodeName:}" failed. No retries permitted until 2026-07-10 00:19:33.430798129 +0000 UTC m=+240.249103628 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3d7b4fce-5123-477b-800c-bec0da7fd372-metrics-certs") pod "router-default-76d8bdc488-dw9ns" (UID: "3d7b4fce-5123-477b-800c-bec0da7fd372") : secret "router-metrics-certs-default" not found Jul 10 00:19:32.932254 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.932235 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3d7b4fce-5123-477b-800c-bec0da7fd372-default-certificate\") pod \"router-default-76d8bdc488-dw9ns\" (UID: \"3d7b4fce-5123-477b-800c-bec0da7fd372\") " pod="openshift-ingress/router-default-76d8bdc488-dw9ns" Jul 10 00:19:32.932663 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.932637 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7946a2a9-a12b-4302-9f33-93692ecf91ff-serving-cert\") pod \"service-ca-operator-7695fc6c8c-dt7gx\" (UID: \"7946a2a9-a12b-4302-9f33-93692ecf91ff\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-dt7gx" Jul 10 00:19:32.933055 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.933039 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3d7b4fce-5123-477b-800c-bec0da7fd372-stats-auth\") pod \"router-default-76d8bdc488-dw9ns\" (UID: \"3d7b4fce-5123-477b-800c-bec0da7fd372\") " pod="openshift-ingress/router-default-76d8bdc488-dw9ns" Jul 10 00:19:32.937702 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.937661 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx6z7\" (UniqueName: \"kubernetes.io/projected/7946a2a9-a12b-4302-9f33-93692ecf91ff-kube-api-access-jx6z7\") pod \"service-ca-operator-7695fc6c8c-dt7gx\" (UID: \"7946a2a9-a12b-4302-9f33-93692ecf91ff\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-dt7gx" Jul 10 00:19:32.938023 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.938000 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnncs\" (UniqueName: \"kubernetes.io/projected/3d7b4fce-5123-477b-800c-bec0da7fd372-kube-api-access-vnncs\") pod \"router-default-76d8bdc488-dw9ns\" (UID: \"3d7b4fce-5123-477b-800c-bec0da7fd372\") " pod="openshift-ingress/router-default-76d8bdc488-dw9ns" Jul 10 00:19:32.970388 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.970351 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-kkbl9" Jul 10 00:19:32.988534 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:32.988465 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-kjhg9"] Jul 10 00:19:32.991526 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:19:32.991498 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4651a90d_ef18_4974_9efd_a77fff3c9dfd.slice/crio-cb18675f550e22ec02550dc4f21863e9450717d29433a7e55682188acacfcec4 WatchSource:0}: Error finding container cb18675f550e22ec02550dc4f21863e9450717d29433a7e55682188acacfcec4: Status 404 returned error can't find the container with id cb18675f550e22ec02550dc4f21863e9450717d29433a7e55682188acacfcec4 Jul 10 00:19:33.066526 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:33.066495 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-dt7gx" Jul 10 00:19:33.090480 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:33.090416 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-kkbl9"] Jul 10 00:19:33.093734 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:19:33.093704 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb6b765c_8264_4e4d_81d3_dc17e7568325.slice/crio-5cfea290f9a1cd2db015876451689f245ecbe0d6150c831f6b930e1347997466 WatchSource:0}: Error finding container 5cfea290f9a1cd2db015876451689f245ecbe0d6150c831f6b930e1347997466: Status 404 returned error can't find the container with id 5cfea290f9a1cd2db015876451689f245ecbe0d6150c831f6b930e1347997466 Jul 10 00:19:33.189790 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:33.189749 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-dt7gx"] Jul 10 00:19:33.192228 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:19:33.192194 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7946a2a9_a12b_4302_9f33_93692ecf91ff.slice/crio-c7000491be9bca757916e2b174f28c8ac46cf703614406eab0177a5f5c49835e WatchSource:0}: Error finding container c7000491be9bca757916e2b174f28c8ac46cf703614406eab0177a5f5c49835e: Status 404 returned error can't find the container with id c7000491be9bca757916e2b174f28c8ac46cf703614406eab0177a5f5c49835e Jul 10 00:19:33.434601 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:33.434509 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d7b4fce-5123-477b-800c-bec0da7fd372-service-ca-bundle\") pod \"router-default-76d8bdc488-dw9ns\" (UID: \"3d7b4fce-5123-477b-800c-bec0da7fd372\") " pod="openshift-ingress/router-default-76d8bdc488-dw9ns" Jul 10 00:19:33.434601 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:33.434548 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d7b4fce-5123-477b-800c-bec0da7fd372-metrics-certs\") pod \"router-default-76d8bdc488-dw9ns\" (UID: \"3d7b4fce-5123-477b-800c-bec0da7fd372\") " pod="openshift-ingress/router-default-76d8bdc488-dw9ns" Jul 10 00:19:33.434785 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:19:33.434640 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 10 00:19:33.434785 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:19:33.434686 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d7b4fce-5123-477b-800c-bec0da7fd372-metrics-certs podName:3d7b4fce-5123-477b-800c-bec0da7fd372 nodeName:}" failed. No retries permitted until 2026-07-10 00:19:34.434673508 +0000 UTC m=+241.252978981 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3d7b4fce-5123-477b-800c-bec0da7fd372-metrics-certs") pod "router-default-76d8bdc488-dw9ns" (UID: "3d7b4fce-5123-477b-800c-bec0da7fd372") : secret "router-metrics-certs-default" not found Jul 10 00:19:33.434785 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:19:33.434701 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3d7b4fce-5123-477b-800c-bec0da7fd372-service-ca-bundle podName:3d7b4fce-5123-477b-800c-bec0da7fd372 nodeName:}" failed. No retries permitted until 2026-07-10 00:19:34.43469309 +0000 UTC m=+241.252998562 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3d7b4fce-5123-477b-800c-bec0da7fd372-service-ca-bundle") pod "router-default-76d8bdc488-dw9ns" (UID: "3d7b4fce-5123-477b-800c-bec0da7fd372") : configmap references non-existent config key: service-ca.crt Jul 10 00:19:33.458136 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:33.458106 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-kkbl9" event={"ID":"fb6b765c-8264-4e4d-81d3-dc17e7568325","Type":"ContainerStarted","Data":"33dd5b5372b160caabdb490dc37bbb7906b160a5b4f9056c28669ea19d8fe1fa"} Jul 10 00:19:33.458327 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:33.458167 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-kkbl9" event={"ID":"fb6b765c-8264-4e4d-81d3-dc17e7568325","Type":"ContainerStarted","Data":"5cfea290f9a1cd2db015876451689f245ecbe0d6150c831f6b930e1347997466"} Jul 10 00:19:33.459202 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:33.459177 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-dt7gx" event={"ID":"7946a2a9-a12b-4302-9f33-93692ecf91ff","Type":"ContainerStarted","Data":"c7000491be9bca757916e2b174f28c8ac46cf703614406eab0177a5f5c49835e"} Jul 10 00:19:33.460162 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:33.460142 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-kjhg9" event={"ID":"4651a90d-ef18-4974-9efd-a77fff3c9dfd","Type":"ContainerStarted","Data":"cb18675f550e22ec02550dc4f21863e9450717d29433a7e55682188acacfcec4"} Jul 10 00:19:34.441851 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:34.441560 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d7b4fce-5123-477b-800c-bec0da7fd372-service-ca-bundle\") pod \"router-default-76d8bdc488-dw9ns\" (UID: \"3d7b4fce-5123-477b-800c-bec0da7fd372\") " pod="openshift-ingress/router-default-76d8bdc488-dw9ns" Jul 10 00:19:34.441851 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:34.441617 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d7b4fce-5123-477b-800c-bec0da7fd372-metrics-certs\") pod \"router-default-76d8bdc488-dw9ns\" (UID: \"3d7b4fce-5123-477b-800c-bec0da7fd372\") " pod="openshift-ingress/router-default-76d8bdc488-dw9ns" Jul 10 00:19:34.441851 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:19:34.441744 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 10 00:19:34.441851 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:19:34.441774 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3d7b4fce-5123-477b-800c-bec0da7fd372-service-ca-bundle podName:3d7b4fce-5123-477b-800c-bec0da7fd372 nodeName:}" failed. No retries permitted until 2026-07-10 00:19:36.441748908 +0000 UTC m=+243.260054406 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3d7b4fce-5123-477b-800c-bec0da7fd372-service-ca-bundle") pod "router-default-76d8bdc488-dw9ns" (UID: "3d7b4fce-5123-477b-800c-bec0da7fd372") : configmap references non-existent config key: service-ca.crt Jul 10 00:19:34.441851 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:19:34.441806 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d7b4fce-5123-477b-800c-bec0da7fd372-metrics-certs podName:3d7b4fce-5123-477b-800c-bec0da7fd372 nodeName:}" failed. No retries permitted until 2026-07-10 00:19:36.44179443 +0000 UTC m=+243.260099914 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3d7b4fce-5123-477b-800c-bec0da7fd372-metrics-certs") pod "router-default-76d8bdc488-dw9ns" (UID: "3d7b4fce-5123-477b-800c-bec0da7fd372") : secret "router-metrics-certs-default" not found Jul 10 00:19:36.457524 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:36.457482 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d7b4fce-5123-477b-800c-bec0da7fd372-service-ca-bundle\") pod \"router-default-76d8bdc488-dw9ns\" (UID: \"3d7b4fce-5123-477b-800c-bec0da7fd372\") " pod="openshift-ingress/router-default-76d8bdc488-dw9ns" Jul 10 00:19:36.457524 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:36.457523 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d7b4fce-5123-477b-800c-bec0da7fd372-metrics-certs\") pod \"router-default-76d8bdc488-dw9ns\" (UID: \"3d7b4fce-5123-477b-800c-bec0da7fd372\") " pod="openshift-ingress/router-default-76d8bdc488-dw9ns" Jul 10 00:19:36.458062 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:19:36.457625 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 10 00:19:36.458062 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:19:36.457654 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3d7b4fce-5123-477b-800c-bec0da7fd372-service-ca-bundle podName:3d7b4fce-5123-477b-800c-bec0da7fd372 nodeName:}" failed. No retries permitted until 2026-07-10 00:19:40.457634976 +0000 UTC m=+247.275940449 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3d7b4fce-5123-477b-800c-bec0da7fd372-service-ca-bundle") pod "router-default-76d8bdc488-dw9ns" (UID: "3d7b4fce-5123-477b-800c-bec0da7fd372") : configmap references non-existent config key: service-ca.crt Jul 10 00:19:36.458062 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:19:36.457677 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d7b4fce-5123-477b-800c-bec0da7fd372-metrics-certs podName:3d7b4fce-5123-477b-800c-bec0da7fd372 nodeName:}" failed. No retries permitted until 2026-07-10 00:19:40.457669619 +0000 UTC m=+247.275975092 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3d7b4fce-5123-477b-800c-bec0da7fd372-metrics-certs") pod "router-default-76d8bdc488-dw9ns" (UID: "3d7b4fce-5123-477b-800c-bec0da7fd372") : secret "router-metrics-certs-default" not found Jul 10 00:19:36.472040 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:36.472007 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-dt7gx" event={"ID":"7946a2a9-a12b-4302-9f33-93692ecf91ff","Type":"ContainerStarted","Data":"2676c20ba5d6ba901fa3022981a91e9ee87ee527cf242f114e6d05f9eacc4985"} Jul 10 00:19:36.473655 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:36.473627 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kjhg9_4651a90d-ef18-4974-9efd-a77fff3c9dfd/console-operator/0.log" Jul 10 00:19:36.473773 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:36.473672 2562 generic.go:356] "Generic (PLEG): container finished" podID="4651a90d-ef18-4974-9efd-a77fff3c9dfd" containerID="26063738946ab6ceff7a121b61131e4d23fef45dac7187a36aab638502dbd24e" exitCode=255 Jul 10 00:19:36.473773 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:36.473702 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-kjhg9" event={"ID":"4651a90d-ef18-4974-9efd-a77fff3c9dfd","Type":"ContainerDied","Data":"26063738946ab6ceff7a121b61131e4d23fef45dac7187a36aab638502dbd24e"} Jul 10 00:19:36.473902 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:36.473888 2562 scope.go:117] "RemoveContainer" containerID="26063738946ab6ceff7a121b61131e4d23fef45dac7187a36aab638502dbd24e" Jul 10 00:19:36.486446 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:36.486394 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-kkbl9" podStartSLOduration=4.48637484 podStartE2EDuration="4.48637484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:19:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:19:33.470679614 +0000 UTC m=+240.288985111" watchObservedRunningTime="2026-07-10 00:19:36.48637484 +0000 UTC m=+243.304680318" Jul 10 00:19:36.486695 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:36.486670 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-dt7gx" podStartSLOduration=1.801015165 podStartE2EDuration="4.486663718s" totalImagesPullingTime="2.685648553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:19:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:19:36.485791656 +0000 UTC m=+243.304097152" watchObservedRunningTime="2026-07-10 00:19:36.486663718 +0000 UTC m=+243.304969214" Jul 10 00:19:36.664067 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:36.664030 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-tff8n"] Jul 10 00:19:36.666545 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:36.666524 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tff8n" Jul 10 00:19:36.669157 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:36.669135 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 10 00:19:36.669291 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:36.669158 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 10 00:19:36.669291 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:36.669252 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 10 00:19:36.669291 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:36.669259 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 10 00:19:36.669445 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:36.669390 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2znzl\"" Jul 10 00:19:36.680181 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:36.680155 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tff8n"] Jul 10 00:19:36.760667 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:36.760561 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53c5a23d-3991-4e70-b665-a45e2d68121e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tff8n\" (UID: \"53c5a23d-3991-4e70-b665-a45e2d68121e\") " pod="openshift-insights/insights-runtime-extractor-tff8n" Jul 10 00:19:36.760667 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:36.760602 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53c5a23d-3991-4e70-b665-a45e2d68121e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tff8n\" (UID: \"53c5a23d-3991-4e70-b665-a45e2d68121e\") " pod="openshift-insights/insights-runtime-extractor-tff8n" Jul 10 00:19:36.760667 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:36.760633 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/53c5a23d-3991-4e70-b665-a45e2d68121e-crio-socket\") pod \"insights-runtime-extractor-tff8n\" (UID: \"53c5a23d-3991-4e70-b665-a45e2d68121e\") " pod="openshift-insights/insights-runtime-extractor-tff8n" Jul 10 00:19:36.760896 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:36.760678 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqb85\" (UniqueName: \"kubernetes.io/projected/53c5a23d-3991-4e70-b665-a45e2d68121e-kube-api-access-bqb85\") pod \"insights-runtime-extractor-tff8n\" (UID: \"53c5a23d-3991-4e70-b665-a45e2d68121e\") " pod="openshift-insights/insights-runtime-extractor-tff8n" Jul 10 00:19:36.760896 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:36.760711 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/53c5a23d-3991-4e70-b665-a45e2d68121e-data-volume\") pod \"insights-runtime-extractor-tff8n\" (UID: \"53c5a23d-3991-4e70-b665-a45e2d68121e\") " pod="openshift-insights/insights-runtime-extractor-tff8n" Jul 10 00:19:36.861208 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:36.861167 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53c5a23d-3991-4e70-b665-a45e2d68121e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tff8n\" (UID: \"53c5a23d-3991-4e70-b665-a45e2d68121e\") " pod="openshift-insights/insights-runtime-extractor-tff8n" Jul 10 00:19:36.861208 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:36.861211 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53c5a23d-3991-4e70-b665-a45e2d68121e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tff8n\" (UID: \"53c5a23d-3991-4e70-b665-a45e2d68121e\") " pod="openshift-insights/insights-runtime-extractor-tff8n" Jul 10 00:19:36.861406 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:36.861249 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/53c5a23d-3991-4e70-b665-a45e2d68121e-crio-socket\") pod \"insights-runtime-extractor-tff8n\" (UID: \"53c5a23d-3991-4e70-b665-a45e2d68121e\") " pod="openshift-insights/insights-runtime-extractor-tff8n" Jul 10 00:19:36.861406 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:36.861280 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqb85\" (UniqueName: \"kubernetes.io/projected/53c5a23d-3991-4e70-b665-a45e2d68121e-kube-api-access-bqb85\") pod \"insights-runtime-extractor-tff8n\" (UID: \"53c5a23d-3991-4e70-b665-a45e2d68121e\") " pod="openshift-insights/insights-runtime-extractor-tff8n" Jul 10 00:19:36.861406 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:19:36.861317 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 00:19:36.861406 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:19:36.861397 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/53c5a23d-3991-4e70-b665-a45e2d68121e-kube-rbac-proxy-cm podName:53c5a23d-3991-4e70-b665-a45e2d68121e nodeName:}" failed. No retries permitted until 2026-07-10 00:19:37.361375441 +0000 UTC m=+244.179680918 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/53c5a23d-3991-4e70-b665-a45e2d68121e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tff8n" (UID: "53c5a23d-3991-4e70-b665-a45e2d68121e") : configmap "kube-rbac-proxy" not found Jul 10 00:19:36.861601 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:19:36.861405 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 00:19:36.861601 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:36.861328 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/53c5a23d-3991-4e70-b665-a45e2d68121e-data-volume\") pod \"insights-runtime-extractor-tff8n\" (UID: \"53c5a23d-3991-4e70-b665-a45e2d68121e\") " pod="openshift-insights/insights-runtime-extractor-tff8n" Jul 10 00:19:36.861601 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:36.861407 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/53c5a23d-3991-4e70-b665-a45e2d68121e-crio-socket\") pod \"insights-runtime-extractor-tff8n\" (UID: \"53c5a23d-3991-4e70-b665-a45e2d68121e\") " pod="openshift-insights/insights-runtime-extractor-tff8n" Jul 10 00:19:36.861601 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:19:36.861461 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53c5a23d-3991-4e70-b665-a45e2d68121e-insights-runtime-extractor-tls podName:53c5a23d-3991-4e70-b665-a45e2d68121e nodeName:}" failed. No retries permitted until 2026-07-10 00:19:37.361445748 +0000 UTC m=+244.179751242 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/53c5a23d-3991-4e70-b665-a45e2d68121e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tff8n" (UID: "53c5a23d-3991-4e70-b665-a45e2d68121e") : secret "insights-runtime-extractor-tls" not found Jul 10 00:19:36.861799 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:36.861753 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/53c5a23d-3991-4e70-b665-a45e2d68121e-data-volume\") pod \"insights-runtime-extractor-tff8n\" (UID: \"53c5a23d-3991-4e70-b665-a45e2d68121e\") " pod="openshift-insights/insights-runtime-extractor-tff8n" Jul 10 00:19:36.873016 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:36.872988 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqb85\" (UniqueName: \"kubernetes.io/projected/53c5a23d-3991-4e70-b665-a45e2d68121e-kube-api-access-bqb85\") pod \"insights-runtime-extractor-tff8n\" (UID: \"53c5a23d-3991-4e70-b665-a45e2d68121e\") " pod="openshift-insights/insights-runtime-extractor-tff8n" Jul 10 00:19:37.365812 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:37.365775 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53c5a23d-3991-4e70-b665-a45e2d68121e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tff8n\" (UID: \"53c5a23d-3991-4e70-b665-a45e2d68121e\") " pod="openshift-insights/insights-runtime-extractor-tff8n" Jul 10 00:19:37.365812 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:37.365811 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53c5a23d-3991-4e70-b665-a45e2d68121e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tff8n\" (UID: \"53c5a23d-3991-4e70-b665-a45e2d68121e\") " pod="openshift-insights/insights-runtime-extractor-tff8n" Jul 10 00:19:37.366027 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:19:37.365904 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 00:19:37.366027 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:19:37.365931 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 00:19:37.366027 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:19:37.365978 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/53c5a23d-3991-4e70-b665-a45e2d68121e-kube-rbac-proxy-cm podName:53c5a23d-3991-4e70-b665-a45e2d68121e nodeName:}" failed. No retries permitted until 2026-07-10 00:19:38.365958065 +0000 UTC m=+245.184263537 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/53c5a23d-3991-4e70-b665-a45e2d68121e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tff8n" (UID: "53c5a23d-3991-4e70-b665-a45e2d68121e") : configmap "kube-rbac-proxy" not found Jul 10 00:19:37.366027 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:19:37.365993 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53c5a23d-3991-4e70-b665-a45e2d68121e-insights-runtime-extractor-tls podName:53c5a23d-3991-4e70-b665-a45e2d68121e nodeName:}" failed. No retries permitted until 2026-07-10 00:19:38.365987354 +0000 UTC m=+245.184292826 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/53c5a23d-3991-4e70-b665-a45e2d68121e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tff8n" (UID: "53c5a23d-3991-4e70-b665-a45e2d68121e") : secret "insights-runtime-extractor-tls" not found Jul 10 00:19:37.471908 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:37.471877 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-9w2qg"] Jul 10 00:19:37.473921 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:37.473905 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-9w2qg" Jul 10 00:19:37.476597 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:37.476577 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 10 00:19:37.476597 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:37.476591 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 10 00:19:37.476776 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:37.476646 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-jxrkx\"" Jul 10 00:19:37.482864 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:37.482843 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kjhg9_4651a90d-ef18-4974-9efd-a77fff3c9dfd/console-operator/1.log" Jul 10 00:19:37.483366 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:37.483343 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kjhg9_4651a90d-ef18-4974-9efd-a77fff3c9dfd/console-operator/0.log" Jul 10 00:19:37.483478 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:37.483391 2562 generic.go:356] "Generic (PLEG): container finished" podID="4651a90d-ef18-4974-9efd-a77fff3c9dfd" containerID="0bd4f659cdab52afe6250f448db52473766dc7b9c9569d67ee2d69dd80c40c02" exitCode=255 Jul 10 00:19:37.483478 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:37.483435 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-kjhg9" event={"ID":"4651a90d-ef18-4974-9efd-a77fff3c9dfd","Type":"ContainerDied","Data":"0bd4f659cdab52afe6250f448db52473766dc7b9c9569d67ee2d69dd80c40c02"} Jul 10 00:19:37.483591 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:37.483550 2562 scope.go:117] "RemoveContainer" containerID="26063738946ab6ceff7a121b61131e4d23fef45dac7187a36aab638502dbd24e" Jul 10 00:19:37.483923 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:37.483908 2562 scope.go:117] "RemoveContainer" containerID="0bd4f659cdab52afe6250f448db52473766dc7b9c9569d67ee2d69dd80c40c02" Jul 10 00:19:37.484284 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:19:37.484257 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-kjhg9_openshift-console-operator(4651a90d-ef18-4974-9efd-a77fff3c9dfd)\"" pod="openshift-console-operator/console-operator-7866497494-kjhg9" podUID="4651a90d-ef18-4974-9efd-a77fff3c9dfd" Jul 10 00:19:37.485580 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:37.485379 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-9w2qg"] Jul 10 00:19:37.567734 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:37.567687 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vw4b\" (UniqueName: \"kubernetes.io/projected/0064d533-5099-4a49-afef-12125cba31ad-kube-api-access-2vw4b\") pod \"migrator-7f59b54f6c-9w2qg\" (UID: \"0064d533-5099-4a49-afef-12125cba31ad\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-9w2qg" Jul 10 00:19:37.669273 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:37.669182 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vw4b\" (UniqueName: \"kubernetes.io/projected/0064d533-5099-4a49-afef-12125cba31ad-kube-api-access-2vw4b\") pod \"migrator-7f59b54f6c-9w2qg\" (UID: \"0064d533-5099-4a49-afef-12125cba31ad\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-9w2qg" Jul 10 00:19:37.677226 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:37.677198 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vw4b\" (UniqueName: \"kubernetes.io/projected/0064d533-5099-4a49-afef-12125cba31ad-kube-api-access-2vw4b\") pod \"migrator-7f59b54f6c-9w2qg\" (UID: \"0064d533-5099-4a49-afef-12125cba31ad\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-9w2qg" Jul 10 00:19:37.787963 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:37.787932 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-9w2qg" Jul 10 00:19:37.910250 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:37.910217 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-9w2qg"] Jul 10 00:19:37.913496 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:19:37.913455 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0064d533_5099_4a49_afef_12125cba31ad.slice/crio-88b308f44124acc6a0a9d5e5c1b121ef1a203a4b1efb519e2e74e8dd701acde6 WatchSource:0}: Error finding container 88b308f44124acc6a0a9d5e5c1b121ef1a203a4b1efb519e2e74e8dd701acde6: Status 404 returned error can't find the container with id 88b308f44124acc6a0a9d5e5c1b121ef1a203a4b1efb519e2e74e8dd701acde6 Jul 10 00:19:38.375448 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:38.375401 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53c5a23d-3991-4e70-b665-a45e2d68121e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tff8n\" (UID: \"53c5a23d-3991-4e70-b665-a45e2d68121e\") " pod="openshift-insights/insights-runtime-extractor-tff8n" Jul 10 00:19:38.375605 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:38.375459 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53c5a23d-3991-4e70-b665-a45e2d68121e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tff8n\" (UID: \"53c5a23d-3991-4e70-b665-a45e2d68121e\") " pod="openshift-insights/insights-runtime-extractor-tff8n" Jul 10 00:19:38.375605 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:19:38.375533 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 00:19:38.375605 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:19:38.375603 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 00:19:38.375726 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:19:38.375618 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/53c5a23d-3991-4e70-b665-a45e2d68121e-kube-rbac-proxy-cm podName:53c5a23d-3991-4e70-b665-a45e2d68121e nodeName:}" failed. No retries permitted until 2026-07-10 00:19:40.375598572 +0000 UTC m=+247.193904055 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/53c5a23d-3991-4e70-b665-a45e2d68121e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tff8n" (UID: "53c5a23d-3991-4e70-b665-a45e2d68121e") : configmap "kube-rbac-proxy" not found Jul 10 00:19:38.375726 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:19:38.375637 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53c5a23d-3991-4e70-b665-a45e2d68121e-insights-runtime-extractor-tls podName:53c5a23d-3991-4e70-b665-a45e2d68121e nodeName:}" failed. No retries permitted until 2026-07-10 00:19:40.375627248 +0000 UTC m=+247.193932725 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/53c5a23d-3991-4e70-b665-a45e2d68121e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tff8n" (UID: "53c5a23d-3991-4e70-b665-a45e2d68121e") : secret "insights-runtime-extractor-tls" not found Jul 10 00:19:38.462011 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:38.461978 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2mpf9_0da6eed2-eb77-4e83-bb07-a6163eaca53a/dns-node-resolver/0.log" Jul 10 00:19:38.488827 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:38.488788 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kjhg9_4651a90d-ef18-4974-9efd-a77fff3c9dfd/console-operator/1.log" Jul 10 00:19:38.489288 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:38.489219 2562 scope.go:117] "RemoveContainer" containerID="0bd4f659cdab52afe6250f448db52473766dc7b9c9569d67ee2d69dd80c40c02" Jul 10 00:19:38.489435 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:19:38.489405 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-kjhg9_openshift-console-operator(4651a90d-ef18-4974-9efd-a77fff3c9dfd)\"" pod="openshift-console-operator/console-operator-7866497494-kjhg9" podUID="4651a90d-ef18-4974-9efd-a77fff3c9dfd" Jul 10 00:19:38.490289 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:38.490250 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-9w2qg" event={"ID":"0064d533-5099-4a49-afef-12125cba31ad","Type":"ContainerStarted","Data":"88b308f44124acc6a0a9d5e5c1b121ef1a203a4b1efb519e2e74e8dd701acde6"} Jul 10 00:19:39.462518 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:39.462490 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gjk2r_03b7fa42-b493-40ce-a0d1-ce267379f3df/node-ca/0.log" Jul 10 00:19:39.495178 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:39.495145 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-9w2qg" event={"ID":"0064d533-5099-4a49-afef-12125cba31ad","Type":"ContainerStarted","Data":"86d6074f303ee0b3c5edf016506b95cc29d6a0dc259005049dbc8daa0bb5c8ad"} Jul 10 00:19:39.495554 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:39.495201 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-9w2qg" event={"ID":"0064d533-5099-4a49-afef-12125cba31ad","Type":"ContainerStarted","Data":"e7e883b83259e19c8fb5b624b217c0f43fb0d65e2adc201cb7b4f102fe243bd8"} Jul 10 00:19:39.509382 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:39.509328 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-9w2qg" podStartSLOduration=1.118191442 podStartE2EDuration="2.509307843s" totalImagesPullingTime="1.391116401s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:19:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:19:39.508339924 +0000 UTC m=+246.326645420" watchObservedRunningTime="2026-07-10 00:19:39.509307843 +0000 UTC m=+246.327613338" Jul 10 00:19:40.392805 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:40.392763 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53c5a23d-3991-4e70-b665-a45e2d68121e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tff8n\" (UID: \"53c5a23d-3991-4e70-b665-a45e2d68121e\") " pod="openshift-insights/insights-runtime-extractor-tff8n" Jul 10 00:19:40.392805 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:40.392809 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53c5a23d-3991-4e70-b665-a45e2d68121e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tff8n\" (UID: \"53c5a23d-3991-4e70-b665-a45e2d68121e\") " pod="openshift-insights/insights-runtime-extractor-tff8n" Jul 10 00:19:40.393104 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:19:40.392903 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 00:19:40.393104 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:19:40.392931 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 00:19:40.393104 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:19:40.392976 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/53c5a23d-3991-4e70-b665-a45e2d68121e-kube-rbac-proxy-cm podName:53c5a23d-3991-4e70-b665-a45e2d68121e nodeName:}" failed. No retries permitted until 2026-07-10 00:19:44.392957665 +0000 UTC m=+251.211263144 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/53c5a23d-3991-4e70-b665-a45e2d68121e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tff8n" (UID: "53c5a23d-3991-4e70-b665-a45e2d68121e") : configmap "kube-rbac-proxy" not found Jul 10 00:19:40.393104 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:19:40.392994 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53c5a23d-3991-4e70-b665-a45e2d68121e-insights-runtime-extractor-tls podName:53c5a23d-3991-4e70-b665-a45e2d68121e nodeName:}" failed. No retries permitted until 2026-07-10 00:19:44.392985578 +0000 UTC m=+251.211291052 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/53c5a23d-3991-4e70-b665-a45e2d68121e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tff8n" (UID: "53c5a23d-3991-4e70-b665-a45e2d68121e") : secret "insights-runtime-extractor-tls" not found Jul 10 00:19:40.462287 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:40.462254 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tjh96_55dcffca-f335-4b01-a6b8-4b2356d2fe57/serve-healthcheck-canary/0.log" Jul 10 00:19:40.493520 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:40.493482 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d7b4fce-5123-477b-800c-bec0da7fd372-service-ca-bundle\") pod \"router-default-76d8bdc488-dw9ns\" (UID: \"3d7b4fce-5123-477b-800c-bec0da7fd372\") " pod="openshift-ingress/router-default-76d8bdc488-dw9ns" Jul 10 00:19:40.493658 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:40.493531 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d7b4fce-5123-477b-800c-bec0da7fd372-metrics-certs\") pod \"router-default-76d8bdc488-dw9ns\" (UID: \"3d7b4fce-5123-477b-800c-bec0da7fd372\") " pod="openshift-ingress/router-default-76d8bdc488-dw9ns" Jul 10 00:19:40.493707 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:19:40.493668 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3d7b4fce-5123-477b-800c-bec0da7fd372-service-ca-bundle podName:3d7b4fce-5123-477b-800c-bec0da7fd372 nodeName:}" failed. No retries permitted until 2026-07-10 00:19:48.493647076 +0000 UTC m=+255.311952549 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3d7b4fce-5123-477b-800c-bec0da7fd372-service-ca-bundle") pod "router-default-76d8bdc488-dw9ns" (UID: "3d7b4fce-5123-477b-800c-bec0da7fd372") : configmap references non-existent config key: service-ca.crt Jul 10 00:19:40.493707 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:19:40.493669 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 10 00:19:40.493707 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:19:40.493708 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d7b4fce-5123-477b-800c-bec0da7fd372-metrics-certs podName:3d7b4fce-5123-477b-800c-bec0da7fd372 nodeName:}" failed. No retries permitted until 2026-07-10 00:19:48.493700271 +0000 UTC m=+255.312005743 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3d7b4fce-5123-477b-800c-bec0da7fd372-metrics-certs") pod "router-default-76d8bdc488-dw9ns" (UID: "3d7b4fce-5123-477b-800c-bec0da7fd372") : secret "router-metrics-certs-default" not found Jul 10 00:19:42.869120 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:42.869008 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-kjhg9" Jul 10 00:19:42.869120 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:42.869108 2562 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-kjhg9" Jul 10 00:19:42.869505 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:42.869455 2562 scope.go:117] "RemoveContainer" containerID="0bd4f659cdab52afe6250f448db52473766dc7b9c9569d67ee2d69dd80c40c02" Jul 10 00:19:42.869635 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:19:42.869612 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-kjhg9_openshift-console-operator(4651a90d-ef18-4974-9efd-a77fff3c9dfd)\"" pod="openshift-console-operator/console-operator-7866497494-kjhg9" podUID="4651a90d-ef18-4974-9efd-a77fff3c9dfd" Jul 10 00:19:44.425587 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:44.425553 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53c5a23d-3991-4e70-b665-a45e2d68121e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tff8n\" (UID: \"53c5a23d-3991-4e70-b665-a45e2d68121e\") " pod="openshift-insights/insights-runtime-extractor-tff8n" Jul 10 00:19:44.425587 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:44.425589 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53c5a23d-3991-4e70-b665-a45e2d68121e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tff8n\" (UID: \"53c5a23d-3991-4e70-b665-a45e2d68121e\") " pod="openshift-insights/insights-runtime-extractor-tff8n" Jul 10 00:19:44.425996 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:19:44.425679 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 00:19:44.425996 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:19:44.425729 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 00:19:44.425996 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:19:44.425749 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/53c5a23d-3991-4e70-b665-a45e2d68121e-kube-rbac-proxy-cm podName:53c5a23d-3991-4e70-b665-a45e2d68121e nodeName:}" failed. No retries permitted until 2026-07-10 00:19:52.425732832 +0000 UTC m=+259.244038304 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/53c5a23d-3991-4e70-b665-a45e2d68121e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tff8n" (UID: "53c5a23d-3991-4e70-b665-a45e2d68121e") : configmap "kube-rbac-proxy" not found Jul 10 00:19:44.425996 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:19:44.425771 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53c5a23d-3991-4e70-b665-a45e2d68121e-insights-runtime-extractor-tls podName:53c5a23d-3991-4e70-b665-a45e2d68121e nodeName:}" failed. No retries permitted until 2026-07-10 00:19:52.42575959 +0000 UTC m=+259.244065062 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/53c5a23d-3991-4e70-b665-a45e2d68121e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tff8n" (UID: "53c5a23d-3991-4e70-b665-a45e2d68121e") : secret "insights-runtime-extractor-tls" not found Jul 10 00:19:45.536044 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:45.536012 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62933c8a-818d-4580-a6c9-6d26dbdfd50d-metrics-certs\") pod \"network-metrics-daemon-x8v89\" (UID: \"62933c8a-818d-4580-a6c9-6d26dbdfd50d\") " pod="openshift-multus/network-metrics-daemon-x8v89" Jul 10 00:19:45.536431 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:19:45.536137 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 10 00:19:45.536431 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:19:45.536192 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62933c8a-818d-4580-a6c9-6d26dbdfd50d-metrics-certs podName:62933c8a-818d-4580-a6c9-6d26dbdfd50d nodeName:}" failed. No retries permitted until 2026-07-10 00:21:47.536175653 +0000 UTC m=+374.354481126 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/62933c8a-818d-4580-a6c9-6d26dbdfd50d-metrics-certs") pod "network-metrics-daemon-x8v89" (UID: "62933c8a-818d-4580-a6c9-6d26dbdfd50d") : secret "metrics-daemon-secret" not found Jul 10 00:19:48.561976 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:48.561943 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d7b4fce-5123-477b-800c-bec0da7fd372-metrics-certs\") pod \"router-default-76d8bdc488-dw9ns\" (UID: \"3d7b4fce-5123-477b-800c-bec0da7fd372\") " pod="openshift-ingress/router-default-76d8bdc488-dw9ns" Jul 10 00:19:48.562456 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:48.562051 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d7b4fce-5123-477b-800c-bec0da7fd372-service-ca-bundle\") pod \"router-default-76d8bdc488-dw9ns\" (UID: \"3d7b4fce-5123-477b-800c-bec0da7fd372\") " pod="openshift-ingress/router-default-76d8bdc488-dw9ns" Jul 10 00:19:48.562456 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:19:48.562109 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 10 00:19:48.562456 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:19:48.562170 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d7b4fce-5123-477b-800c-bec0da7fd372-metrics-certs podName:3d7b4fce-5123-477b-800c-bec0da7fd372 nodeName:}" failed. No retries permitted until 2026-07-10 00:20:04.562155029 +0000 UTC m=+271.380460506 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3d7b4fce-5123-477b-800c-bec0da7fd372-metrics-certs") pod "router-default-76d8bdc488-dw9ns" (UID: "3d7b4fce-5123-477b-800c-bec0da7fd372") : secret "router-metrics-certs-default" not found Jul 10 00:19:48.562456 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:19:48.562184 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3d7b4fce-5123-477b-800c-bec0da7fd372-service-ca-bundle podName:3d7b4fce-5123-477b-800c-bec0da7fd372 nodeName:}" failed. No retries permitted until 2026-07-10 00:20:04.562178366 +0000 UTC m=+271.380483839 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3d7b4fce-5123-477b-800c-bec0da7fd372-service-ca-bundle") pod "router-default-76d8bdc488-dw9ns" (UID: "3d7b4fce-5123-477b-800c-bec0da7fd372") : configmap references non-existent config key: service-ca.crt Jul 10 00:19:52.489891 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:52.489835 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53c5a23d-3991-4e70-b665-a45e2d68121e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tff8n\" (UID: \"53c5a23d-3991-4e70-b665-a45e2d68121e\") " pod="openshift-insights/insights-runtime-extractor-tff8n" Jul 10 00:19:52.489891 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:52.489894 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53c5a23d-3991-4e70-b665-a45e2d68121e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tff8n\" (UID: \"53c5a23d-3991-4e70-b665-a45e2d68121e\") " pod="openshift-insights/insights-runtime-extractor-tff8n" Jul 10 00:19:52.490374 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:19:52.490071 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 00:19:52.490374 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:19:52.490156 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53c5a23d-3991-4e70-b665-a45e2d68121e-insights-runtime-extractor-tls podName:53c5a23d-3991-4e70-b665-a45e2d68121e nodeName:}" failed. No retries permitted until 2026-07-10 00:20:08.490140149 +0000 UTC m=+275.308445621 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/53c5a23d-3991-4e70-b665-a45e2d68121e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tff8n" (UID: "53c5a23d-3991-4e70-b665-a45e2d68121e") : secret "insights-runtime-extractor-tls" not found Jul 10 00:19:52.490374 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:52.490362 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53c5a23d-3991-4e70-b665-a45e2d68121e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tff8n\" (UID: \"53c5a23d-3991-4e70-b665-a45e2d68121e\") " pod="openshift-insights/insights-runtime-extractor-tff8n" Jul 10 00:19:55.734040 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:55.734009 2562 scope.go:117] "RemoveContainer" containerID="0bd4f659cdab52afe6250f448db52473766dc7b9c9569d67ee2d69dd80c40c02" Jul 10 00:19:56.548918 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:56.548884 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kjhg9_4651a90d-ef18-4974-9efd-a77fff3c9dfd/console-operator/1.log" Jul 10 00:19:56.549121 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:56.548946 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-kjhg9" event={"ID":"4651a90d-ef18-4974-9efd-a77fff3c9dfd","Type":"ContainerStarted","Data":"5097e4acc429e48839daae05a94912d638c33dc7ec9fa77b5b34c3f4733e10e2"} Jul 10 00:19:56.549243 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:56.549223 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-kjhg9" Jul 10 00:19:56.563988 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:56.563942 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7866497494-kjhg9" podStartSLOduration=21.675412379 podStartE2EDuration="24.56392884s" totalImagesPullingTime="2.888516461s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:19:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:19:56.563421934 +0000 UTC m=+263.381727430" watchObservedRunningTime="2026-07-10 00:19:56.56392884 +0000 UTC m=+263.382234338" Jul 10 00:19:56.867902 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:19:56.867823 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7866497494-kjhg9" Jul 10 00:20:04.589575 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:04.589542 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d7b4fce-5123-477b-800c-bec0da7fd372-service-ca-bundle\") pod \"router-default-76d8bdc488-dw9ns\" (UID: \"3d7b4fce-5123-477b-800c-bec0da7fd372\") " pod="openshift-ingress/router-default-76d8bdc488-dw9ns" Jul 10 00:20:04.589575 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:04.589582 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d7b4fce-5123-477b-800c-bec0da7fd372-metrics-certs\") pod \"router-default-76d8bdc488-dw9ns\" (UID: \"3d7b4fce-5123-477b-800c-bec0da7fd372\") " pod="openshift-ingress/router-default-76d8bdc488-dw9ns" Jul 10 00:20:04.590204 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:04.590184 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d7b4fce-5123-477b-800c-bec0da7fd372-service-ca-bundle\") pod \"router-default-76d8bdc488-dw9ns\" (UID: \"3d7b4fce-5123-477b-800c-bec0da7fd372\") " pod="openshift-ingress/router-default-76d8bdc488-dw9ns" Jul 10 00:20:04.591883 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:04.591864 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d7b4fce-5123-477b-800c-bec0da7fd372-metrics-certs\") pod \"router-default-76d8bdc488-dw9ns\" (UID: \"3d7b4fce-5123-477b-800c-bec0da7fd372\") " pod="openshift-ingress/router-default-76d8bdc488-dw9ns" Jul 10 00:20:04.874301 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:04.874219 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-qgwtt\"" Jul 10 00:20:04.882890 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:04.882864 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-76d8bdc488-dw9ns" Jul 10 00:20:05.005040 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:05.004948 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-76d8bdc488-dw9ns"] Jul 10 00:20:05.007190 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:20:05.007158 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d7b4fce_5123_477b_800c_bec0da7fd372.slice/crio-17692ebababdc4a9a4057d7896fe0394357d0a70f21f526125c10eb5ee824ecc WatchSource:0}: Error finding container 17692ebababdc4a9a4057d7896fe0394357d0a70f21f526125c10eb5ee824ecc: Status 404 returned error can't find the container with id 17692ebababdc4a9a4057d7896fe0394357d0a70f21f526125c10eb5ee824ecc Jul 10 00:20:05.576422 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:05.576388 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-76d8bdc488-dw9ns" event={"ID":"3d7b4fce-5123-477b-800c-bec0da7fd372","Type":"ContainerStarted","Data":"34906d8ee5775308aaa5a23787959a497ec6bb7f8a126b8f48d8c6cbc14a4a0d"} Jul 10 00:20:05.576589 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:05.576440 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-76d8bdc488-dw9ns" event={"ID":"3d7b4fce-5123-477b-800c-bec0da7fd372","Type":"ContainerStarted","Data":"17692ebababdc4a9a4057d7896fe0394357d0a70f21f526125c10eb5ee824ecc"} Jul 10 00:20:05.594941 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:05.594896 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-76d8bdc488-dw9ns" podStartSLOduration=33.594882446 podStartE2EDuration="33.594882446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:19:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:20:05.593802295 +0000 UTC m=+272.412107791" watchObservedRunningTime="2026-07-10 00:20:05.594882446 +0000 UTC m=+272.413187941" Jul 10 00:20:05.883644 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:05.883572 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-76d8bdc488-dw9ns" Jul 10 00:20:05.886104 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:05.886063 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-76d8bdc488-dw9ns" Jul 10 00:20:06.580008 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:06.579977 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-76d8bdc488-dw9ns" Jul 10 00:20:06.581269 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:06.581247 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-76d8bdc488-dw9ns" Jul 10 00:20:08.520269 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:08.520230 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53c5a23d-3991-4e70-b665-a45e2d68121e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tff8n\" (UID: \"53c5a23d-3991-4e70-b665-a45e2d68121e\") " pod="openshift-insights/insights-runtime-extractor-tff8n" Jul 10 00:20:08.522627 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:08.522601 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53c5a23d-3991-4e70-b665-a45e2d68121e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tff8n\" (UID: \"53c5a23d-3991-4e70-b665-a45e2d68121e\") " pod="openshift-insights/insights-runtime-extractor-tff8n" Jul 10 00:20:08.775252 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:08.775178 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tff8n" Jul 10 00:20:08.895350 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:08.895327 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tff8n"] Jul 10 00:20:08.897307 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:20:08.897275 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53c5a23d_3991_4e70_b665_a45e2d68121e.slice/crio-224846ab352f21ea2bdb64d987c809bdd6e077218cbe8de7940692d536515c06 WatchSource:0}: Error finding container 224846ab352f21ea2bdb64d987c809bdd6e077218cbe8de7940692d536515c06: Status 404 returned error can't find the container with id 224846ab352f21ea2bdb64d987c809bdd6e077218cbe8de7940692d536515c06 Jul 10 00:20:09.592793 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:09.592751 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tff8n" event={"ID":"53c5a23d-3991-4e70-b665-a45e2d68121e","Type":"ContainerStarted","Data":"224846ab352f21ea2bdb64d987c809bdd6e077218cbe8de7940692d536515c06"} Jul 10 00:20:12.360380 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:12.360344 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-84d8c57647-m8t5j"] Jul 10 00:20:12.363815 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:12.363794 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84d8c57647-m8t5j" Jul 10 00:20:12.365833 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:12.365797 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 10 00:20:12.365947 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:12.365798 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 10 00:20:12.365947 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:12.365902 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 10 00:20:12.366186 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:12.366165 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 10 00:20:12.366745 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:12.366724 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 10 00:20:12.366854 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:12.366753 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 10 00:20:12.366854 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:12.366764 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 10 00:20:12.367427 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:12.367399 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-g7w2x\"" Jul 10 00:20:12.373142 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:12.373120 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84d8c57647-m8t5j"] Jul 10 00:20:12.454886 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:12.454853 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b54ecf35-9fb0-417d-be21-87a463f92bb4-service-ca\") pod \"console-84d8c57647-m8t5j\" (UID: \"b54ecf35-9fb0-417d-be21-87a463f92bb4\") " pod="openshift-console/console-84d8c57647-m8t5j" Jul 10 00:20:12.455111 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:12.454903 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b54ecf35-9fb0-417d-be21-87a463f92bb4-console-oauth-config\") pod \"console-84d8c57647-m8t5j\" (UID: \"b54ecf35-9fb0-417d-be21-87a463f92bb4\") " pod="openshift-console/console-84d8c57647-m8t5j" Jul 10 00:20:12.455111 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:12.455013 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b54ecf35-9fb0-417d-be21-87a463f92bb4-console-serving-cert\") pod \"console-84d8c57647-m8t5j\" (UID: \"b54ecf35-9fb0-417d-be21-87a463f92bb4\") " pod="openshift-console/console-84d8c57647-m8t5j" Jul 10 00:20:12.455212 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:12.455135 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b54ecf35-9fb0-417d-be21-87a463f92bb4-oauth-serving-cert\") pod \"console-84d8c57647-m8t5j\" (UID: \"b54ecf35-9fb0-417d-be21-87a463f92bb4\") " pod="openshift-console/console-84d8c57647-m8t5j" Jul 10 00:20:12.455212 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:12.455172 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b54ecf35-9fb0-417d-be21-87a463f92bb4-console-config\") pod \"console-84d8c57647-m8t5j\" (UID: \"b54ecf35-9fb0-417d-be21-87a463f92bb4\") " pod="openshift-console/console-84d8c57647-m8t5j" Jul 10 00:20:12.455308 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:12.455260 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b885\" (UniqueName: \"kubernetes.io/projected/b54ecf35-9fb0-417d-be21-87a463f92bb4-kube-api-access-8b885\") pod \"console-84d8c57647-m8t5j\" (UID: \"b54ecf35-9fb0-417d-be21-87a463f92bb4\") " pod="openshift-console/console-84d8c57647-m8t5j" Jul 10 00:20:12.556621 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:12.556586 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b54ecf35-9fb0-417d-be21-87a463f92bb4-service-ca\") pod \"console-84d8c57647-m8t5j\" (UID: \"b54ecf35-9fb0-417d-be21-87a463f92bb4\") " pod="openshift-console/console-84d8c57647-m8t5j" Jul 10 00:20:12.556621 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:12.556627 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b54ecf35-9fb0-417d-be21-87a463f92bb4-console-oauth-config\") pod \"console-84d8c57647-m8t5j\" (UID: \"b54ecf35-9fb0-417d-be21-87a463f92bb4\") " pod="openshift-console/console-84d8c57647-m8t5j" Jul 10 00:20:12.556855 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:12.556665 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b54ecf35-9fb0-417d-be21-87a463f92bb4-console-serving-cert\") pod \"console-84d8c57647-m8t5j\" (UID: \"b54ecf35-9fb0-417d-be21-87a463f92bb4\") " pod="openshift-console/console-84d8c57647-m8t5j" Jul 10 00:20:12.556855 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:12.556823 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b54ecf35-9fb0-417d-be21-87a463f92bb4-oauth-serving-cert\") pod \"console-84d8c57647-m8t5j\" (UID: \"b54ecf35-9fb0-417d-be21-87a463f92bb4\") " pod="openshift-console/console-84d8c57647-m8t5j" Jul 10 00:20:12.556855 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:12.556854 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b54ecf35-9fb0-417d-be21-87a463f92bb4-console-config\") pod \"console-84d8c57647-m8t5j\" (UID: \"b54ecf35-9fb0-417d-be21-87a463f92bb4\") " pod="openshift-console/console-84d8c57647-m8t5j" Jul 10 00:20:12.556999 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:12.556885 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b885\" (UniqueName: \"kubernetes.io/projected/b54ecf35-9fb0-417d-be21-87a463f92bb4-kube-api-access-8b885\") pod \"console-84d8c57647-m8t5j\" (UID: \"b54ecf35-9fb0-417d-be21-87a463f92bb4\") " pod="openshift-console/console-84d8c57647-m8t5j" Jul 10 00:20:12.557443 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:12.557416 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b54ecf35-9fb0-417d-be21-87a463f92bb4-service-ca\") pod \"console-84d8c57647-m8t5j\" (UID: \"b54ecf35-9fb0-417d-be21-87a463f92bb4\") " pod="openshift-console/console-84d8c57647-m8t5j" Jul 10 00:20:12.557551 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:12.557533 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b54ecf35-9fb0-417d-be21-87a463f92bb4-console-config\") pod \"console-84d8c57647-m8t5j\" (UID: \"b54ecf35-9fb0-417d-be21-87a463f92bb4\") " pod="openshift-console/console-84d8c57647-m8t5j" Jul 10 00:20:12.557593 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:12.557564 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b54ecf35-9fb0-417d-be21-87a463f92bb4-oauth-serving-cert\") pod \"console-84d8c57647-m8t5j\" (UID: \"b54ecf35-9fb0-417d-be21-87a463f92bb4\") " pod="openshift-console/console-84d8c57647-m8t5j" Jul 10 00:20:12.559169 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:12.559139 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b54ecf35-9fb0-417d-be21-87a463f92bb4-console-oauth-config\") pod \"console-84d8c57647-m8t5j\" (UID: \"b54ecf35-9fb0-417d-be21-87a463f92bb4\") " pod="openshift-console/console-84d8c57647-m8t5j" Jul 10 00:20:12.559169 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:12.559164 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b54ecf35-9fb0-417d-be21-87a463f92bb4-console-serving-cert\") pod \"console-84d8c57647-m8t5j\" (UID: \"b54ecf35-9fb0-417d-be21-87a463f92bb4\") " pod="openshift-console/console-84d8c57647-m8t5j" Jul 10 00:20:12.564547 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:12.564524 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b885\" (UniqueName: \"kubernetes.io/projected/b54ecf35-9fb0-417d-be21-87a463f92bb4-kube-api-access-8b885\") pod \"console-84d8c57647-m8t5j\" (UID: \"b54ecf35-9fb0-417d-be21-87a463f92bb4\") " pod="openshift-console/console-84d8c57647-m8t5j" Jul 10 00:20:12.678498 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:12.678464 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84d8c57647-m8t5j" Jul 10 00:20:12.807669 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:12.807585 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84d8c57647-m8t5j"] Jul 10 00:20:12.831461 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:20:12.831433 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb54ecf35_9fb0_417d_be21_87a463f92bb4.slice/crio-2269289079e358361de3dbe399d97e8863fce5d969fa38ffbd1cacd345125de1 WatchSource:0}: Error finding container 2269289079e358361de3dbe399d97e8863fce5d969fa38ffbd1cacd345125de1: Status 404 returned error can't find the container with id 2269289079e358361de3dbe399d97e8863fce5d969fa38ffbd1cacd345125de1 Jul 10 00:20:13.607232 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:13.607196 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tff8n" event={"ID":"53c5a23d-3991-4e70-b665-a45e2d68121e","Type":"ContainerStarted","Data":"86f1aac716dbe220592b9b9747993503588d6a20f9a7bb2e0d4f40a9527cb016"} Jul 10 00:20:13.607630 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:13.607255 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tff8n" event={"ID":"53c5a23d-3991-4e70-b665-a45e2d68121e","Type":"ContainerStarted","Data":"06a2ac482e8b364b107354e1234eadac70dcec85f054b6a2853159f4771cfb9f"} Jul 10 00:20:13.608308 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:13.608285 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84d8c57647-m8t5j" event={"ID":"b54ecf35-9fb0-417d-be21-87a463f92bb4","Type":"ContainerStarted","Data":"2269289079e358361de3dbe399d97e8863fce5d969fa38ffbd1cacd345125de1"} Jul 10 00:20:17.623706 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:17.623662 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84d8c57647-m8t5j" event={"ID":"b54ecf35-9fb0-417d-be21-87a463f92bb4","Type":"ContainerStarted","Data":"893e800b1840b3c520c77ca5eb4658ef0eb75eed21b8a96ade3e7bece509f39e"} Jul 10 00:20:17.625589 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:17.625563 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tff8n" event={"ID":"53c5a23d-3991-4e70-b665-a45e2d68121e","Type":"ContainerStarted","Data":"ce990089335df2f3efdf9d8748133c3bc7121981b91667c94fb202311404a4bd"} Jul 10 00:20:17.639432 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:17.639378 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-84d8c57647-m8t5j" podStartSLOduration=1.51380086 podStartE2EDuration="5.639359952s" totalImagesPullingTime="4.125559092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:20:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:20:17.638423616 +0000 UTC m=+284.456729110" watchObservedRunningTime="2026-07-10 00:20:17.639359952 +0000 UTC m=+284.457665449" Jul 10 00:20:17.653683 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:17.653628 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-tff8n" podStartSLOduration=33.711634817 podStartE2EDuration="41.653610118s" totalImagesPullingTime="7.941975301s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 00:19:36 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:20:17.653186398 +0000 UTC m=+284.471491894" watchObservedRunningTime="2026-07-10 00:20:17.653610118 +0000 UTC m=+284.471915608" Jul 10 00:20:18.057843 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:18.057798 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-5nckm"] Jul 10 00:20:18.061346 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:18.061321 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5nckm" Jul 10 00:20:18.063724 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:18.063694 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 10 00:20:18.063724 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:18.063717 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 10 00:20:18.063922 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:18.063736 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 10 00:20:18.063922 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:18.063706 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 10 00:20:18.064524 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:18.064507 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-xhc2d\"" Jul 10 00:20:18.064584 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:18.064532 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 10 00:20:18.071447 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:18.071418 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-5nckm"] Jul 10 00:20:18.095739 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:18.095707 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-k76q6"] Jul 10 00:20:18.099159 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:18.099139 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-k76q6" Jul 10 00:20:18.101383 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:18.101344 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 10 00:20:18.101383 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:18.101371 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dtnhw\"" Jul 10 00:20:18.101560 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:18.101361 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 10 00:20:18.101637 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:18.101623 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 10 00:20:18.103048 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:18.103024 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l6ww\" (UniqueName: \"kubernetes.io/projected/51841204-01e4-47f8-8bc0-83f66203bc07-kube-api-access-9l6ww\") pod \"openshift-state-metrics-84bd68b896-5nckm\" (UID: \"51841204-01e4-47f8-8bc0-83f66203bc07\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5nckm" Jul 10 00:20:18.103152 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:18.103108 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/51841204-01e4-47f8-8bc0-83f66203bc07-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-5nckm\" (UID: \"51841204-01e4-47f8-8bc0-83f66203bc07\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5nckm" Jul 10 00:20:18.103214 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:18.103164 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/51841204-01e4-47f8-8bc0-83f66203bc07-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-5nckm\" (UID: \"51841204-01e4-47f8-8bc0-83f66203bc07\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5nckm" Jul 10 00:20:18.103214 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:18.103204 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/51841204-01e4-47f8-8bc0-83f66203bc07-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-5nckm\" (UID: \"51841204-01e4-47f8-8bc0-83f66203bc07\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5nckm" Jul 10 00:20:18.204385 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:18.204351 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l6ww\" (UniqueName: \"kubernetes.io/projected/51841204-01e4-47f8-8bc0-83f66203bc07-kube-api-access-9l6ww\") pod \"openshift-state-metrics-84bd68b896-5nckm\" (UID: \"51841204-01e4-47f8-8bc0-83f66203bc07\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5nckm" Jul 10 00:20:18.204385 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:18.204394 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlnlv\" (UniqueName: \"kubernetes.io/projected/38181004-3a43-475c-81e5-3cc031197de2-kube-api-access-wlnlv\") pod \"node-exporter-k76q6\" (UID: \"38181004-3a43-475c-81e5-3cc031197de2\") " pod="openshift-monitoring/node-exporter-k76q6" Jul 10 00:20:18.204659 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:18.204435 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/38181004-3a43-475c-81e5-3cc031197de2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-k76q6\" (UID: \"38181004-3a43-475c-81e5-3cc031197de2\") " pod="openshift-monitoring/node-exporter-k76q6" Jul 10 00:20:18.204659 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:18.204482 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/38181004-3a43-475c-81e5-3cc031197de2-root\") pod \"node-exporter-k76q6\" (UID: \"38181004-3a43-475c-81e5-3cc031197de2\") " pod="openshift-monitoring/node-exporter-k76q6" Jul 10 00:20:18.204659 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:18.204566 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/38181004-3a43-475c-81e5-3cc031197de2-metrics-client-ca\") pod \"node-exporter-k76q6\" (UID: \"38181004-3a43-475c-81e5-3cc031197de2\") " pod="openshift-monitoring/node-exporter-k76q6" Jul 10 00:20:18.204659 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:18.204617 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/38181004-3a43-475c-81e5-3cc031197de2-node-exporter-tls\") pod \"node-exporter-k76q6\" (UID: \"38181004-3a43-475c-81e5-3cc031197de2\") " pod="openshift-monitoring/node-exporter-k76q6" Jul 10 00:20:18.204659 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:18.204657 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/51841204-01e4-47f8-8bc0-83f66203bc07-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-5nckm\" (UID: \"51841204-01e4-47f8-8bc0-83f66203bc07\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5nckm" Jul 10 00:20:18.204916 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:18.204702 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/38181004-3a43-475c-81e5-3cc031197de2-node-exporter-textfile\") pod \"node-exporter-k76q6\" (UID: \"38181004-3a43-475c-81e5-3cc031197de2\") " pod="openshift-monitoring/node-exporter-k76q6" Jul 10 00:20:18.204916 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:18.204728 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/38181004-3a43-475c-81e5-3cc031197de2-node-exporter-accelerators-collector-config\") pod \"node-exporter-k76q6\" (UID: \"38181004-3a43-475c-81e5-3cc031197de2\") " pod="openshift-monitoring/node-exporter-k76q6" Jul 10 00:20:18.204916 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:18.204783 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/51841204-01e4-47f8-8bc0-83f66203bc07-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-5nckm\" (UID: \"51841204-01e4-47f8-8bc0-83f66203bc07\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5nckm" Jul 10 00:20:18.204916 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:18.204849 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/51841204-01e4-47f8-8bc0-83f66203bc07-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-5nckm\" (UID: \"51841204-01e4-47f8-8bc0-83f66203bc07\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5nckm" Jul 10 00:20:18.204916 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:18.204877 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/38181004-3a43-475c-81e5-3cc031197de2-sys\") pod \"node-exporter-k76q6\" (UID: \"38181004-3a43-475c-81e5-3cc031197de2\") " pod="openshift-monitoring/node-exporter-k76q6" Jul 10 00:20:18.205116 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:18.204920 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/38181004-3a43-475c-81e5-3cc031197de2-node-exporter-wtmp\") pod \"node-exporter-k76q6\" (UID: \"38181004-3a43-475c-81e5-3cc031197de2\") " pod="openshift-monitoring/node-exporter-k76q6" Jul 10 00:20:18.205507 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:18.205477 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/51841204-01e4-47f8-8bc0-83f66203bc07-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-5nckm\" (UID: \"51841204-01e4-47f8-8bc0-83f66203bc07\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5nckm" Jul 10 00:20:18.207154 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:18.207132 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/51841204-01e4-47f8-8bc0-83f66203bc07-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-5nckm\" (UID: \"51841204-01e4-47f8-8bc0-83f66203bc07\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5nckm" Jul 10 00:20:18.207248 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:18.207224 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/51841204-01e4-47f8-8bc0-83f66203bc07-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-5nckm\" (UID: \"51841204-01e4-47f8-8bc0-83f66203bc07\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5nckm" Jul 10 00:20:18.214851 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:18.214827 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l6ww\" (UniqueName: \"kubernetes.io/projected/51841204-01e4-47f8-8bc0-83f66203bc07-kube-api-access-9l6ww\") pod \"openshift-state-metrics-84bd68b896-5nckm\" (UID: \"51841204-01e4-47f8-8bc0-83f66203bc07\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5nckm" Jul 10 00:20:18.305432 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:18.305391 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/38181004-3a43-475c-81e5-3cc031197de2-node-exporter-wtmp\") pod \"node-exporter-k76q6\" (UID: \"38181004-3a43-475c-81e5-3cc031197de2\") " pod="openshift-monitoring/node-exporter-k76q6" Jul 10 00:20:18.305594 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:18.305469 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlnlv\" (UniqueName: \"kubernetes.io/projected/38181004-3a43-475c-81e5-3cc031197de2-kube-api-access-wlnlv\") pod \"node-exporter-k76q6\" (UID: \"38181004-3a43-475c-81e5-3cc031197de2\") " pod="openshift-monitoring/node-exporter-k76q6" Jul 10 00:20:18.305594 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:18.305511 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/38181004-3a43-475c-81e5-3cc031197de2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-k76q6\" (UID: \"38181004-3a43-475c-81e5-3cc031197de2\") " pod="openshift-monitoring/node-exporter-k76q6" Jul 10 00:20:18.305594 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:18.305546 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/38181004-3a43-475c-81e5-3cc031197de2-root\") pod \"node-exporter-k76q6\" (UID: \"38181004-3a43-475c-81e5-3cc031197de2\") " pod="openshift-monitoring/node-exporter-k76q6" Jul 10 00:20:18.305594 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:18.305580 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/38181004-3a43-475c-81e5-3cc031197de2-metrics-client-ca\") pod \"node-exporter-k76q6\" (UID: \"38181004-3a43-475c-81e5-3cc031197de2\") " pod="openshift-monitoring/node-exporter-k76q6" Jul 10 00:20:18.305827 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:18.305606 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/38181004-3a43-475c-81e5-3cc031197de2-node-exporter-tls\") pod \"node-exporter-k76q6\" (UID: \"38181004-3a43-475c-81e5-3cc031197de2\") " pod="openshift-monitoring/node-exporter-k76q6" Jul 10 00:20:18.305827 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:18.305640 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/38181004-3a43-475c-81e5-3cc031197de2-node-exporter-textfile\") pod \"node-exporter-k76q6\" (UID: \"38181004-3a43-475c-81e5-3cc031197de2\") " pod="openshift-monitoring/node-exporter-k76q6" Jul 10 00:20:18.305827 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:18.305649 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/38181004-3a43-475c-81e5-3cc031197de2-root\") pod \"node-exporter-k76q6\" (UID: \"38181004-3a43-475c-81e5-3cc031197de2\") " pod="openshift-monitoring/node-exporter-k76q6" Jul 10 00:20:18.305827 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:18.305666 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/38181004-3a43-475c-81e5-3cc031197de2-node-exporter-accelerators-collector-config\") pod \"node-exporter-k76q6\" (UID: \"38181004-3a43-475c-81e5-3cc031197de2\") " pod="openshift-monitoring/node-exporter-k76q6" Jul 10 00:20:18.305827 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:18.305748 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/38181004-3a43-475c-81e5-3cc031197de2-sys\") pod \"node-exporter-k76q6\" (UID: \"38181004-3a43-475c-81e5-3cc031197de2\") " pod="openshift-monitoring/node-exporter-k76q6" Jul 10 00:20:18.306062 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:18.305833 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/38181004-3a43-475c-81e5-3cc031197de2-sys\") pod \"node-exporter-k76q6\" (UID: \"38181004-3a43-475c-81e5-3cc031197de2\") " pod="openshift-monitoring/node-exporter-k76q6" Jul 10 00:20:18.306062 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:18.305600 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/38181004-3a43-475c-81e5-3cc031197de2-node-exporter-wtmp\") pod \"node-exporter-k76q6\" (UID: \"38181004-3a43-475c-81e5-3cc031197de2\") " pod="openshift-monitoring/node-exporter-k76q6" Jul 10 00:20:18.306394 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:18.306371 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/38181004-3a43-475c-81e5-3cc031197de2-node-exporter-accelerators-collector-config\") pod \"node-exporter-k76q6\" (UID: \"38181004-3a43-475c-81e5-3cc031197de2\") " pod="openshift-monitoring/node-exporter-k76q6" Jul 10 00:20:18.306546 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:18.306456 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/38181004-3a43-475c-81e5-3cc031197de2-node-exporter-textfile\") pod \"node-exporter-k76q6\" (UID: \"38181004-3a43-475c-81e5-3cc031197de2\") " pod="openshift-monitoring/node-exporter-k76q6" Jul 10 00:20:18.306639 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:18.306566 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/38181004-3a43-475c-81e5-3cc031197de2-metrics-client-ca\") pod \"node-exporter-k76q6\" (UID: \"38181004-3a43-475c-81e5-3cc031197de2\") " pod="openshift-monitoring/node-exporter-k76q6" Jul 10 00:20:18.308887 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:18.308828 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/38181004-3a43-475c-81e5-3cc031197de2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-k76q6\" (UID: \"38181004-3a43-475c-81e5-3cc031197de2\") " pod="openshift-monitoring/node-exporter-k76q6" Jul 10 00:20:18.308887 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:18.308831 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/38181004-3a43-475c-81e5-3cc031197de2-node-exporter-tls\") pod \"node-exporter-k76q6\" (UID: \"38181004-3a43-475c-81e5-3cc031197de2\") " pod="openshift-monitoring/node-exporter-k76q6" Jul 10 00:20:18.313844 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:18.313806 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlnlv\" (UniqueName: \"kubernetes.io/projected/38181004-3a43-475c-81e5-3cc031197de2-kube-api-access-wlnlv\") pod \"node-exporter-k76q6\" (UID: \"38181004-3a43-475c-81e5-3cc031197de2\") " pod="openshift-monitoring/node-exporter-k76q6" Jul 10 00:20:18.370699 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:18.370667 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5nckm" Jul 10 00:20:18.408471 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:18.408432 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-k76q6" Jul 10 00:20:18.419407 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:20:18.419366 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38181004_3a43_475c_81e5_3cc031197de2.slice/crio-3b7e98bd6b4dbd9c17aeb90160671efa7cf2dd80784daa90c48fc187f40e17ae WatchSource:0}: Error finding container 3b7e98bd6b4dbd9c17aeb90160671efa7cf2dd80784daa90c48fc187f40e17ae: Status 404 returned error can't find the container with id 3b7e98bd6b4dbd9c17aeb90160671efa7cf2dd80784daa90c48fc187f40e17ae Jul 10 00:20:18.508797 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:18.508763 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-5nckm"] Jul 10 00:20:18.511739 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:20:18.511712 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51841204_01e4_47f8_8bc0_83f66203bc07.slice/crio-15000fe1dd20c2fd5773d819b05a62fa9708f35e4a2087ca6c396c6459d60b21 WatchSource:0}: Error finding container 15000fe1dd20c2fd5773d819b05a62fa9708f35e4a2087ca6c396c6459d60b21: Status 404 returned error can't find the container with id 15000fe1dd20c2fd5773d819b05a62fa9708f35e4a2087ca6c396c6459d60b21 Jul 10 00:20:18.631299 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:18.631261 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5nckm" event={"ID":"51841204-01e4-47f8-8bc0-83f66203bc07","Type":"ContainerStarted","Data":"879babbd90937204ed1da3b07e50fc315b4080fd31d5cbe6bb47d1f800fc141b"} Jul 10 00:20:18.631701 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:18.631334 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5nckm" event={"ID":"51841204-01e4-47f8-8bc0-83f66203bc07","Type":"ContainerStarted","Data":"15000fe1dd20c2fd5773d819b05a62fa9708f35e4a2087ca6c396c6459d60b21"} Jul 10 00:20:18.632408 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:18.632376 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k76q6" event={"ID":"38181004-3a43-475c-81e5-3cc031197de2","Type":"ContainerStarted","Data":"3b7e98bd6b4dbd9c17aeb90160671efa7cf2dd80784daa90c48fc187f40e17ae"} Jul 10 00:20:19.638279 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:19.638237 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5nckm" event={"ID":"51841204-01e4-47f8-8bc0-83f66203bc07","Type":"ContainerStarted","Data":"b203ab0dfde972e049f8c5d8bd265f023dbbfa45be39a7947b2f961002564206"} Jul 10 00:20:20.270894 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:20:20.270790 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-5d58f56c87-wn9cc" podUID="bbbf0b29-8280-450c-99e6-5965971c89da" Jul 10 00:20:20.270894 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:20:20.270789 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-9mt8s" podUID="b56f3715-4283-4226-8188-d1b3869826db" Jul 10 00:20:20.647895 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:20.647858 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5nckm" event={"ID":"51841204-01e4-47f8-8bc0-83f66203bc07","Type":"ContainerStarted","Data":"387348d66b822b6eb0810cabb58c28068059b9e48377613ed68b8c47afa14b29"} Jul 10 00:20:20.649355 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:20.649334 2562 generic.go:356] "Generic (PLEG): container finished" podID="38181004-3a43-475c-81e5-3cc031197de2" containerID="3b5a05f0c43aef2c0ccd02433801f99bc55cee6c094c963654e0fa9d94ffbead" exitCode=0 Jul 10 00:20:20.649441 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:20.649397 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9mt8s" Jul 10 00:20:20.649441 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:20.649414 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k76q6" event={"ID":"38181004-3a43-475c-81e5-3cc031197de2","Type":"ContainerDied","Data":"3b5a05f0c43aef2c0ccd02433801f99bc55cee6c094c963654e0fa9d94ffbead"} Jul 10 00:20:20.649581 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:20.649570 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d58f56c87-wn9cc" Jul 10 00:20:20.663103 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:20.663043 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5nckm" podStartSLOduration=1.455404742 podStartE2EDuration="2.663029155s" totalImagesPullingTime="1.207624413s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:20:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:20:20.66242617 +0000 UTC m=+287.480731665" watchObservedRunningTime="2026-07-10 00:20:20.663029155 +0000 UTC m=+287.481334650" Jul 10 00:20:21.057814 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:21.057781 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6955559f45-ftdzl"] Jul 10 00:20:21.061371 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:21.061353 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6955559f45-ftdzl" Jul 10 00:20:21.063544 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:21.063514 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 10 00:20:21.063716 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:21.063576 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-8j1ajajmeb9ck\"" Jul 10 00:20:21.063716 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:21.063708 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 10 00:20:21.063849 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:21.063778 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-gc5hv\"" Jul 10 00:20:21.063965 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:21.063943 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 10 00:20:21.064031 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:21.063975 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 10 00:20:21.064031 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:21.064013 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 10 00:20:21.071722 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:21.071701 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6955559f45-ftdzl"] Jul 10 00:20:21.127282 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:21.127256 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm2hc\" (UniqueName: \"kubernetes.io/projected/90f6720d-0564-473d-93d3-68905e69d4ca-kube-api-access-fm2hc\") pod \"thanos-querier-6955559f45-ftdzl\" (UID: \"90f6720d-0564-473d-93d3-68905e69d4ca\") " pod="openshift-monitoring/thanos-querier-6955559f45-ftdzl" Jul 10 00:20:21.127440 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:21.127305 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/90f6720d-0564-473d-93d3-68905e69d4ca-secret-grpc-tls\") pod \"thanos-querier-6955559f45-ftdzl\" (UID: \"90f6720d-0564-473d-93d3-68905e69d4ca\") " pod="openshift-monitoring/thanos-querier-6955559f45-ftdzl" Jul 10 00:20:21.127440 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:21.127330 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/90f6720d-0564-473d-93d3-68905e69d4ca-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6955559f45-ftdzl\" (UID: \"90f6720d-0564-473d-93d3-68905e69d4ca\") " pod="openshift-monitoring/thanos-querier-6955559f45-ftdzl" Jul 10 00:20:21.127440 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:21.127376 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/90f6720d-0564-473d-93d3-68905e69d4ca-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6955559f45-ftdzl\" (UID: \"90f6720d-0564-473d-93d3-68905e69d4ca\") " pod="openshift-monitoring/thanos-querier-6955559f45-ftdzl" Jul 10 00:20:21.127440 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:21.127393 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/90f6720d-0564-473d-93d3-68905e69d4ca-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6955559f45-ftdzl\" (UID: \"90f6720d-0564-473d-93d3-68905e69d4ca\") " pod="openshift-monitoring/thanos-querier-6955559f45-ftdzl" Jul 10 00:20:21.127440 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:21.127413 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90f6720d-0564-473d-93d3-68905e69d4ca-metrics-client-ca\") pod \"thanos-querier-6955559f45-ftdzl\" (UID: \"90f6720d-0564-473d-93d3-68905e69d4ca\") " pod="openshift-monitoring/thanos-querier-6955559f45-ftdzl" Jul 10 00:20:21.127440 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:21.127432 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/90f6720d-0564-473d-93d3-68905e69d4ca-secret-thanos-querier-tls\") pod \"thanos-querier-6955559f45-ftdzl\" (UID: \"90f6720d-0564-473d-93d3-68905e69d4ca\") " pod="openshift-monitoring/thanos-querier-6955559f45-ftdzl" Jul 10 00:20:21.127648 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:21.127490 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/90f6720d-0564-473d-93d3-68905e69d4ca-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6955559f45-ftdzl\" (UID: \"90f6720d-0564-473d-93d3-68905e69d4ca\") " pod="openshift-monitoring/thanos-querier-6955559f45-ftdzl" Jul 10 00:20:21.228808 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:21.228772 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/90f6720d-0564-473d-93d3-68905e69d4ca-secret-grpc-tls\") pod \"thanos-querier-6955559f45-ftdzl\" (UID: \"90f6720d-0564-473d-93d3-68905e69d4ca\") " pod="openshift-monitoring/thanos-querier-6955559f45-ftdzl" Jul 10 00:20:21.228808 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:21.228812 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/90f6720d-0564-473d-93d3-68905e69d4ca-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6955559f45-ftdzl\" (UID: \"90f6720d-0564-473d-93d3-68905e69d4ca\") " pod="openshift-monitoring/thanos-querier-6955559f45-ftdzl" Jul 10 00:20:21.228998 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:21.228873 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/90f6720d-0564-473d-93d3-68905e69d4ca-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6955559f45-ftdzl\" (UID: \"90f6720d-0564-473d-93d3-68905e69d4ca\") " pod="openshift-monitoring/thanos-querier-6955559f45-ftdzl" Jul 10 00:20:21.228998 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:21.228906 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/90f6720d-0564-473d-93d3-68905e69d4ca-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6955559f45-ftdzl\" (UID: \"90f6720d-0564-473d-93d3-68905e69d4ca\") " pod="openshift-monitoring/thanos-querier-6955559f45-ftdzl" Jul 10 00:20:21.228998 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:21.228931 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90f6720d-0564-473d-93d3-68905e69d4ca-metrics-client-ca\") pod \"thanos-querier-6955559f45-ftdzl\" (UID: \"90f6720d-0564-473d-93d3-68905e69d4ca\") " pod="openshift-monitoring/thanos-querier-6955559f45-ftdzl" Jul 10 00:20:21.228998 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:21.228956 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/90f6720d-0564-473d-93d3-68905e69d4ca-secret-thanos-querier-tls\") pod \"thanos-querier-6955559f45-ftdzl\" (UID: \"90f6720d-0564-473d-93d3-68905e69d4ca\") " pod="openshift-monitoring/thanos-querier-6955559f45-ftdzl" Jul 10 00:20:21.228998 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:21.228991 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/90f6720d-0564-473d-93d3-68905e69d4ca-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6955559f45-ftdzl\" (UID: \"90f6720d-0564-473d-93d3-68905e69d4ca\") " pod="openshift-monitoring/thanos-querier-6955559f45-ftdzl" Jul 10 00:20:21.229298 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:21.229138 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm2hc\" (UniqueName: \"kubernetes.io/projected/90f6720d-0564-473d-93d3-68905e69d4ca-kube-api-access-fm2hc\") pod \"thanos-querier-6955559f45-ftdzl\" (UID: \"90f6720d-0564-473d-93d3-68905e69d4ca\") " pod="openshift-monitoring/thanos-querier-6955559f45-ftdzl" Jul 10 00:20:21.229753 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:21.229724 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90f6720d-0564-473d-93d3-68905e69d4ca-metrics-client-ca\") pod \"thanos-querier-6955559f45-ftdzl\" (UID: \"90f6720d-0564-473d-93d3-68905e69d4ca\") " pod="openshift-monitoring/thanos-querier-6955559f45-ftdzl" Jul 10 00:20:21.231669 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:21.231643 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/90f6720d-0564-473d-93d3-68905e69d4ca-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6955559f45-ftdzl\" (UID: \"90f6720d-0564-473d-93d3-68905e69d4ca\") " pod="openshift-monitoring/thanos-querier-6955559f45-ftdzl" Jul 10 00:20:21.231788 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:21.231654 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/90f6720d-0564-473d-93d3-68905e69d4ca-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6955559f45-ftdzl\" (UID: \"90f6720d-0564-473d-93d3-68905e69d4ca\") " pod="openshift-monitoring/thanos-querier-6955559f45-ftdzl" Jul 10 00:20:21.231852 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:21.231828 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/90f6720d-0564-473d-93d3-68905e69d4ca-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6955559f45-ftdzl\" (UID: \"90f6720d-0564-473d-93d3-68905e69d4ca\") " pod="openshift-monitoring/thanos-querier-6955559f45-ftdzl" Jul 10 00:20:21.231910 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:21.231848 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/90f6720d-0564-473d-93d3-68905e69d4ca-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6955559f45-ftdzl\" (UID: \"90f6720d-0564-473d-93d3-68905e69d4ca\") " pod="openshift-monitoring/thanos-querier-6955559f45-ftdzl" Jul 10 00:20:21.232208 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:21.232189 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/90f6720d-0564-473d-93d3-68905e69d4ca-secret-thanos-querier-tls\") pod \"thanos-querier-6955559f45-ftdzl\" (UID: \"90f6720d-0564-473d-93d3-68905e69d4ca\") " pod="openshift-monitoring/thanos-querier-6955559f45-ftdzl" Jul 10 00:20:21.232279 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:21.232239 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/90f6720d-0564-473d-93d3-68905e69d4ca-secret-grpc-tls\") pod \"thanos-querier-6955559f45-ftdzl\" (UID: \"90f6720d-0564-473d-93d3-68905e69d4ca\") " pod="openshift-monitoring/thanos-querier-6955559f45-ftdzl" Jul 10 00:20:21.236930 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:21.236909 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm2hc\" (UniqueName: \"kubernetes.io/projected/90f6720d-0564-473d-93d3-68905e69d4ca-kube-api-access-fm2hc\") pod \"thanos-querier-6955559f45-ftdzl\" (UID: \"90f6720d-0564-473d-93d3-68905e69d4ca\") " pod="openshift-monitoring/thanos-querier-6955559f45-ftdzl" Jul 10 00:20:21.370355 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:21.370260 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6955559f45-ftdzl" Jul 10 00:20:21.499724 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:21.499638 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6955559f45-ftdzl"] Jul 10 00:20:21.501753 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:20:21.501727 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90f6720d_0564_473d_93d3_68905e69d4ca.slice/crio-116d1f32c5d30d8193889835a3844f342a5d4a0980ee24c017d6cd3726dab327 WatchSource:0}: Error finding container 116d1f32c5d30d8193889835a3844f342a5d4a0980ee24c017d6cd3726dab327: Status 404 returned error can't find the container with id 116d1f32c5d30d8193889835a3844f342a5d4a0980ee24c017d6cd3726dab327 Jul 10 00:20:21.655201 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:21.655097 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k76q6" event={"ID":"38181004-3a43-475c-81e5-3cc031197de2","Type":"ContainerStarted","Data":"bb26e19305e2d1bd4c8aee4d60a254871d20789c42dae9f05d02a54b3fc000ae"} Jul 10 00:20:21.655201 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:21.655163 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k76q6" event={"ID":"38181004-3a43-475c-81e5-3cc031197de2","Type":"ContainerStarted","Data":"3adeabee3a8b33f57e60f9e6309edd3a3e130424acad92f02844e33f8e854ba1"} Jul 10 00:20:21.656266 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:21.656244 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6955559f45-ftdzl" event={"ID":"90f6720d-0564-473d-93d3-68905e69d4ca","Type":"ContainerStarted","Data":"116d1f32c5d30d8193889835a3844f342a5d4a0980ee24c017d6cd3726dab327"} Jul 10 00:20:21.674020 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:21.673978 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-k76q6" podStartSLOduration=2.179406149 podStartE2EDuration="3.673963976s" totalImagesPullingTime="1.429367124s" totalInitContainerRuntime="65.190703ms" isStatefulPod=true podCreationTimestamp="2026-07-10 00:20:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:20:21.672466257 +0000 UTC m=+288.490771763" watchObservedRunningTime="2026-07-10 00:20:21.673963976 +0000 UTC m=+288.492269472" Jul 10 00:20:22.364234 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:22.364202 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-594957f4c7-s69q2"] Jul 10 00:20:22.367249 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:22.367232 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-594957f4c7-s69q2" Jul 10 00:20:22.370243 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:22.370202 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 10 00:20:22.370243 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:22.370244 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-pql8k\"" Jul 10 00:20:22.370446 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:22.370247 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 10 00:20:22.370446 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:22.370244 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 10 00:20:22.370555 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:22.370534 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-bgbuq8cgloejb\"" Jul 10 00:20:22.370795 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:22.370778 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 10 00:20:22.377237 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:22.377219 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-594957f4c7-s69q2"] Jul 10 00:20:22.438491 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:22.438459 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ff3ada19-4c2d-492d-9a57-b3b7ffa46306-secret-metrics-server-client-certs\") pod \"metrics-server-594957f4c7-s69q2\" (UID: \"ff3ada19-4c2d-492d-9a57-b3b7ffa46306\") " pod="openshift-monitoring/metrics-server-594957f4c7-s69q2" Jul 10 00:20:22.438491 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:22.438492 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ff3ada19-4c2d-492d-9a57-b3b7ffa46306-secret-metrics-server-tls\") pod \"metrics-server-594957f4c7-s69q2\" (UID: \"ff3ada19-4c2d-492d-9a57-b3b7ffa46306\") " pod="openshift-monitoring/metrics-server-594957f4c7-s69q2" Jul 10 00:20:22.438707 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:22.438538 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff3ada19-4c2d-492d-9a57-b3b7ffa46306-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-594957f4c7-s69q2\" (UID: \"ff3ada19-4c2d-492d-9a57-b3b7ffa46306\") " pod="openshift-monitoring/metrics-server-594957f4c7-s69q2" Jul 10 00:20:22.438707 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:22.438563 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ff3ada19-4c2d-492d-9a57-b3b7ffa46306-audit-log\") pod \"metrics-server-594957f4c7-s69q2\" (UID: \"ff3ada19-4c2d-492d-9a57-b3b7ffa46306\") " pod="openshift-monitoring/metrics-server-594957f4c7-s69q2" Jul 10 00:20:22.438707 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:22.438623 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7rnb\" (UniqueName: \"kubernetes.io/projected/ff3ada19-4c2d-492d-9a57-b3b7ffa46306-kube-api-access-b7rnb\") pod \"metrics-server-594957f4c7-s69q2\" (UID: \"ff3ada19-4c2d-492d-9a57-b3b7ffa46306\") " pod="openshift-monitoring/metrics-server-594957f4c7-s69q2" Jul 10 00:20:22.438707 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:22.438684 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ff3ada19-4c2d-492d-9a57-b3b7ffa46306-client-ca-bundle\") pod \"metrics-server-594957f4c7-s69q2\" (UID: \"ff3ada19-4c2d-492d-9a57-b3b7ffa46306\") " pod="openshift-monitoring/metrics-server-594957f4c7-s69q2" Jul 10 00:20:22.438867 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:22.438731 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ff3ada19-4c2d-492d-9a57-b3b7ffa46306-metrics-server-audit-profiles\") pod \"metrics-server-594957f4c7-s69q2\" (UID: \"ff3ada19-4c2d-492d-9a57-b3b7ffa46306\") " pod="openshift-monitoring/metrics-server-594957f4c7-s69q2" Jul 10 00:20:22.539726 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:22.539692 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff3ada19-4c2d-492d-9a57-b3b7ffa46306-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-594957f4c7-s69q2\" (UID: \"ff3ada19-4c2d-492d-9a57-b3b7ffa46306\") " pod="openshift-monitoring/metrics-server-594957f4c7-s69q2" Jul 10 00:20:22.539726 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:22.539732 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ff3ada19-4c2d-492d-9a57-b3b7ffa46306-audit-log\") pod \"metrics-server-594957f4c7-s69q2\" (UID: \"ff3ada19-4c2d-492d-9a57-b3b7ffa46306\") " pod="openshift-monitoring/metrics-server-594957f4c7-s69q2" Jul 10 00:20:22.539961 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:22.539759 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7rnb\" (UniqueName: \"kubernetes.io/projected/ff3ada19-4c2d-492d-9a57-b3b7ffa46306-kube-api-access-b7rnb\") pod \"metrics-server-594957f4c7-s69q2\" (UID: \"ff3ada19-4c2d-492d-9a57-b3b7ffa46306\") " pod="openshift-monitoring/metrics-server-594957f4c7-s69q2" Jul 10 00:20:22.539961 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:22.539804 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ff3ada19-4c2d-492d-9a57-b3b7ffa46306-client-ca-bundle\") pod \"metrics-server-594957f4c7-s69q2\" (UID: \"ff3ada19-4c2d-492d-9a57-b3b7ffa46306\") " pod="openshift-monitoring/metrics-server-594957f4c7-s69q2" Jul 10 00:20:22.539961 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:22.539838 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ff3ada19-4c2d-492d-9a57-b3b7ffa46306-metrics-server-audit-profiles\") pod \"metrics-server-594957f4c7-s69q2\" (UID: \"ff3ada19-4c2d-492d-9a57-b3b7ffa46306\") " pod="openshift-monitoring/metrics-server-594957f4c7-s69q2" Jul 10 00:20:22.539961 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:22.539915 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ff3ada19-4c2d-492d-9a57-b3b7ffa46306-secret-metrics-server-client-certs\") pod \"metrics-server-594957f4c7-s69q2\" (UID: \"ff3ada19-4c2d-492d-9a57-b3b7ffa46306\") " pod="openshift-monitoring/metrics-server-594957f4c7-s69q2" Jul 10 00:20:22.539961 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:22.539939 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ff3ada19-4c2d-492d-9a57-b3b7ffa46306-secret-metrics-server-tls\") pod \"metrics-server-594957f4c7-s69q2\" (UID: \"ff3ada19-4c2d-492d-9a57-b3b7ffa46306\") " pod="openshift-monitoring/metrics-server-594957f4c7-s69q2" Jul 10 00:20:22.540255 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:22.540206 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ff3ada19-4c2d-492d-9a57-b3b7ffa46306-audit-log\") pod \"metrics-server-594957f4c7-s69q2\" (UID: \"ff3ada19-4c2d-492d-9a57-b3b7ffa46306\") " pod="openshift-monitoring/metrics-server-594957f4c7-s69q2" Jul 10 00:20:22.540683 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:22.540661 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff3ada19-4c2d-492d-9a57-b3b7ffa46306-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-594957f4c7-s69q2\" (UID: \"ff3ada19-4c2d-492d-9a57-b3b7ffa46306\") " pod="openshift-monitoring/metrics-server-594957f4c7-s69q2" Jul 10 00:20:22.541148 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:22.541121 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ff3ada19-4c2d-492d-9a57-b3b7ffa46306-metrics-server-audit-profiles\") pod \"metrics-server-594957f4c7-s69q2\" (UID: \"ff3ada19-4c2d-492d-9a57-b3b7ffa46306\") " pod="openshift-monitoring/metrics-server-594957f4c7-s69q2" Jul 10 00:20:22.542732 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:22.542706 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ff3ada19-4c2d-492d-9a57-b3b7ffa46306-secret-metrics-server-tls\") pod \"metrics-server-594957f4c7-s69q2\" (UID: \"ff3ada19-4c2d-492d-9a57-b3b7ffa46306\") " pod="openshift-monitoring/metrics-server-594957f4c7-s69q2" Jul 10 00:20:22.542855 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:22.542839 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ff3ada19-4c2d-492d-9a57-b3b7ffa46306-client-ca-bundle\") pod \"metrics-server-594957f4c7-s69q2\" (UID: \"ff3ada19-4c2d-492d-9a57-b3b7ffa46306\") " pod="openshift-monitoring/metrics-server-594957f4c7-s69q2" Jul 10 00:20:22.542990 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:22.542963 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ff3ada19-4c2d-492d-9a57-b3b7ffa46306-secret-metrics-server-client-certs\") pod \"metrics-server-594957f4c7-s69q2\" (UID: \"ff3ada19-4c2d-492d-9a57-b3b7ffa46306\") " pod="openshift-monitoring/metrics-server-594957f4c7-s69q2" Jul 10 00:20:22.547938 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:22.547915 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7rnb\" (UniqueName: \"kubernetes.io/projected/ff3ada19-4c2d-492d-9a57-b3b7ffa46306-kube-api-access-b7rnb\") pod \"metrics-server-594957f4c7-s69q2\" (UID: \"ff3ada19-4c2d-492d-9a57-b3b7ffa46306\") " pod="openshift-monitoring/metrics-server-594957f4c7-s69q2" Jul 10 00:20:22.677661 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:22.677581 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-594957f4c7-s69q2" Jul 10 00:20:22.678780 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:22.678680 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-84d8c57647-m8t5j" Jul 10 00:20:22.678780 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:22.678781 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-84d8c57647-m8t5j" Jul 10 00:20:22.680258 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:22.680220 2562 patch_prober.go:28] interesting pod/console-84d8c57647-m8t5j container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.132.0.16:8443/health\": dial tcp 10.132.0.16:8443: connect: connection refused" start-of-body= Jul 10 00:20:22.680379 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:22.680275 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-84d8c57647-m8t5j" podUID="b54ecf35-9fb0-417d-be21-87a463f92bb4" containerName="console" probeResult="failure" output="Get \"https://10.132.0.16:8443/health\": dial tcp 10.132.0.16:8443: connect: connection refused" Jul 10 00:20:22.828932 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:22.828908 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-594957f4c7-s69q2"] Jul 10 00:20:22.831723 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:20:22.831693 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff3ada19_4c2d_492d_9a57_b3b7ffa46306.slice/crio-cd28b521e68d8947700b372ed35bd20e78cd124f166fe2d8527f2932dcdb6d6e WatchSource:0}: Error finding container cd28b521e68d8947700b372ed35bd20e78cd124f166fe2d8527f2932dcdb6d6e: Status 404 returned error can't find the container with id cd28b521e68d8947700b372ed35bd20e78cd124f166fe2d8527f2932dcdb6d6e Jul 10 00:20:23.549711 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:23.549668 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b56f3715-4283-4226-8188-d1b3869826db-metrics-tls\") pod \"dns-default-9mt8s\" (UID: \"b56f3715-4283-4226-8188-d1b3869826db\") " pod="openshift-dns/dns-default-9mt8s" Jul 10 00:20:23.549915 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:23.549765 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bbbf0b29-8280-450c-99e6-5965971c89da-registry-tls\") pod \"image-registry-5d58f56c87-wn9cc\" (UID: \"bbbf0b29-8280-450c-99e6-5965971c89da\") " pod="openshift-image-registry/image-registry-5d58f56c87-wn9cc" Jul 10 00:20:23.552635 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:23.552610 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b56f3715-4283-4226-8188-d1b3869826db-metrics-tls\") pod \"dns-default-9mt8s\" (UID: \"b56f3715-4283-4226-8188-d1b3869826db\") " pod="openshift-dns/dns-default-9mt8s" Jul 10 00:20:23.552757 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:23.552639 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bbbf0b29-8280-450c-99e6-5965971c89da-registry-tls\") pod \"image-registry-5d58f56c87-wn9cc\" (UID: \"bbbf0b29-8280-450c-99e6-5965971c89da\") " pod="openshift-image-registry/image-registry-5d58f56c87-wn9cc" Jul 10 00:20:23.652859 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:23.652821 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-s69wz\"" Jul 10 00:20:23.653064 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:23.652822 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-crl8x\"" Jul 10 00:20:23.660777 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:23.660753 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9mt8s" Jul 10 00:20:23.660894 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:23.660879 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d58f56c87-wn9cc" Jul 10 00:20:23.672287 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:23.672262 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-594957f4c7-s69q2" event={"ID":"ff3ada19-4c2d-492d-9a57-b3b7ffa46306","Type":"ContainerStarted","Data":"cd28b521e68d8947700b372ed35bd20e78cd124f166fe2d8527f2932dcdb6d6e"} Jul 10 00:20:24.169447 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:24.169411 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84d8c57647-m8t5j"] Jul 10 00:20:24.547993 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:24.547405 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9mt8s"] Jul 10 00:20:24.554152 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:20:24.554127 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb56f3715_4283_4226_8188_d1b3869826db.slice/crio-5b9fc4d3d7f5e00fe5653bf52f70fdcfdb4ddd6c0ba03a8e565c4f55d1a3b77a WatchSource:0}: Error finding container 5b9fc4d3d7f5e00fe5653bf52f70fdcfdb4ddd6c0ba03a8e565c4f55d1a3b77a: Status 404 returned error can't find the container with id 5b9fc4d3d7f5e00fe5653bf52f70fdcfdb4ddd6c0ba03a8e565c4f55d1a3b77a Jul 10 00:20:24.568432 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:24.568405 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d58f56c87-wn9cc"] Jul 10 00:20:24.571396 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:20:24.571374 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbbf0b29_8280_450c_99e6_5965971c89da.slice/crio-09f703348e1f220549e25f2bb680fded18d8717ca7e1b1ac5f8a6e6a1df80b14 WatchSource:0}: Error finding container 09f703348e1f220549e25f2bb680fded18d8717ca7e1b1ac5f8a6e6a1df80b14: Status 404 returned error can't find the container with id 09f703348e1f220549e25f2bb680fded18d8717ca7e1b1ac5f8a6e6a1df80b14 Jul 10 00:20:24.678433 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:24.678349 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d58f56c87-wn9cc" event={"ID":"bbbf0b29-8280-450c-99e6-5965971c89da","Type":"ContainerStarted","Data":"20c2afcc17ce9e0226fbf836eb0e9e069f79213a547fa7071b2cdd9b3605a0e6"} Jul 10 00:20:24.678433 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:24.678406 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d58f56c87-wn9cc" event={"ID":"bbbf0b29-8280-450c-99e6-5965971c89da","Type":"ContainerStarted","Data":"09f703348e1f220549e25f2bb680fded18d8717ca7e1b1ac5f8a6e6a1df80b14"} Jul 10 00:20:24.678433 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:24.678431 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5d58f56c87-wn9cc" Jul 10 00:20:24.679641 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:24.679576 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9mt8s" event={"ID":"b56f3715-4283-4226-8188-d1b3869826db","Type":"ContainerStarted","Data":"5b9fc4d3d7f5e00fe5653bf52f70fdcfdb4ddd6c0ba03a8e565c4f55d1a3b77a"} Jul 10 00:20:24.680967 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:24.680941 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-594957f4c7-s69q2" event={"ID":"ff3ada19-4c2d-492d-9a57-b3b7ffa46306","Type":"ContainerStarted","Data":"0d371e142e19c29758d65632458a6328b1a8770057454466f26e18c26032477e"} Jul 10 00:20:24.683449 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:24.683411 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6955559f45-ftdzl" event={"ID":"90f6720d-0564-473d-93d3-68905e69d4ca","Type":"ContainerStarted","Data":"457c7e986f0461fb3537131e7c018aa7f36689eb1ec2ef7fd5a12db6bd2e3c59"} Jul 10 00:20:24.683565 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:24.683469 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6955559f45-ftdzl" event={"ID":"90f6720d-0564-473d-93d3-68905e69d4ca","Type":"ContainerStarted","Data":"4e31d77947e31d50fd369d4783f419ac98ff2ed3eab946e01dd2f747e3b4a9f9"} Jul 10 00:20:24.683565 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:24.683497 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6955559f45-ftdzl" event={"ID":"90f6720d-0564-473d-93d3-68905e69d4ca","Type":"ContainerStarted","Data":"5f478c8c3d923603b0659ae38944c091819bf467fff8fce02f3f40806c417f89"} Jul 10 00:20:24.696781 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:24.696735 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5d58f56c87-wn9cc" podStartSLOduration=289.696717843 podStartE2EDuration="4m49.696717843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:15:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:20:24.695965932 +0000 UTC m=+291.514271428" watchObservedRunningTime="2026-07-10 00:20:24.696717843 +0000 UTC m=+291.515023341" Jul 10 00:20:24.713282 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:24.713236 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-594957f4c7-s69q2" podStartSLOduration=1.14128834 podStartE2EDuration="2.713223081s" totalImagesPullingTime="1.571934741s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:20:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:20:24.71178497 +0000 UTC m=+291.530090496" watchObservedRunningTime="2026-07-10 00:20:24.713223081 +0000 UTC m=+291.531528576" Jul 10 00:20:26.695770 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:26.695740 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6955559f45-ftdzl" event={"ID":"90f6720d-0564-473d-93d3-68905e69d4ca","Type":"ContainerStarted","Data":"34a02a0f7e9e91f69153a38eabee67d50aafbf0a49100f22f7372007511db75c"} Jul 10 00:20:26.696241 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:26.695805 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6955559f45-ftdzl" event={"ID":"90f6720d-0564-473d-93d3-68905e69d4ca","Type":"ContainerStarted","Data":"e5587b81712e90d0a32f6f2df53f1050479a584194aa5715edcb44097542c6d9"} Jul 10 00:20:26.696241 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:26.695844 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6955559f45-ftdzl" event={"ID":"90f6720d-0564-473d-93d3-68905e69d4ca","Type":"ContainerStarted","Data":"8f371f3935375ae798ebf737189a5bf377de3916d2d308f3de58c4ff0b8bd750"} Jul 10 00:20:26.696241 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:26.695957 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6955559f45-ftdzl" Jul 10 00:20:26.715527 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:26.715477 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6955559f45-ftdzl" podStartSLOduration=1.773766244 podStartE2EDuration="5.71546162s" totalImagesPullingTime="3.941695376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:20:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:20:26.713843136 +0000 UTC m=+293.532148632" watchObservedRunningTime="2026-07-10 00:20:26.71546162 +0000 UTC m=+293.533767113" Jul 10 00:20:27.208337 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:27.208312 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d8b644f74-v576j"] Jul 10 00:20:27.212631 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:27.212615 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d8b644f74-v576j" Jul 10 00:20:27.218827 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:27.218807 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 10 00:20:27.222176 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:27.222154 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d8b644f74-v576j"] Jul 10 00:20:27.281635 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:27.281601 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/583565d6-101c-4012-9586-230727b59066-service-ca\") pod \"console-5d8b644f74-v576j\" (UID: \"583565d6-101c-4012-9586-230727b59066\") " pod="openshift-console/console-5d8b644f74-v576j" Jul 10 00:20:27.281756 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:27.281741 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/583565d6-101c-4012-9586-230727b59066-console-config\") pod \"console-5d8b644f74-v576j\" (UID: \"583565d6-101c-4012-9586-230727b59066\") " pod="openshift-console/console-5d8b644f74-v576j" Jul 10 00:20:27.281800 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:27.281774 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/583565d6-101c-4012-9586-230727b59066-trusted-ca-bundle\") pod \"console-5d8b644f74-v576j\" (UID: \"583565d6-101c-4012-9586-230727b59066\") " pod="openshift-console/console-5d8b644f74-v576j" Jul 10 00:20:27.281836 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:27.281800 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/583565d6-101c-4012-9586-230727b59066-oauth-serving-cert\") pod \"console-5d8b644f74-v576j\" (UID: \"583565d6-101c-4012-9586-230727b59066\") " pod="openshift-console/console-5d8b644f74-v576j" Jul 10 00:20:27.281886 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:27.281871 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/583565d6-101c-4012-9586-230727b59066-console-oauth-config\") pod \"console-5d8b644f74-v576j\" (UID: \"583565d6-101c-4012-9586-230727b59066\") " pod="openshift-console/console-5d8b644f74-v576j" Jul 10 00:20:27.281937 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:27.281906 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/583565d6-101c-4012-9586-230727b59066-console-serving-cert\") pod \"console-5d8b644f74-v576j\" (UID: \"583565d6-101c-4012-9586-230727b59066\") " pod="openshift-console/console-5d8b644f74-v576j" Jul 10 00:20:27.281937 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:27.281924 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqvpz\" (UniqueName: \"kubernetes.io/projected/583565d6-101c-4012-9586-230727b59066-kube-api-access-fqvpz\") pod \"console-5d8b644f74-v576j\" (UID: \"583565d6-101c-4012-9586-230727b59066\") " pod="openshift-console/console-5d8b644f74-v576j" Jul 10 00:20:27.382943 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:27.382882 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/583565d6-101c-4012-9586-230727b59066-service-ca\") pod \"console-5d8b644f74-v576j\" (UID: \"583565d6-101c-4012-9586-230727b59066\") " pod="openshift-console/console-5d8b644f74-v576j" Jul 10 00:20:27.383099 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:27.382948 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/583565d6-101c-4012-9586-230727b59066-console-config\") pod \"console-5d8b644f74-v576j\" (UID: \"583565d6-101c-4012-9586-230727b59066\") " pod="openshift-console/console-5d8b644f74-v576j" Jul 10 00:20:27.383099 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:27.382967 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/583565d6-101c-4012-9586-230727b59066-trusted-ca-bundle\") pod \"console-5d8b644f74-v576j\" (UID: \"583565d6-101c-4012-9586-230727b59066\") " pod="openshift-console/console-5d8b644f74-v576j" Jul 10 00:20:27.383099 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:27.382983 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/583565d6-101c-4012-9586-230727b59066-oauth-serving-cert\") pod \"console-5d8b644f74-v576j\" (UID: \"583565d6-101c-4012-9586-230727b59066\") " pod="openshift-console/console-5d8b644f74-v576j" Jul 10 00:20:27.383099 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:27.383028 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/583565d6-101c-4012-9586-230727b59066-console-oauth-config\") pod \"console-5d8b644f74-v576j\" (UID: \"583565d6-101c-4012-9586-230727b59066\") " pod="openshift-console/console-5d8b644f74-v576j" Jul 10 00:20:27.383099 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:27.383054 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/583565d6-101c-4012-9586-230727b59066-console-serving-cert\") pod \"console-5d8b644f74-v576j\" (UID: \"583565d6-101c-4012-9586-230727b59066\") " pod="openshift-console/console-5d8b644f74-v576j" Jul 10 00:20:27.383099 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:27.383099 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqvpz\" (UniqueName: \"kubernetes.io/projected/583565d6-101c-4012-9586-230727b59066-kube-api-access-fqvpz\") pod \"console-5d8b644f74-v576j\" (UID: \"583565d6-101c-4012-9586-230727b59066\") " pod="openshift-console/console-5d8b644f74-v576j" Jul 10 00:20:27.383757 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:27.383735 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/583565d6-101c-4012-9586-230727b59066-console-config\") pod \"console-5d8b644f74-v576j\" (UID: \"583565d6-101c-4012-9586-230727b59066\") " pod="openshift-console/console-5d8b644f74-v576j" Jul 10 00:20:27.383858 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:27.383782 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/583565d6-101c-4012-9586-230727b59066-service-ca\") pod \"console-5d8b644f74-v576j\" (UID: \"583565d6-101c-4012-9586-230727b59066\") " pod="openshift-console/console-5d8b644f74-v576j" Jul 10 00:20:27.383924 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:27.383848 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/583565d6-101c-4012-9586-230727b59066-oauth-serving-cert\") pod \"console-5d8b644f74-v576j\" (UID: \"583565d6-101c-4012-9586-230727b59066\") " pod="openshift-console/console-5d8b644f74-v576j" Jul 10 00:20:27.384228 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:27.384209 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/583565d6-101c-4012-9586-230727b59066-trusted-ca-bundle\") pod \"console-5d8b644f74-v576j\" (UID: \"583565d6-101c-4012-9586-230727b59066\") " pod="openshift-console/console-5d8b644f74-v576j" Jul 10 00:20:27.385984 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:27.385965 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/583565d6-101c-4012-9586-230727b59066-console-serving-cert\") pod \"console-5d8b644f74-v576j\" (UID: \"583565d6-101c-4012-9586-230727b59066\") " pod="openshift-console/console-5d8b644f74-v576j" Jul 10 00:20:27.386057 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:27.385981 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/583565d6-101c-4012-9586-230727b59066-console-oauth-config\") pod \"console-5d8b644f74-v576j\" (UID: \"583565d6-101c-4012-9586-230727b59066\") " pod="openshift-console/console-5d8b644f74-v576j" Jul 10 00:20:27.389938 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:27.389916 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqvpz\" (UniqueName: \"kubernetes.io/projected/583565d6-101c-4012-9586-230727b59066-kube-api-access-fqvpz\") pod \"console-5d8b644f74-v576j\" (UID: \"583565d6-101c-4012-9586-230727b59066\") " pod="openshift-console/console-5d8b644f74-v576j" Jul 10 00:20:27.547323 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:27.547299 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d8b644f74-v576j" Jul 10 00:20:27.670357 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:27.670281 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d8b644f74-v576j"] Jul 10 00:20:27.672855 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:20:27.672831 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod583565d6_101c_4012_9586_230727b59066.slice/crio-5e61c69d24a623bb68c52c7828be64e8313b12d07f5557699ee23c034afba6f6 WatchSource:0}: Error finding container 5e61c69d24a623bb68c52c7828be64e8313b12d07f5557699ee23c034afba6f6: Status 404 returned error can't find the container with id 5e61c69d24a623bb68c52c7828be64e8313b12d07f5557699ee23c034afba6f6 Jul 10 00:20:27.700668 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:27.700638 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d8b644f74-v576j" event={"ID":"583565d6-101c-4012-9586-230727b59066","Type":"ContainerStarted","Data":"5e61c69d24a623bb68c52c7828be64e8313b12d07f5557699ee23c034afba6f6"} Jul 10 00:20:27.702214 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:27.702191 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9mt8s" event={"ID":"b56f3715-4283-4226-8188-d1b3869826db","Type":"ContainerStarted","Data":"7d691bfc97fa4f0e8a6c41abc6929c19b62b3093182e6555d8ea973f198197ad"} Jul 10 00:20:27.702293 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:27.702236 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9mt8s" event={"ID":"b56f3715-4283-4226-8188-d1b3869826db","Type":"ContainerStarted","Data":"12030753f70ae86d8ec13fad63472836db7a5641af31ca8b024a381d7958473b"} Jul 10 00:20:27.702475 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:27.702458 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9mt8s" Jul 10 00:20:27.716982 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:27.716939 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-9mt8s" podStartSLOduration=252.199375157 podStartE2EDuration="4m14.716924602s" totalImagesPullingTime="2.517549445s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:16:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:20:27.71632802 +0000 UTC m=+294.534633518" watchObservedRunningTime="2026-07-10 00:20:27.716924602 +0000 UTC m=+294.535230101" Jul 10 00:20:28.708459 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:28.708417 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d8b644f74-v576j" event={"ID":"583565d6-101c-4012-9586-230727b59066","Type":"ContainerStarted","Data":"2fd2ee2ecd2683ccda85de05b780cdd0b8e3a3f5edf6b3e61b046544e49a647a"} Jul 10 00:20:28.725936 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:28.725882 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5d8b644f74-v576j" podStartSLOduration=1.725868813 podStartE2EDuration="1.725868813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:20:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:20:28.72431153 +0000 UTC m=+295.542617024" watchObservedRunningTime="2026-07-10 00:20:28.725868813 +0000 UTC m=+295.544174307" Jul 10 00:20:31.730638 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:20:31.730601 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-x8v89" podUID="62933c8a-818d-4580-a6c9-6d26dbdfd50d" Jul 10 00:20:32.708122 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:32.708096 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6955559f45-ftdzl" Jul 10 00:20:33.603436 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:33.603409 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kjhg9_4651a90d-ef18-4974-9efd-a77fff3c9dfd/console-operator/1.log" Jul 10 00:20:33.603888 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:33.603616 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kjhg9_4651a90d-ef18-4974-9efd-a77fff3c9dfd/console-operator/1.log" Jul 10 00:20:33.607370 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:33.607349 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h5g9w_f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32/ovn-acl-logging/0.log" Jul 10 00:20:33.607627 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:33.607612 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h5g9w_f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32/ovn-acl-logging/0.log" Jul 10 00:20:33.613954 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:33.613937 2562 kubelet.go:1648] "Image garbage collection succeeded" Jul 10 00:20:37.547937 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:37.547906 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5d8b644f74-v576j" Jul 10 00:20:37.552624 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:37.547972 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d8b644f74-v576j" Jul 10 00:20:37.552624 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:37.549069 2562 patch_prober.go:28] interesting pod/console-5d8b644f74-v576j container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.132.0.20:8443/health\": dial tcp 10.132.0.20:8443: connect: connection refused" start-of-body= Jul 10 00:20:37.552624 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:37.549125 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-5d8b644f74-v576j" podUID="583565d6-101c-4012-9586-230727b59066" containerName="console" probeResult="failure" output="Get \"https://10.132.0.20:8443/health\": dial tcp 10.132.0.20:8443: connect: connection refused" Jul 10 00:20:37.710691 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:37.710670 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9mt8s" Jul 10 00:20:42.678636 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:42.678601 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-594957f4c7-s69q2" Jul 10 00:20:42.679040 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:42.678677 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-594957f4c7-s69q2" Jul 10 00:20:42.729117 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:42.729092 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x8v89" Jul 10 00:20:43.666964 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:43.666930 2562 patch_prober.go:28] interesting pod/image-registry-5d58f56c87-wn9cc container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 10 00:20:43.667144 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:43.666985 2562 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5d58f56c87-wn9cc" podUID="bbbf0b29-8280-450c-99e6-5965971c89da" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 00:20:45.691954 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:45.691925 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5d58f56c87-wn9cc" Jul 10 00:20:47.552453 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:47.552424 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5d8b644f74-v576j" Jul 10 00:20:47.556276 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:47.556254 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5d8b644f74-v576j" Jul 10 00:20:49.192162 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:49.192096 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-84d8c57647-m8t5j" podUID="b54ecf35-9fb0-417d-be21-87a463f92bb4" containerName="console" containerID="cri-o://893e800b1840b3c520c77ca5eb4658ef0eb75eed21b8a96ade3e7bece509f39e" gracePeriod=15 Jul 10 00:20:49.786161 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:49.786132 2562 generic.go:356] "Generic (PLEG): container finished" podID="b54ecf35-9fb0-417d-be21-87a463f92bb4" containerID="893e800b1840b3c520c77ca5eb4658ef0eb75eed21b8a96ade3e7bece509f39e" exitCode=0 Jul 10 00:20:49.786334 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:49.786183 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84d8c57647-m8t5j" event={"ID":"b54ecf35-9fb0-417d-be21-87a463f92bb4","Type":"ContainerDied","Data":"893e800b1840b3c520c77ca5eb4658ef0eb75eed21b8a96ade3e7bece509f39e"} Jul 10 00:20:49.957156 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:49.957134 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84d8c57647-m8t5j" Jul 10 00:20:50.074119 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:50.074093 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b54ecf35-9fb0-417d-be21-87a463f92bb4-service-ca\") pod \"b54ecf35-9fb0-417d-be21-87a463f92bb4\" (UID: \"b54ecf35-9fb0-417d-be21-87a463f92bb4\") " Jul 10 00:20:50.074249 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:50.074131 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b54ecf35-9fb0-417d-be21-87a463f92bb4-console-config\") pod \"b54ecf35-9fb0-417d-be21-87a463f92bb4\" (UID: \"b54ecf35-9fb0-417d-be21-87a463f92bb4\") " Jul 10 00:20:50.074249 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:50.074180 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8b885\" (UniqueName: \"kubernetes.io/projected/b54ecf35-9fb0-417d-be21-87a463f92bb4-kube-api-access-8b885\") pod \"b54ecf35-9fb0-417d-be21-87a463f92bb4\" (UID: \"b54ecf35-9fb0-417d-be21-87a463f92bb4\") " Jul 10 00:20:50.074249 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:50.074216 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b54ecf35-9fb0-417d-be21-87a463f92bb4-console-serving-cert\") pod \"b54ecf35-9fb0-417d-be21-87a463f92bb4\" (UID: \"b54ecf35-9fb0-417d-be21-87a463f92bb4\") " Jul 10 00:20:50.074446 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:50.074259 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b54ecf35-9fb0-417d-be21-87a463f92bb4-oauth-serving-cert\") pod \"b54ecf35-9fb0-417d-be21-87a463f92bb4\" (UID: \"b54ecf35-9fb0-417d-be21-87a463f92bb4\") " Jul 10 00:20:50.074446 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:50.074298 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b54ecf35-9fb0-417d-be21-87a463f92bb4-console-oauth-config\") pod \"b54ecf35-9fb0-417d-be21-87a463f92bb4\" (UID: \"b54ecf35-9fb0-417d-be21-87a463f92bb4\") " Jul 10 00:20:50.074598 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:50.074577 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b54ecf35-9fb0-417d-be21-87a463f92bb4-console-config" (OuterVolumeSpecName: "console-config") pod "b54ecf35-9fb0-417d-be21-87a463f92bb4" (UID: "b54ecf35-9fb0-417d-be21-87a463f92bb4"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:20:50.074598 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:50.074495 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b54ecf35-9fb0-417d-be21-87a463f92bb4-service-ca" (OuterVolumeSpecName: "service-ca") pod "b54ecf35-9fb0-417d-be21-87a463f92bb4" (UID: "b54ecf35-9fb0-417d-be21-87a463f92bb4"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:20:50.074723 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:50.074621 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b54ecf35-9fb0-417d-be21-87a463f92bb4-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b54ecf35-9fb0-417d-be21-87a463f92bb4" (UID: "b54ecf35-9fb0-417d-be21-87a463f92bb4"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:20:50.076471 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:50.076442 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b54ecf35-9fb0-417d-be21-87a463f92bb4-kube-api-access-8b885" (OuterVolumeSpecName: "kube-api-access-8b885") pod "b54ecf35-9fb0-417d-be21-87a463f92bb4" (UID: "b54ecf35-9fb0-417d-be21-87a463f92bb4"). InnerVolumeSpecName "kube-api-access-8b885". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:20:50.076582 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:50.076473 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b54ecf35-9fb0-417d-be21-87a463f92bb4-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b54ecf35-9fb0-417d-be21-87a463f92bb4" (UID: "b54ecf35-9fb0-417d-be21-87a463f92bb4"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:20:50.076582 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:50.076537 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b54ecf35-9fb0-417d-be21-87a463f92bb4-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b54ecf35-9fb0-417d-be21-87a463f92bb4" (UID: "b54ecf35-9fb0-417d-be21-87a463f92bb4"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:20:50.175261 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:50.175235 2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b54ecf35-9fb0-417d-be21-87a463f92bb4-oauth-serving-cert\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:20:50.175261 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:50.175258 2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b54ecf35-9fb0-417d-be21-87a463f92bb4-console-oauth-config\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:20:50.175428 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:50.175273 2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b54ecf35-9fb0-417d-be21-87a463f92bb4-service-ca\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:20:50.175428 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:50.175287 2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b54ecf35-9fb0-417d-be21-87a463f92bb4-console-config\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:20:50.175428 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:50.175299 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b885\" (UniqueName: \"kubernetes.io/projected/b54ecf35-9fb0-417d-be21-87a463f92bb4-kube-api-access-8b885\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:20:50.175428 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:50.175311 2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b54ecf35-9fb0-417d-be21-87a463f92bb4-console-serving-cert\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:20:50.790909 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:50.790886 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84d8c57647-m8t5j" Jul 10 00:20:50.791310 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:50.790881 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84d8c57647-m8t5j" event={"ID":"b54ecf35-9fb0-417d-be21-87a463f92bb4","Type":"ContainerDied","Data":"2269289079e358361de3dbe399d97e8863fce5d969fa38ffbd1cacd345125de1"} Jul 10 00:20:50.791310 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:50.791024 2562 scope.go:117] "RemoveContainer" containerID="893e800b1840b3c520c77ca5eb4658ef0eb75eed21b8a96ade3e7bece509f39e" Jul 10 00:20:50.809882 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:50.809857 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84d8c57647-m8t5j"] Jul 10 00:20:50.811248 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:50.811226 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-84d8c57647-m8t5j"] Jul 10 00:20:50.954717 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:50.954686 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5d58f56c87-wn9cc"] Jul 10 00:20:51.734538 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:20:51.734495 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b54ecf35-9fb0-417d-be21-87a463f92bb4" path="/var/lib/kubelet/pods/b54ecf35-9fb0-417d-be21-87a463f92bb4/volumes" Jul 10 00:21:02.683903 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:21:02.683868 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-594957f4c7-s69q2" Jul 10 00:21:02.687787 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:21:02.687763 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-594957f4c7-s69q2" Jul 10 00:21:12.649851 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:21:12.649822 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5d58f56c87-wn9cc_bbbf0b29-8280-450c-99e6-5965971c89da/registry/0.log" Jul 10 00:21:12.872317 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:21:12.872280 2562 generic.go:356] "Generic (PLEG): container finished" podID="7946a2a9-a12b-4302-9f33-93692ecf91ff" containerID="2676c20ba5d6ba901fa3022981a91e9ee87ee527cf242f114e6d05f9eacc4985" exitCode=0 Jul 10 00:21:12.872518 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:21:12.872358 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-dt7gx" event={"ID":"7946a2a9-a12b-4302-9f33-93692ecf91ff","Type":"ContainerDied","Data":"2676c20ba5d6ba901fa3022981a91e9ee87ee527cf242f114e6d05f9eacc4985"} Jul 10 00:21:12.872730 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:21:12.872713 2562 scope.go:117] "RemoveContainer" containerID="2676c20ba5d6ba901fa3022981a91e9ee87ee527cf242f114e6d05f9eacc4985" Jul 10 00:21:12.927882 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:21:12.927864 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gjk2r_03b7fa42-b493-40ce-a0d1-ce267379f3df/node-ca/0.log" Jul 10 00:21:13.526060 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:21:13.526033 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-76d8bdc488-dw9ns_3d7b4fce-5123-477b-800c-bec0da7fd372/router/0.log" Jul 10 00:21:13.877722 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:21:13.877689 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-dt7gx" event={"ID":"7946a2a9-a12b-4302-9f33-93692ecf91ff","Type":"ContainerStarted","Data":"a30fc78c51775bc770c85dfe3480cc01ac922b98029ff5d16f884a7ccb33b587"} Jul 10 00:21:13.924930 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:21:13.924906 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tjh96_55dcffca-f335-4b01-a6b8-4b2356d2fe57/serve-healthcheck-canary/0.log" Jul 10 00:21:15.974259 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:21:15.974211 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5d58f56c87-wn9cc" podUID="bbbf0b29-8280-450c-99e6-5965971c89da" containerName="registry" containerID="cri-o://20c2afcc17ce9e0226fbf836eb0e9e069f79213a547fa7071b2cdd9b3605a0e6" gracePeriod=30 Jul 10 00:21:17.218669 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:21:17.218644 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d58f56c87-wn9cc" Jul 10 00:21:17.300662 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:21:17.300631 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bbbf0b29-8280-450c-99e6-5965971c89da-image-registry-private-configuration\") pod \"bbbf0b29-8280-450c-99e6-5965971c89da\" (UID: \"bbbf0b29-8280-450c-99e6-5965971c89da\") " Jul 10 00:21:17.300800 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:21:17.300685 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bbbf0b29-8280-450c-99e6-5965971c89da-registry-tls\") pod \"bbbf0b29-8280-450c-99e6-5965971c89da\" (UID: \"bbbf0b29-8280-450c-99e6-5965971c89da\") " Jul 10 00:21:17.300852 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:21:17.300804 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bbbf0b29-8280-450c-99e6-5965971c89da-bound-sa-token\") pod \"bbbf0b29-8280-450c-99e6-5965971c89da\" (UID: \"bbbf0b29-8280-450c-99e6-5965971c89da\") " Jul 10 00:21:17.300912 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:21:17.300849 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bbbf0b29-8280-450c-99e6-5965971c89da-installation-pull-secrets\") pod \"bbbf0b29-8280-450c-99e6-5965971c89da\" (UID: \"bbbf0b29-8280-450c-99e6-5965971c89da\") " Jul 10 00:21:17.300912 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:21:17.300886 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bbbf0b29-8280-450c-99e6-5965971c89da-ca-trust-extracted\") pod \"bbbf0b29-8280-450c-99e6-5965971c89da\" (UID: \"bbbf0b29-8280-450c-99e6-5965971c89da\") " Jul 10 00:21:17.300912 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:21:17.300907 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tcdpg\" (UniqueName: \"kubernetes.io/projected/bbbf0b29-8280-450c-99e6-5965971c89da-kube-api-access-tcdpg\") pod \"bbbf0b29-8280-450c-99e6-5965971c89da\" (UID: \"bbbf0b29-8280-450c-99e6-5965971c89da\") " Jul 10 00:21:17.301030 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:21:17.300929 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bbbf0b29-8280-450c-99e6-5965971c89da-registry-certificates\") pod \"bbbf0b29-8280-450c-99e6-5965971c89da\" (UID: \"bbbf0b29-8280-450c-99e6-5965971c89da\") " Jul 10 00:21:17.301030 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:21:17.300979 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bbbf0b29-8280-450c-99e6-5965971c89da-trusted-ca\") pod \"bbbf0b29-8280-450c-99e6-5965971c89da\" (UID: \"bbbf0b29-8280-450c-99e6-5965971c89da\") " Jul 10 00:21:17.301405 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:21:17.301357 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbbf0b29-8280-450c-99e6-5965971c89da-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "bbbf0b29-8280-450c-99e6-5965971c89da" (UID: "bbbf0b29-8280-450c-99e6-5965971c89da"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:21:17.301632 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:21:17.301567 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbbf0b29-8280-450c-99e6-5965971c89da-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "bbbf0b29-8280-450c-99e6-5965971c89da" (UID: "bbbf0b29-8280-450c-99e6-5965971c89da"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:21:17.303644 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:21:17.303604 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbbf0b29-8280-450c-99e6-5965971c89da-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "bbbf0b29-8280-450c-99e6-5965971c89da" (UID: "bbbf0b29-8280-450c-99e6-5965971c89da"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:21:17.303807 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:21:17.303781 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbbf0b29-8280-450c-99e6-5965971c89da-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "bbbf0b29-8280-450c-99e6-5965971c89da" (UID: "bbbf0b29-8280-450c-99e6-5965971c89da"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:21:17.303886 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:21:17.303817 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbbf0b29-8280-450c-99e6-5965971c89da-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "bbbf0b29-8280-450c-99e6-5965971c89da" (UID: "bbbf0b29-8280-450c-99e6-5965971c89da"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:21:17.303928 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:21:17.303871 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbbf0b29-8280-450c-99e6-5965971c89da-kube-api-access-tcdpg" (OuterVolumeSpecName: "kube-api-access-tcdpg") pod "bbbf0b29-8280-450c-99e6-5965971c89da" (UID: "bbbf0b29-8280-450c-99e6-5965971c89da"). InnerVolumeSpecName "kube-api-access-tcdpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:21:17.303928 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:21:17.303906 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbbf0b29-8280-450c-99e6-5965971c89da-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "bbbf0b29-8280-450c-99e6-5965971c89da" (UID: "bbbf0b29-8280-450c-99e6-5965971c89da"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:21:17.310712 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:21:17.310683 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbbf0b29-8280-450c-99e6-5965971c89da-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "bbbf0b29-8280-450c-99e6-5965971c89da" (UID: "bbbf0b29-8280-450c-99e6-5965971c89da"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:21:17.402430 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:21:17.402393 2562 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bbbf0b29-8280-450c-99e6-5965971c89da-bound-sa-token\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:21:17.402430 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:21:17.402420 2562 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bbbf0b29-8280-450c-99e6-5965971c89da-installation-pull-secrets\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:21:17.402430 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:21:17.402432 2562 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bbbf0b29-8280-450c-99e6-5965971c89da-ca-trust-extracted\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:21:17.402430 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:21:17.402440 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcdpg\" (UniqueName: \"kubernetes.io/projected/bbbf0b29-8280-450c-99e6-5965971c89da-kube-api-access-tcdpg\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:21:17.402686 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:21:17.402449 2562 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bbbf0b29-8280-450c-99e6-5965971c89da-registry-certificates\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:21:17.402686 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:21:17.402458 2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bbbf0b29-8280-450c-99e6-5965971c89da-trusted-ca\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:21:17.402686 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:21:17.402466 2562 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bbbf0b29-8280-450c-99e6-5965971c89da-image-registry-private-configuration\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:21:17.402686 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:21:17.402475 2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bbbf0b29-8280-450c-99e6-5965971c89da-registry-tls\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:21:17.893699 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:21:17.893661 2562 generic.go:356] "Generic (PLEG): container finished" podID="bbbf0b29-8280-450c-99e6-5965971c89da" containerID="20c2afcc17ce9e0226fbf836eb0e9e069f79213a547fa7071b2cdd9b3605a0e6" exitCode=0 Jul 10 00:21:17.893861 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:21:17.893707 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d58f56c87-wn9cc" Jul 10 00:21:17.893861 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:21:17.893728 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d58f56c87-wn9cc" event={"ID":"bbbf0b29-8280-450c-99e6-5965971c89da","Type":"ContainerDied","Data":"20c2afcc17ce9e0226fbf836eb0e9e069f79213a547fa7071b2cdd9b3605a0e6"} Jul 10 00:21:17.893861 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:21:17.893778 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d58f56c87-wn9cc" event={"ID":"bbbf0b29-8280-450c-99e6-5965971c89da","Type":"ContainerDied","Data":"09f703348e1f220549e25f2bb680fded18d8717ca7e1b1ac5f8a6e6a1df80b14"} Jul 10 00:21:17.893861 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:21:17.893807 2562 scope.go:117] "RemoveContainer" containerID="20c2afcc17ce9e0226fbf836eb0e9e069f79213a547fa7071b2cdd9b3605a0e6" Jul 10 00:21:17.902110 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:21:17.902066 2562 scope.go:117] "RemoveContainer" containerID="20c2afcc17ce9e0226fbf836eb0e9e069f79213a547fa7071b2cdd9b3605a0e6" Jul 10 00:21:17.902365 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:21:17.902344 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20c2afcc17ce9e0226fbf836eb0e9e069f79213a547fa7071b2cdd9b3605a0e6\": container with ID starting with 20c2afcc17ce9e0226fbf836eb0e9e069f79213a547fa7071b2cdd9b3605a0e6 not found: ID does not exist" containerID="20c2afcc17ce9e0226fbf836eb0e9e069f79213a547fa7071b2cdd9b3605a0e6" Jul 10 00:21:17.902441 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:21:17.902376 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20c2afcc17ce9e0226fbf836eb0e9e069f79213a547fa7071b2cdd9b3605a0e6"} err="failed to get container status \"20c2afcc17ce9e0226fbf836eb0e9e069f79213a547fa7071b2cdd9b3605a0e6\": rpc error: code = NotFound desc = could not find container \"20c2afcc17ce9e0226fbf836eb0e9e069f79213a547fa7071b2cdd9b3605a0e6\": container with ID starting with 20c2afcc17ce9e0226fbf836eb0e9e069f79213a547fa7071b2cdd9b3605a0e6 not found: ID does not exist" Jul 10 00:21:17.908164 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:21:17.908142 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5d58f56c87-wn9cc"] Jul 10 00:21:17.911472 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:21:17.911452 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5d58f56c87-wn9cc"] Jul 10 00:21:19.734198 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:21:19.734163 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbbf0b29-8280-450c-99e6-5965971c89da" path="/var/lib/kubelet/pods/bbbf0b29-8280-450c-99e6-5965971c89da/volumes" Jul 10 00:21:47.552973 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:21:47.552928 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62933c8a-818d-4580-a6c9-6d26dbdfd50d-metrics-certs\") pod \"network-metrics-daemon-x8v89\" (UID: \"62933c8a-818d-4580-a6c9-6d26dbdfd50d\") " pod="openshift-multus/network-metrics-daemon-x8v89" Jul 10 00:21:47.555348 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:21:47.555320 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62933c8a-818d-4580-a6c9-6d26dbdfd50d-metrics-certs\") pod \"network-metrics-daemon-x8v89\" (UID: \"62933c8a-818d-4580-a6c9-6d26dbdfd50d\") " pod="openshift-multus/network-metrics-daemon-x8v89" Jul 10 00:21:47.832644 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:21:47.832560 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qn6qf\"" Jul 10 00:21:47.841171 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:21:47.841145 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x8v89" Jul 10 00:21:47.970824 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:21:47.970779 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-x8v89"] Jul 10 00:21:47.973826 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:21:47.973801 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62933c8a_818d_4580_a6c9_6d26dbdfd50d.slice/crio-d055e775ab2d04841c4fe8d2355be1a3d5385b2b29acab709a61699d62dce533 WatchSource:0}: Error finding container d055e775ab2d04841c4fe8d2355be1a3d5385b2b29acab709a61699d62dce533: Status 404 returned error can't find the container with id d055e775ab2d04841c4fe8d2355be1a3d5385b2b29acab709a61699d62dce533 Jul 10 00:21:47.975933 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:21:47.975915 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 00:21:48.004684 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:21:48.004651 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-x8v89" event={"ID":"62933c8a-818d-4580-a6c9-6d26dbdfd50d","Type":"ContainerStarted","Data":"d055e775ab2d04841c4fe8d2355be1a3d5385b2b29acab709a61699d62dce533"} Jul 10 00:21:50.013625 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:21:50.013581 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-x8v89" event={"ID":"62933c8a-818d-4580-a6c9-6d26dbdfd50d","Type":"ContainerStarted","Data":"febd88265e5a8d649f04fc5fd514418c63a6cf097741cdbc3d39c853dbad98e9"} Jul 10 00:21:50.014149 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:21:50.013646 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-x8v89" event={"ID":"62933c8a-818d-4580-a6c9-6d26dbdfd50d","Type":"ContainerStarted","Data":"3007e69ef4a0eca6b6e21d7924b7689a353a26548b21d77cb9a5a75322577e91"} Jul 10 00:21:50.029350 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:21:50.029294 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-x8v89" podStartSLOduration=374.718308508 podStartE2EDuration="6m16.029276706s" totalImagesPullingTime="1.310968198s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:15:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:21:50.028292584 +0000 UTC m=+376.846598079" watchObservedRunningTime="2026-07-10 00:21:50.029276706 +0000 UTC m=+376.847582200" Jul 10 00:21:51.446899 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:21:51.446861 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d8b644f74-v576j"] Jul 10 00:22:16.474327 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:16.474280 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5d8b644f74-v576j" podUID="583565d6-101c-4012-9586-230727b59066" containerName="console" containerID="cri-o://2fd2ee2ecd2683ccda85de05b780cdd0b8e3a3f5edf6b3e61b046544e49a647a" gracePeriod=15 Jul 10 00:22:16.708943 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:16.708904 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d8b644f74-v576j" Jul 10 00:22:16.778756 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:16.778660 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/583565d6-101c-4012-9586-230727b59066-console-oauth-config\") pod \"583565d6-101c-4012-9586-230727b59066\" (UID: \"583565d6-101c-4012-9586-230727b59066\") " Jul 10 00:22:16.778756 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:16.778736 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fqvpz\" (UniqueName: \"kubernetes.io/projected/583565d6-101c-4012-9586-230727b59066-kube-api-access-fqvpz\") pod \"583565d6-101c-4012-9586-230727b59066\" (UID: \"583565d6-101c-4012-9586-230727b59066\") " Jul 10 00:22:16.778978 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:16.778771 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/583565d6-101c-4012-9586-230727b59066-trusted-ca-bundle\") pod \"583565d6-101c-4012-9586-230727b59066\" (UID: \"583565d6-101c-4012-9586-230727b59066\") " Jul 10 00:22:16.778978 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:16.778797 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/583565d6-101c-4012-9586-230727b59066-service-ca\") pod \"583565d6-101c-4012-9586-230727b59066\" (UID: \"583565d6-101c-4012-9586-230727b59066\") " Jul 10 00:22:16.778978 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:16.778819 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/583565d6-101c-4012-9586-230727b59066-console-config\") pod \"583565d6-101c-4012-9586-230727b59066\" (UID: \"583565d6-101c-4012-9586-230727b59066\") " Jul 10 00:22:16.778978 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:16.778842 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/583565d6-101c-4012-9586-230727b59066-console-serving-cert\") pod \"583565d6-101c-4012-9586-230727b59066\" (UID: \"583565d6-101c-4012-9586-230727b59066\") " Jul 10 00:22:16.778978 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:16.778870 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/583565d6-101c-4012-9586-230727b59066-oauth-serving-cert\") pod \"583565d6-101c-4012-9586-230727b59066\" (UID: \"583565d6-101c-4012-9586-230727b59066\") " Jul 10 00:22:16.779282 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:16.779252 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/583565d6-101c-4012-9586-230727b59066-service-ca" (OuterVolumeSpecName: "service-ca") pod "583565d6-101c-4012-9586-230727b59066" (UID: "583565d6-101c-4012-9586-230727b59066"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:22:16.779282 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:16.779265 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/583565d6-101c-4012-9586-230727b59066-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "583565d6-101c-4012-9586-230727b59066" (UID: "583565d6-101c-4012-9586-230727b59066"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:22:16.779382 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:16.779293 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/583565d6-101c-4012-9586-230727b59066-console-config" (OuterVolumeSpecName: "console-config") pod "583565d6-101c-4012-9586-230727b59066" (UID: "583565d6-101c-4012-9586-230727b59066"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:22:16.779591 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:16.779573 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/583565d6-101c-4012-9586-230727b59066-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "583565d6-101c-4012-9586-230727b59066" (UID: "583565d6-101c-4012-9586-230727b59066"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:22:16.780893 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:16.780869 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/583565d6-101c-4012-9586-230727b59066-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "583565d6-101c-4012-9586-230727b59066" (UID: "583565d6-101c-4012-9586-230727b59066"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:22:16.781427 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:16.781402 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/583565d6-101c-4012-9586-230727b59066-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "583565d6-101c-4012-9586-230727b59066" (UID: "583565d6-101c-4012-9586-230727b59066"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:22:16.781517 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:16.781438 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/583565d6-101c-4012-9586-230727b59066-kube-api-access-fqvpz" (OuterVolumeSpecName: "kube-api-access-fqvpz") pod "583565d6-101c-4012-9586-230727b59066" (UID: "583565d6-101c-4012-9586-230727b59066"). InnerVolumeSpecName "kube-api-access-fqvpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:22:16.879744 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:16.879709 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqvpz\" (UniqueName: \"kubernetes.io/projected/583565d6-101c-4012-9586-230727b59066-kube-api-access-fqvpz\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:22:16.879744 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:16.879738 2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/583565d6-101c-4012-9586-230727b59066-trusted-ca-bundle\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:22:16.879744 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:16.879748 2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/583565d6-101c-4012-9586-230727b59066-service-ca\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:22:16.879744 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:16.879757 2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/583565d6-101c-4012-9586-230727b59066-console-config\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:22:16.879998 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:16.879766 2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/583565d6-101c-4012-9586-230727b59066-console-serving-cert\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:22:16.879998 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:16.879774 2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/583565d6-101c-4012-9586-230727b59066-oauth-serving-cert\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:22:16.879998 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:16.879782 2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/583565d6-101c-4012-9586-230727b59066-console-oauth-config\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:22:17.118335 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:17.118301 2562 generic.go:356] "Generic (PLEG): container finished" podID="583565d6-101c-4012-9586-230727b59066" containerID="2fd2ee2ecd2683ccda85de05b780cdd0b8e3a3f5edf6b3e61b046544e49a647a" exitCode=0 Jul 10 00:22:17.118517 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:17.118349 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d8b644f74-v576j" Jul 10 00:22:17.118517 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:17.118362 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d8b644f74-v576j" event={"ID":"583565d6-101c-4012-9586-230727b59066","Type":"ContainerDied","Data":"2fd2ee2ecd2683ccda85de05b780cdd0b8e3a3f5edf6b3e61b046544e49a647a"} Jul 10 00:22:17.118517 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:17.118407 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d8b644f74-v576j" event={"ID":"583565d6-101c-4012-9586-230727b59066","Type":"ContainerDied","Data":"5e61c69d24a623bb68c52c7828be64e8313b12d07f5557699ee23c034afba6f6"} Jul 10 00:22:17.118517 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:17.118427 2562 scope.go:117] "RemoveContainer" containerID="2fd2ee2ecd2683ccda85de05b780cdd0b8e3a3f5edf6b3e61b046544e49a647a" Jul 10 00:22:17.126849 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:17.126831 2562 scope.go:117] "RemoveContainer" containerID="2fd2ee2ecd2683ccda85de05b780cdd0b8e3a3f5edf6b3e61b046544e49a647a" Jul 10 00:22:17.127117 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:22:17.127071 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fd2ee2ecd2683ccda85de05b780cdd0b8e3a3f5edf6b3e61b046544e49a647a\": container with ID starting with 2fd2ee2ecd2683ccda85de05b780cdd0b8e3a3f5edf6b3e61b046544e49a647a not found: ID does not exist" containerID="2fd2ee2ecd2683ccda85de05b780cdd0b8e3a3f5edf6b3e61b046544e49a647a" Jul 10 00:22:17.127195 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:17.127130 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fd2ee2ecd2683ccda85de05b780cdd0b8e3a3f5edf6b3e61b046544e49a647a"} err="failed to get container status \"2fd2ee2ecd2683ccda85de05b780cdd0b8e3a3f5edf6b3e61b046544e49a647a\": rpc error: code = NotFound desc = could not find container \"2fd2ee2ecd2683ccda85de05b780cdd0b8e3a3f5edf6b3e61b046544e49a647a\": container with ID starting with 2fd2ee2ecd2683ccda85de05b780cdd0b8e3a3f5edf6b3e61b046544e49a647a not found: ID does not exist" Jul 10 00:22:17.137842 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:17.137816 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d8b644f74-v576j"] Jul 10 00:22:17.141376 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:17.141354 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5d8b644f74-v576j"] Jul 10 00:22:17.734069 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:17.734033 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="583565d6-101c-4012-9586-230727b59066" path="/var/lib/kubelet/pods/583565d6-101c-4012-9586-230727b59066/volumes" Jul 10 00:22:37.006160 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:37.006123 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754024gf4"] Jul 10 00:22:37.006592 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:37.006445 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b54ecf35-9fb0-417d-be21-87a463f92bb4" containerName="console" Jul 10 00:22:37.006592 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:37.006459 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b54ecf35-9fb0-417d-be21-87a463f92bb4" containerName="console" Jul 10 00:22:37.006592 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:37.006467 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="583565d6-101c-4012-9586-230727b59066" containerName="console" Jul 10 00:22:37.006592 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:37.006472 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="583565d6-101c-4012-9586-230727b59066" containerName="console" Jul 10 00:22:37.006592 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:37.006482 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbbf0b29-8280-450c-99e6-5965971c89da" containerName="registry" Jul 10 00:22:37.006592 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:37.006487 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbbf0b29-8280-450c-99e6-5965971c89da" containerName="registry" Jul 10 00:22:37.006592 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:37.006529 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="583565d6-101c-4012-9586-230727b59066" containerName="console" Jul 10 00:22:37.006592 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:37.006536 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbbf0b29-8280-450c-99e6-5965971c89da" containerName="registry" Jul 10 00:22:37.006592 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:37.006544 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b54ecf35-9fb0-417d-be21-87a463f92bb4" containerName="console" Jul 10 00:22:37.010012 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:37.009996 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754024gf4" Jul 10 00:22:37.012229 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:37.012201 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 10 00:22:37.012229 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:37.012222 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 10 00:22:37.012774 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:37.012757 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-vm8fl\"" Jul 10 00:22:37.018366 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:37.018344 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754024gf4"] Jul 10 00:22:37.144497 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:37.144448 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c6e9ffbe-49ec-43f2-8662-4ad2fdad2137-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754024gf4\" (UID: \"c6e9ffbe-49ec-43f2-8662-4ad2fdad2137\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754024gf4" Jul 10 00:22:37.144497 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:37.144492 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c6e9ffbe-49ec-43f2-8662-4ad2fdad2137-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754024gf4\" (UID: \"c6e9ffbe-49ec-43f2-8662-4ad2fdad2137\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754024gf4" Jul 10 00:22:37.144735 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:37.144561 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mp7n\" (UniqueName: \"kubernetes.io/projected/c6e9ffbe-49ec-43f2-8662-4ad2fdad2137-kube-api-access-4mp7n\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754024gf4\" (UID: \"c6e9ffbe-49ec-43f2-8662-4ad2fdad2137\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754024gf4" Jul 10 00:22:37.245447 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:37.245409 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c6e9ffbe-49ec-43f2-8662-4ad2fdad2137-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754024gf4\" (UID: \"c6e9ffbe-49ec-43f2-8662-4ad2fdad2137\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754024gf4" Jul 10 00:22:37.245447 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:37.245448 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c6e9ffbe-49ec-43f2-8662-4ad2fdad2137-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754024gf4\" (UID: \"c6e9ffbe-49ec-43f2-8662-4ad2fdad2137\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754024gf4" Jul 10 00:22:37.245666 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:37.245484 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mp7n\" (UniqueName: \"kubernetes.io/projected/c6e9ffbe-49ec-43f2-8662-4ad2fdad2137-kube-api-access-4mp7n\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754024gf4\" (UID: \"c6e9ffbe-49ec-43f2-8662-4ad2fdad2137\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754024gf4" Jul 10 00:22:37.245898 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:37.245873 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c6e9ffbe-49ec-43f2-8662-4ad2fdad2137-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754024gf4\" (UID: \"c6e9ffbe-49ec-43f2-8662-4ad2fdad2137\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754024gf4" Jul 10 00:22:37.245934 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:37.245879 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c6e9ffbe-49ec-43f2-8662-4ad2fdad2137-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754024gf4\" (UID: \"c6e9ffbe-49ec-43f2-8662-4ad2fdad2137\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754024gf4" Jul 10 00:22:37.253640 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:37.253609 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mp7n\" (UniqueName: \"kubernetes.io/projected/c6e9ffbe-49ec-43f2-8662-4ad2fdad2137-kube-api-access-4mp7n\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754024gf4\" (UID: \"c6e9ffbe-49ec-43f2-8662-4ad2fdad2137\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754024gf4" Jul 10 00:22:37.320182 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:37.320140 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754024gf4" Jul 10 00:22:37.450448 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:37.450427 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754024gf4"] Jul 10 00:22:37.452784 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:22:37.452750 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6e9ffbe_49ec_43f2_8662_4ad2fdad2137.slice/crio-d804d875471a52363e72cbd8d69f6948954b24d96f73b8eb294f7d3245649baf WatchSource:0}: Error finding container d804d875471a52363e72cbd8d69f6948954b24d96f73b8eb294f7d3245649baf: Status 404 returned error can't find the container with id d804d875471a52363e72cbd8d69f6948954b24d96f73b8eb294f7d3245649baf Jul 10 00:22:38.197136 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:38.197091 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754024gf4" event={"ID":"c6e9ffbe-49ec-43f2-8662-4ad2fdad2137","Type":"ContainerStarted","Data":"d804d875471a52363e72cbd8d69f6948954b24d96f73b8eb294f7d3245649baf"} Jul 10 00:22:43.220855 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:43.220818 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754024gf4" event={"ID":"c6e9ffbe-49ec-43f2-8662-4ad2fdad2137","Type":"ContainerStarted","Data":"a8ba4cb1a37e10c8f8c37535872ceab6451edd366e28aa2c7b66dc2448cb9858"} Jul 10 00:22:44.225500 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:44.225463 2562 generic.go:356] "Generic (PLEG): container finished" podID="c6e9ffbe-49ec-43f2-8662-4ad2fdad2137" containerID="a8ba4cb1a37e10c8f8c37535872ceab6451edd366e28aa2c7b66dc2448cb9858" exitCode=0 Jul 10 00:22:44.225975 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:44.225548 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754024gf4" event={"ID":"c6e9ffbe-49ec-43f2-8662-4ad2fdad2137","Type":"ContainerDied","Data":"a8ba4cb1a37e10c8f8c37535872ceab6451edd366e28aa2c7b66dc2448cb9858"} Jul 10 00:22:47.239218 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:47.239178 2562 generic.go:356] "Generic (PLEG): container finished" podID="c6e9ffbe-49ec-43f2-8662-4ad2fdad2137" containerID="e91bf3a273d037ce6d65c6a6cc814b944dda7a1c1700e766a4e58e86b82cdd66" exitCode=0 Jul 10 00:22:47.239666 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:47.239260 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754024gf4" event={"ID":"c6e9ffbe-49ec-43f2-8662-4ad2fdad2137","Type":"ContainerDied","Data":"e91bf3a273d037ce6d65c6a6cc814b944dda7a1c1700e766a4e58e86b82cdd66"} Jul 10 00:22:55.276829 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:55.276796 2562 generic.go:356] "Generic (PLEG): container finished" podID="c6e9ffbe-49ec-43f2-8662-4ad2fdad2137" containerID="87fc0d2d6e13a5e550c6c990c1ac12d92c28955f5309265dac121a26eb8854d3" exitCode=0 Jul 10 00:22:55.277249 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:55.276862 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754024gf4" event={"ID":"c6e9ffbe-49ec-43f2-8662-4ad2fdad2137","Type":"ContainerDied","Data":"87fc0d2d6e13a5e550c6c990c1ac12d92c28955f5309265dac121a26eb8854d3"} Jul 10 00:22:56.409212 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:56.409183 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754024gf4" Jul 10 00:22:56.506180 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:56.506143 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c6e9ffbe-49ec-43f2-8662-4ad2fdad2137-util\") pod \"c6e9ffbe-49ec-43f2-8662-4ad2fdad2137\" (UID: \"c6e9ffbe-49ec-43f2-8662-4ad2fdad2137\") " Jul 10 00:22:56.506381 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:56.506199 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c6e9ffbe-49ec-43f2-8662-4ad2fdad2137-bundle\") pod \"c6e9ffbe-49ec-43f2-8662-4ad2fdad2137\" (UID: \"c6e9ffbe-49ec-43f2-8662-4ad2fdad2137\") " Jul 10 00:22:56.506381 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:56.506278 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4mp7n\" (UniqueName: \"kubernetes.io/projected/c6e9ffbe-49ec-43f2-8662-4ad2fdad2137-kube-api-access-4mp7n\") pod \"c6e9ffbe-49ec-43f2-8662-4ad2fdad2137\" (UID: \"c6e9ffbe-49ec-43f2-8662-4ad2fdad2137\") " Jul 10 00:22:56.506908 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:56.506886 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6e9ffbe-49ec-43f2-8662-4ad2fdad2137-bundle" (OuterVolumeSpecName: "bundle") pod "c6e9ffbe-49ec-43f2-8662-4ad2fdad2137" (UID: "c6e9ffbe-49ec-43f2-8662-4ad2fdad2137"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:22:56.508439 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:56.508409 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6e9ffbe-49ec-43f2-8662-4ad2fdad2137-kube-api-access-4mp7n" (OuterVolumeSpecName: "kube-api-access-4mp7n") pod "c6e9ffbe-49ec-43f2-8662-4ad2fdad2137" (UID: "c6e9ffbe-49ec-43f2-8662-4ad2fdad2137"). InnerVolumeSpecName "kube-api-access-4mp7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:22:56.510210 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:56.510184 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6e9ffbe-49ec-43f2-8662-4ad2fdad2137-util" (OuterVolumeSpecName: "util") pod "c6e9ffbe-49ec-43f2-8662-4ad2fdad2137" (UID: "c6e9ffbe-49ec-43f2-8662-4ad2fdad2137"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:22:56.607536 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:56.607500 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mp7n\" (UniqueName: \"kubernetes.io/projected/c6e9ffbe-49ec-43f2-8662-4ad2fdad2137-kube-api-access-4mp7n\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:22:56.607536 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:56.607532 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c6e9ffbe-49ec-43f2-8662-4ad2fdad2137-util\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:22:56.607536 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:56.607541 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c6e9ffbe-49ec-43f2-8662-4ad2fdad2137-bundle\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:22:57.286581 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:57.286551 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754024gf4" Jul 10 00:22:57.286581 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:57.286554 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754024gf4" event={"ID":"c6e9ffbe-49ec-43f2-8662-4ad2fdad2137","Type":"ContainerDied","Data":"d804d875471a52363e72cbd8d69f6948954b24d96f73b8eb294f7d3245649baf"} Jul 10 00:22:57.286841 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:22:57.286599 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d804d875471a52363e72cbd8d69f6948954b24d96f73b8eb294f7d3245649baf" Jul 10 00:23:04.762517 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:04.762472 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pfgws"] Jul 10 00:23:04.763005 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:04.762869 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6e9ffbe-49ec-43f2-8662-4ad2fdad2137" containerName="pull" Jul 10 00:23:04.763005 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:04.762886 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6e9ffbe-49ec-43f2-8662-4ad2fdad2137" containerName="pull" Jul 10 00:23:04.763005 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:04.762902 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6e9ffbe-49ec-43f2-8662-4ad2fdad2137" containerName="extract" Jul 10 00:23:04.763005 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:04.762910 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6e9ffbe-49ec-43f2-8662-4ad2fdad2137" containerName="extract" Jul 10 00:23:04.763005 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:04.762932 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6e9ffbe-49ec-43f2-8662-4ad2fdad2137" containerName="util" Jul 10 00:23:04.763005 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:04.762939 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6e9ffbe-49ec-43f2-8662-4ad2fdad2137" containerName="util" Jul 10 00:23:04.763332 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:04.763031 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6e9ffbe-49ec-43f2-8662-4ad2fdad2137" containerName="extract" Jul 10 00:23:04.767286 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:04.767264 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pfgws" Jul 10 00:23:04.769306 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:04.769279 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 10 00:23:04.769410 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:04.769314 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 10 00:23:04.769410 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:04.769332 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-js75p\"" Jul 10 00:23:04.777532 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:04.777504 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pfgws"] Jul 10 00:23:04.870199 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:04.870163 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2mxv\" (UniqueName: \"kubernetes.io/projected/cf38ffd7-80d3-4107-b60b-9d6e31758ddf-kube-api-access-h2mxv\") pod \"cert-manager-operator-controller-manager-55c444bf86-pfgws\" (UID: \"cf38ffd7-80d3-4107-b60b-9d6e31758ddf\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pfgws" Jul 10 00:23:04.870199 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:04.870206 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cf38ffd7-80d3-4107-b60b-9d6e31758ddf-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-pfgws\" (UID: \"cf38ffd7-80d3-4107-b60b-9d6e31758ddf\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pfgws" Jul 10 00:23:04.970554 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:04.970508 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2mxv\" (UniqueName: \"kubernetes.io/projected/cf38ffd7-80d3-4107-b60b-9d6e31758ddf-kube-api-access-h2mxv\") pod \"cert-manager-operator-controller-manager-55c444bf86-pfgws\" (UID: \"cf38ffd7-80d3-4107-b60b-9d6e31758ddf\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pfgws" Jul 10 00:23:04.970554 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:04.970567 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cf38ffd7-80d3-4107-b60b-9d6e31758ddf-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-pfgws\" (UID: \"cf38ffd7-80d3-4107-b60b-9d6e31758ddf\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pfgws" Jul 10 00:23:04.970901 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:04.970886 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cf38ffd7-80d3-4107-b60b-9d6e31758ddf-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-pfgws\" (UID: \"cf38ffd7-80d3-4107-b60b-9d6e31758ddf\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pfgws" Jul 10 00:23:04.977906 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:04.977881 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2mxv\" (UniqueName: \"kubernetes.io/projected/cf38ffd7-80d3-4107-b60b-9d6e31758ddf-kube-api-access-h2mxv\") pod \"cert-manager-operator-controller-manager-55c444bf86-pfgws\" (UID: \"cf38ffd7-80d3-4107-b60b-9d6e31758ddf\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pfgws" Jul 10 00:23:05.076353 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:05.076300 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pfgws" Jul 10 00:23:05.207263 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:05.207233 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pfgws"] Jul 10 00:23:05.209910 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:23:05.209883 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf38ffd7_80d3_4107_b60b_9d6e31758ddf.slice/crio-91fd85b41a476df9bd046809a97f5c06d96265284687220d822f5da4f853f7ac WatchSource:0}: Error finding container 91fd85b41a476df9bd046809a97f5c06d96265284687220d822f5da4f853f7ac: Status 404 returned error can't find the container with id 91fd85b41a476df9bd046809a97f5c06d96265284687220d822f5da4f853f7ac Jul 10 00:23:05.318733 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:05.318698 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pfgws" event={"ID":"cf38ffd7-80d3-4107-b60b-9d6e31758ddf","Type":"ContainerStarted","Data":"91fd85b41a476df9bd046809a97f5c06d96265284687220d822f5da4f853f7ac"} Jul 10 00:23:07.329427 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:07.329392 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pfgws" event={"ID":"cf38ffd7-80d3-4107-b60b-9d6e31758ddf","Type":"ContainerStarted","Data":"ad2d72f71f5da939ed4258a5ee9472ea1ba801b8c2d36f1a93cfacb9b5a15003"} Jul 10 00:23:07.348796 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:07.348743 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pfgws" podStartSLOduration=1.848753441 podStartE2EDuration="3.348728651s" totalImagesPullingTime="1.49997521s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:23:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:23:07.346776553 +0000 UTC m=+454.165082058" watchObservedRunningTime="2026-07-10 00:23:07.348728651 +0000 UTC m=+454.167034144" Jul 10 00:23:08.654430 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:08.654396 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9pwgw"] Jul 10 00:23:08.656767 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:08.656751 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9pwgw" Jul 10 00:23:08.658664 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:08.658636 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 10 00:23:08.659380 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:08.659360 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 10 00:23:08.659435 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:08.659373 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-vm8fl\"" Jul 10 00:23:08.664641 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:08.664611 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9pwgw"] Jul 10 00:23:08.701463 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:08.701431 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r8kz\" (UniqueName: \"kubernetes.io/projected/b228a367-455b-435f-be62-518fd8c33d86-kube-api-access-9r8kz\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9pwgw\" (UID: \"b228a367-455b-435f-be62-518fd8c33d86\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9pwgw" Jul 10 00:23:08.701629 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:08.701473 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b228a367-455b-435f-be62-518fd8c33d86-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9pwgw\" (UID: \"b228a367-455b-435f-be62-518fd8c33d86\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9pwgw" Jul 10 00:23:08.701629 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:08.701498 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b228a367-455b-435f-be62-518fd8c33d86-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9pwgw\" (UID: \"b228a367-455b-435f-be62-518fd8c33d86\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9pwgw" Jul 10 00:23:08.802344 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:08.802308 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b228a367-455b-435f-be62-518fd8c33d86-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9pwgw\" (UID: \"b228a367-455b-435f-be62-518fd8c33d86\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9pwgw" Jul 10 00:23:08.802520 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:08.802353 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b228a367-455b-435f-be62-518fd8c33d86-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9pwgw\" (UID: \"b228a367-455b-435f-be62-518fd8c33d86\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9pwgw" Jul 10 00:23:08.802584 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:08.802533 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r8kz\" (UniqueName: \"kubernetes.io/projected/b228a367-455b-435f-be62-518fd8c33d86-kube-api-access-9r8kz\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9pwgw\" (UID: \"b228a367-455b-435f-be62-518fd8c33d86\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9pwgw" Jul 10 00:23:08.802751 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:08.802728 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b228a367-455b-435f-be62-518fd8c33d86-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9pwgw\" (UID: \"b228a367-455b-435f-be62-518fd8c33d86\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9pwgw" Jul 10 00:23:08.802828 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:08.802739 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b228a367-455b-435f-be62-518fd8c33d86-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9pwgw\" (UID: \"b228a367-455b-435f-be62-518fd8c33d86\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9pwgw" Jul 10 00:23:08.809611 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:08.809586 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r8kz\" (UniqueName: \"kubernetes.io/projected/b228a367-455b-435f-be62-518fd8c33d86-kube-api-access-9r8kz\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9pwgw\" (UID: \"b228a367-455b-435f-be62-518fd8c33d86\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9pwgw" Jul 10 00:23:08.966752 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:08.966666 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9pwgw" Jul 10 00:23:09.092511 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:09.092351 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9pwgw"] Jul 10 00:23:09.094632 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:23:09.094589 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb228a367_455b_435f_be62_518fd8c33d86.slice/crio-9ad3bdc737f5e5930b1afa3523e2c1f7d810ffeed7c01908f785e92f5beda73a WatchSource:0}: Error finding container 9ad3bdc737f5e5930b1afa3523e2c1f7d810ffeed7c01908f785e92f5beda73a: Status 404 returned error can't find the container with id 9ad3bdc737f5e5930b1afa3523e2c1f7d810ffeed7c01908f785e92f5beda73a Jul 10 00:23:09.338786 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:09.338753 2562 generic.go:356] "Generic (PLEG): container finished" podID="b228a367-455b-435f-be62-518fd8c33d86" containerID="5c6cecf4404e9699fc7487d8630354285757d084391db51a0344a7be7a41a141" exitCode=0 Jul 10 00:23:09.338973 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:09.338796 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9pwgw" event={"ID":"b228a367-455b-435f-be62-518fd8c33d86","Type":"ContainerDied","Data":"5c6cecf4404e9699fc7487d8630354285757d084391db51a0344a7be7a41a141"} Jul 10 00:23:09.338973 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:09.338848 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9pwgw" event={"ID":"b228a367-455b-435f-be62-518fd8c33d86","Type":"ContainerStarted","Data":"9ad3bdc737f5e5930b1afa3523e2c1f7d810ffeed7c01908f785e92f5beda73a"} Jul 10 00:23:09.566094 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:09.566037 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-9mf6t"] Jul 10 00:23:09.568373 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:09.568354 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-9mf6t" Jul 10 00:23:09.570322 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:09.570300 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 10 00:23:09.570417 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:09.570379 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 10 00:23:09.570601 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:09.570589 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-j4649\"" Jul 10 00:23:09.576366 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:09.576345 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-9mf6t"] Jul 10 00:23:09.611054 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:09.610982 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7d31884b-b2f5-4757-aa56-78d053fb1eb2-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-9mf6t\" (UID: \"7d31884b-b2f5-4757-aa56-78d053fb1eb2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-9mf6t" Jul 10 00:23:09.611054 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:09.611041 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvq4w\" (UniqueName: \"kubernetes.io/projected/7d31884b-b2f5-4757-aa56-78d053fb1eb2-kube-api-access-vvq4w\") pod \"cert-manager-webhook-6486ffd997-9mf6t\" (UID: \"7d31884b-b2f5-4757-aa56-78d053fb1eb2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-9mf6t" Jul 10 00:23:09.712099 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:09.712046 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvq4w\" (UniqueName: \"kubernetes.io/projected/7d31884b-b2f5-4757-aa56-78d053fb1eb2-kube-api-access-vvq4w\") pod \"cert-manager-webhook-6486ffd997-9mf6t\" (UID: \"7d31884b-b2f5-4757-aa56-78d053fb1eb2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-9mf6t" Jul 10 00:23:09.712456 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:09.712114 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7d31884b-b2f5-4757-aa56-78d053fb1eb2-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-9mf6t\" (UID: \"7d31884b-b2f5-4757-aa56-78d053fb1eb2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-9mf6t" Jul 10 00:23:09.719001 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:09.718977 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7d31884b-b2f5-4757-aa56-78d053fb1eb2-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-9mf6t\" (UID: \"7d31884b-b2f5-4757-aa56-78d053fb1eb2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-9mf6t" Jul 10 00:23:09.719131 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:09.719037 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvq4w\" (UniqueName: \"kubernetes.io/projected/7d31884b-b2f5-4757-aa56-78d053fb1eb2-kube-api-access-vvq4w\") pod \"cert-manager-webhook-6486ffd997-9mf6t\" (UID: \"7d31884b-b2f5-4757-aa56-78d053fb1eb2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-9mf6t" Jul 10 00:23:09.886255 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:09.886164 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-9mf6t" Jul 10 00:23:10.018051 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:10.017943 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-9mf6t"] Jul 10 00:23:10.020326 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:23:10.020293 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7d31884b_b2f5_4757_aa56_78d053fb1eb2.slice/crio-40a221b0bd936b1e2c958fc1d221fc361a5121d5ef1d0bce3f1a10e1a7bed092 WatchSource:0}: Error finding container 40a221b0bd936b1e2c958fc1d221fc361a5121d5ef1d0bce3f1a10e1a7bed092: Status 404 returned error can't find the container with id 40a221b0bd936b1e2c958fc1d221fc361a5121d5ef1d0bce3f1a10e1a7bed092 Jul 10 00:23:10.343647 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:10.343606 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-9mf6t" event={"ID":"7d31884b-b2f5-4757-aa56-78d053fb1eb2","Type":"ContainerStarted","Data":"40a221b0bd936b1e2c958fc1d221fc361a5121d5ef1d0bce3f1a10e1a7bed092"} Jul 10 00:23:13.359678 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:13.359645 2562 generic.go:356] "Generic (PLEG): container finished" podID="b228a367-455b-435f-be62-518fd8c33d86" containerID="fca446534e4811da0a68d4d5f0e2656e2bbc8142253c90eeeac024ceae4fbcf5" exitCode=0 Jul 10 00:23:13.360138 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:13.359731 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9pwgw" event={"ID":"b228a367-455b-435f-be62-518fd8c33d86","Type":"ContainerDied","Data":"fca446534e4811da0a68d4d5f0e2656e2bbc8142253c90eeeac024ceae4fbcf5"} Jul 10 00:23:13.361382 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:13.361364 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-9mf6t" event={"ID":"7d31884b-b2f5-4757-aa56-78d053fb1eb2","Type":"ContainerStarted","Data":"1edbf29db25c253233e69c44db82b4c742437f49bb538b516fe17598bc8d796f"} Jul 10 00:23:13.361504 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:13.361482 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-9mf6t" Jul 10 00:23:13.388133 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:13.388061 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-9mf6t" podStartSLOduration=1.503668363 podStartE2EDuration="4.388045618s" totalImagesPullingTime="2.884377255s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:23:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:23:13.387067542 +0000 UTC m=+460.205373040" watchObservedRunningTime="2026-07-10 00:23:13.388045618 +0000 UTC m=+460.206351111" Jul 10 00:23:14.367407 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:14.367372 2562 generic.go:356] "Generic (PLEG): container finished" podID="b228a367-455b-435f-be62-518fd8c33d86" containerID="faee56ec378ee42db079ad0fa15c4b16e63055fbfd053db6e350f7a1ac4dc547" exitCode=0 Jul 10 00:23:14.367880 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:14.367447 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9pwgw" event={"ID":"b228a367-455b-435f-be62-518fd8c33d86","Type":"ContainerDied","Data":"faee56ec378ee42db079ad0fa15c4b16e63055fbfd053db6e350f7a1ac4dc547"} Jul 10 00:23:15.495383 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:15.495359 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9pwgw" Jul 10 00:23:15.558409 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:15.558360 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9r8kz\" (UniqueName: \"kubernetes.io/projected/b228a367-455b-435f-be62-518fd8c33d86-kube-api-access-9r8kz\") pod \"b228a367-455b-435f-be62-518fd8c33d86\" (UID: \"b228a367-455b-435f-be62-518fd8c33d86\") " Jul 10 00:23:15.558409 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:15.558415 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b228a367-455b-435f-be62-518fd8c33d86-bundle\") pod \"b228a367-455b-435f-be62-518fd8c33d86\" (UID: \"b228a367-455b-435f-be62-518fd8c33d86\") " Jul 10 00:23:15.558629 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:15.558493 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b228a367-455b-435f-be62-518fd8c33d86-util\") pod \"b228a367-455b-435f-be62-518fd8c33d86\" (UID: \"b228a367-455b-435f-be62-518fd8c33d86\") " Jul 10 00:23:15.558804 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:15.558782 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b228a367-455b-435f-be62-518fd8c33d86-bundle" (OuterVolumeSpecName: "bundle") pod "b228a367-455b-435f-be62-518fd8c33d86" (UID: "b228a367-455b-435f-be62-518fd8c33d86"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:23:15.560485 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:15.560463 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b228a367-455b-435f-be62-518fd8c33d86-kube-api-access-9r8kz" (OuterVolumeSpecName: "kube-api-access-9r8kz") pod "b228a367-455b-435f-be62-518fd8c33d86" (UID: "b228a367-455b-435f-be62-518fd8c33d86"). InnerVolumeSpecName "kube-api-access-9r8kz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:23:15.562832 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:15.562794 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b228a367-455b-435f-be62-518fd8c33d86-util" (OuterVolumeSpecName: "util") pod "b228a367-455b-435f-be62-518fd8c33d86" (UID: "b228a367-455b-435f-be62-518fd8c33d86"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:23:15.659047 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:15.658949 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b228a367-455b-435f-be62-518fd8c33d86-util\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:23:15.659047 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:15.658985 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r8kz\" (UniqueName: \"kubernetes.io/projected/b228a367-455b-435f-be62-518fd8c33d86-kube-api-access-9r8kz\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:23:15.659047 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:15.658997 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b228a367-455b-435f-be62-518fd8c33d86-bundle\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:23:16.377649 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:16.377610 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9pwgw" event={"ID":"b228a367-455b-435f-be62-518fd8c33d86","Type":"ContainerDied","Data":"9ad3bdc737f5e5930b1afa3523e2c1f7d810ffeed7c01908f785e92f5beda73a"} Jul 10 00:23:16.377649 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:16.377644 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9pwgw" Jul 10 00:23:16.377856 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:16.377662 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ad3bdc737f5e5930b1afa3523e2c1f7d810ffeed7c01908f785e92f5beda73a" Jul 10 00:23:19.370660 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:19.370627 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-9mf6t" Jul 10 00:23:26.853951 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:26.853910 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fll5x"] Jul 10 00:23:26.854395 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:26.854225 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b228a367-455b-435f-be62-518fd8c33d86" containerName="pull" Jul 10 00:23:26.854395 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:26.854236 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b228a367-455b-435f-be62-518fd8c33d86" containerName="pull" Jul 10 00:23:26.854395 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:26.854245 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b228a367-455b-435f-be62-518fd8c33d86" containerName="extract" Jul 10 00:23:26.854395 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:26.854251 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b228a367-455b-435f-be62-518fd8c33d86" containerName="extract" Jul 10 00:23:26.854395 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:26.854271 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b228a367-455b-435f-be62-518fd8c33d86" containerName="util" Jul 10 00:23:26.854395 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:26.854276 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b228a367-455b-435f-be62-518fd8c33d86" containerName="util" Jul 10 00:23:26.854395 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:26.854331 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b228a367-455b-435f-be62-518fd8c33d86" containerName="extract" Jul 10 00:23:26.857882 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:26.857864 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fll5x" Jul 10 00:23:26.860206 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:26.860181 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 10 00:23:26.860371 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:26.860232 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 10 00:23:26.860790 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:26.860774 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-vn4k5\"" Jul 10 00:23:26.864967 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:26.864624 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fll5x"] Jul 10 00:23:26.958734 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:26.958701 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a854a498-ae6d-4153-b9ee-64927e960a9b-tmp\") pod \"openshift-lws-operator-fd8ccff4c-fll5x\" (UID: \"a854a498-ae6d-4153-b9ee-64927e960a9b\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fll5x" Jul 10 00:23:26.958910 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:26.958766 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhgn7\" (UniqueName: \"kubernetes.io/projected/a854a498-ae6d-4153-b9ee-64927e960a9b-kube-api-access-jhgn7\") pod \"openshift-lws-operator-fd8ccff4c-fll5x\" (UID: \"a854a498-ae6d-4153-b9ee-64927e960a9b\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fll5x" Jul 10 00:23:27.059790 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:27.059752 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a854a498-ae6d-4153-b9ee-64927e960a9b-tmp\") pod \"openshift-lws-operator-fd8ccff4c-fll5x\" (UID: \"a854a498-ae6d-4153-b9ee-64927e960a9b\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fll5x" Jul 10 00:23:27.059984 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:27.059804 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhgn7\" (UniqueName: \"kubernetes.io/projected/a854a498-ae6d-4153-b9ee-64927e960a9b-kube-api-access-jhgn7\") pod \"openshift-lws-operator-fd8ccff4c-fll5x\" (UID: \"a854a498-ae6d-4153-b9ee-64927e960a9b\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fll5x" Jul 10 00:23:27.060222 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:27.060198 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a854a498-ae6d-4153-b9ee-64927e960a9b-tmp\") pod \"openshift-lws-operator-fd8ccff4c-fll5x\" (UID: \"a854a498-ae6d-4153-b9ee-64927e960a9b\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fll5x" Jul 10 00:23:27.067000 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:27.066974 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhgn7\" (UniqueName: \"kubernetes.io/projected/a854a498-ae6d-4153-b9ee-64927e960a9b-kube-api-access-jhgn7\") pod \"openshift-lws-operator-fd8ccff4c-fll5x\" (UID: \"a854a498-ae6d-4153-b9ee-64927e960a9b\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fll5x" Jul 10 00:23:27.167362 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:27.167280 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fll5x" Jul 10 00:23:27.293244 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:27.293217 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fll5x"] Jul 10 00:23:27.295543 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:23:27.295519 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda854a498_ae6d_4153_b9ee_64927e960a9b.slice/crio-1036bc4b0ddbc716f91f356a3894fe3672ad88c7f0c649154c74dad665ed701a WatchSource:0}: Error finding container 1036bc4b0ddbc716f91f356a3894fe3672ad88c7f0c649154c74dad665ed701a: Status 404 returned error can't find the container with id 1036bc4b0ddbc716f91f356a3894fe3672ad88c7f0c649154c74dad665ed701a Jul 10 00:23:27.425194 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:27.425105 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fll5x" event={"ID":"a854a498-ae6d-4153-b9ee-64927e960a9b","Type":"ContainerStarted","Data":"1036bc4b0ddbc716f91f356a3894fe3672ad88c7f0c649154c74dad665ed701a"} Jul 10 00:23:29.101208 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:29.101174 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-pkkr7"] Jul 10 00:23:29.117232 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:29.117207 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-pkkr7"] Jul 10 00:23:29.117357 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:29.117291 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-pkkr7" Jul 10 00:23:29.119306 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:29.119286 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-ks4hq\"" Jul 10 00:23:29.276997 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:29.276957 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4d8411be-fdd0-4451-9611-559bc36b5329-bound-sa-token\") pod \"cert-manager-55b4d954d5-pkkr7\" (UID: \"4d8411be-fdd0-4451-9611-559bc36b5329\") " pod="cert-manager/cert-manager-55b4d954d5-pkkr7" Jul 10 00:23:29.277181 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:29.277013 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9q7v\" (UniqueName: \"kubernetes.io/projected/4d8411be-fdd0-4451-9611-559bc36b5329-kube-api-access-n9q7v\") pod \"cert-manager-55b4d954d5-pkkr7\" (UID: \"4d8411be-fdd0-4451-9611-559bc36b5329\") " pod="cert-manager/cert-manager-55b4d954d5-pkkr7" Jul 10 00:23:29.378401 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:29.378324 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9q7v\" (UniqueName: \"kubernetes.io/projected/4d8411be-fdd0-4451-9611-559bc36b5329-kube-api-access-n9q7v\") pod \"cert-manager-55b4d954d5-pkkr7\" (UID: \"4d8411be-fdd0-4451-9611-559bc36b5329\") " pod="cert-manager/cert-manager-55b4d954d5-pkkr7" Jul 10 00:23:29.378565 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:29.378421 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4d8411be-fdd0-4451-9611-559bc36b5329-bound-sa-token\") pod \"cert-manager-55b4d954d5-pkkr7\" (UID: \"4d8411be-fdd0-4451-9611-559bc36b5329\") " pod="cert-manager/cert-manager-55b4d954d5-pkkr7" Jul 10 00:23:29.388256 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:29.388230 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4d8411be-fdd0-4451-9611-559bc36b5329-bound-sa-token\") pod \"cert-manager-55b4d954d5-pkkr7\" (UID: \"4d8411be-fdd0-4451-9611-559bc36b5329\") " pod="cert-manager/cert-manager-55b4d954d5-pkkr7" Jul 10 00:23:29.388411 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:29.388390 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9q7v\" (UniqueName: \"kubernetes.io/projected/4d8411be-fdd0-4451-9611-559bc36b5329-kube-api-access-n9q7v\") pod \"cert-manager-55b4d954d5-pkkr7\" (UID: \"4d8411be-fdd0-4451-9611-559bc36b5329\") " pod="cert-manager/cert-manager-55b4d954d5-pkkr7" Jul 10 00:23:29.430159 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:29.430127 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-pkkr7" Jul 10 00:23:29.639826 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:29.639802 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-pkkr7"] Jul 10 00:23:29.641775 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:23:29.641750 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d8411be_fdd0_4451_9611_559bc36b5329.slice/crio-24bff159320373862213f8d481011d5ea8e0d6bda92214d947e15cf214fbe428 WatchSource:0}: Error finding container 24bff159320373862213f8d481011d5ea8e0d6bda92214d947e15cf214fbe428: Status 404 returned error can't find the container with id 24bff159320373862213f8d481011d5ea8e0d6bda92214d947e15cf214fbe428 Jul 10 00:23:30.438857 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:30.438807 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-pkkr7" event={"ID":"4d8411be-fdd0-4451-9611-559bc36b5329","Type":"ContainerStarted","Data":"606f235e1bdca77c69b1a920fa8a44f3d3574e134aee48ca9e00defcb642c164"} Jul 10 00:23:30.439324 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:30.438870 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-pkkr7" event={"ID":"4d8411be-fdd0-4451-9611-559bc36b5329","Type":"ContainerStarted","Data":"24bff159320373862213f8d481011d5ea8e0d6bda92214d947e15cf214fbe428"} Jul 10 00:23:30.440332 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:30.440308 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fll5x" event={"ID":"a854a498-ae6d-4153-b9ee-64927e960a9b","Type":"ContainerStarted","Data":"41c9bfbc6ffa7729e32fb187d7a42a1c93afaa739a27bb768f9fb6350f0954bb"} Jul 10 00:23:30.452119 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:30.452049 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-pkkr7" podStartSLOduration=1.452031486 podStartE2EDuration="1.452031486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:23:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:23:30.451751289 +0000 UTC m=+477.270056794" watchObservedRunningTime="2026-07-10 00:23:30.452031486 +0000 UTC m=+477.270336981" Jul 10 00:23:30.467041 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:30.466998 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fll5x" podStartSLOduration=2.245693027 podStartE2EDuration="4.466986921s" totalImagesPullingTime="2.221293894s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:23:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:23:30.465525195 +0000 UTC m=+477.283830690" watchObservedRunningTime="2026-07-10 00:23:30.466986921 +0000 UTC m=+477.285292415" Jul 10 00:23:34.722259 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:34.722222 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7xcvx"] Jul 10 00:23:34.725284 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:34.725261 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7xcvx" Jul 10 00:23:34.727283 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:34.727253 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 10 00:23:34.727406 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:34.727296 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 10 00:23:34.727406 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:34.727339 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-vm8fl\"" Jul 10 00:23:34.740390 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:34.740361 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7xcvx"] Jul 10 00:23:34.821211 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:34.821161 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v679z\" (UniqueName: \"kubernetes.io/projected/7fdec8f5-a4a5-4ae0-8f3f-f7bd72a7884b-kube-api-access-v679z\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7xcvx\" (UID: \"7fdec8f5-a4a5-4ae0-8f3f-f7bd72a7884b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7xcvx" Jul 10 00:23:34.821399 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:34.821298 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7fdec8f5-a4a5-4ae0-8f3f-f7bd72a7884b-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7xcvx\" (UID: \"7fdec8f5-a4a5-4ae0-8f3f-f7bd72a7884b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7xcvx" Jul 10 00:23:34.821399 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:34.821331 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7fdec8f5-a4a5-4ae0-8f3f-f7bd72a7884b-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7xcvx\" (UID: \"7fdec8f5-a4a5-4ae0-8f3f-f7bd72a7884b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7xcvx" Jul 10 00:23:34.922166 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:34.922135 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7fdec8f5-a4a5-4ae0-8f3f-f7bd72a7884b-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7xcvx\" (UID: \"7fdec8f5-a4a5-4ae0-8f3f-f7bd72a7884b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7xcvx" Jul 10 00:23:34.922166 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:34.922173 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7fdec8f5-a4a5-4ae0-8f3f-f7bd72a7884b-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7xcvx\" (UID: \"7fdec8f5-a4a5-4ae0-8f3f-f7bd72a7884b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7xcvx" Jul 10 00:23:34.922361 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:34.922214 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v679z\" (UniqueName: \"kubernetes.io/projected/7fdec8f5-a4a5-4ae0-8f3f-f7bd72a7884b-kube-api-access-v679z\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7xcvx\" (UID: \"7fdec8f5-a4a5-4ae0-8f3f-f7bd72a7884b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7xcvx" Jul 10 00:23:34.922512 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:34.922491 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7fdec8f5-a4a5-4ae0-8f3f-f7bd72a7884b-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7xcvx\" (UID: \"7fdec8f5-a4a5-4ae0-8f3f-f7bd72a7884b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7xcvx" Jul 10 00:23:34.922578 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:34.922559 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7fdec8f5-a4a5-4ae0-8f3f-f7bd72a7884b-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7xcvx\" (UID: \"7fdec8f5-a4a5-4ae0-8f3f-f7bd72a7884b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7xcvx" Jul 10 00:23:34.930049 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:34.930016 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v679z\" (UniqueName: \"kubernetes.io/projected/7fdec8f5-a4a5-4ae0-8f3f-f7bd72a7884b-kube-api-access-v679z\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7xcvx\" (UID: \"7fdec8f5-a4a5-4ae0-8f3f-f7bd72a7884b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7xcvx" Jul 10 00:23:35.035884 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:35.035788 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7xcvx" Jul 10 00:23:35.167066 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:35.167041 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7xcvx"] Jul 10 00:23:35.169150 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:23:35.169121 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fdec8f5_a4a5_4ae0_8f3f_f7bd72a7884b.slice/crio-80d6fd765d9d2dfbdff347091d590ac53eae1e5019ba3b15a491fa14eeb6288c WatchSource:0}: Error finding container 80d6fd765d9d2dfbdff347091d590ac53eae1e5019ba3b15a491fa14eeb6288c: Status 404 returned error can't find the container with id 80d6fd765d9d2dfbdff347091d590ac53eae1e5019ba3b15a491fa14eeb6288c Jul 10 00:23:35.463461 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:35.463424 2562 generic.go:356] "Generic (PLEG): container finished" podID="7fdec8f5-a4a5-4ae0-8f3f-f7bd72a7884b" containerID="37811a5d04e818fd045b62bc4f57a2bfa11671d9d47dbb49b7f0ca2ad09b1d85" exitCode=0 Jul 10 00:23:35.463683 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:35.463499 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7xcvx" event={"ID":"7fdec8f5-a4a5-4ae0-8f3f-f7bd72a7884b","Type":"ContainerDied","Data":"37811a5d04e818fd045b62bc4f57a2bfa11671d9d47dbb49b7f0ca2ad09b1d85"} Jul 10 00:23:35.463683 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:35.463546 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7xcvx" event={"ID":"7fdec8f5-a4a5-4ae0-8f3f-f7bd72a7884b","Type":"ContainerStarted","Data":"80d6fd765d9d2dfbdff347091d590ac53eae1e5019ba3b15a491fa14eeb6288c"} Jul 10 00:23:36.469220 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:36.469182 2562 generic.go:356] "Generic (PLEG): container finished" podID="7fdec8f5-a4a5-4ae0-8f3f-f7bd72a7884b" containerID="25701c696ff885df3c75ca7e7faf16d4fb50564749d1de2c22bea335b80b2c84" exitCode=0 Jul 10 00:23:36.469596 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:36.469243 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7xcvx" event={"ID":"7fdec8f5-a4a5-4ae0-8f3f-f7bd72a7884b","Type":"ContainerDied","Data":"25701c696ff885df3c75ca7e7faf16d4fb50564749d1de2c22bea335b80b2c84"} Jul 10 00:23:37.475606 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:37.475569 2562 generic.go:356] "Generic (PLEG): container finished" podID="7fdec8f5-a4a5-4ae0-8f3f-f7bd72a7884b" containerID="3edba7db3fd1ffd2bcd80cdd356ea2e110ea5bde53837c8392ac40d276a7f0d0" exitCode=0 Jul 10 00:23:37.475967 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:37.475650 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7xcvx" event={"ID":"7fdec8f5-a4a5-4ae0-8f3f-f7bd72a7884b","Type":"ContainerDied","Data":"3edba7db3fd1ffd2bcd80cdd356ea2e110ea5bde53837c8392ac40d276a7f0d0"} Jul 10 00:23:38.606060 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:38.606029 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7xcvx" Jul 10 00:23:38.752607 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:38.752518 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7fdec8f5-a4a5-4ae0-8f3f-f7bd72a7884b-bundle\") pod \"7fdec8f5-a4a5-4ae0-8f3f-f7bd72a7884b\" (UID: \"7fdec8f5-a4a5-4ae0-8f3f-f7bd72a7884b\") " Jul 10 00:23:38.752744 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:38.752619 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v679z\" (UniqueName: \"kubernetes.io/projected/7fdec8f5-a4a5-4ae0-8f3f-f7bd72a7884b-kube-api-access-v679z\") pod \"7fdec8f5-a4a5-4ae0-8f3f-f7bd72a7884b\" (UID: \"7fdec8f5-a4a5-4ae0-8f3f-f7bd72a7884b\") " Jul 10 00:23:38.752744 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:38.752643 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7fdec8f5-a4a5-4ae0-8f3f-f7bd72a7884b-util\") pod \"7fdec8f5-a4a5-4ae0-8f3f-f7bd72a7884b\" (UID: \"7fdec8f5-a4a5-4ae0-8f3f-f7bd72a7884b\") " Jul 10 00:23:38.753354 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:38.753305 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fdec8f5-a4a5-4ae0-8f3f-f7bd72a7884b-bundle" (OuterVolumeSpecName: "bundle") pod "7fdec8f5-a4a5-4ae0-8f3f-f7bd72a7884b" (UID: "7fdec8f5-a4a5-4ae0-8f3f-f7bd72a7884b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:23:38.754724 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:38.754698 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fdec8f5-a4a5-4ae0-8f3f-f7bd72a7884b-kube-api-access-v679z" (OuterVolumeSpecName: "kube-api-access-v679z") pod "7fdec8f5-a4a5-4ae0-8f3f-f7bd72a7884b" (UID: "7fdec8f5-a4a5-4ae0-8f3f-f7bd72a7884b"). InnerVolumeSpecName "kube-api-access-v679z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:23:38.758249 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:38.758214 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fdec8f5-a4a5-4ae0-8f3f-f7bd72a7884b-util" (OuterVolumeSpecName: "util") pod "7fdec8f5-a4a5-4ae0-8f3f-f7bd72a7884b" (UID: "7fdec8f5-a4a5-4ae0-8f3f-f7bd72a7884b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:23:38.854045 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:38.854011 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v679z\" (UniqueName: \"kubernetes.io/projected/7fdec8f5-a4a5-4ae0-8f3f-f7bd72a7884b-kube-api-access-v679z\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:23:38.854045 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:38.854039 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7fdec8f5-a4a5-4ae0-8f3f-f7bd72a7884b-util\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:23:38.854045 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:38.854050 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7fdec8f5-a4a5-4ae0-8f3f-f7bd72a7884b-bundle\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:23:39.486498 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:39.486472 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7xcvx" Jul 10 00:23:39.486665 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:39.486465 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7xcvx" event={"ID":"7fdec8f5-a4a5-4ae0-8f3f-f7bd72a7884b","Type":"ContainerDied","Data":"80d6fd765d9d2dfbdff347091d590ac53eae1e5019ba3b15a491fa14eeb6288c"} Jul 10 00:23:39.486665 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:39.486595 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80d6fd765d9d2dfbdff347091d590ac53eae1e5019ba3b15a491fa14eeb6288c" Jul 10 00:23:47.171905 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:47.171872 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-559fc4999-lqpff"] Jul 10 00:23:47.172378 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:47.172175 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fdec8f5-a4a5-4ae0-8f3f-f7bd72a7884b" containerName="pull" Jul 10 00:23:47.172378 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:47.172186 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fdec8f5-a4a5-4ae0-8f3f-f7bd72a7884b" containerName="pull" Jul 10 00:23:47.172378 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:47.172199 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fdec8f5-a4a5-4ae0-8f3f-f7bd72a7884b" containerName="util" Jul 10 00:23:47.172378 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:47.172204 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fdec8f5-a4a5-4ae0-8f3f-f7bd72a7884b" containerName="util" Jul 10 00:23:47.172378 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:47.172216 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fdec8f5-a4a5-4ae0-8f3f-f7bd72a7884b" containerName="extract" Jul 10 00:23:47.172378 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:47.172222 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fdec8f5-a4a5-4ae0-8f3f-f7bd72a7884b" containerName="extract" Jul 10 00:23:47.172378 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:47.172284 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fdec8f5-a4a5-4ae0-8f3f-f7bd72a7884b" containerName="extract" Jul 10 00:23:47.196011 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:47.195962 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-559fc4999-lqpff"] Jul 10 00:23:47.196213 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:47.196131 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-559fc4999-lqpff" Jul 10 00:23:47.198935 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:47.198903 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 10 00:23:47.199087 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:47.199038 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-hdnzg\"" Jul 10 00:23:47.199164 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:47.199114 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 10 00:23:47.199278 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:47.199209 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 10 00:23:47.199466 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:47.199444 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 10 00:23:47.325297 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:47.325255 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/22d431a4-4965-4605-b603-6773ffd80c0d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-559fc4999-lqpff\" (UID: \"22d431a4-4965-4605-b603-6773ffd80c0d\") " pod="opendatahub/opendatahub-operator-controller-manager-559fc4999-lqpff" Jul 10 00:23:47.325484 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:47.325316 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cn94d\" (UniqueName: \"kubernetes.io/projected/22d431a4-4965-4605-b603-6773ffd80c0d-kube-api-access-cn94d\") pod \"opendatahub-operator-controller-manager-559fc4999-lqpff\" (UID: \"22d431a4-4965-4605-b603-6773ffd80c0d\") " pod="opendatahub/opendatahub-operator-controller-manager-559fc4999-lqpff" Jul 10 00:23:47.325484 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:47.325405 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/22d431a4-4965-4605-b603-6773ffd80c0d-tmp\") pod \"opendatahub-operator-controller-manager-559fc4999-lqpff\" (UID: \"22d431a4-4965-4605-b603-6773ffd80c0d\") " pod="opendatahub/opendatahub-operator-controller-manager-559fc4999-lqpff" Jul 10 00:23:47.325484 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:47.325433 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/22d431a4-4965-4605-b603-6773ffd80c0d-webhook-cert\") pod \"opendatahub-operator-controller-manager-559fc4999-lqpff\" (UID: \"22d431a4-4965-4605-b603-6773ffd80c0d\") " pod="opendatahub/opendatahub-operator-controller-manager-559fc4999-lqpff" Jul 10 00:23:47.325484 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:47.325479 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/22d431a4-4965-4605-b603-6773ffd80c0d-manifests\") pod \"opendatahub-operator-controller-manager-559fc4999-lqpff\" (UID: \"22d431a4-4965-4605-b603-6773ffd80c0d\") " pod="opendatahub/opendatahub-operator-controller-manager-559fc4999-lqpff" Jul 10 00:23:47.426753 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:47.426665 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/22d431a4-4965-4605-b603-6773ffd80c0d-tmp\") pod \"opendatahub-operator-controller-manager-559fc4999-lqpff\" (UID: \"22d431a4-4965-4605-b603-6773ffd80c0d\") " pod="opendatahub/opendatahub-operator-controller-manager-559fc4999-lqpff" Jul 10 00:23:47.426753 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:47.426705 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/22d431a4-4965-4605-b603-6773ffd80c0d-webhook-cert\") pod \"opendatahub-operator-controller-manager-559fc4999-lqpff\" (UID: \"22d431a4-4965-4605-b603-6773ffd80c0d\") " pod="opendatahub/opendatahub-operator-controller-manager-559fc4999-lqpff" Jul 10 00:23:47.426948 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:47.426755 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/22d431a4-4965-4605-b603-6773ffd80c0d-manifests\") pod \"opendatahub-operator-controller-manager-559fc4999-lqpff\" (UID: \"22d431a4-4965-4605-b603-6773ffd80c0d\") " pod="opendatahub/opendatahub-operator-controller-manager-559fc4999-lqpff" Jul 10 00:23:47.426948 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:47.426798 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/22d431a4-4965-4605-b603-6773ffd80c0d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-559fc4999-lqpff\" (UID: \"22d431a4-4965-4605-b603-6773ffd80c0d\") " pod="opendatahub/opendatahub-operator-controller-manager-559fc4999-lqpff" Jul 10 00:23:47.426948 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:47.426851 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn94d\" (UniqueName: \"kubernetes.io/projected/22d431a4-4965-4605-b603-6773ffd80c0d-kube-api-access-cn94d\") pod \"opendatahub-operator-controller-manager-559fc4999-lqpff\" (UID: \"22d431a4-4965-4605-b603-6773ffd80c0d\") " pod="opendatahub/opendatahub-operator-controller-manager-559fc4999-lqpff" Jul 10 00:23:47.427281 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:47.427249 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/22d431a4-4965-4605-b603-6773ffd80c0d-manifests\") pod \"opendatahub-operator-controller-manager-559fc4999-lqpff\" (UID: \"22d431a4-4965-4605-b603-6773ffd80c0d\") " pod="opendatahub/opendatahub-operator-controller-manager-559fc4999-lqpff" Jul 10 00:23:47.429031 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:47.429005 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/22d431a4-4965-4605-b603-6773ffd80c0d-tmp\") pod \"opendatahub-operator-controller-manager-559fc4999-lqpff\" (UID: \"22d431a4-4965-4605-b603-6773ffd80c0d\") " pod="opendatahub/opendatahub-operator-controller-manager-559fc4999-lqpff" Jul 10 00:23:47.429318 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:47.429300 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/22d431a4-4965-4605-b603-6773ffd80c0d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-559fc4999-lqpff\" (UID: \"22d431a4-4965-4605-b603-6773ffd80c0d\") " pod="opendatahub/opendatahub-operator-controller-manager-559fc4999-lqpff" Jul 10 00:23:47.429367 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:47.429330 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/22d431a4-4965-4605-b603-6773ffd80c0d-webhook-cert\") pod \"opendatahub-operator-controller-manager-559fc4999-lqpff\" (UID: \"22d431a4-4965-4605-b603-6773ffd80c0d\") " pod="opendatahub/opendatahub-operator-controller-manager-559fc4999-lqpff" Jul 10 00:23:47.434229 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:47.434208 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn94d\" (UniqueName: \"kubernetes.io/projected/22d431a4-4965-4605-b603-6773ffd80c0d-kube-api-access-cn94d\") pod \"opendatahub-operator-controller-manager-559fc4999-lqpff\" (UID: \"22d431a4-4965-4605-b603-6773ffd80c0d\") " pod="opendatahub/opendatahub-operator-controller-manager-559fc4999-lqpff" Jul 10 00:23:47.506770 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:47.506731 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-559fc4999-lqpff" Jul 10 00:23:47.667330 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:47.667305 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-559fc4999-lqpff"] Jul 10 00:23:47.670046 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:23:47.670016 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22d431a4_4965_4605_b603_6773ffd80c0d.slice/crio-db051f60e803ff4c7a7af06ef0c7f6e331cd7f457da8496961b5180f6be83403 WatchSource:0}: Error finding container db051f60e803ff4c7a7af06ef0c7f6e331cd7f457da8496961b5180f6be83403: Status 404 returned error can't find the container with id db051f60e803ff4c7a7af06ef0c7f6e331cd7f457da8496961b5180f6be83403 Jul 10 00:23:48.530226 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:48.530159 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-559fc4999-lqpff" event={"ID":"22d431a4-4965-4605-b603-6773ffd80c0d","Type":"ContainerStarted","Data":"db051f60e803ff4c7a7af06ef0c7f6e331cd7f457da8496961b5180f6be83403"} Jul 10 00:23:51.549420 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:51.549384 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-559fc4999-lqpff" event={"ID":"22d431a4-4965-4605-b603-6773ffd80c0d","Type":"ContainerStarted","Data":"8d4022f87b752729b19b6b3f1c5f9ce46033eff629bf49fe70df88a6b7892ab8"} Jul 10 00:23:52.555165 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:52.555126 2562 generic.go:356] "Generic (PLEG): container finished" podID="22d431a4-4965-4605-b603-6773ffd80c0d" containerID="8d4022f87b752729b19b6b3f1c5f9ce46033eff629bf49fe70df88a6b7892ab8" exitCode=0 Jul 10 00:23:52.555554 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:52.555221 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-559fc4999-lqpff" event={"ID":"22d431a4-4965-4605-b603-6773ffd80c0d","Type":"ContainerDied","Data":"8d4022f87b752729b19b6b3f1c5f9ce46033eff629bf49fe70df88a6b7892ab8"} Jul 10 00:23:53.562050 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:53.562014 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-559fc4999-lqpff" event={"ID":"22d431a4-4965-4605-b603-6773ffd80c0d","Type":"ContainerStarted","Data":"30c0471f0662066b7ea0fbc797859ce19bce09d3a55349c5f95f4f90c272e264"} Jul 10 00:23:53.562461 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:53.562111 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-559fc4999-lqpff" Jul 10 00:23:53.581374 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:23:53.581325 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-559fc4999-lqpff" podStartSLOduration=2.119402013 podStartE2EDuration="6.581309876s" totalImagesPullingTime="3.308662066s" totalInitContainerRuntime="1.153245797s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:23:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:23:53.5793951 +0000 UTC m=+500.397700596" watchObservedRunningTime="2026-07-10 00:23:53.581309876 +0000 UTC m=+500.399615371" Jul 10 00:24:04.569999 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:04.569966 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-559fc4999-lqpff" Jul 10 00:24:06.710629 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:06.710591 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-b9f4c495f-zv55m"] Jul 10 00:24:06.716047 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:06.716025 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-b9f4c495f-zv55m" Jul 10 00:24:06.718463 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:06.718441 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 10 00:24:06.718803 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:06.718780 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 10 00:24:06.718909 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:06.718827 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-76m8r\"" Jul 10 00:24:06.723652 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:06.723402 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-b9f4c495f-zv55m"] Jul 10 00:24:06.788287 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:06.788249 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f066a2dc-a4c4-4f25-94db-472621e7fc95-tmp\") pod \"kube-auth-proxy-b9f4c495f-zv55m\" (UID: \"f066a2dc-a4c4-4f25-94db-472621e7fc95\") " pod="openshift-ingress/kube-auth-proxy-b9f4c495f-zv55m" Jul 10 00:24:06.788475 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:06.788328 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f066a2dc-a4c4-4f25-94db-472621e7fc95-tls-certs\") pod \"kube-auth-proxy-b9f4c495f-zv55m\" (UID: \"f066a2dc-a4c4-4f25-94db-472621e7fc95\") " pod="openshift-ingress/kube-auth-proxy-b9f4c495f-zv55m" Jul 10 00:24:06.788475 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:06.788350 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw4nz\" (UniqueName: \"kubernetes.io/projected/f066a2dc-a4c4-4f25-94db-472621e7fc95-kube-api-access-rw4nz\") pod \"kube-auth-proxy-b9f4c495f-zv55m\" (UID: \"f066a2dc-a4c4-4f25-94db-472621e7fc95\") " pod="openshift-ingress/kube-auth-proxy-b9f4c495f-zv55m" Jul 10 00:24:06.889592 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:06.889556 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f066a2dc-a4c4-4f25-94db-472621e7fc95-tls-certs\") pod \"kube-auth-proxy-b9f4c495f-zv55m\" (UID: \"f066a2dc-a4c4-4f25-94db-472621e7fc95\") " pod="openshift-ingress/kube-auth-proxy-b9f4c495f-zv55m" Jul 10 00:24:06.889592 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:06.889598 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw4nz\" (UniqueName: \"kubernetes.io/projected/f066a2dc-a4c4-4f25-94db-472621e7fc95-kube-api-access-rw4nz\") pod \"kube-auth-proxy-b9f4c495f-zv55m\" (UID: \"f066a2dc-a4c4-4f25-94db-472621e7fc95\") " pod="openshift-ingress/kube-auth-proxy-b9f4c495f-zv55m" Jul 10 00:24:06.889828 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:06.889714 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f066a2dc-a4c4-4f25-94db-472621e7fc95-tmp\") pod \"kube-auth-proxy-b9f4c495f-zv55m\" (UID: \"f066a2dc-a4c4-4f25-94db-472621e7fc95\") " pod="openshift-ingress/kube-auth-proxy-b9f4c495f-zv55m" Jul 10 00:24:06.891930 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:06.891901 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f066a2dc-a4c4-4f25-94db-472621e7fc95-tmp\") pod \"kube-auth-proxy-b9f4c495f-zv55m\" (UID: \"f066a2dc-a4c4-4f25-94db-472621e7fc95\") " pod="openshift-ingress/kube-auth-proxy-b9f4c495f-zv55m" Jul 10 00:24:06.892169 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:06.892148 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f066a2dc-a4c4-4f25-94db-472621e7fc95-tls-certs\") pod \"kube-auth-proxy-b9f4c495f-zv55m\" (UID: \"f066a2dc-a4c4-4f25-94db-472621e7fc95\") " pod="openshift-ingress/kube-auth-proxy-b9f4c495f-zv55m" Jul 10 00:24:06.904283 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:06.904258 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw4nz\" (UniqueName: \"kubernetes.io/projected/f066a2dc-a4c4-4f25-94db-472621e7fc95-kube-api-access-rw4nz\") pod \"kube-auth-proxy-b9f4c495f-zv55m\" (UID: \"f066a2dc-a4c4-4f25-94db-472621e7fc95\") " pod="openshift-ingress/kube-auth-proxy-b9f4c495f-zv55m" Jul 10 00:24:07.026842 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:07.026766 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-b9f4c495f-zv55m" Jul 10 00:24:07.175669 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:07.175633 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-b9f4c495f-zv55m"] Jul 10 00:24:07.177818 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:24:07.177790 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf066a2dc_a4c4_4f25_94db_472621e7fc95.slice/crio-e6b53fe77602eb68c9c518f46b532369d6e2f6f9830464636ea0c53108ff5429 WatchSource:0}: Error finding container e6b53fe77602eb68c9c518f46b532369d6e2f6f9830464636ea0c53108ff5429: Status 404 returned error can't find the container with id e6b53fe77602eb68c9c518f46b532369d6e2f6f9830464636ea0c53108ff5429 Jul 10 00:24:07.635269 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:07.635232 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-b9f4c495f-zv55m" event={"ID":"f066a2dc-a4c4-4f25-94db-472621e7fc95","Type":"ContainerStarted","Data":"e6b53fe77602eb68c9c518f46b532369d6e2f6f9830464636ea0c53108ff5429"} Jul 10 00:24:10.651476 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:10.651387 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-b9f4c495f-zv55m" event={"ID":"f066a2dc-a4c4-4f25-94db-472621e7fc95","Type":"ContainerStarted","Data":"ba2288ae4c952f04548da919940abf3ebe6e5f13eeeafea602fc7d14ef2f81bf"} Jul 10 00:24:10.667247 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:10.667193 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-b9f4c495f-zv55m" podStartSLOduration=1.444260173 podStartE2EDuration="4.667175699s" totalImagesPullingTime="3.222915526s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:24:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:24:10.666641369 +0000 UTC m=+517.484946864" watchObservedRunningTime="2026-07-10 00:24:10.667175699 +0000 UTC m=+517.485481195" Jul 10 00:24:14.604673 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:14.604643 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47czbk"] Jul 10 00:24:14.608347 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:14.608323 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47czbk" Jul 10 00:24:14.610458 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:14.610434 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 10 00:24:14.610586 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:14.610478 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-65mdb\"" Jul 10 00:24:14.621427 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:14.621404 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47czbk"] Jul 10 00:24:14.658819 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:14.658788 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0ba5141-fd5f-4e01-8865-037d6e6743d6-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf47czbk\" (UID: \"d0ba5141-fd5f-4e01-8865-037d6e6743d6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47czbk" Jul 10 00:24:14.658977 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:14.658828 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0ba5141-fd5f-4e01-8865-037d6e6743d6-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf47czbk\" (UID: \"d0ba5141-fd5f-4e01-8865-037d6e6743d6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47czbk" Jul 10 00:24:14.658977 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:14.658857 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0ba5141-fd5f-4e01-8865-037d6e6743d6-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf47czbk\" (UID: \"d0ba5141-fd5f-4e01-8865-037d6e6743d6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47czbk" Jul 10 00:24:14.658977 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:14.658900 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0ba5141-fd5f-4e01-8865-037d6e6743d6-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf47czbk\" (UID: \"d0ba5141-fd5f-4e01-8865-037d6e6743d6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47czbk" Jul 10 00:24:14.658977 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:14.658958 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtw6f\" (UniqueName: \"kubernetes.io/projected/d0ba5141-fd5f-4e01-8865-037d6e6743d6-kube-api-access-vtw6f\") pod \"data-science-gateway-data-science-gateway-class-6576859cf47czbk\" (UID: \"d0ba5141-fd5f-4e01-8865-037d6e6743d6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47czbk" Jul 10 00:24:14.659154 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:14.658984 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0ba5141-fd5f-4e01-8865-037d6e6743d6-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf47czbk\" (UID: \"d0ba5141-fd5f-4e01-8865-037d6e6743d6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47czbk" Jul 10 00:24:14.659154 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:14.659000 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0ba5141-fd5f-4e01-8865-037d6e6743d6-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf47czbk\" (UID: \"d0ba5141-fd5f-4e01-8865-037d6e6743d6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47czbk" Jul 10 00:24:14.659154 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:14.659022 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0ba5141-fd5f-4e01-8865-037d6e6743d6-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf47czbk\" (UID: \"d0ba5141-fd5f-4e01-8865-037d6e6743d6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47czbk" Jul 10 00:24:14.659154 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:14.659067 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0ba5141-fd5f-4e01-8865-037d6e6743d6-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf47czbk\" (UID: \"d0ba5141-fd5f-4e01-8865-037d6e6743d6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47czbk" Jul 10 00:24:14.760300 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:14.760262 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0ba5141-fd5f-4e01-8865-037d6e6743d6-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf47czbk\" (UID: \"d0ba5141-fd5f-4e01-8865-037d6e6743d6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47czbk" Jul 10 00:24:14.760300 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:14.760297 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0ba5141-fd5f-4e01-8865-037d6e6743d6-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf47czbk\" (UID: \"d0ba5141-fd5f-4e01-8865-037d6e6743d6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47czbk" Jul 10 00:24:14.760559 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:14.760317 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0ba5141-fd5f-4e01-8865-037d6e6743d6-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf47czbk\" (UID: \"d0ba5141-fd5f-4e01-8865-037d6e6743d6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47czbk" Jul 10 00:24:14.760559 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:14.760339 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0ba5141-fd5f-4e01-8865-037d6e6743d6-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf47czbk\" (UID: \"d0ba5141-fd5f-4e01-8865-037d6e6743d6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47czbk" Jul 10 00:24:14.760559 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:14.760449 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0ba5141-fd5f-4e01-8865-037d6e6743d6-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf47czbk\" (UID: \"d0ba5141-fd5f-4e01-8865-037d6e6743d6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47czbk" Jul 10 00:24:14.760559 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:14.760472 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0ba5141-fd5f-4e01-8865-037d6e6743d6-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf47czbk\" (UID: \"d0ba5141-fd5f-4e01-8865-037d6e6743d6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47czbk" Jul 10 00:24:14.760559 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:14.760498 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0ba5141-fd5f-4e01-8865-037d6e6743d6-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf47czbk\" (UID: \"d0ba5141-fd5f-4e01-8865-037d6e6743d6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47czbk" Jul 10 00:24:14.760559 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:14.760518 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0ba5141-fd5f-4e01-8865-037d6e6743d6-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf47czbk\" (UID: \"d0ba5141-fd5f-4e01-8865-037d6e6743d6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47czbk" Jul 10 00:24:14.760559 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:14.760556 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtw6f\" (UniqueName: \"kubernetes.io/projected/d0ba5141-fd5f-4e01-8865-037d6e6743d6-kube-api-access-vtw6f\") pod \"data-science-gateway-data-science-gateway-class-6576859cf47czbk\" (UID: \"d0ba5141-fd5f-4e01-8865-037d6e6743d6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47czbk" Jul 10 00:24:14.760962 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:14.760728 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0ba5141-fd5f-4e01-8865-037d6e6743d6-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf47czbk\" (UID: \"d0ba5141-fd5f-4e01-8865-037d6e6743d6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47czbk" Jul 10 00:24:14.760962 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:14.760812 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0ba5141-fd5f-4e01-8865-037d6e6743d6-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf47czbk\" (UID: \"d0ba5141-fd5f-4e01-8865-037d6e6743d6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47czbk" Jul 10 00:24:14.761179 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:14.761042 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0ba5141-fd5f-4e01-8865-037d6e6743d6-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf47czbk\" (UID: \"d0ba5141-fd5f-4e01-8865-037d6e6743d6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47czbk" Jul 10 00:24:14.761179 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:14.761169 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0ba5141-fd5f-4e01-8865-037d6e6743d6-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf47czbk\" (UID: \"d0ba5141-fd5f-4e01-8865-037d6e6743d6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47czbk" Jul 10 00:24:14.761332 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:14.761309 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0ba5141-fd5f-4e01-8865-037d6e6743d6-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf47czbk\" (UID: \"d0ba5141-fd5f-4e01-8865-037d6e6743d6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47czbk" Jul 10 00:24:14.762948 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:14.762926 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0ba5141-fd5f-4e01-8865-037d6e6743d6-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf47czbk\" (UID: \"d0ba5141-fd5f-4e01-8865-037d6e6743d6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47czbk" Jul 10 00:24:14.763213 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:14.763193 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0ba5141-fd5f-4e01-8865-037d6e6743d6-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf47czbk\" (UID: \"d0ba5141-fd5f-4e01-8865-037d6e6743d6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47czbk" Jul 10 00:24:14.775716 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:14.775695 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0ba5141-fd5f-4e01-8865-037d6e6743d6-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf47czbk\" (UID: \"d0ba5141-fd5f-4e01-8865-037d6e6743d6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47czbk" Jul 10 00:24:14.775864 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:14.775842 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtw6f\" (UniqueName: \"kubernetes.io/projected/d0ba5141-fd5f-4e01-8865-037d6e6743d6-kube-api-access-vtw6f\") pod \"data-science-gateway-data-science-gateway-class-6576859cf47czbk\" (UID: \"d0ba5141-fd5f-4e01-8865-037d6e6743d6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47czbk" Jul 10 00:24:14.919113 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:14.918987 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47czbk" Jul 10 00:24:15.094020 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:15.093992 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47czbk"] Jul 10 00:24:15.095818 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:24:15.095785 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0ba5141_fd5f_4e01_8865_037d6e6743d6.slice/crio-ce69a6a88387318d0f30cbe1ce06a6fa95ff5ca35d5ff7d168765acba865b782 WatchSource:0}: Error finding container ce69a6a88387318d0f30cbe1ce06a6fa95ff5ca35d5ff7d168765acba865b782: Status 404 returned error can't find the container with id ce69a6a88387318d0f30cbe1ce06a6fa95ff5ca35d5ff7d168765acba865b782 Jul 10 00:24:15.676244 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:15.676207 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47czbk" event={"ID":"d0ba5141-fd5f-4e01-8865-037d6e6743d6","Type":"ContainerStarted","Data":"ce69a6a88387318d0f30cbe1ce06a6fa95ff5ca35d5ff7d168765acba865b782"} Jul 10 00:24:16.491922 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:16.491883 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7b4d8999df-x6scl"] Jul 10 00:24:16.495742 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:16.495711 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7b4d8999df-x6scl" Jul 10 00:24:16.499717 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:16.499687 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 10 00:24:16.499875 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:16.499687 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 10 00:24:16.499875 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:16.499695 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 10 00:24:16.500108 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:16.500070 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-wgrz8\"" Jul 10 00:24:16.511466 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:16.511438 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7b4d8999df-x6scl"] Jul 10 00:24:16.579355 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:16.579317 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0d056fdd-af45-4a37-bacb-9daa0b022eeb-metrics-cert\") pod \"lws-controller-manager-7b4d8999df-x6scl\" (UID: \"0d056fdd-af45-4a37-bacb-9daa0b022eeb\") " pod="openshift-lws-operator/lws-controller-manager-7b4d8999df-x6scl" Jul 10 00:24:16.579569 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:16.579397 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0d056fdd-af45-4a37-bacb-9daa0b022eeb-cert\") pod \"lws-controller-manager-7b4d8999df-x6scl\" (UID: \"0d056fdd-af45-4a37-bacb-9daa0b022eeb\") " pod="openshift-lws-operator/lws-controller-manager-7b4d8999df-x6scl" Jul 10 00:24:16.579569 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:16.579432 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0d056fdd-af45-4a37-bacb-9daa0b022eeb-manager-config\") pod \"lws-controller-manager-7b4d8999df-x6scl\" (UID: \"0d056fdd-af45-4a37-bacb-9daa0b022eeb\") " pod="openshift-lws-operator/lws-controller-manager-7b4d8999df-x6scl" Jul 10 00:24:16.579693 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:16.579581 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snhzc\" (UniqueName: \"kubernetes.io/projected/0d056fdd-af45-4a37-bacb-9daa0b022eeb-kube-api-access-snhzc\") pod \"lws-controller-manager-7b4d8999df-x6scl\" (UID: \"0d056fdd-af45-4a37-bacb-9daa0b022eeb\") " pod="openshift-lws-operator/lws-controller-manager-7b4d8999df-x6scl" Jul 10 00:24:16.680320 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:16.680287 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0d056fdd-af45-4a37-bacb-9daa0b022eeb-cert\") pod \"lws-controller-manager-7b4d8999df-x6scl\" (UID: \"0d056fdd-af45-4a37-bacb-9daa0b022eeb\") " pod="openshift-lws-operator/lws-controller-manager-7b4d8999df-x6scl" Jul 10 00:24:16.680320 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:16.680332 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0d056fdd-af45-4a37-bacb-9daa0b022eeb-manager-config\") pod \"lws-controller-manager-7b4d8999df-x6scl\" (UID: \"0d056fdd-af45-4a37-bacb-9daa0b022eeb\") " pod="openshift-lws-operator/lws-controller-manager-7b4d8999df-x6scl" Jul 10 00:24:16.680877 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:16.680381 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snhzc\" (UniqueName: \"kubernetes.io/projected/0d056fdd-af45-4a37-bacb-9daa0b022eeb-kube-api-access-snhzc\") pod \"lws-controller-manager-7b4d8999df-x6scl\" (UID: \"0d056fdd-af45-4a37-bacb-9daa0b022eeb\") " pod="openshift-lws-operator/lws-controller-manager-7b4d8999df-x6scl" Jul 10 00:24:16.680877 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:16.680489 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0d056fdd-af45-4a37-bacb-9daa0b022eeb-metrics-cert\") pod \"lws-controller-manager-7b4d8999df-x6scl\" (UID: \"0d056fdd-af45-4a37-bacb-9daa0b022eeb\") " pod="openshift-lws-operator/lws-controller-manager-7b4d8999df-x6scl" Jul 10 00:24:16.681327 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:16.681302 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0d056fdd-af45-4a37-bacb-9daa0b022eeb-manager-config\") pod \"lws-controller-manager-7b4d8999df-x6scl\" (UID: \"0d056fdd-af45-4a37-bacb-9daa0b022eeb\") " pod="openshift-lws-operator/lws-controller-manager-7b4d8999df-x6scl" Jul 10 00:24:16.683169 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:16.683138 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0d056fdd-af45-4a37-bacb-9daa0b022eeb-cert\") pod \"lws-controller-manager-7b4d8999df-x6scl\" (UID: \"0d056fdd-af45-4a37-bacb-9daa0b022eeb\") " pod="openshift-lws-operator/lws-controller-manager-7b4d8999df-x6scl" Jul 10 00:24:16.683300 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:16.683144 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0d056fdd-af45-4a37-bacb-9daa0b022eeb-metrics-cert\") pod \"lws-controller-manager-7b4d8999df-x6scl\" (UID: \"0d056fdd-af45-4a37-bacb-9daa0b022eeb\") " pod="openshift-lws-operator/lws-controller-manager-7b4d8999df-x6scl" Jul 10 00:24:16.696459 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:16.696425 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snhzc\" (UniqueName: \"kubernetes.io/projected/0d056fdd-af45-4a37-bacb-9daa0b022eeb-kube-api-access-snhzc\") pod \"lws-controller-manager-7b4d8999df-x6scl\" (UID: \"0d056fdd-af45-4a37-bacb-9daa0b022eeb\") " pod="openshift-lws-operator/lws-controller-manager-7b4d8999df-x6scl" Jul 10 00:24:16.809180 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:16.809141 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7b4d8999df-x6scl" Jul 10 00:24:16.981063 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:16.981035 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7b4d8999df-x6scl"] Jul 10 00:24:17.868985 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:24:17.868948 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d056fdd_af45_4a37_bacb_9daa0b022eeb.slice/crio-02cef284fd9c5df6ba0b12439f73c49e95a7ff1867aefc4b4c47fa332aaa5e74 WatchSource:0}: Error finding container 02cef284fd9c5df6ba0b12439f73c49e95a7ff1867aefc4b4c47fa332aaa5e74: Status 404 returned error can't find the container with id 02cef284fd9c5df6ba0b12439f73c49e95a7ff1867aefc4b4c47fa332aaa5e74 Jul 10 00:24:17.930838 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:17.930801 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:24:17.930933 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:17.930877 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:24:17.930933 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:17.930908 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:24:18.694466 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:18.694423 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7b4d8999df-x6scl" event={"ID":"0d056fdd-af45-4a37-bacb-9daa0b022eeb","Type":"ContainerStarted","Data":"02cef284fd9c5df6ba0b12439f73c49e95a7ff1867aefc4b4c47fa332aaa5e74"} Jul 10 00:24:18.696210 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:18.696173 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47czbk" event={"ID":"d0ba5141-fd5f-4e01-8865-037d6e6743d6","Type":"ContainerStarted","Data":"14ee2d0e6db87193722f62ee8297767176772cbfd3983ee20a63ac64449c9d75"} Jul 10 00:24:18.716585 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:18.716521 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47czbk" podStartSLOduration=1.883848352 podStartE2EDuration="4.716501536s" totalImagesPullingTime="2.832653184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:24:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:24:18.715166097 +0000 UTC m=+525.533471594" watchObservedRunningTime="2026-07-10 00:24:18.716501536 +0000 UTC m=+525.534807030" Jul 10 00:24:18.919290 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:18.919254 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47czbk" Jul 10 00:24:18.924612 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:18.924579 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47czbk" Jul 10 00:24:19.702537 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:19.702446 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7b4d8999df-x6scl" event={"ID":"0d056fdd-af45-4a37-bacb-9daa0b022eeb","Type":"ContainerStarted","Data":"d90529236cd6a5185219e002528dd9e589ec16d114e271df73df1499185febe7"} Jul 10 00:24:19.702712 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:19.702687 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7b4d8999df-x6scl" Jul 10 00:24:19.702776 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:19.702737 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47czbk" Jul 10 00:24:19.703804 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:19.703788 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47czbk" Jul 10 00:24:19.717705 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:19.717657 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7b4d8999df-x6scl" podStartSLOduration=2.247420904 podStartE2EDuration="3.71764011s" totalImagesPullingTime="1.470219206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:24:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:24:19.71758223 +0000 UTC m=+526.535887727" watchObservedRunningTime="2026-07-10 00:24:19.71764011 +0000 UTC m=+526.535945609" Jul 10 00:24:30.709652 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:24:30.709564 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7b4d8999df-x6scl" Jul 10 00:25:20.119782 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:20.119747 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk85r"] Jul 10 00:25:20.122884 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:20.122868 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk85r" Jul 10 00:25:20.124758 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:20.124731 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 10 00:25:20.124758 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:20.124744 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 10 00:25:20.125357 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:20.125340 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-vzvcw\"" Jul 10 00:25:20.129031 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:20.129008 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk85r"] Jul 10 00:25:20.217744 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:20.217710 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/080517b7-abe3-4d7f-9937-f3839e9a2417-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk85r\" (UID: \"080517b7-abe3-4d7f-9937-f3839e9a2417\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk85r" Jul 10 00:25:20.217940 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:20.217758 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h5kg\" (UniqueName: \"kubernetes.io/projected/080517b7-abe3-4d7f-9937-f3839e9a2417-kube-api-access-8h5kg\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk85r\" (UID: \"080517b7-abe3-4d7f-9937-f3839e9a2417\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk85r" Jul 10 00:25:20.217940 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:20.217783 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/080517b7-abe3-4d7f-9937-f3839e9a2417-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk85r\" (UID: \"080517b7-abe3-4d7f-9937-f3839e9a2417\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk85r" Jul 10 00:25:20.319057 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:20.319014 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/080517b7-abe3-4d7f-9937-f3839e9a2417-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk85r\" (UID: \"080517b7-abe3-4d7f-9937-f3839e9a2417\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk85r" Jul 10 00:25:20.319265 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:20.319094 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h5kg\" (UniqueName: \"kubernetes.io/projected/080517b7-abe3-4d7f-9937-f3839e9a2417-kube-api-access-8h5kg\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk85r\" (UID: \"080517b7-abe3-4d7f-9937-f3839e9a2417\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk85r" Jul 10 00:25:20.319265 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:20.319208 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/080517b7-abe3-4d7f-9937-f3839e9a2417-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk85r\" (UID: \"080517b7-abe3-4d7f-9937-f3839e9a2417\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk85r" Jul 10 00:25:20.319457 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:20.319439 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/080517b7-abe3-4d7f-9937-f3839e9a2417-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk85r\" (UID: \"080517b7-abe3-4d7f-9937-f3839e9a2417\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk85r" Jul 10 00:25:20.319535 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:20.319518 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/080517b7-abe3-4d7f-9937-f3839e9a2417-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk85r\" (UID: \"080517b7-abe3-4d7f-9937-f3839e9a2417\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk85r" Jul 10 00:25:20.326497 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:20.326475 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h5kg\" (UniqueName: \"kubernetes.io/projected/080517b7-abe3-4d7f-9937-f3839e9a2417-kube-api-access-8h5kg\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk85r\" (UID: \"080517b7-abe3-4d7f-9937-f3839e9a2417\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk85r" Jul 10 00:25:20.433188 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:20.433109 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk85r" Jul 10 00:25:20.589867 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:20.589842 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk85r"] Jul 10 00:25:20.592130 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:25:20.592097 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod080517b7_abe3_4d7f_9937_f3839e9a2417.slice/crio-c9ac78df71053cd5c4eb85d7f4987ceed02baceb65881fec2d1d18bff9bcf447 WatchSource:0}: Error finding container c9ac78df71053cd5c4eb85d7f4987ceed02baceb65881fec2d1d18bff9bcf447: Status 404 returned error can't find the container with id c9ac78df71053cd5c4eb85d7f4987ceed02baceb65881fec2d1d18bff9bcf447 Jul 10 00:25:20.719848 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:20.719816 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7362l79"] Jul 10 00:25:20.723290 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:20.723274 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7362l79" Jul 10 00:25:20.729555 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:20.729531 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7362l79"] Jul 10 00:25:20.826765 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:20.826730 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/815d1fa6-90ea-4254-af1a-4be28806be87-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7362l79\" (UID: \"815d1fa6-90ea-4254-af1a-4be28806be87\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7362l79" Jul 10 00:25:20.826765 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:20.826766 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/815d1fa6-90ea-4254-af1a-4be28806be87-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7362l79\" (UID: \"815d1fa6-90ea-4254-af1a-4be28806be87\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7362l79" Jul 10 00:25:20.826953 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:20.826786 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7hln\" (UniqueName: \"kubernetes.io/projected/815d1fa6-90ea-4254-af1a-4be28806be87-kube-api-access-h7hln\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7362l79\" (UID: \"815d1fa6-90ea-4254-af1a-4be28806be87\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7362l79" Jul 10 00:25:20.927910 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:20.927873 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/815d1fa6-90ea-4254-af1a-4be28806be87-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7362l79\" (UID: \"815d1fa6-90ea-4254-af1a-4be28806be87\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7362l79" Jul 10 00:25:20.927910 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:20.927912 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/815d1fa6-90ea-4254-af1a-4be28806be87-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7362l79\" (UID: \"815d1fa6-90ea-4254-af1a-4be28806be87\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7362l79" Jul 10 00:25:20.928152 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:20.927934 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7hln\" (UniqueName: \"kubernetes.io/projected/815d1fa6-90ea-4254-af1a-4be28806be87-kube-api-access-h7hln\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7362l79\" (UID: \"815d1fa6-90ea-4254-af1a-4be28806be87\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7362l79" Jul 10 00:25:20.928268 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:20.928248 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/815d1fa6-90ea-4254-af1a-4be28806be87-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7362l79\" (UID: \"815d1fa6-90ea-4254-af1a-4be28806be87\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7362l79" Jul 10 00:25:20.928303 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:20.928278 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/815d1fa6-90ea-4254-af1a-4be28806be87-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7362l79\" (UID: \"815d1fa6-90ea-4254-af1a-4be28806be87\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7362l79" Jul 10 00:25:20.935194 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:20.935168 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7hln\" (UniqueName: \"kubernetes.io/projected/815d1fa6-90ea-4254-af1a-4be28806be87-kube-api-access-h7hln\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7362l79\" (UID: \"815d1fa6-90ea-4254-af1a-4be28806be87\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7362l79" Jul 10 00:25:20.983763 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:20.983689 2562 generic.go:356] "Generic (PLEG): container finished" podID="080517b7-abe3-4d7f-9937-f3839e9a2417" containerID="83d75e8d06ede84eca56c8073e5f69eed4fc3834f95b1af9bc387606ad0782f6" exitCode=0 Jul 10 00:25:20.983877 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:20.983760 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk85r" event={"ID":"080517b7-abe3-4d7f-9937-f3839e9a2417","Type":"ContainerDied","Data":"83d75e8d06ede84eca56c8073e5f69eed4fc3834f95b1af9bc387606ad0782f6"} Jul 10 00:25:20.983877 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:20.983810 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk85r" event={"ID":"080517b7-abe3-4d7f-9937-f3839e9a2417","Type":"ContainerStarted","Data":"c9ac78df71053cd5c4eb85d7f4987ceed02baceb65881fec2d1d18bff9bcf447"} Jul 10 00:25:21.065915 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:21.065884 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7362l79" Jul 10 00:25:21.192787 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:21.192672 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7362l79"] Jul 10 00:25:21.194853 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:25:21.194822 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod815d1fa6_90ea_4254_af1a_4be28806be87.slice/crio-3f7b115c3363b57896b57cf41b628697cec9741bbf64f14c07eb91362e28b1df WatchSource:0}: Error finding container 3f7b115c3363b57896b57cf41b628697cec9741bbf64f14c07eb91362e28b1df: Status 404 returned error can't find the container with id 3f7b115c3363b57896b57cf41b628697cec9741bbf64f14c07eb91362e28b1df Jul 10 00:25:21.320397 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:21.320368 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwhx6"] Jul 10 00:25:21.323701 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:21.323685 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwhx6" Jul 10 00:25:21.330448 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:21.330426 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwhx6"] Jul 10 00:25:21.432617 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:21.432583 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b8c42013-1e7f-4f1c-a84b-d9f05a557e4d-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwhx6\" (UID: \"b8c42013-1e7f-4f1c-a84b-d9f05a557e4d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwhx6" Jul 10 00:25:21.432745 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:21.432627 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b8c42013-1e7f-4f1c-a84b-d9f05a557e4d-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwhx6\" (UID: \"b8c42013-1e7f-4f1c-a84b-d9f05a557e4d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwhx6" Jul 10 00:25:21.432745 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:21.432719 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gktdc\" (UniqueName: \"kubernetes.io/projected/b8c42013-1e7f-4f1c-a84b-d9f05a557e4d-kube-api-access-gktdc\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwhx6\" (UID: \"b8c42013-1e7f-4f1c-a84b-d9f05a557e4d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwhx6" Jul 10 00:25:21.533660 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:21.533585 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b8c42013-1e7f-4f1c-a84b-d9f05a557e4d-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwhx6\" (UID: \"b8c42013-1e7f-4f1c-a84b-d9f05a557e4d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwhx6" Jul 10 00:25:21.533773 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:21.533658 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gktdc\" (UniqueName: \"kubernetes.io/projected/b8c42013-1e7f-4f1c-a84b-d9f05a557e4d-kube-api-access-gktdc\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwhx6\" (UID: \"b8c42013-1e7f-4f1c-a84b-d9f05a557e4d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwhx6" Jul 10 00:25:21.533773 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:21.533697 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b8c42013-1e7f-4f1c-a84b-d9f05a557e4d-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwhx6\" (UID: \"b8c42013-1e7f-4f1c-a84b-d9f05a557e4d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwhx6" Jul 10 00:25:21.533953 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:21.533935 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b8c42013-1e7f-4f1c-a84b-d9f05a557e4d-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwhx6\" (UID: \"b8c42013-1e7f-4f1c-a84b-d9f05a557e4d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwhx6" Jul 10 00:25:21.534006 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:21.533991 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b8c42013-1e7f-4f1c-a84b-d9f05a557e4d-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwhx6\" (UID: \"b8c42013-1e7f-4f1c-a84b-d9f05a557e4d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwhx6" Jul 10 00:25:21.541354 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:21.541337 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gktdc\" (UniqueName: \"kubernetes.io/projected/b8c42013-1e7f-4f1c-a84b-d9f05a557e4d-kube-api-access-gktdc\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwhx6\" (UID: \"b8c42013-1e7f-4f1c-a84b-d9f05a557e4d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwhx6" Jul 10 00:25:21.671491 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:21.671458 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwhx6" Jul 10 00:25:21.806553 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:21.806529 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwhx6"] Jul 10 00:25:21.851630 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:25:21.851603 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8c42013_1e7f_4f1c_a84b_d9f05a557e4d.slice/crio-f13153bed067ba9cadbbc777489a58ac79d4b4889afb584a207c87cfec5ea510 WatchSource:0}: Error finding container f13153bed067ba9cadbbc777489a58ac79d4b4889afb584a207c87cfec5ea510: Status 404 returned error can't find the container with id f13153bed067ba9cadbbc777489a58ac79d4b4889afb584a207c87cfec5ea510 Jul 10 00:25:21.922969 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:21.922945 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e087669"] Jul 10 00:25:21.927531 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:21.927515 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e087669" Jul 10 00:25:21.933369 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:21.933343 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e087669"] Jul 10 00:25:21.991212 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:21.991185 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk85r" event={"ID":"080517b7-abe3-4d7f-9937-f3839e9a2417","Type":"ContainerStarted","Data":"2524117513caa060a294a990159e0e116ddf375cb39ee814fff027e9663b0126"} Jul 10 00:25:21.992695 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:21.992667 2562 generic.go:356] "Generic (PLEG): container finished" podID="b8c42013-1e7f-4f1c-a84b-d9f05a557e4d" containerID="c9458da0ddeabca2e29f7067d28fcebe16cebf07e808a467a21c79426782177c" exitCode=0 Jul 10 00:25:21.992817 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:21.992750 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwhx6" event={"ID":"b8c42013-1e7f-4f1c-a84b-d9f05a557e4d","Type":"ContainerDied","Data":"c9458da0ddeabca2e29f7067d28fcebe16cebf07e808a467a21c79426782177c"} Jul 10 00:25:21.992817 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:21.992800 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwhx6" event={"ID":"b8c42013-1e7f-4f1c-a84b-d9f05a557e4d","Type":"ContainerStarted","Data":"f13153bed067ba9cadbbc777489a58ac79d4b4889afb584a207c87cfec5ea510"} Jul 10 00:25:21.994766 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:21.994745 2562 generic.go:356] "Generic (PLEG): container finished" podID="815d1fa6-90ea-4254-af1a-4be28806be87" containerID="22f369230ce882953c91f35ccd064f8090a2597c4765460b971fe3813c9101b0" exitCode=0 Jul 10 00:25:21.994870 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:21.994800 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7362l79" event={"ID":"815d1fa6-90ea-4254-af1a-4be28806be87","Type":"ContainerDied","Data":"22f369230ce882953c91f35ccd064f8090a2597c4765460b971fe3813c9101b0"} Jul 10 00:25:21.994870 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:21.994834 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7362l79" event={"ID":"815d1fa6-90ea-4254-af1a-4be28806be87","Type":"ContainerStarted","Data":"3f7b115c3363b57896b57cf41b628697cec9741bbf64f14c07eb91362e28b1df"} Jul 10 00:25:22.037358 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:22.037336 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27fbf742-b6f0-448f-a86a-e4f40ef8b985-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e087669\" (UID: \"27fbf742-b6f0-448f-a86a-e4f40ef8b985\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e087669" Jul 10 00:25:22.037473 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:22.037392 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27fbf742-b6f0-448f-a86a-e4f40ef8b985-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e087669\" (UID: \"27fbf742-b6f0-448f-a86a-e4f40ef8b985\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e087669" Jul 10 00:25:22.037473 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:22.037428 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwms8\" (UniqueName: \"kubernetes.io/projected/27fbf742-b6f0-448f-a86a-e4f40ef8b985-kube-api-access-bwms8\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e087669\" (UID: \"27fbf742-b6f0-448f-a86a-e4f40ef8b985\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e087669" Jul 10 00:25:22.138482 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:22.138401 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwms8\" (UniqueName: \"kubernetes.io/projected/27fbf742-b6f0-448f-a86a-e4f40ef8b985-kube-api-access-bwms8\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e087669\" (UID: \"27fbf742-b6f0-448f-a86a-e4f40ef8b985\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e087669" Jul 10 00:25:22.138482 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:22.138454 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27fbf742-b6f0-448f-a86a-e4f40ef8b985-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e087669\" (UID: \"27fbf742-b6f0-448f-a86a-e4f40ef8b985\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e087669" Jul 10 00:25:22.138652 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:22.138534 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27fbf742-b6f0-448f-a86a-e4f40ef8b985-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e087669\" (UID: \"27fbf742-b6f0-448f-a86a-e4f40ef8b985\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e087669" Jul 10 00:25:22.138823 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:22.138804 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27fbf742-b6f0-448f-a86a-e4f40ef8b985-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e087669\" (UID: \"27fbf742-b6f0-448f-a86a-e4f40ef8b985\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e087669" Jul 10 00:25:22.138897 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:22.138878 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27fbf742-b6f0-448f-a86a-e4f40ef8b985-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e087669\" (UID: \"27fbf742-b6f0-448f-a86a-e4f40ef8b985\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e087669" Jul 10 00:25:22.145477 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:22.145452 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwms8\" (UniqueName: \"kubernetes.io/projected/27fbf742-b6f0-448f-a86a-e4f40ef8b985-kube-api-access-bwms8\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e087669\" (UID: \"27fbf742-b6f0-448f-a86a-e4f40ef8b985\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e087669" Jul 10 00:25:22.339299 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:22.339264 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e087669" Jul 10 00:25:22.467922 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:22.467894 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e087669"] Jul 10 00:25:22.470560 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:25:22.470531 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27fbf742_b6f0_448f_a86a_e4f40ef8b985.slice/crio-f86b30140361d2074d1a87f9fdfee392a74706f53b091e4676088be745810573 WatchSource:0}: Error finding container f86b30140361d2074d1a87f9fdfee392a74706f53b091e4676088be745810573: Status 404 returned error can't find the container with id f86b30140361d2074d1a87f9fdfee392a74706f53b091e4676088be745810573 Jul 10 00:25:23.005958 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:23.005846 2562 generic.go:356] "Generic (PLEG): container finished" podID="080517b7-abe3-4d7f-9937-f3839e9a2417" containerID="2524117513caa060a294a990159e0e116ddf375cb39ee814fff027e9663b0126" exitCode=0 Jul 10 00:25:23.005958 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:23.005925 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk85r" event={"ID":"080517b7-abe3-4d7f-9937-f3839e9a2417","Type":"ContainerDied","Data":"2524117513caa060a294a990159e0e116ddf375cb39ee814fff027e9663b0126"} Jul 10 00:25:23.016298 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:23.016047 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwhx6" event={"ID":"b8c42013-1e7f-4f1c-a84b-d9f05a557e4d","Type":"ContainerStarted","Data":"34e8a19cc0a7cec7eaf65b302a9b666d0745f040cc9d8a22c29d958df237d97b"} Jul 10 00:25:23.020363 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:23.020290 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7362l79" event={"ID":"815d1fa6-90ea-4254-af1a-4be28806be87","Type":"ContainerStarted","Data":"ae9c43c4ebc314990255861344389691fba9a5bdb3a449d2495ad1b73dbbfda6"} Jul 10 00:25:23.022243 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:23.022209 2562 generic.go:356] "Generic (PLEG): container finished" podID="27fbf742-b6f0-448f-a86a-e4f40ef8b985" containerID="3db78a4dda77bf5af27371b82ea4bbe6eb5e63b2df62e85d70a29140002585a7" exitCode=0 Jul 10 00:25:23.022347 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:23.022289 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e087669" event={"ID":"27fbf742-b6f0-448f-a86a-e4f40ef8b985","Type":"ContainerDied","Data":"3db78a4dda77bf5af27371b82ea4bbe6eb5e63b2df62e85d70a29140002585a7"} Jul 10 00:25:23.022347 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:23.022327 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e087669" event={"ID":"27fbf742-b6f0-448f-a86a-e4f40ef8b985","Type":"ContainerStarted","Data":"f86b30140361d2074d1a87f9fdfee392a74706f53b091e4676088be745810573"} Jul 10 00:25:24.029521 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:24.029434 2562 generic.go:356] "Generic (PLEG): container finished" podID="080517b7-abe3-4d7f-9937-f3839e9a2417" containerID="f6ee64faef7eb3690e02caefadd2d6ba5ff16a7e75eeb14957782436e07e12bc" exitCode=0 Jul 10 00:25:24.029916 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:24.029523 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk85r" event={"ID":"080517b7-abe3-4d7f-9937-f3839e9a2417","Type":"ContainerDied","Data":"f6ee64faef7eb3690e02caefadd2d6ba5ff16a7e75eeb14957782436e07e12bc"} Jul 10 00:25:24.031166 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:24.031145 2562 generic.go:356] "Generic (PLEG): container finished" podID="b8c42013-1e7f-4f1c-a84b-d9f05a557e4d" containerID="34e8a19cc0a7cec7eaf65b302a9b666d0745f040cc9d8a22c29d958df237d97b" exitCode=0 Jul 10 00:25:24.031259 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:24.031229 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwhx6" event={"ID":"b8c42013-1e7f-4f1c-a84b-d9f05a557e4d","Type":"ContainerDied","Data":"34e8a19cc0a7cec7eaf65b302a9b666d0745f040cc9d8a22c29d958df237d97b"} Jul 10 00:25:24.033149 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:24.033130 2562 generic.go:356] "Generic (PLEG): container finished" podID="815d1fa6-90ea-4254-af1a-4be28806be87" containerID="ae9c43c4ebc314990255861344389691fba9a5bdb3a449d2495ad1b73dbbfda6" exitCode=0 Jul 10 00:25:24.033232 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:24.033208 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7362l79" event={"ID":"815d1fa6-90ea-4254-af1a-4be28806be87","Type":"ContainerDied","Data":"ae9c43c4ebc314990255861344389691fba9a5bdb3a449d2495ad1b73dbbfda6"} Jul 10 00:25:24.034964 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:24.034943 2562 generic.go:356] "Generic (PLEG): container finished" podID="27fbf742-b6f0-448f-a86a-e4f40ef8b985" containerID="664d9051eaa5cdf436bc3024c618d44b4eb584736a1a097d454d86838ea12117" exitCode=0 Jul 10 00:25:24.035045 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:24.034986 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e087669" event={"ID":"27fbf742-b6f0-448f-a86a-e4f40ef8b985","Type":"ContainerDied","Data":"664d9051eaa5cdf436bc3024c618d44b4eb584736a1a097d454d86838ea12117"} Jul 10 00:25:25.041850 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:25.041812 2562 generic.go:356] "Generic (PLEG): container finished" podID="815d1fa6-90ea-4254-af1a-4be28806be87" containerID="609a46cdb7e14146f920a2478bca8713d7196fc63be146627c0f6915129d1b4c" exitCode=0 Jul 10 00:25:25.042351 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:25.041902 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7362l79" event={"ID":"815d1fa6-90ea-4254-af1a-4be28806be87","Type":"ContainerDied","Data":"609a46cdb7e14146f920a2478bca8713d7196fc63be146627c0f6915129d1b4c"} Jul 10 00:25:25.043719 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:25.043696 2562 generic.go:356] "Generic (PLEG): container finished" podID="27fbf742-b6f0-448f-a86a-e4f40ef8b985" containerID="146630d5a9d504d0a754ec4e7776ace07577442ea52e142d631b1f598f9a25d5" exitCode=0 Jul 10 00:25:25.043811 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:25.043772 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e087669" event={"ID":"27fbf742-b6f0-448f-a86a-e4f40ef8b985","Type":"ContainerDied","Data":"146630d5a9d504d0a754ec4e7776ace07577442ea52e142d631b1f598f9a25d5"} Jul 10 00:25:25.045669 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:25.045648 2562 generic.go:356] "Generic (PLEG): container finished" podID="b8c42013-1e7f-4f1c-a84b-d9f05a557e4d" containerID="674371c068d8770300c96388d5074146a055ef89e808474470b48d7e36f7a7a9" exitCode=0 Jul 10 00:25:25.045751 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:25.045724 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwhx6" event={"ID":"b8c42013-1e7f-4f1c-a84b-d9f05a557e4d","Type":"ContainerDied","Data":"674371c068d8770300c96388d5074146a055ef89e808474470b48d7e36f7a7a9"} Jul 10 00:25:25.172867 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:25.172844 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk85r" Jul 10 00:25:25.263163 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:25.263134 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8h5kg\" (UniqueName: \"kubernetes.io/projected/080517b7-abe3-4d7f-9937-f3839e9a2417-kube-api-access-8h5kg\") pod \"080517b7-abe3-4d7f-9937-f3839e9a2417\" (UID: \"080517b7-abe3-4d7f-9937-f3839e9a2417\") " Jul 10 00:25:25.263163 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:25.263170 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/080517b7-abe3-4d7f-9937-f3839e9a2417-bundle\") pod \"080517b7-abe3-4d7f-9937-f3839e9a2417\" (UID: \"080517b7-abe3-4d7f-9937-f3839e9a2417\") " Jul 10 00:25:25.263394 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:25.263207 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/080517b7-abe3-4d7f-9937-f3839e9a2417-util\") pod \"080517b7-abe3-4d7f-9937-f3839e9a2417\" (UID: \"080517b7-abe3-4d7f-9937-f3839e9a2417\") " Jul 10 00:25:25.263667 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:25.263630 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/080517b7-abe3-4d7f-9937-f3839e9a2417-bundle" (OuterVolumeSpecName: "bundle") pod "080517b7-abe3-4d7f-9937-f3839e9a2417" (UID: "080517b7-abe3-4d7f-9937-f3839e9a2417"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:25:25.265360 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:25.265335 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/080517b7-abe3-4d7f-9937-f3839e9a2417-kube-api-access-8h5kg" (OuterVolumeSpecName: "kube-api-access-8h5kg") pod "080517b7-abe3-4d7f-9937-f3839e9a2417" (UID: "080517b7-abe3-4d7f-9937-f3839e9a2417"). InnerVolumeSpecName "kube-api-access-8h5kg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:25:25.268763 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:25.268730 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/080517b7-abe3-4d7f-9937-f3839e9a2417-util" (OuterVolumeSpecName: "util") pod "080517b7-abe3-4d7f-9937-f3839e9a2417" (UID: "080517b7-abe3-4d7f-9937-f3839e9a2417"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:25:25.364649 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:25.364624 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8h5kg\" (UniqueName: \"kubernetes.io/projected/080517b7-abe3-4d7f-9937-f3839e9a2417-kube-api-access-8h5kg\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:25:25.364649 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:25.364648 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/080517b7-abe3-4d7f-9937-f3839e9a2417-bundle\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:25:25.364789 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:25.364658 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/080517b7-abe3-4d7f-9937-f3839e9a2417-util\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:25:26.052681 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:26.052658 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk85r" Jul 10 00:25:26.053154 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:26.052675 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk85r" event={"ID":"080517b7-abe3-4d7f-9937-f3839e9a2417","Type":"ContainerDied","Data":"c9ac78df71053cd5c4eb85d7f4987ceed02baceb65881fec2d1d18bff9bcf447"} Jul 10 00:25:26.053154 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:26.052732 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9ac78df71053cd5c4eb85d7f4987ceed02baceb65881fec2d1d18bff9bcf447" Jul 10 00:25:26.203772 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:26.203743 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwhx6" Jul 10 00:25:26.235116 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:26.235092 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e087669" Jul 10 00:25:26.238117 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:26.238096 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7362l79" Jul 10 00:25:26.374647 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:26.374573 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/815d1fa6-90ea-4254-af1a-4be28806be87-util\") pod \"815d1fa6-90ea-4254-af1a-4be28806be87\" (UID: \"815d1fa6-90ea-4254-af1a-4be28806be87\") " Jul 10 00:25:26.374647 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:26.374605 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b8c42013-1e7f-4f1c-a84b-d9f05a557e4d-util\") pod \"b8c42013-1e7f-4f1c-a84b-d9f05a557e4d\" (UID: \"b8c42013-1e7f-4f1c-a84b-d9f05a557e4d\") " Jul 10 00:25:26.374647 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:26.374626 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27fbf742-b6f0-448f-a86a-e4f40ef8b985-util\") pod \"27fbf742-b6f0-448f-a86a-e4f40ef8b985\" (UID: \"27fbf742-b6f0-448f-a86a-e4f40ef8b985\") " Jul 10 00:25:26.374895 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:26.374660 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h7hln\" (UniqueName: \"kubernetes.io/projected/815d1fa6-90ea-4254-af1a-4be28806be87-kube-api-access-h7hln\") pod \"815d1fa6-90ea-4254-af1a-4be28806be87\" (UID: \"815d1fa6-90ea-4254-af1a-4be28806be87\") " Jul 10 00:25:26.374895 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:26.374692 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gktdc\" (UniqueName: \"kubernetes.io/projected/b8c42013-1e7f-4f1c-a84b-d9f05a557e4d-kube-api-access-gktdc\") pod \"b8c42013-1e7f-4f1c-a84b-d9f05a557e4d\" (UID: \"b8c42013-1e7f-4f1c-a84b-d9f05a557e4d\") " Jul 10 00:25:26.374895 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:26.374730 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/815d1fa6-90ea-4254-af1a-4be28806be87-bundle\") pod \"815d1fa6-90ea-4254-af1a-4be28806be87\" (UID: \"815d1fa6-90ea-4254-af1a-4be28806be87\") " Jul 10 00:25:26.374895 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:26.374749 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bwms8\" (UniqueName: \"kubernetes.io/projected/27fbf742-b6f0-448f-a86a-e4f40ef8b985-kube-api-access-bwms8\") pod \"27fbf742-b6f0-448f-a86a-e4f40ef8b985\" (UID: \"27fbf742-b6f0-448f-a86a-e4f40ef8b985\") " Jul 10 00:25:26.375225 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:26.374915 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27fbf742-b6f0-448f-a86a-e4f40ef8b985-bundle\") pod \"27fbf742-b6f0-448f-a86a-e4f40ef8b985\" (UID: \"27fbf742-b6f0-448f-a86a-e4f40ef8b985\") " Jul 10 00:25:26.375225 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:26.374980 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b8c42013-1e7f-4f1c-a84b-d9f05a557e4d-bundle\") pod \"b8c42013-1e7f-4f1c-a84b-d9f05a557e4d\" (UID: \"b8c42013-1e7f-4f1c-a84b-d9f05a557e4d\") " Jul 10 00:25:26.375865 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:26.375759 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/815d1fa6-90ea-4254-af1a-4be28806be87-bundle" (OuterVolumeSpecName: "bundle") pod "815d1fa6-90ea-4254-af1a-4be28806be87" (UID: "815d1fa6-90ea-4254-af1a-4be28806be87"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:25:26.376236 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:26.376178 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8c42013-1e7f-4f1c-a84b-d9f05a557e4d-bundle" (OuterVolumeSpecName: "bundle") pod "b8c42013-1e7f-4f1c-a84b-d9f05a557e4d" (UID: "b8c42013-1e7f-4f1c-a84b-d9f05a557e4d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:25:26.376407 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:26.376388 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27fbf742-b6f0-448f-a86a-e4f40ef8b985-bundle" (OuterVolumeSpecName: "bundle") pod "27fbf742-b6f0-448f-a86a-e4f40ef8b985" (UID: "27fbf742-b6f0-448f-a86a-e4f40ef8b985"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:25:26.377316 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:26.377282 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/815d1fa6-90ea-4254-af1a-4be28806be87-kube-api-access-h7hln" (OuterVolumeSpecName: "kube-api-access-h7hln") pod "815d1fa6-90ea-4254-af1a-4be28806be87" (UID: "815d1fa6-90ea-4254-af1a-4be28806be87"). InnerVolumeSpecName "kube-api-access-h7hln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:25:26.377417 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:26.377398 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8c42013-1e7f-4f1c-a84b-d9f05a557e4d-kube-api-access-gktdc" (OuterVolumeSpecName: "kube-api-access-gktdc") pod "b8c42013-1e7f-4f1c-a84b-d9f05a557e4d" (UID: "b8c42013-1e7f-4f1c-a84b-d9f05a557e4d"). InnerVolumeSpecName "kube-api-access-gktdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:25:26.377472 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:26.377431 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27fbf742-b6f0-448f-a86a-e4f40ef8b985-kube-api-access-bwms8" (OuterVolumeSpecName: "kube-api-access-bwms8") pod "27fbf742-b6f0-448f-a86a-e4f40ef8b985" (UID: "27fbf742-b6f0-448f-a86a-e4f40ef8b985"). InnerVolumeSpecName "kube-api-access-bwms8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:25:26.381091 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:26.381042 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8c42013-1e7f-4f1c-a84b-d9f05a557e4d-util" (OuterVolumeSpecName: "util") pod "b8c42013-1e7f-4f1c-a84b-d9f05a557e4d" (UID: "b8c42013-1e7f-4f1c-a84b-d9f05a557e4d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:25:26.384066 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:26.384041 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27fbf742-b6f0-448f-a86a-e4f40ef8b985-util" (OuterVolumeSpecName: "util") pod "27fbf742-b6f0-448f-a86a-e4f40ef8b985" (UID: "27fbf742-b6f0-448f-a86a-e4f40ef8b985"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:25:26.384168 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:26.384112 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/815d1fa6-90ea-4254-af1a-4be28806be87-util" (OuterVolumeSpecName: "util") pod "815d1fa6-90ea-4254-af1a-4be28806be87" (UID: "815d1fa6-90ea-4254-af1a-4be28806be87"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:25:26.476037 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:26.476012 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b8c42013-1e7f-4f1c-a84b-d9f05a557e4d-bundle\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:25:26.476037 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:26.476034 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/815d1fa6-90ea-4254-af1a-4be28806be87-util\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:25:26.476037 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:26.476042 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b8c42013-1e7f-4f1c-a84b-d9f05a557e4d-util\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:25:26.476239 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:26.476050 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27fbf742-b6f0-448f-a86a-e4f40ef8b985-util\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:25:26.476239 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:26.476058 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7hln\" (UniqueName: \"kubernetes.io/projected/815d1fa6-90ea-4254-af1a-4be28806be87-kube-api-access-h7hln\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:25:26.476239 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:26.476068 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gktdc\" (UniqueName: \"kubernetes.io/projected/b8c42013-1e7f-4f1c-a84b-d9f05a557e4d-kube-api-access-gktdc\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:25:26.476239 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:26.476096 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/815d1fa6-90ea-4254-af1a-4be28806be87-bundle\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:25:26.476239 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:26.476104 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwms8\" (UniqueName: \"kubernetes.io/projected/27fbf742-b6f0-448f-a86a-e4f40ef8b985-kube-api-access-bwms8\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:25:26.476239 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:26.476111 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27fbf742-b6f0-448f-a86a-e4f40ef8b985-bundle\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:25:27.059434 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:27.059403 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwhx6" Jul 10 00:25:27.059829 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:27.059428 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwhx6" event={"ID":"b8c42013-1e7f-4f1c-a84b-d9f05a557e4d","Type":"ContainerDied","Data":"f13153bed067ba9cadbbc777489a58ac79d4b4889afb584a207c87cfec5ea510"} Jul 10 00:25:27.059829 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:27.059483 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f13153bed067ba9cadbbc777489a58ac79d4b4889afb584a207c87cfec5ea510" Jul 10 00:25:27.061344 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:27.061321 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7362l79" Jul 10 00:25:27.061474 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:27.061315 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7362l79" event={"ID":"815d1fa6-90ea-4254-af1a-4be28806be87","Type":"ContainerDied","Data":"3f7b115c3363b57896b57cf41b628697cec9741bbf64f14c07eb91362e28b1df"} Jul 10 00:25:27.061474 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:27.061427 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f7b115c3363b57896b57cf41b628697cec9741bbf64f14c07eb91362e28b1df" Jul 10 00:25:27.063244 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:27.063226 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e087669" event={"ID":"27fbf742-b6f0-448f-a86a-e4f40ef8b985","Type":"ContainerDied","Data":"f86b30140361d2074d1a87f9fdfee392a74706f53b091e4676088be745810573"} Jul 10 00:25:27.063354 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:27.063252 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e087669" Jul 10 00:25:27.063354 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:27.063257 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f86b30140361d2074d1a87f9fdfee392a74706f53b091e4676088be745810573" Jul 10 00:25:33.638949 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:33.638920 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kjhg9_4651a90d-ef18-4974-9efd-a77fff3c9dfd/console-operator/1.log" Jul 10 00:25:33.639502 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:33.639150 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kjhg9_4651a90d-ef18-4974-9efd-a77fff3c9dfd/console-operator/1.log" Jul 10 00:25:33.642808 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:33.642783 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h5g9w_f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32/ovn-acl-logging/0.log" Jul 10 00:25:33.642949 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:25:33.642819 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h5g9w_f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32/ovn-acl-logging/0.log" Jul 10 00:26:32.048875 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:32.048843 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lhjrd"] Jul 10 00:26:32.049371 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:32.049188 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="080517b7-abe3-4d7f-9937-f3839e9a2417" containerName="extract" Jul 10 00:26:32.049371 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:32.049200 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="080517b7-abe3-4d7f-9937-f3839e9a2417" containerName="extract" Jul 10 00:26:32.049371 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:32.049208 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27fbf742-b6f0-448f-a86a-e4f40ef8b985" containerName="pull" Jul 10 00:26:32.049371 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:32.049213 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="27fbf742-b6f0-448f-a86a-e4f40ef8b985" containerName="pull" Jul 10 00:26:32.049371 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:32.049222 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8c42013-1e7f-4f1c-a84b-d9f05a557e4d" containerName="extract" Jul 10 00:26:32.049371 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:32.049228 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8c42013-1e7f-4f1c-a84b-d9f05a557e4d" containerName="extract" Jul 10 00:26:32.049371 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:32.049240 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="080517b7-abe3-4d7f-9937-f3839e9a2417" containerName="util" Jul 10 00:26:32.049371 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:32.049245 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="080517b7-abe3-4d7f-9937-f3839e9a2417" containerName="util" Jul 10 00:26:32.049371 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:32.049251 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="815d1fa6-90ea-4254-af1a-4be28806be87" containerName="util" Jul 10 00:26:32.049371 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:32.049257 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="815d1fa6-90ea-4254-af1a-4be28806be87" containerName="util" Jul 10 00:26:32.049371 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:32.049263 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8c42013-1e7f-4f1c-a84b-d9f05a557e4d" containerName="pull" Jul 10 00:26:32.049371 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:32.049268 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8c42013-1e7f-4f1c-a84b-d9f05a557e4d" containerName="pull" Jul 10 00:26:32.049371 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:32.049274 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27fbf742-b6f0-448f-a86a-e4f40ef8b985" containerName="extract" Jul 10 00:26:32.049371 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:32.049279 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="27fbf742-b6f0-448f-a86a-e4f40ef8b985" containerName="extract" Jul 10 00:26:32.049371 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:32.049286 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="815d1fa6-90ea-4254-af1a-4be28806be87" containerName="extract" Jul 10 00:26:32.049371 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:32.049291 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="815d1fa6-90ea-4254-af1a-4be28806be87" containerName="extract" Jul 10 00:26:32.049371 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:32.049302 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8c42013-1e7f-4f1c-a84b-d9f05a557e4d" containerName="util" Jul 10 00:26:32.049371 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:32.049306 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8c42013-1e7f-4f1c-a84b-d9f05a557e4d" containerName="util" Jul 10 00:26:32.049371 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:32.049311 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="080517b7-abe3-4d7f-9937-f3839e9a2417" containerName="pull" Jul 10 00:26:32.049371 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:32.049316 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="080517b7-abe3-4d7f-9937-f3839e9a2417" containerName="pull" Jul 10 00:26:32.049371 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:32.049325 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27fbf742-b6f0-448f-a86a-e4f40ef8b985" containerName="util" Jul 10 00:26:32.049371 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:32.049333 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="27fbf742-b6f0-448f-a86a-e4f40ef8b985" containerName="util" Jul 10 00:26:32.049371 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:32.049339 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="815d1fa6-90ea-4254-af1a-4be28806be87" containerName="pull" Jul 10 00:26:32.049371 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:32.049343 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="815d1fa6-90ea-4254-af1a-4be28806be87" containerName="pull" Jul 10 00:26:32.050203 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:32.049390 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="27fbf742-b6f0-448f-a86a-e4f40ef8b985" containerName="extract" Jul 10 00:26:32.050203 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:32.049398 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8c42013-1e7f-4f1c-a84b-d9f05a557e4d" containerName="extract" Jul 10 00:26:32.050203 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:32.049404 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="815d1fa6-90ea-4254-af1a-4be28806be87" containerName="extract" Jul 10 00:26:32.050203 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:32.049411 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="080517b7-abe3-4d7f-9937-f3839e9a2417" containerName="extract" Jul 10 00:26:32.052299 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:32.052280 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lhjrd" Jul 10 00:26:32.054194 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:32.054175 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-x2fql\"" Jul 10 00:26:32.063483 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:32.063455 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lhjrd"] Jul 10 00:26:32.133896 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:32.133862 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92786366-f55e-4e7b-b107-8cd178d52054-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-lhjrd\" (UID: \"92786366-f55e-4e7b-b107-8cd178d52054\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lhjrd" Jul 10 00:26:32.133896 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:32.133899 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92786366-f55e-4e7b-b107-8cd178d52054-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-lhjrd\" (UID: \"92786366-f55e-4e7b-b107-8cd178d52054\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lhjrd" Jul 10 00:26:32.134195 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:32.133926 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92786366-f55e-4e7b-b107-8cd178d52054-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-lhjrd\" (UID: \"92786366-f55e-4e7b-b107-8cd178d52054\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lhjrd" Jul 10 00:26:32.134195 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:32.134036 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92786366-f55e-4e7b-b107-8cd178d52054-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-lhjrd\" (UID: \"92786366-f55e-4e7b-b107-8cd178d52054\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lhjrd" Jul 10 00:26:32.134195 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:32.134070 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45mmn\" (UniqueName: \"kubernetes.io/projected/92786366-f55e-4e7b-b107-8cd178d52054-kube-api-access-45mmn\") pod \"maas-default-gateway-openshift-default-56654dcf7-lhjrd\" (UID: \"92786366-f55e-4e7b-b107-8cd178d52054\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lhjrd" Jul 10 00:26:32.134195 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:32.134153 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92786366-f55e-4e7b-b107-8cd178d52054-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-lhjrd\" (UID: \"92786366-f55e-4e7b-b107-8cd178d52054\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lhjrd" Jul 10 00:26:32.134195 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:32.134180 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92786366-f55e-4e7b-b107-8cd178d52054-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-lhjrd\" (UID: \"92786366-f55e-4e7b-b107-8cd178d52054\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lhjrd" Jul 10 00:26:32.134391 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:32.134211 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92786366-f55e-4e7b-b107-8cd178d52054-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-lhjrd\" (UID: \"92786366-f55e-4e7b-b107-8cd178d52054\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lhjrd" Jul 10 00:26:32.134391 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:32.134233 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92786366-f55e-4e7b-b107-8cd178d52054-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-lhjrd\" (UID: \"92786366-f55e-4e7b-b107-8cd178d52054\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lhjrd" Jul 10 00:26:32.234942 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:32.234909 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92786366-f55e-4e7b-b107-8cd178d52054-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-lhjrd\" (UID: \"92786366-f55e-4e7b-b107-8cd178d52054\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lhjrd" Jul 10 00:26:32.234942 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:32.234941 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45mmn\" (UniqueName: \"kubernetes.io/projected/92786366-f55e-4e7b-b107-8cd178d52054-kube-api-access-45mmn\") pod \"maas-default-gateway-openshift-default-56654dcf7-lhjrd\" (UID: \"92786366-f55e-4e7b-b107-8cd178d52054\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lhjrd" Jul 10 00:26:32.235212 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:32.234975 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92786366-f55e-4e7b-b107-8cd178d52054-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-lhjrd\" (UID: \"92786366-f55e-4e7b-b107-8cd178d52054\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lhjrd" Jul 10 00:26:32.235212 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:32.234990 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92786366-f55e-4e7b-b107-8cd178d52054-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-lhjrd\" (UID: \"92786366-f55e-4e7b-b107-8cd178d52054\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lhjrd" Jul 10 00:26:32.235212 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:32.235012 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92786366-f55e-4e7b-b107-8cd178d52054-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-lhjrd\" (UID: \"92786366-f55e-4e7b-b107-8cd178d52054\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lhjrd" Jul 10 00:26:32.235212 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:32.235030 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92786366-f55e-4e7b-b107-8cd178d52054-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-lhjrd\" (UID: \"92786366-f55e-4e7b-b107-8cd178d52054\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lhjrd" Jul 10 00:26:32.235212 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:32.235062 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92786366-f55e-4e7b-b107-8cd178d52054-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-lhjrd\" (UID: \"92786366-f55e-4e7b-b107-8cd178d52054\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lhjrd" Jul 10 00:26:32.235212 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:32.235122 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92786366-f55e-4e7b-b107-8cd178d52054-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-lhjrd\" (UID: \"92786366-f55e-4e7b-b107-8cd178d52054\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lhjrd" Jul 10 00:26:32.235212 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:32.235167 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92786366-f55e-4e7b-b107-8cd178d52054-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-lhjrd\" (UID: \"92786366-f55e-4e7b-b107-8cd178d52054\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lhjrd" Jul 10 00:26:32.235535 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:32.235364 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92786366-f55e-4e7b-b107-8cd178d52054-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-lhjrd\" (UID: \"92786366-f55e-4e7b-b107-8cd178d52054\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lhjrd" Jul 10 00:26:32.235535 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:32.235522 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92786366-f55e-4e7b-b107-8cd178d52054-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-lhjrd\" (UID: \"92786366-f55e-4e7b-b107-8cd178d52054\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lhjrd" Jul 10 00:26:32.235668 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:32.235545 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92786366-f55e-4e7b-b107-8cd178d52054-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-lhjrd\" (UID: \"92786366-f55e-4e7b-b107-8cd178d52054\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lhjrd" Jul 10 00:26:32.235668 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:32.235580 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92786366-f55e-4e7b-b107-8cd178d52054-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-lhjrd\" (UID: \"92786366-f55e-4e7b-b107-8cd178d52054\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lhjrd" Jul 10 00:26:32.235903 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:32.235883 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92786366-f55e-4e7b-b107-8cd178d52054-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-lhjrd\" (UID: \"92786366-f55e-4e7b-b107-8cd178d52054\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lhjrd" Jul 10 00:26:32.237274 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:32.237254 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92786366-f55e-4e7b-b107-8cd178d52054-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-lhjrd\" (UID: \"92786366-f55e-4e7b-b107-8cd178d52054\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lhjrd" Jul 10 00:26:32.237583 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:32.237565 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92786366-f55e-4e7b-b107-8cd178d52054-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-lhjrd\" (UID: \"92786366-f55e-4e7b-b107-8cd178d52054\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lhjrd" Jul 10 00:26:32.241661 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:32.241639 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92786366-f55e-4e7b-b107-8cd178d52054-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-lhjrd\" (UID: \"92786366-f55e-4e7b-b107-8cd178d52054\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lhjrd" Jul 10 00:26:32.242336 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:32.242319 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45mmn\" (UniqueName: \"kubernetes.io/projected/92786366-f55e-4e7b-b107-8cd178d52054-kube-api-access-45mmn\") pod \"maas-default-gateway-openshift-default-56654dcf7-lhjrd\" (UID: \"92786366-f55e-4e7b-b107-8cd178d52054\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lhjrd" Jul 10 00:26:32.362604 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:32.362522 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lhjrd" Jul 10 00:26:32.512867 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:32.512826 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lhjrd"] Jul 10 00:26:32.515756 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:26:32.515730 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92786366_f55e_4e7b_b107_8cd178d52054.slice/crio-96ad6e47aa139f8d92229c74370e6038e2c30cdd4c14e8a059749eb60b2bd8e3 WatchSource:0}: Error finding container 96ad6e47aa139f8d92229c74370e6038e2c30cdd4c14e8a059749eb60b2bd8e3: Status 404 returned error can't find the container with id 96ad6e47aa139f8d92229c74370e6038e2c30cdd4c14e8a059749eb60b2bd8e3 Jul 10 00:26:32.517825 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:32.517793 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:26:32.517914 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:32.517869 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:26:32.517969 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:32.517916 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:26:33.394607 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:33.394565 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lhjrd" event={"ID":"92786366-f55e-4e7b-b107-8cd178d52054","Type":"ContainerStarted","Data":"8417f9336f413e14e7ac2653f9884407b7c30e8de925d6f23fba7657aaaff355"} Jul 10 00:26:33.395006 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:33.394627 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lhjrd" event={"ID":"92786366-f55e-4e7b-b107-8cd178d52054","Type":"ContainerStarted","Data":"96ad6e47aa139f8d92229c74370e6038e2c30cdd4c14e8a059749eb60b2bd8e3"} Jul 10 00:26:33.414749 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:33.414687 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lhjrd" podStartSLOduration=1.414669178 podStartE2EDuration="1.414669178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:26:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:26:33.411037832 +0000 UTC m=+660.229343347" watchObservedRunningTime="2026-07-10 00:26:33.414669178 +0000 UTC m=+660.232974672" Jul 10 00:26:34.362963 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:34.362928 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lhjrd" Jul 10 00:26:34.367340 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:34.367318 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lhjrd" Jul 10 00:26:34.399481 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:34.399452 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lhjrd" Jul 10 00:26:34.400192 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:34.400173 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lhjrd" Jul 10 00:26:55.521748 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:55.521712 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-t298z"] Jul 10 00:26:55.525473 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:55.525454 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-t298z" Jul 10 00:26:55.527563 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:55.527538 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 10 00:26:55.528429 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:55.528401 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-7kws9\"" Jul 10 00:26:55.528568 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:55.528422 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 10 00:26:55.530706 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:55.530680 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-t298z"] Jul 10 00:26:55.636518 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:55.636481 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw7x2\" (UniqueName: \"kubernetes.io/projected/097b66ea-2cf0-43ef-ad5a-ef92af653270-kube-api-access-fw7x2\") pod \"authorino-75d556bd77-t298z\" (UID: \"097b66ea-2cf0-43ef-ad5a-ef92af653270\") " pod="kuadrant-system/authorino-75d556bd77-t298z" Jul 10 00:26:55.737648 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:55.737603 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw7x2\" (UniqueName: \"kubernetes.io/projected/097b66ea-2cf0-43ef-ad5a-ef92af653270-kube-api-access-fw7x2\") pod \"authorino-75d556bd77-t298z\" (UID: \"097b66ea-2cf0-43ef-ad5a-ef92af653270\") " pod="kuadrant-system/authorino-75d556bd77-t298z" Jul 10 00:26:55.744590 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:55.744564 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw7x2\" (UniqueName: \"kubernetes.io/projected/097b66ea-2cf0-43ef-ad5a-ef92af653270-kube-api-access-fw7x2\") pod \"authorino-75d556bd77-t298z\" (UID: \"097b66ea-2cf0-43ef-ad5a-ef92af653270\") " pod="kuadrant-system/authorino-75d556bd77-t298z" Jul 10 00:26:55.835755 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:55.835712 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-t298z" Jul 10 00:26:55.964673 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:55.964649 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-t298z"] Jul 10 00:26:55.966370 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:26:55.966344 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod097b66ea_2cf0_43ef_ad5a_ef92af653270.slice/crio-6b081a815248afc862ae2d006ec9763af6e1df4cf8bb2b81224ea62f54a4f422 WatchSource:0}: Error finding container 6b081a815248afc862ae2d006ec9763af6e1df4cf8bb2b81224ea62f54a4f422: Status 404 returned error can't find the container with id 6b081a815248afc862ae2d006ec9763af6e1df4cf8bb2b81224ea62f54a4f422 Jul 10 00:26:55.967654 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:55.967634 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 00:26:56.506525 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:56.506486 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-t298z" event={"ID":"097b66ea-2cf0-43ef-ad5a-ef92af653270","Type":"ContainerStarted","Data":"6b081a815248afc862ae2d006ec9763af6e1df4cf8bb2b81224ea62f54a4f422"} Jul 10 00:26:59.534518 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:59.534474 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-t298z" event={"ID":"097b66ea-2cf0-43ef-ad5a-ef92af653270","Type":"ContainerStarted","Data":"0a3c257aa931f7cc171d9d056bbc82d2725c4b3b22d37685a23da802a3e74266"} Jul 10 00:26:59.548799 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:26:59.548733 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-t298z" podStartSLOduration=1.213281306 podStartE2EDuration="4.548716823s" totalImagesPullingTime="3.335435517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:26:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:26:59.548248139 +0000 UTC m=+686.366553631" watchObservedRunningTime="2026-07-10 00:26:59.548716823 +0000 UTC m=+686.367022319" Jul 10 00:27:41.069030 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:27:41.068950 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-t298z"] Jul 10 00:27:41.069526 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:27:41.069164 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-t298z" podUID="097b66ea-2cf0-43ef-ad5a-ef92af653270" containerName="authorino" containerID="cri-o://0a3c257aa931f7cc171d9d056bbc82d2725c4b3b22d37685a23da802a3e74266" gracePeriod=30 Jul 10 00:27:41.315515 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:27:41.315488 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-t298z" Jul 10 00:27:41.439910 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:27:41.439824 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fw7x2\" (UniqueName: \"kubernetes.io/projected/097b66ea-2cf0-43ef-ad5a-ef92af653270-kube-api-access-fw7x2\") pod \"097b66ea-2cf0-43ef-ad5a-ef92af653270\" (UID: \"097b66ea-2cf0-43ef-ad5a-ef92af653270\") " Jul 10 00:27:41.441827 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:27:41.441796 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/097b66ea-2cf0-43ef-ad5a-ef92af653270-kube-api-access-fw7x2" (OuterVolumeSpecName: "kube-api-access-fw7x2") pod "097b66ea-2cf0-43ef-ad5a-ef92af653270" (UID: "097b66ea-2cf0-43ef-ad5a-ef92af653270"). InnerVolumeSpecName "kube-api-access-fw7x2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:27:41.540557 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:27:41.540518 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw7x2\" (UniqueName: \"kubernetes.io/projected/097b66ea-2cf0-43ef-ad5a-ef92af653270-kube-api-access-fw7x2\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:27:41.753807 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:27:41.753769 2562 generic.go:356] "Generic (PLEG): container finished" podID="097b66ea-2cf0-43ef-ad5a-ef92af653270" containerID="0a3c257aa931f7cc171d9d056bbc82d2725c4b3b22d37685a23da802a3e74266" exitCode=0 Jul 10 00:27:41.753807 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:27:41.753805 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-t298z" Jul 10 00:27:41.754003 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:27:41.753848 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-t298z" event={"ID":"097b66ea-2cf0-43ef-ad5a-ef92af653270","Type":"ContainerDied","Data":"0a3c257aa931f7cc171d9d056bbc82d2725c4b3b22d37685a23da802a3e74266"} Jul 10 00:27:41.754003 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:27:41.753884 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-t298z" event={"ID":"097b66ea-2cf0-43ef-ad5a-ef92af653270","Type":"ContainerDied","Data":"6b081a815248afc862ae2d006ec9763af6e1df4cf8bb2b81224ea62f54a4f422"} Jul 10 00:27:41.754003 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:27:41.753900 2562 scope.go:117] "RemoveContainer" containerID="0a3c257aa931f7cc171d9d056bbc82d2725c4b3b22d37685a23da802a3e74266" Jul 10 00:27:41.762225 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:27:41.762198 2562 scope.go:117] "RemoveContainer" containerID="0a3c257aa931f7cc171d9d056bbc82d2725c4b3b22d37685a23da802a3e74266" Jul 10 00:27:41.762495 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:27:41.762471 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a3c257aa931f7cc171d9d056bbc82d2725c4b3b22d37685a23da802a3e74266\": container with ID starting with 0a3c257aa931f7cc171d9d056bbc82d2725c4b3b22d37685a23da802a3e74266 not found: ID does not exist" containerID="0a3c257aa931f7cc171d9d056bbc82d2725c4b3b22d37685a23da802a3e74266" Jul 10 00:27:41.762565 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:27:41.762499 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a3c257aa931f7cc171d9d056bbc82d2725c4b3b22d37685a23da802a3e74266"} err="failed to get container status \"0a3c257aa931f7cc171d9d056bbc82d2725c4b3b22d37685a23da802a3e74266\": rpc error: code = NotFound desc = could not find container \"0a3c257aa931f7cc171d9d056bbc82d2725c4b3b22d37685a23da802a3e74266\": container with ID starting with 0a3c257aa931f7cc171d9d056bbc82d2725c4b3b22d37685a23da802a3e74266 not found: ID does not exist" Jul 10 00:27:41.770820 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:27:41.770798 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-t298z"] Jul 10 00:27:41.776717 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:27:41.776695 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-t298z"] Jul 10 00:27:43.735435 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:27:43.735393 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="097b66ea-2cf0-43ef-ad5a-ef92af653270" path="/var/lib/kubelet/pods/097b66ea-2cf0-43ef-ad5a-ef92af653270/volumes" Jul 10 00:28:32.006107 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:28:32.006059 2562 generic.go:356] "Generic (PLEG): container finished" podID="92786366-f55e-4e7b-b107-8cd178d52054" containerID="8417f9336f413e14e7ac2653f9884407b7c30e8de925d6f23fba7657aaaff355" exitCode=0 Jul 10 00:28:32.006500 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:28:32.006135 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lhjrd" event={"ID":"92786366-f55e-4e7b-b107-8cd178d52054","Type":"ContainerDied","Data":"8417f9336f413e14e7ac2653f9884407b7c30e8de925d6f23fba7657aaaff355"} Jul 10 00:28:32.006592 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:28:32.006576 2562 scope.go:117] "RemoveContainer" containerID="8417f9336f413e14e7ac2653f9884407b7c30e8de925d6f23fba7657aaaff355" Jul 10 00:28:32.007594 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:28:32.007562 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:28:32.007645 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:28:32.007623 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:28:32.007679 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:28:32.007656 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:28:33.021516 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:28:33.021462 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lhjrd" event={"ID":"92786366-f55e-4e7b-b107-8cd178d52054","Type":"ContainerStarted","Data":"ef82f5a4d0462d531265dd4d1ebd6858ce9463f7c9757f5e2ed8a1a8a3a57886"} Jul 10 00:28:33.363035 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:28:33.362999 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lhjrd" Jul 10 00:28:33.364236 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:28:33.364214 2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-lhjrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Jul 10 00:28:33.364292 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:28:33.364261 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lhjrd" podUID="92786366-f55e-4e7b-b107-8cd178d52054" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Jul 10 00:28:34.363161 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:28:34.363125 2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-lhjrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Jul 10 00:28:34.363553 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:28:34.363190 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lhjrd" podUID="92786366-f55e-4e7b-b107-8cd178d52054" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Jul 10 00:28:34.400182 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:28:34.400155 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lhjrd" Jul 10 00:28:35.363795 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:28:35.363758 2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-lhjrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Jul 10 00:28:35.364309 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:28:35.363823 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lhjrd" podUID="92786366-f55e-4e7b-b107-8cd178d52054" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Jul 10 00:28:36.363052 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:28:36.363019 2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-lhjrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Jul 10 00:28:36.363236 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:28:36.363069 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lhjrd" podUID="92786366-f55e-4e7b-b107-8cd178d52054" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Jul 10 00:28:37.367672 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:28:37.367645 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lhjrd" Jul 10 00:28:37.368710 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:28:37.368689 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lhjrd" Jul 10 00:29:36.382477 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:36.382298 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-wrxqq"] Jul 10 00:29:36.383072 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:36.382840 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="097b66ea-2cf0-43ef-ad5a-ef92af653270" containerName="authorino" Jul 10 00:29:36.383072 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:36.382858 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="097b66ea-2cf0-43ef-ad5a-ef92af653270" containerName="authorino" Jul 10 00:29:36.383072 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:36.382959 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="097b66ea-2cf0-43ef-ad5a-ef92af653270" containerName="authorino" Jul 10 00:29:36.385538 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:36.385513 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wrxqq" Jul 10 00:29:36.387546 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:36.387524 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-nzpmf\"" Jul 10 00:29:36.387671 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:36.387535 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 10 00:29:36.388337 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:36.388315 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 10 00:29:36.388447 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:36.388388 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 10 00:29:36.394715 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:36.394694 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-wrxqq"] Jul 10 00:29:36.498936 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:36.498903 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3227fbbd-9453-4705-96a2-5d62ff23379d-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wrxqq\" (UID: \"3227fbbd-9453-4705-96a2-5d62ff23379d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wrxqq" Jul 10 00:29:36.499172 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:36.498946 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3227fbbd-9453-4705-96a2-5d62ff23379d-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wrxqq\" (UID: \"3227fbbd-9453-4705-96a2-5d62ff23379d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wrxqq" Jul 10 00:29:36.499172 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:36.499003 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3227fbbd-9453-4705-96a2-5d62ff23379d-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wrxqq\" (UID: \"3227fbbd-9453-4705-96a2-5d62ff23379d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wrxqq" Jul 10 00:29:36.499172 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:36.499134 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3227fbbd-9453-4705-96a2-5d62ff23379d-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wrxqq\" (UID: \"3227fbbd-9453-4705-96a2-5d62ff23379d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wrxqq" Jul 10 00:29:36.499172 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:36.499161 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3227fbbd-9453-4705-96a2-5d62ff23379d-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wrxqq\" (UID: \"3227fbbd-9453-4705-96a2-5d62ff23379d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wrxqq" Jul 10 00:29:36.499379 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:36.499232 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhmbh\" (UniqueName: \"kubernetes.io/projected/3227fbbd-9453-4705-96a2-5d62ff23379d-kube-api-access-vhmbh\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wrxqq\" (UID: \"3227fbbd-9453-4705-96a2-5d62ff23379d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wrxqq" Jul 10 00:29:36.599859 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:36.599823 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhmbh\" (UniqueName: \"kubernetes.io/projected/3227fbbd-9453-4705-96a2-5d62ff23379d-kube-api-access-vhmbh\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wrxqq\" (UID: \"3227fbbd-9453-4705-96a2-5d62ff23379d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wrxqq" Jul 10 00:29:36.600058 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:36.599876 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3227fbbd-9453-4705-96a2-5d62ff23379d-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wrxqq\" (UID: \"3227fbbd-9453-4705-96a2-5d62ff23379d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wrxqq" Jul 10 00:29:36.600058 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:36.599917 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3227fbbd-9453-4705-96a2-5d62ff23379d-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wrxqq\" (UID: \"3227fbbd-9453-4705-96a2-5d62ff23379d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wrxqq" Jul 10 00:29:36.600058 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:36.599953 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3227fbbd-9453-4705-96a2-5d62ff23379d-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wrxqq\" (UID: \"3227fbbd-9453-4705-96a2-5d62ff23379d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wrxqq" Jul 10 00:29:36.600058 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:36.600010 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3227fbbd-9453-4705-96a2-5d62ff23379d-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wrxqq\" (UID: \"3227fbbd-9453-4705-96a2-5d62ff23379d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wrxqq" Jul 10 00:29:36.600058 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:36.600033 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3227fbbd-9453-4705-96a2-5d62ff23379d-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wrxqq\" (UID: \"3227fbbd-9453-4705-96a2-5d62ff23379d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wrxqq" Jul 10 00:29:36.600389 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:36.600368 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3227fbbd-9453-4705-96a2-5d62ff23379d-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wrxqq\" (UID: \"3227fbbd-9453-4705-96a2-5d62ff23379d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wrxqq" Jul 10 00:29:36.600465 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:36.600443 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3227fbbd-9453-4705-96a2-5d62ff23379d-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wrxqq\" (UID: \"3227fbbd-9453-4705-96a2-5d62ff23379d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wrxqq" Jul 10 00:29:36.600507 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:36.600471 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3227fbbd-9453-4705-96a2-5d62ff23379d-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wrxqq\" (UID: \"3227fbbd-9453-4705-96a2-5d62ff23379d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wrxqq" Jul 10 00:29:36.602219 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:36.602190 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3227fbbd-9453-4705-96a2-5d62ff23379d-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wrxqq\" (UID: \"3227fbbd-9453-4705-96a2-5d62ff23379d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wrxqq" Jul 10 00:29:36.602471 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:36.602455 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3227fbbd-9453-4705-96a2-5d62ff23379d-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wrxqq\" (UID: \"3227fbbd-9453-4705-96a2-5d62ff23379d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wrxqq" Jul 10 00:29:36.606722 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:36.606699 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhmbh\" (UniqueName: \"kubernetes.io/projected/3227fbbd-9453-4705-96a2-5d62ff23379d-kube-api-access-vhmbh\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wrxqq\" (UID: \"3227fbbd-9453-4705-96a2-5d62ff23379d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wrxqq" Jul 10 00:29:36.697139 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:36.697046 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wrxqq" Jul 10 00:29:36.825322 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:36.825295 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-wrxqq"] Jul 10 00:29:36.826820 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:29:36.826791 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3227fbbd_9453_4705_96a2_5d62ff23379d.slice/crio-29d8a6ba64d6168f1d30929b7095b10637797454886b3edc331e610acbd21c77 WatchSource:0}: Error finding container 29d8a6ba64d6168f1d30929b7095b10637797454886b3edc331e610acbd21c77: Status 404 returned error can't find the container with id 29d8a6ba64d6168f1d30929b7095b10637797454886b3edc331e610acbd21c77 Jul 10 00:29:37.341143 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:37.341101 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wrxqq" event={"ID":"3227fbbd-9453-4705-96a2-5d62ff23379d","Type":"ContainerStarted","Data":"29d8a6ba64d6168f1d30929b7095b10637797454886b3edc331e610acbd21c77"} Jul 10 00:29:38.347392 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:38.347350 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wrxqq" event={"ID":"3227fbbd-9453-4705-96a2-5d62ff23379d","Type":"ContainerStarted","Data":"618a2d71c74640ba40168c50ef494eaa29fe4093cb6b110f679fc82cbdde47d3"} Jul 10 00:29:38.363643 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:38.363594 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wrxqq" podStartSLOduration=1.3098272340000001 podStartE2EDuration="2.363580833s" totalImagesPullingTime="1.053753599s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:29:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:29:38.362322542 +0000 UTC m=+845.180628063" watchObservedRunningTime="2026-07-10 00:29:38.363580833 +0000 UTC m=+845.181886354" Jul 10 00:29:43.085206 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:43.085171 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-6k2xw"] Jul 10 00:29:43.088591 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:43.088569 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-6k2xw" Jul 10 00:29:43.090550 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:43.090528 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 10 00:29:43.097124 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:43.097098 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-6k2xw"] Jul 10 00:29:43.159860 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:43.159827 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twgps\" (UniqueName: \"kubernetes.io/projected/408be265-42a3-4565-abd5-f133b31f0aec-kube-api-access-twgps\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-6k2xw\" (UID: \"408be265-42a3-4565-abd5-f133b31f0aec\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-6k2xw" Jul 10 00:29:43.159860 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:43.159867 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/408be265-42a3-4565-abd5-f133b31f0aec-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-6k2xw\" (UID: \"408be265-42a3-4565-abd5-f133b31f0aec\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-6k2xw" Jul 10 00:29:43.160100 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:43.159900 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/408be265-42a3-4565-abd5-f133b31f0aec-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-6k2xw\" (UID: \"408be265-42a3-4565-abd5-f133b31f0aec\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-6k2xw" Jul 10 00:29:43.160100 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:43.159944 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/408be265-42a3-4565-abd5-f133b31f0aec-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-6k2xw\" (UID: \"408be265-42a3-4565-abd5-f133b31f0aec\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-6k2xw" Jul 10 00:29:43.160100 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:43.160005 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/408be265-42a3-4565-abd5-f133b31f0aec-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-6k2xw\" (UID: \"408be265-42a3-4565-abd5-f133b31f0aec\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-6k2xw" Jul 10 00:29:43.160100 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:43.160037 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/408be265-42a3-4565-abd5-f133b31f0aec-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-6k2xw\" (UID: \"408be265-42a3-4565-abd5-f133b31f0aec\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-6k2xw" Jul 10 00:29:43.260962 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:43.260881 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/408be265-42a3-4565-abd5-f133b31f0aec-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-6k2xw\" (UID: \"408be265-42a3-4565-abd5-f133b31f0aec\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-6k2xw" Jul 10 00:29:43.260962 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:43.260959 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/408be265-42a3-4565-abd5-f133b31f0aec-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-6k2xw\" (UID: \"408be265-42a3-4565-abd5-f133b31f0aec\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-6k2xw" Jul 10 00:29:43.261268 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:43.261029 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/408be265-42a3-4565-abd5-f133b31f0aec-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-6k2xw\" (UID: \"408be265-42a3-4565-abd5-f133b31f0aec\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-6k2xw" Jul 10 00:29:43.261268 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:43.261070 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/408be265-42a3-4565-abd5-f133b31f0aec-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-6k2xw\" (UID: \"408be265-42a3-4565-abd5-f133b31f0aec\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-6k2xw" Jul 10 00:29:43.261268 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:43.261155 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twgps\" (UniqueName: \"kubernetes.io/projected/408be265-42a3-4565-abd5-f133b31f0aec-kube-api-access-twgps\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-6k2xw\" (UID: \"408be265-42a3-4565-abd5-f133b31f0aec\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-6k2xw" Jul 10 00:29:43.261268 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:43.261185 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/408be265-42a3-4565-abd5-f133b31f0aec-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-6k2xw\" (UID: \"408be265-42a3-4565-abd5-f133b31f0aec\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-6k2xw" Jul 10 00:29:43.261502 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:43.261456 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/408be265-42a3-4565-abd5-f133b31f0aec-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-6k2xw\" (UID: \"408be265-42a3-4565-abd5-f133b31f0aec\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-6k2xw" Jul 10 00:29:43.261897 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:43.261862 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/408be265-42a3-4565-abd5-f133b31f0aec-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-6k2xw\" (UID: \"408be265-42a3-4565-abd5-f133b31f0aec\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-6k2xw" Jul 10 00:29:43.261897 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:43.261880 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/408be265-42a3-4565-abd5-f133b31f0aec-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-6k2xw\" (UID: \"408be265-42a3-4565-abd5-f133b31f0aec\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-6k2xw" Jul 10 00:29:43.263248 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:43.263223 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/408be265-42a3-4565-abd5-f133b31f0aec-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-6k2xw\" (UID: \"408be265-42a3-4565-abd5-f133b31f0aec\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-6k2xw" Jul 10 00:29:43.263787 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:43.263765 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/408be265-42a3-4565-abd5-f133b31f0aec-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-6k2xw\" (UID: \"408be265-42a3-4565-abd5-f133b31f0aec\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-6k2xw" Jul 10 00:29:43.274786 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:43.274763 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twgps\" (UniqueName: \"kubernetes.io/projected/408be265-42a3-4565-abd5-f133b31f0aec-kube-api-access-twgps\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-6k2xw\" (UID: \"408be265-42a3-4565-abd5-f133b31f0aec\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-6k2xw" Jul 10 00:29:43.398420 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:43.398332 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-6k2xw" Jul 10 00:29:43.523731 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:43.523705 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-6k2xw"] Jul 10 00:29:43.524777 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:29:43.524750 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod408be265_42a3_4565_abd5_f133b31f0aec.slice/crio-0cff14e931ecfa9fc1c1c7b21910b2d70d5461b7f9f0c9681e0961ae6333c23d WatchSource:0}: Error finding container 0cff14e931ecfa9fc1c1c7b21910b2d70d5461b7f9f0c9681e0961ae6333c23d: Status 404 returned error can't find the container with id 0cff14e931ecfa9fc1c1c7b21910b2d70d5461b7f9f0c9681e0961ae6333c23d Jul 10 00:29:44.383808 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:44.383767 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-6k2xw" event={"ID":"408be265-42a3-4565-abd5-f133b31f0aec","Type":"ContainerStarted","Data":"b961f022a620ea2316138c3fe5539a958cb099a8ebc3c21137ba9eb6ede3cdfd"} Jul 10 00:29:44.383808 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:44.383815 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-6k2xw" event={"ID":"408be265-42a3-4565-abd5-f133b31f0aec","Type":"ContainerStarted","Data":"0cff14e931ecfa9fc1c1c7b21910b2d70d5461b7f9f0c9681e0961ae6333c23d"} Jul 10 00:29:44.399117 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:44.399046 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-6k2xw" podStartSLOduration=1.242235151 podStartE2EDuration="1.399030582s" totalImagesPullingTime="156.795431ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:29:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:29:44.397818415 +0000 UTC m=+851.216123909" watchObservedRunningTime="2026-07-10 00:29:44.399030582 +0000 UTC m=+851.217336076" Jul 10 00:29:46.697670 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:46.697622 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wrxqq" Jul 10 00:29:46.698141 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:46.697729 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wrxqq" Jul 10 00:29:46.710553 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:46.710524 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wrxqq" Jul 10 00:29:47.412300 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:47.412272 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wrxqq" Jul 10 00:29:53.398632 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:53.398588 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-6k2xw" Jul 10 00:29:53.399066 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:53.398648 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-6k2xw" Jul 10 00:29:53.411102 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:53.411050 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-6k2xw" Jul 10 00:29:53.443068 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:29:53.443043 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-6k2xw" Jul 10 00:30:33.673497 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:30:33.673423 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kjhg9_4651a90d-ef18-4974-9efd-a77fff3c9dfd/console-operator/1.log" Jul 10 00:30:33.675802 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:30:33.675769 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kjhg9_4651a90d-ef18-4974-9efd-a77fff3c9dfd/console-operator/1.log" Jul 10 00:30:33.677940 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:30:33.677918 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h5g9w_f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32/ovn-acl-logging/0.log" Jul 10 00:30:33.679674 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:30:33.679657 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h5g9w_f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32/ovn-acl-logging/0.log" Jul 10 00:30:45.393409 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:30:45.393373 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-9c4cbf5d7-n88zq"] Jul 10 00:30:45.395841 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:30:45.395822 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-9c4cbf5d7-n88zq" Jul 10 00:30:45.397858 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:30:45.397834 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 10 00:30:45.397982 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:30:45.397882 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 10 00:30:45.398469 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:30:45.398448 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 10 00:30:45.398571 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:30:45.398465 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-7kws9\"" Jul 10 00:30:45.404215 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:30:45.404195 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-9c4cbf5d7-n88zq"] Jul 10 00:30:45.494613 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:30:45.494575 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7be3328e-4d6e-4dee-a615-90d6ade4f629-tls-cert\") pod \"authorino-9c4cbf5d7-n88zq\" (UID: \"7be3328e-4d6e-4dee-a615-90d6ade4f629\") " pod="kuadrant-system/authorino-9c4cbf5d7-n88zq" Jul 10 00:30:45.494613 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:30:45.494622 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-859ks\" (UniqueName: \"kubernetes.io/projected/7be3328e-4d6e-4dee-a615-90d6ade4f629-kube-api-access-859ks\") pod \"authorino-9c4cbf5d7-n88zq\" (UID: \"7be3328e-4d6e-4dee-a615-90d6ade4f629\") " pod="kuadrant-system/authorino-9c4cbf5d7-n88zq" Jul 10 00:30:45.595716 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:30:45.595676 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7be3328e-4d6e-4dee-a615-90d6ade4f629-tls-cert\") pod \"authorino-9c4cbf5d7-n88zq\" (UID: \"7be3328e-4d6e-4dee-a615-90d6ade4f629\") " pod="kuadrant-system/authorino-9c4cbf5d7-n88zq" Jul 10 00:30:45.595878 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:30:45.595725 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-859ks\" (UniqueName: \"kubernetes.io/projected/7be3328e-4d6e-4dee-a615-90d6ade4f629-kube-api-access-859ks\") pod \"authorino-9c4cbf5d7-n88zq\" (UID: \"7be3328e-4d6e-4dee-a615-90d6ade4f629\") " pod="kuadrant-system/authorino-9c4cbf5d7-n88zq" Jul 10 00:30:45.598166 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:30:45.598143 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7be3328e-4d6e-4dee-a615-90d6ade4f629-tls-cert\") pod \"authorino-9c4cbf5d7-n88zq\" (UID: \"7be3328e-4d6e-4dee-a615-90d6ade4f629\") " pod="kuadrant-system/authorino-9c4cbf5d7-n88zq" Jul 10 00:30:45.602350 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:30:45.602320 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-859ks\" (UniqueName: \"kubernetes.io/projected/7be3328e-4d6e-4dee-a615-90d6ade4f629-kube-api-access-859ks\") pod \"authorino-9c4cbf5d7-n88zq\" (UID: \"7be3328e-4d6e-4dee-a615-90d6ade4f629\") " pod="kuadrant-system/authorino-9c4cbf5d7-n88zq" Jul 10 00:30:45.706042 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:30:45.705958 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-9c4cbf5d7-n88zq" Jul 10 00:30:45.834439 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:30:45.834415 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-9c4cbf5d7-n88zq"] Jul 10 00:30:45.835980 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:30:45.835952 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7be3328e_4d6e_4dee_a615_90d6ade4f629.slice/crio-5b165bbd3329bacfcd43b6b257ec9575cb26af36edd5b87d4d6c2087fdaf04cf WatchSource:0}: Error finding container 5b165bbd3329bacfcd43b6b257ec9575cb26af36edd5b87d4d6c2087fdaf04cf: Status 404 returned error can't find the container with id 5b165bbd3329bacfcd43b6b257ec9575cb26af36edd5b87d4d6c2087fdaf04cf Jul 10 00:30:46.706654 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:30:46.706611 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-9c4cbf5d7-n88zq" event={"ID":"7be3328e-4d6e-4dee-a615-90d6ade4f629","Type":"ContainerStarted","Data":"eae04581a1c1b22d278f78516cfe0e52e0023d048f0eaaf7e0c7a8bbead312f5"} Jul 10 00:30:46.707220 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:30:46.706668 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-9c4cbf5d7-n88zq" event={"ID":"7be3328e-4d6e-4dee-a615-90d6ade4f629","Type":"ContainerStarted","Data":"5b165bbd3329bacfcd43b6b257ec9575cb26af36edd5b87d4d6c2087fdaf04cf"} Jul 10 00:30:46.721291 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:30:46.721231 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-9c4cbf5d7-n88zq" podStartSLOduration=1.287326056 podStartE2EDuration="1.721212379s" totalImagesPullingTime="433.886323ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:30:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:30:46.720597629 +0000 UTC m=+913.538903126" watchObservedRunningTime="2026-07-10 00:30:46.721212379 +0000 UTC m=+913.539517874" Jul 10 00:35:33.711585 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:35:33.711555 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kjhg9_4651a90d-ef18-4974-9efd-a77fff3c9dfd/console-operator/1.log" Jul 10 00:35:33.714093 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:35:33.714059 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kjhg9_4651a90d-ef18-4974-9efd-a77fff3c9dfd/console-operator/1.log" Jul 10 00:35:33.715271 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:35:33.715249 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h5g9w_f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32/ovn-acl-logging/0.log" Jul 10 00:35:33.717906 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:35:33.717889 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h5g9w_f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32/ovn-acl-logging/0.log" Jul 10 00:40:33.747453 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:40:33.747425 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kjhg9_4651a90d-ef18-4974-9efd-a77fff3c9dfd/console-operator/1.log" Jul 10 00:40:33.750957 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:40:33.750936 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kjhg9_4651a90d-ef18-4974-9efd-a77fff3c9dfd/console-operator/1.log" Jul 10 00:40:33.751310 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:40:33.751293 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h5g9w_f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32/ovn-acl-logging/0.log" Jul 10 00:40:33.754372 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:40:33.754353 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h5g9w_f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32/ovn-acl-logging/0.log" Jul 10 00:44:05.318971 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:05.318938 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-95438d51-openshift-default-64655477f7-xhxvm"] Jul 10 00:44:05.321492 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:05.321469 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-95438d51-openshift-default-64655477f7-xhxvm" Jul 10 00:44:05.323544 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:05.323523 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-95438d51-openshift-default-dockercfg-t8p8j\"" Jul 10 00:44:05.344138 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:05.344107 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-95438d51-openshift-default-64655477f7-xhxvm"] Jul 10 00:44:05.411524 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:05.411484 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f883386-cb0c-4d2b-b1aa-dbdef14944e6-istio-envoy\") pod \"e2e-ait-95438d51-openshift-default-64655477f7-xhxvm\" (UID: \"1f883386-cb0c-4d2b-b1aa-dbdef14944e6\") " pod="openshift-ingress/e2e-ait-95438d51-openshift-default-64655477f7-xhxvm" Jul 10 00:44:05.411524 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:05.411526 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f883386-cb0c-4d2b-b1aa-dbdef14944e6-workload-certs\") pod \"e2e-ait-95438d51-openshift-default-64655477f7-xhxvm\" (UID: \"1f883386-cb0c-4d2b-b1aa-dbdef14944e6\") " pod="openshift-ingress/e2e-ait-95438d51-openshift-default-64655477f7-xhxvm" Jul 10 00:44:05.411754 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:05.411568 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f883386-cb0c-4d2b-b1aa-dbdef14944e6-istio-token\") pod \"e2e-ait-95438d51-openshift-default-64655477f7-xhxvm\" (UID: \"1f883386-cb0c-4d2b-b1aa-dbdef14944e6\") " pod="openshift-ingress/e2e-ait-95438d51-openshift-default-64655477f7-xhxvm" Jul 10 00:44:05.411754 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:05.411601 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f883386-cb0c-4d2b-b1aa-dbdef14944e6-istiod-ca-cert\") pod \"e2e-ait-95438d51-openshift-default-64655477f7-xhxvm\" (UID: \"1f883386-cb0c-4d2b-b1aa-dbdef14944e6\") " pod="openshift-ingress/e2e-ait-95438d51-openshift-default-64655477f7-xhxvm" Jul 10 00:44:05.411754 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:05.411631 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f883386-cb0c-4d2b-b1aa-dbdef14944e6-credential-socket\") pod \"e2e-ait-95438d51-openshift-default-64655477f7-xhxvm\" (UID: \"1f883386-cb0c-4d2b-b1aa-dbdef14944e6\") " pod="openshift-ingress/e2e-ait-95438d51-openshift-default-64655477f7-xhxvm" Jul 10 00:44:05.411754 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:05.411672 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f883386-cb0c-4d2b-b1aa-dbdef14944e6-istio-data\") pod \"e2e-ait-95438d51-openshift-default-64655477f7-xhxvm\" (UID: \"1f883386-cb0c-4d2b-b1aa-dbdef14944e6\") " pod="openshift-ingress/e2e-ait-95438d51-openshift-default-64655477f7-xhxvm" Jul 10 00:44:05.411754 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:05.411733 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f883386-cb0c-4d2b-b1aa-dbdef14944e6-workload-socket\") pod \"e2e-ait-95438d51-openshift-default-64655477f7-xhxvm\" (UID: \"1f883386-cb0c-4d2b-b1aa-dbdef14944e6\") " pod="openshift-ingress/e2e-ait-95438d51-openshift-default-64655477f7-xhxvm" Jul 10 00:44:05.411911 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:05.411782 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bz2w\" (UniqueName: \"kubernetes.io/projected/1f883386-cb0c-4d2b-b1aa-dbdef14944e6-kube-api-access-2bz2w\") pod \"e2e-ait-95438d51-openshift-default-64655477f7-xhxvm\" (UID: \"1f883386-cb0c-4d2b-b1aa-dbdef14944e6\") " pod="openshift-ingress/e2e-ait-95438d51-openshift-default-64655477f7-xhxvm" Jul 10 00:44:05.411911 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:05.411815 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f883386-cb0c-4d2b-b1aa-dbdef14944e6-istio-podinfo\") pod \"e2e-ait-95438d51-openshift-default-64655477f7-xhxvm\" (UID: \"1f883386-cb0c-4d2b-b1aa-dbdef14944e6\") " pod="openshift-ingress/e2e-ait-95438d51-openshift-default-64655477f7-xhxvm" Jul 10 00:44:05.512484 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:05.512440 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f883386-cb0c-4d2b-b1aa-dbdef14944e6-istio-envoy\") pod \"e2e-ait-95438d51-openshift-default-64655477f7-xhxvm\" (UID: \"1f883386-cb0c-4d2b-b1aa-dbdef14944e6\") " pod="openshift-ingress/e2e-ait-95438d51-openshift-default-64655477f7-xhxvm" Jul 10 00:44:05.512484 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:05.512484 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f883386-cb0c-4d2b-b1aa-dbdef14944e6-workload-certs\") pod \"e2e-ait-95438d51-openshift-default-64655477f7-xhxvm\" (UID: \"1f883386-cb0c-4d2b-b1aa-dbdef14944e6\") " pod="openshift-ingress/e2e-ait-95438d51-openshift-default-64655477f7-xhxvm" Jul 10 00:44:05.512755 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:05.512504 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f883386-cb0c-4d2b-b1aa-dbdef14944e6-istio-token\") pod \"e2e-ait-95438d51-openshift-default-64655477f7-xhxvm\" (UID: \"1f883386-cb0c-4d2b-b1aa-dbdef14944e6\") " pod="openshift-ingress/e2e-ait-95438d51-openshift-default-64655477f7-xhxvm" Jul 10 00:44:05.512755 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:05.512529 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f883386-cb0c-4d2b-b1aa-dbdef14944e6-istiod-ca-cert\") pod \"e2e-ait-95438d51-openshift-default-64655477f7-xhxvm\" (UID: \"1f883386-cb0c-4d2b-b1aa-dbdef14944e6\") " pod="openshift-ingress/e2e-ait-95438d51-openshift-default-64655477f7-xhxvm" Jul 10 00:44:05.512755 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:05.512551 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f883386-cb0c-4d2b-b1aa-dbdef14944e6-credential-socket\") pod \"e2e-ait-95438d51-openshift-default-64655477f7-xhxvm\" (UID: \"1f883386-cb0c-4d2b-b1aa-dbdef14944e6\") " pod="openshift-ingress/e2e-ait-95438d51-openshift-default-64655477f7-xhxvm" Jul 10 00:44:05.512755 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:05.512566 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f883386-cb0c-4d2b-b1aa-dbdef14944e6-istio-data\") pod \"e2e-ait-95438d51-openshift-default-64655477f7-xhxvm\" (UID: \"1f883386-cb0c-4d2b-b1aa-dbdef14944e6\") " pod="openshift-ingress/e2e-ait-95438d51-openshift-default-64655477f7-xhxvm" Jul 10 00:44:05.512755 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:05.512588 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f883386-cb0c-4d2b-b1aa-dbdef14944e6-workload-socket\") pod \"e2e-ait-95438d51-openshift-default-64655477f7-xhxvm\" (UID: \"1f883386-cb0c-4d2b-b1aa-dbdef14944e6\") " pod="openshift-ingress/e2e-ait-95438d51-openshift-default-64655477f7-xhxvm" Jul 10 00:44:05.512755 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:05.512629 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bz2w\" (UniqueName: \"kubernetes.io/projected/1f883386-cb0c-4d2b-b1aa-dbdef14944e6-kube-api-access-2bz2w\") pod \"e2e-ait-95438d51-openshift-default-64655477f7-xhxvm\" (UID: \"1f883386-cb0c-4d2b-b1aa-dbdef14944e6\") " pod="openshift-ingress/e2e-ait-95438d51-openshift-default-64655477f7-xhxvm" Jul 10 00:44:05.512755 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:05.512655 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f883386-cb0c-4d2b-b1aa-dbdef14944e6-istio-podinfo\") pod \"e2e-ait-95438d51-openshift-default-64655477f7-xhxvm\" (UID: \"1f883386-cb0c-4d2b-b1aa-dbdef14944e6\") " pod="openshift-ingress/e2e-ait-95438d51-openshift-default-64655477f7-xhxvm" Jul 10 00:44:05.513128 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:05.512998 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f883386-cb0c-4d2b-b1aa-dbdef14944e6-workload-socket\") pod \"e2e-ait-95438d51-openshift-default-64655477f7-xhxvm\" (UID: \"1f883386-cb0c-4d2b-b1aa-dbdef14944e6\") " pod="openshift-ingress/e2e-ait-95438d51-openshift-default-64655477f7-xhxvm" Jul 10 00:44:05.513128 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:05.513027 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f883386-cb0c-4d2b-b1aa-dbdef14944e6-credential-socket\") pod \"e2e-ait-95438d51-openshift-default-64655477f7-xhxvm\" (UID: \"1f883386-cb0c-4d2b-b1aa-dbdef14944e6\") " pod="openshift-ingress/e2e-ait-95438d51-openshift-default-64655477f7-xhxvm" Jul 10 00:44:05.513128 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:05.513057 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f883386-cb0c-4d2b-b1aa-dbdef14944e6-workload-certs\") pod \"e2e-ait-95438d51-openshift-default-64655477f7-xhxvm\" (UID: \"1f883386-cb0c-4d2b-b1aa-dbdef14944e6\") " pod="openshift-ingress/e2e-ait-95438d51-openshift-default-64655477f7-xhxvm" Jul 10 00:44:05.513286 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:05.513166 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f883386-cb0c-4d2b-b1aa-dbdef14944e6-istio-data\") pod \"e2e-ait-95438d51-openshift-default-64655477f7-xhxvm\" (UID: \"1f883386-cb0c-4d2b-b1aa-dbdef14944e6\") " pod="openshift-ingress/e2e-ait-95438d51-openshift-default-64655477f7-xhxvm" Jul 10 00:44:05.513482 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:05.513462 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f883386-cb0c-4d2b-b1aa-dbdef14944e6-istiod-ca-cert\") pod \"e2e-ait-95438d51-openshift-default-64655477f7-xhxvm\" (UID: \"1f883386-cb0c-4d2b-b1aa-dbdef14944e6\") " pod="openshift-ingress/e2e-ait-95438d51-openshift-default-64655477f7-xhxvm" Jul 10 00:44:05.514811 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:05.514787 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f883386-cb0c-4d2b-b1aa-dbdef14944e6-istio-envoy\") pod \"e2e-ait-95438d51-openshift-default-64655477f7-xhxvm\" (UID: \"1f883386-cb0c-4d2b-b1aa-dbdef14944e6\") " pod="openshift-ingress/e2e-ait-95438d51-openshift-default-64655477f7-xhxvm" Jul 10 00:44:05.515034 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:05.515016 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f883386-cb0c-4d2b-b1aa-dbdef14944e6-istio-podinfo\") pod \"e2e-ait-95438d51-openshift-default-64655477f7-xhxvm\" (UID: \"1f883386-cb0c-4d2b-b1aa-dbdef14944e6\") " pod="openshift-ingress/e2e-ait-95438d51-openshift-default-64655477f7-xhxvm" Jul 10 00:44:05.519610 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:05.519586 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f883386-cb0c-4d2b-b1aa-dbdef14944e6-istio-token\") pod \"e2e-ait-95438d51-openshift-default-64655477f7-xhxvm\" (UID: \"1f883386-cb0c-4d2b-b1aa-dbdef14944e6\") " pod="openshift-ingress/e2e-ait-95438d51-openshift-default-64655477f7-xhxvm" Jul 10 00:44:05.519726 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:05.519615 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bz2w\" (UniqueName: \"kubernetes.io/projected/1f883386-cb0c-4d2b-b1aa-dbdef14944e6-kube-api-access-2bz2w\") pod \"e2e-ait-95438d51-openshift-default-64655477f7-xhxvm\" (UID: \"1f883386-cb0c-4d2b-b1aa-dbdef14944e6\") " pod="openshift-ingress/e2e-ait-95438d51-openshift-default-64655477f7-xhxvm" Jul 10 00:44:05.632884 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:05.632777 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-95438d51-openshift-default-64655477f7-xhxvm" Jul 10 00:44:05.763694 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:05.763671 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-95438d51-openshift-default-64655477f7-xhxvm"] Jul 10 00:44:05.766090 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:44:05.766050 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f883386_cb0c_4d2b_b1aa_dbdef14944e6.slice/crio-5120cb5a2398840920171dfd7ffa984955d803e183001b6e441860693b07f1d8 WatchSource:0}: Error finding container 5120cb5a2398840920171dfd7ffa984955d803e183001b6e441860693b07f1d8: Status 404 returned error can't find the container with id 5120cb5a2398840920171dfd7ffa984955d803e183001b6e441860693b07f1d8 Jul 10 00:44:05.768480 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:05.768448 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:44:05.768533 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:05.768506 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:44:05.768570 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:05.768541 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:44:05.870376 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:05.870338 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-95438d51-openshift-default-64655477f7-xhxvm" event={"ID":"1f883386-cb0c-4d2b-b1aa-dbdef14944e6","Type":"ContainerStarted","Data":"159aa2d148d618abae342e90bbb42144a6a807ea1221f14dcb476f78eeb94d8f"} Jul 10 00:44:05.870573 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:05.870408 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-95438d51-openshift-default-64655477f7-xhxvm" event={"ID":"1f883386-cb0c-4d2b-b1aa-dbdef14944e6","Type":"ContainerStarted","Data":"5120cb5a2398840920171dfd7ffa984955d803e183001b6e441860693b07f1d8"} Jul 10 00:44:05.889072 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:05.888977 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-95438d51-openshift-default-64655477f7-xhxvm" podStartSLOduration=0.888962345 podStartE2EDuration="888.962345ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:44:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:44:05.886918417 +0000 UTC m=+1712.705223945" watchObservedRunningTime="2026-07-10 00:44:05.888962345 +0000 UTC m=+1712.707267839" Jul 10 00:44:06.632956 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:06.632916 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-95438d51-openshift-default-64655477f7-xhxvm" Jul 10 00:44:06.638303 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:06.638276 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-95438d51-openshift-default-64655477f7-xhxvm" Jul 10 00:44:06.875457 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:06.875427 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-95438d51-openshift-default-64655477f7-xhxvm" Jul 10 00:44:06.876444 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:06.876419 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-95438d51-openshift-default-64655477f7-xhxvm" Jul 10 00:44:11.816366 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:11.816312 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-95438d51-openshift-default-64655477f7-xhxvm"] Jul 10 00:44:11.817541 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:11.817507 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-95438d51-openshift-default-64655477f7-xhxvm" podUID="1f883386-cb0c-4d2b-b1aa-dbdef14944e6" containerName="istio-proxy" containerID="cri-o://159aa2d148d618abae342e90bbb42144a6a807ea1221f14dcb476f78eeb94d8f" gracePeriod=30 Jul 10 00:44:17.952979 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:17.952945 2562 generic.go:356] "Generic (PLEG): container finished" podID="1f883386-cb0c-4d2b-b1aa-dbdef14944e6" containerID="159aa2d148d618abae342e90bbb42144a6a807ea1221f14dcb476f78eeb94d8f" exitCode=0 Jul 10 00:44:17.953407 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:17.953017 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-95438d51-openshift-default-64655477f7-xhxvm" event={"ID":"1f883386-cb0c-4d2b-b1aa-dbdef14944e6","Type":"ContainerDied","Data":"159aa2d148d618abae342e90bbb42144a6a807ea1221f14dcb476f78eeb94d8f"} Jul 10 00:44:18.068210 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:18.068184 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-95438d51-openshift-default-64655477f7-xhxvm" Jul 10 00:44:18.234260 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:18.234175 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f883386-cb0c-4d2b-b1aa-dbdef14944e6-workload-socket\") pod \"1f883386-cb0c-4d2b-b1aa-dbdef14944e6\" (UID: \"1f883386-cb0c-4d2b-b1aa-dbdef14944e6\") " Jul 10 00:44:18.234260 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:18.234220 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f883386-cb0c-4d2b-b1aa-dbdef14944e6-credential-socket\") pod \"1f883386-cb0c-4d2b-b1aa-dbdef14944e6\" (UID: \"1f883386-cb0c-4d2b-b1aa-dbdef14944e6\") " Jul 10 00:44:18.234260 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:18.234260 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f883386-cb0c-4d2b-b1aa-dbdef14944e6-istio-envoy\") pod \"1f883386-cb0c-4d2b-b1aa-dbdef14944e6\" (UID: \"1f883386-cb0c-4d2b-b1aa-dbdef14944e6\") " Jul 10 00:44:18.234966 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:18.234281 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f883386-cb0c-4d2b-b1aa-dbdef14944e6-istio-token\") pod \"1f883386-cb0c-4d2b-b1aa-dbdef14944e6\" (UID: \"1f883386-cb0c-4d2b-b1aa-dbdef14944e6\") " Jul 10 00:44:18.234966 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:18.234296 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f883386-cb0c-4d2b-b1aa-dbdef14944e6-istio-data\") pod \"1f883386-cb0c-4d2b-b1aa-dbdef14944e6\" (UID: \"1f883386-cb0c-4d2b-b1aa-dbdef14944e6\") " Jul 10 00:44:18.234966 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:18.234326 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2bz2w\" (UniqueName: \"kubernetes.io/projected/1f883386-cb0c-4d2b-b1aa-dbdef14944e6-kube-api-access-2bz2w\") pod \"1f883386-cb0c-4d2b-b1aa-dbdef14944e6\" (UID: \"1f883386-cb0c-4d2b-b1aa-dbdef14944e6\") " Jul 10 00:44:18.234966 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:18.234388 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f883386-cb0c-4d2b-b1aa-dbdef14944e6-istiod-ca-cert\") pod \"1f883386-cb0c-4d2b-b1aa-dbdef14944e6\" (UID: \"1f883386-cb0c-4d2b-b1aa-dbdef14944e6\") " Jul 10 00:44:18.234966 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:18.234418 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f883386-cb0c-4d2b-b1aa-dbdef14944e6-workload-certs\") pod \"1f883386-cb0c-4d2b-b1aa-dbdef14944e6\" (UID: \"1f883386-cb0c-4d2b-b1aa-dbdef14944e6\") " Jul 10 00:44:18.234966 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:18.234440 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f883386-cb0c-4d2b-b1aa-dbdef14944e6-istio-podinfo\") pod \"1f883386-cb0c-4d2b-b1aa-dbdef14944e6\" (UID: \"1f883386-cb0c-4d2b-b1aa-dbdef14944e6\") " Jul 10 00:44:18.234966 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:18.234499 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f883386-cb0c-4d2b-b1aa-dbdef14944e6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1f883386-cb0c-4d2b-b1aa-dbdef14944e6" (UID: "1f883386-cb0c-4d2b-b1aa-dbdef14944e6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:44:18.234966 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:18.234580 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f883386-cb0c-4d2b-b1aa-dbdef14944e6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1f883386-cb0c-4d2b-b1aa-dbdef14944e6" (UID: "1f883386-cb0c-4d2b-b1aa-dbdef14944e6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:44:18.234966 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:18.234778 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f883386-cb0c-4d2b-b1aa-dbdef14944e6-workload-socket\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:44:18.234966 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:18.234783 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f883386-cb0c-4d2b-b1aa-dbdef14944e6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1f883386-cb0c-4d2b-b1aa-dbdef14944e6" (UID: "1f883386-cb0c-4d2b-b1aa-dbdef14944e6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:44:18.234966 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:18.234797 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f883386-cb0c-4d2b-b1aa-dbdef14944e6-credential-socket\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:44:18.234966 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:18.234859 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f883386-cb0c-4d2b-b1aa-dbdef14944e6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1f883386-cb0c-4d2b-b1aa-dbdef14944e6" (UID: "1f883386-cb0c-4d2b-b1aa-dbdef14944e6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:44:18.235673 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:18.235287 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f883386-cb0c-4d2b-b1aa-dbdef14944e6-istio-data" (OuterVolumeSpecName: "istio-data") pod "1f883386-cb0c-4d2b-b1aa-dbdef14944e6" (UID: "1f883386-cb0c-4d2b-b1aa-dbdef14944e6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:44:18.236625 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:18.236604 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f883386-cb0c-4d2b-b1aa-dbdef14944e6-istio-token" (OuterVolumeSpecName: "istio-token") pod "1f883386-cb0c-4d2b-b1aa-dbdef14944e6" (UID: "1f883386-cb0c-4d2b-b1aa-dbdef14944e6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:44:18.236706 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:18.236687 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f883386-cb0c-4d2b-b1aa-dbdef14944e6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1f883386-cb0c-4d2b-b1aa-dbdef14944e6" (UID: "1f883386-cb0c-4d2b-b1aa-dbdef14944e6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 00:44:18.236869 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:18.236834 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f883386-cb0c-4d2b-b1aa-dbdef14944e6-kube-api-access-2bz2w" (OuterVolumeSpecName: "kube-api-access-2bz2w") pod "1f883386-cb0c-4d2b-b1aa-dbdef14944e6" (UID: "1f883386-cb0c-4d2b-b1aa-dbdef14944e6"). InnerVolumeSpecName "kube-api-access-2bz2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:44:18.236964 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:18.236901 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f883386-cb0c-4d2b-b1aa-dbdef14944e6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1f883386-cb0c-4d2b-b1aa-dbdef14944e6" (UID: "1f883386-cb0c-4d2b-b1aa-dbdef14944e6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:44:18.336287 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:18.336251 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f883386-cb0c-4d2b-b1aa-dbdef14944e6-istio-envoy\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:44:18.336287 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:18.336279 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f883386-cb0c-4d2b-b1aa-dbdef14944e6-istio-token\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:44:18.336287 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:18.336288 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f883386-cb0c-4d2b-b1aa-dbdef14944e6-istio-data\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:44:18.336520 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:18.336316 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bz2w\" (UniqueName: \"kubernetes.io/projected/1f883386-cb0c-4d2b-b1aa-dbdef14944e6-kube-api-access-2bz2w\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:44:18.336520 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:18.336326 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f883386-cb0c-4d2b-b1aa-dbdef14944e6-istiod-ca-cert\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:44:18.336520 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:18.336335 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f883386-cb0c-4d2b-b1aa-dbdef14944e6-workload-certs\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:44:18.336520 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:18.336343 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f883386-cb0c-4d2b-b1aa-dbdef14944e6-istio-podinfo\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:44:18.959890 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:18.959852 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-95438d51-openshift-default-64655477f7-xhxvm" Jul 10 00:44:18.960359 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:18.959858 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-95438d51-openshift-default-64655477f7-xhxvm" event={"ID":"1f883386-cb0c-4d2b-b1aa-dbdef14944e6","Type":"ContainerDied","Data":"5120cb5a2398840920171dfd7ffa984955d803e183001b6e441860693b07f1d8"} Jul 10 00:44:18.960359 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:18.960003 2562 scope.go:117] "RemoveContainer" containerID="159aa2d148d618abae342e90bbb42144a6a807ea1221f14dcb476f78eeb94d8f" Jul 10 00:44:18.981131 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:18.981108 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-95438d51-openshift-default-64655477f7-xhxvm"] Jul 10 00:44:18.987251 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:18.987226 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-95438d51-openshift-default-64655477f7-xhxvm"] Jul 10 00:44:19.735061 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:19.735027 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f883386-cb0c-4d2b-b1aa-dbdef14944e6" path="/var/lib/kubelet/pods/1f883386-cb0c-4d2b-b1aa-dbdef14944e6/volumes" Jul 10 00:44:36.459326 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:36.459285 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v"] Jul 10 00:44:36.462734 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:36.460247 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f883386-cb0c-4d2b-b1aa-dbdef14944e6" containerName="istio-proxy" Jul 10 00:44:36.462734 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:36.460273 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f883386-cb0c-4d2b-b1aa-dbdef14944e6" containerName="istio-proxy" Jul 10 00:44:36.462734 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:36.460509 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f883386-cb0c-4d2b-b1aa-dbdef14944e6" containerName="istio-proxy" Jul 10 00:44:36.463855 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:36.463816 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v" Jul 10 00:44:36.466728 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:36.466705 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-055f1f02-openshift-default-dockercfg-zpx77\"" Jul 10 00:44:36.475217 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:36.475195 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v"] Jul 10 00:44:36.504688 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:36.504653 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a866f6f-9374-4ff8-992f-b72b985b1f09-istio-data\") pod \"e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v\" (UID: \"5a866f6f-9374-4ff8-992f-b72b985b1f09\") " pod="openshift-ingress/e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v" Jul 10 00:44:36.504827 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:36.504714 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a866f6f-9374-4ff8-992f-b72b985b1f09-istio-token\") pod \"e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v\" (UID: \"5a866f6f-9374-4ff8-992f-b72b985b1f09\") " pod="openshift-ingress/e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v" Jul 10 00:44:36.504827 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:36.504771 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxgd7\" (UniqueName: \"kubernetes.io/projected/5a866f6f-9374-4ff8-992f-b72b985b1f09-kube-api-access-sxgd7\") pod \"e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v\" (UID: \"5a866f6f-9374-4ff8-992f-b72b985b1f09\") " pod="openshift-ingress/e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v" Jul 10 00:44:36.504827 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:36.504807 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a866f6f-9374-4ff8-992f-b72b985b1f09-istio-podinfo\") pod \"e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v\" (UID: \"5a866f6f-9374-4ff8-992f-b72b985b1f09\") " pod="openshift-ingress/e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v" Jul 10 00:44:36.504827 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:36.504823 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a866f6f-9374-4ff8-992f-b72b985b1f09-istiod-ca-cert\") pod \"e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v\" (UID: \"5a866f6f-9374-4ff8-992f-b72b985b1f09\") " pod="openshift-ingress/e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v" Jul 10 00:44:36.505000 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:36.504865 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a866f6f-9374-4ff8-992f-b72b985b1f09-workload-socket\") pod \"e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v\" (UID: \"5a866f6f-9374-4ff8-992f-b72b985b1f09\") " pod="openshift-ingress/e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v" Jul 10 00:44:36.505000 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:36.504922 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a866f6f-9374-4ff8-992f-b72b985b1f09-istio-envoy\") pod \"e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v\" (UID: \"5a866f6f-9374-4ff8-992f-b72b985b1f09\") " pod="openshift-ingress/e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v" Jul 10 00:44:36.505069 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:36.505004 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a866f6f-9374-4ff8-992f-b72b985b1f09-workload-certs\") pod \"e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v\" (UID: \"5a866f6f-9374-4ff8-992f-b72b985b1f09\") " pod="openshift-ingress/e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v" Jul 10 00:44:36.505069 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:36.505032 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a866f6f-9374-4ff8-992f-b72b985b1f09-credential-socket\") pod \"e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v\" (UID: \"5a866f6f-9374-4ff8-992f-b72b985b1f09\") " pod="openshift-ingress/e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v" Jul 10 00:44:36.605539 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:36.605500 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a866f6f-9374-4ff8-992f-b72b985b1f09-workload-certs\") pod \"e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v\" (UID: \"5a866f6f-9374-4ff8-992f-b72b985b1f09\") " pod="openshift-ingress/e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v" Jul 10 00:44:36.605539 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:36.605544 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a866f6f-9374-4ff8-992f-b72b985b1f09-credential-socket\") pod \"e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v\" (UID: \"5a866f6f-9374-4ff8-992f-b72b985b1f09\") " pod="openshift-ingress/e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v" Jul 10 00:44:36.605795 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:36.605582 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a866f6f-9374-4ff8-992f-b72b985b1f09-istio-data\") pod \"e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v\" (UID: \"5a866f6f-9374-4ff8-992f-b72b985b1f09\") " pod="openshift-ingress/e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v" Jul 10 00:44:36.605795 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:36.605616 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a866f6f-9374-4ff8-992f-b72b985b1f09-istio-token\") pod \"e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v\" (UID: \"5a866f6f-9374-4ff8-992f-b72b985b1f09\") " pod="openshift-ingress/e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v" Jul 10 00:44:36.605795 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:36.605638 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxgd7\" (UniqueName: \"kubernetes.io/projected/5a866f6f-9374-4ff8-992f-b72b985b1f09-kube-api-access-sxgd7\") pod \"e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v\" (UID: \"5a866f6f-9374-4ff8-992f-b72b985b1f09\") " pod="openshift-ingress/e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v" Jul 10 00:44:36.605795 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:36.605667 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a866f6f-9374-4ff8-992f-b72b985b1f09-istio-podinfo\") pod \"e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v\" (UID: \"5a866f6f-9374-4ff8-992f-b72b985b1f09\") " pod="openshift-ingress/e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v" Jul 10 00:44:36.605795 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:36.605693 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a866f6f-9374-4ff8-992f-b72b985b1f09-istiod-ca-cert\") pod \"e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v\" (UID: \"5a866f6f-9374-4ff8-992f-b72b985b1f09\") " pod="openshift-ingress/e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v" Jul 10 00:44:36.605795 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:36.605771 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a866f6f-9374-4ff8-992f-b72b985b1f09-workload-socket\") pod \"e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v\" (UID: \"5a866f6f-9374-4ff8-992f-b72b985b1f09\") " pod="openshift-ingress/e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v" Jul 10 00:44:36.606220 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:36.605817 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a866f6f-9374-4ff8-992f-b72b985b1f09-istio-envoy\") pod \"e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v\" (UID: \"5a866f6f-9374-4ff8-992f-b72b985b1f09\") " pod="openshift-ingress/e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v" Jul 10 00:44:36.606220 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:36.605947 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a866f6f-9374-4ff8-992f-b72b985b1f09-workload-certs\") pod \"e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v\" (UID: \"5a866f6f-9374-4ff8-992f-b72b985b1f09\") " pod="openshift-ingress/e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v" Jul 10 00:44:36.606220 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:36.606110 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a866f6f-9374-4ff8-992f-b72b985b1f09-credential-socket\") pod \"e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v\" (UID: \"5a866f6f-9374-4ff8-992f-b72b985b1f09\") " pod="openshift-ingress/e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v" Jul 10 00:44:36.606220 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:36.606171 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a866f6f-9374-4ff8-992f-b72b985b1f09-workload-socket\") pod \"e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v\" (UID: \"5a866f6f-9374-4ff8-992f-b72b985b1f09\") " pod="openshift-ingress/e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v" Jul 10 00:44:36.606387 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:36.606340 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a866f6f-9374-4ff8-992f-b72b985b1f09-istio-data\") pod \"e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v\" (UID: \"5a866f6f-9374-4ff8-992f-b72b985b1f09\") " pod="openshift-ingress/e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v" Jul 10 00:44:36.606483 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:36.606459 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a866f6f-9374-4ff8-992f-b72b985b1f09-istiod-ca-cert\") pod \"e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v\" (UID: \"5a866f6f-9374-4ff8-992f-b72b985b1f09\") " pod="openshift-ingress/e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v" Jul 10 00:44:36.608091 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:36.608059 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a866f6f-9374-4ff8-992f-b72b985b1f09-istio-envoy\") pod \"e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v\" (UID: \"5a866f6f-9374-4ff8-992f-b72b985b1f09\") " pod="openshift-ingress/e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v" Jul 10 00:44:36.608409 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:36.608388 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a866f6f-9374-4ff8-992f-b72b985b1f09-istio-podinfo\") pod \"e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v\" (UID: \"5a866f6f-9374-4ff8-992f-b72b985b1f09\") " pod="openshift-ingress/e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v" Jul 10 00:44:36.612522 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:36.612503 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a866f6f-9374-4ff8-992f-b72b985b1f09-istio-token\") pod \"e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v\" (UID: \"5a866f6f-9374-4ff8-992f-b72b985b1f09\") " pod="openshift-ingress/e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v" Jul 10 00:44:36.612821 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:36.612791 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxgd7\" (UniqueName: \"kubernetes.io/projected/5a866f6f-9374-4ff8-992f-b72b985b1f09-kube-api-access-sxgd7\") pod \"e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v\" (UID: \"5a866f6f-9374-4ff8-992f-b72b985b1f09\") " pod="openshift-ingress/e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v" Jul 10 00:44:36.777622 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:36.777521 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v" Jul 10 00:44:36.908787 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:36.908760 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v"] Jul 10 00:44:36.909890 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:44:36.909866 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a866f6f_9374_4ff8_992f_b72b985b1f09.slice/crio-f65131ad8aeccdb49ec593d472d971e03de5a9f7e6c4d52d566f7842d3ed3e61 WatchSource:0}: Error finding container f65131ad8aeccdb49ec593d472d971e03de5a9f7e6c4d52d566f7842d3ed3e61: Status 404 returned error can't find the container with id f65131ad8aeccdb49ec593d472d971e03de5a9f7e6c4d52d566f7842d3ed3e61 Jul 10 00:44:36.911947 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:36.911914 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:44:36.912041 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:36.911975 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:44:36.912041 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:36.912003 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:44:37.055017 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:37.054974 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v" event={"ID":"5a866f6f-9374-4ff8-992f-b72b985b1f09","Type":"ContainerStarted","Data":"8fd7bed37dbcdcfea25e00f01866f1c4adb1c9dfb926b16c28d8fe826c6a3cb6"} Jul 10 00:44:37.055253 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:37.055037 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v" event={"ID":"5a866f6f-9374-4ff8-992f-b72b985b1f09","Type":"ContainerStarted","Data":"f65131ad8aeccdb49ec593d472d971e03de5a9f7e6c4d52d566f7842d3ed3e61"} Jul 10 00:44:37.072620 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:37.072560 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v" podStartSLOduration=1.072541861 podStartE2EDuration="1.072541861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:44:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:44:37.071539198 +0000 UTC m=+1743.889844692" watchObservedRunningTime="2026-07-10 00:44:37.072541861 +0000 UTC m=+1743.890847357" Jul 10 00:44:37.777724 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:37.777689 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v" Jul 10 00:44:37.783804 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:37.783778 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v" Jul 10 00:44:38.060244 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:38.060211 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v" Jul 10 00:44:38.061185 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:38.061163 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v" Jul 10 00:44:42.233429 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:42.233394 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v"] Jul 10 00:44:42.233822 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:42.233570 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v" podUID="5a866f6f-9374-4ff8-992f-b72b985b1f09" containerName="istio-proxy" containerID="cri-o://8fd7bed37dbcdcfea25e00f01866f1c4adb1c9dfb926b16c28d8fe826c6a3cb6" gracePeriod=30 Jul 10 00:44:46.882963 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:46.882909 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v" Jul 10 00:44:47.009487 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:47.009399 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a866f6f-9374-4ff8-992f-b72b985b1f09-istio-envoy\") pod \"5a866f6f-9374-4ff8-992f-b72b985b1f09\" (UID: \"5a866f6f-9374-4ff8-992f-b72b985b1f09\") " Jul 10 00:44:47.009487 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:47.009468 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a866f6f-9374-4ff8-992f-b72b985b1f09-workload-certs\") pod \"5a866f6f-9374-4ff8-992f-b72b985b1f09\" (UID: \"5a866f6f-9374-4ff8-992f-b72b985b1f09\") " Jul 10 00:44:47.009725 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:47.009512 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a866f6f-9374-4ff8-992f-b72b985b1f09-istio-podinfo\") pod \"5a866f6f-9374-4ff8-992f-b72b985b1f09\" (UID: \"5a866f6f-9374-4ff8-992f-b72b985b1f09\") " Jul 10 00:44:47.009725 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:47.009534 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a866f6f-9374-4ff8-992f-b72b985b1f09-credential-socket\") pod \"5a866f6f-9374-4ff8-992f-b72b985b1f09\" (UID: \"5a866f6f-9374-4ff8-992f-b72b985b1f09\") " Jul 10 00:44:47.009725 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:47.009598 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a866f6f-9374-4ff8-992f-b72b985b1f09-istio-data\") pod \"5a866f6f-9374-4ff8-992f-b72b985b1f09\" (UID: \"5a866f6f-9374-4ff8-992f-b72b985b1f09\") " Jul 10 00:44:47.009725 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:47.009623 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a866f6f-9374-4ff8-992f-b72b985b1f09-istiod-ca-cert\") pod \"5a866f6f-9374-4ff8-992f-b72b985b1f09\" (UID: \"5a866f6f-9374-4ff8-992f-b72b985b1f09\") " Jul 10 00:44:47.009725 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:47.009655 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sxgd7\" (UniqueName: \"kubernetes.io/projected/5a866f6f-9374-4ff8-992f-b72b985b1f09-kube-api-access-sxgd7\") pod \"5a866f6f-9374-4ff8-992f-b72b985b1f09\" (UID: \"5a866f6f-9374-4ff8-992f-b72b985b1f09\") " Jul 10 00:44:47.009725 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:47.009719 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a866f6f-9374-4ff8-992f-b72b985b1f09-workload-socket\") pod \"5a866f6f-9374-4ff8-992f-b72b985b1f09\" (UID: \"5a866f6f-9374-4ff8-992f-b72b985b1f09\") " Jul 10 00:44:47.010034 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:47.009747 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a866f6f-9374-4ff8-992f-b72b985b1f09-istio-token\") pod \"5a866f6f-9374-4ff8-992f-b72b985b1f09\" (UID: \"5a866f6f-9374-4ff8-992f-b72b985b1f09\") " Jul 10 00:44:47.010458 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:47.010170 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a866f6f-9374-4ff8-992f-b72b985b1f09-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5a866f6f-9374-4ff8-992f-b72b985b1f09" (UID: "5a866f6f-9374-4ff8-992f-b72b985b1f09"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:44:47.011519 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:47.011465 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a866f6f-9374-4ff8-992f-b72b985b1f09-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5a866f6f-9374-4ff8-992f-b72b985b1f09" (UID: "5a866f6f-9374-4ff8-992f-b72b985b1f09"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:44:47.011711 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:47.011672 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a866f6f-9374-4ff8-992f-b72b985b1f09-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5a866f6f-9374-4ff8-992f-b72b985b1f09" (UID: "5a866f6f-9374-4ff8-992f-b72b985b1f09"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:44:47.011789 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:47.011706 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a866f6f-9374-4ff8-992f-b72b985b1f09-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5a866f6f-9374-4ff8-992f-b72b985b1f09" (UID: "5a866f6f-9374-4ff8-992f-b72b985b1f09"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:44:47.011853 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:47.011799 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a866f6f-9374-4ff8-992f-b72b985b1f09-istio-data" (OuterVolumeSpecName: "istio-data") pod "5a866f6f-9374-4ff8-992f-b72b985b1f09" (UID: "5a866f6f-9374-4ff8-992f-b72b985b1f09"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:44:47.011986 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:47.011963 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a866f6f-9374-4ff8-992f-b72b985b1f09-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5a866f6f-9374-4ff8-992f-b72b985b1f09" (UID: "5a866f6f-9374-4ff8-992f-b72b985b1f09"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:44:47.012297 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:47.012270 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a866f6f-9374-4ff8-992f-b72b985b1f09-istio-token" (OuterVolumeSpecName: "istio-token") pod "5a866f6f-9374-4ff8-992f-b72b985b1f09" (UID: "5a866f6f-9374-4ff8-992f-b72b985b1f09"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:44:47.012661 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:47.012614 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a866f6f-9374-4ff8-992f-b72b985b1f09-kube-api-access-sxgd7" (OuterVolumeSpecName: "kube-api-access-sxgd7") pod "5a866f6f-9374-4ff8-992f-b72b985b1f09" (UID: "5a866f6f-9374-4ff8-992f-b72b985b1f09"). InnerVolumeSpecName "kube-api-access-sxgd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:44:47.013477 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:47.013459 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a866f6f-9374-4ff8-992f-b72b985b1f09-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5a866f6f-9374-4ff8-992f-b72b985b1f09" (UID: "5a866f6f-9374-4ff8-992f-b72b985b1f09"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 00:44:47.107940 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:47.107904 2562 generic.go:356] "Generic (PLEG): container finished" podID="5a866f6f-9374-4ff8-992f-b72b985b1f09" containerID="8fd7bed37dbcdcfea25e00f01866f1c4adb1c9dfb926b16c28d8fe826c6a3cb6" exitCode=0 Jul 10 00:44:47.108139 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:47.107951 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v" Jul 10 00:44:47.108139 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:47.107996 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v" event={"ID":"5a866f6f-9374-4ff8-992f-b72b985b1f09","Type":"ContainerDied","Data":"8fd7bed37dbcdcfea25e00f01866f1c4adb1c9dfb926b16c28d8fe826c6a3cb6"} Jul 10 00:44:47.108139 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:47.108062 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v" event={"ID":"5a866f6f-9374-4ff8-992f-b72b985b1f09","Type":"ContainerDied","Data":"f65131ad8aeccdb49ec593d472d971e03de5a9f7e6c4d52d566f7842d3ed3e61"} Jul 10 00:44:47.108139 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:47.108097 2562 scope.go:117] "RemoveContainer" containerID="8fd7bed37dbcdcfea25e00f01866f1c4adb1c9dfb926b16c28d8fe826c6a3cb6" Jul 10 00:44:47.110542 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:47.110519 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a866f6f-9374-4ff8-992f-b72b985b1f09-istio-data\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:44:47.110632 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:47.110547 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a866f6f-9374-4ff8-992f-b72b985b1f09-istiod-ca-cert\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:44:47.110632 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:47.110561 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxgd7\" (UniqueName: \"kubernetes.io/projected/5a866f6f-9374-4ff8-992f-b72b985b1f09-kube-api-access-sxgd7\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:44:47.110632 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:47.110577 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a866f6f-9374-4ff8-992f-b72b985b1f09-workload-socket\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:44:47.110632 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:47.110590 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a866f6f-9374-4ff8-992f-b72b985b1f09-istio-token\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:44:47.110632 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:47.110603 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a866f6f-9374-4ff8-992f-b72b985b1f09-istio-envoy\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:44:47.110632 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:47.110616 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a866f6f-9374-4ff8-992f-b72b985b1f09-workload-certs\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:44:47.110632 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:47.110628 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a866f6f-9374-4ff8-992f-b72b985b1f09-istio-podinfo\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:44:47.110923 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:47.110641 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a866f6f-9374-4ff8-992f-b72b985b1f09-credential-socket\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:44:47.117464 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:47.117443 2562 scope.go:117] "RemoveContainer" containerID="8fd7bed37dbcdcfea25e00f01866f1c4adb1c9dfb926b16c28d8fe826c6a3cb6" Jul 10 00:44:47.117729 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:44:47.117711 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fd7bed37dbcdcfea25e00f01866f1c4adb1c9dfb926b16c28d8fe826c6a3cb6\": container with ID starting with 8fd7bed37dbcdcfea25e00f01866f1c4adb1c9dfb926b16c28d8fe826c6a3cb6 not found: ID does not exist" containerID="8fd7bed37dbcdcfea25e00f01866f1c4adb1c9dfb926b16c28d8fe826c6a3cb6" Jul 10 00:44:47.117797 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:47.117738 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fd7bed37dbcdcfea25e00f01866f1c4adb1c9dfb926b16c28d8fe826c6a3cb6"} err="failed to get container status \"8fd7bed37dbcdcfea25e00f01866f1c4adb1c9dfb926b16c28d8fe826c6a3cb6\": rpc error: code = NotFound desc = could not find container \"8fd7bed37dbcdcfea25e00f01866f1c4adb1c9dfb926b16c28d8fe826c6a3cb6\": container with ID starting with 8fd7bed37dbcdcfea25e00f01866f1c4adb1c9dfb926b16c28d8fe826c6a3cb6 not found: ID does not exist" Jul 10 00:44:47.129070 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:47.129045 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v"] Jul 10 00:44:47.132614 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:47.132587 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-055f1f02-openshift-default-666fd68b7-v8j7v"] Jul 10 00:44:47.736860 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:44:47.736812 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a866f6f-9374-4ff8-992f-b72b985b1f09" path="/var/lib/kubelet/pods/5a866f6f-9374-4ff8-992f-b72b985b1f09/volumes" Jul 10 00:45:33.779263 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:45:33.779236 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kjhg9_4651a90d-ef18-4974-9efd-a77fff3c9dfd/console-operator/1.log" Jul 10 00:45:33.782781 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:45:33.782761 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h5g9w_f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32/ovn-acl-logging/0.log" Jul 10 00:45:33.788433 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:45:33.788412 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kjhg9_4651a90d-ef18-4974-9efd-a77fff3c9dfd/console-operator/1.log" Jul 10 00:45:33.792003 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:45:33.791986 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h5g9w_f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32/ovn-acl-logging/0.log" Jul 10 00:50:23.087478 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:50:23.087449 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-b8817a/test-model-b8817a-kserve-7fbd497899-2l8j6"] Jul 10 00:50:23.089915 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:50:23.087784 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a866f6f-9374-4ff8-992f-b72b985b1f09" containerName="istio-proxy" Jul 10 00:50:23.089915 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:50:23.087795 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a866f6f-9374-4ff8-992f-b72b985b1f09" containerName="istio-proxy" Jul 10 00:50:23.089915 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:50:23.087847 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a866f6f-9374-4ff8-992f-b72b985b1f09" containerName="istio-proxy" Jul 10 00:50:23.090724 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:50:23.090707 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b8817a/test-model-b8817a-kserve-7fbd497899-2l8j6" Jul 10 00:50:23.092809 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:50:23.092784 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-b8817a\"/\"openshift-service-ca.crt\"" Jul 10 00:50:23.093477 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:50:23.093455 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-b8817a\"/\"kube-root-ca.crt\"" Jul 10 00:50:23.093582 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:50:23.093492 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-b8817a\"/\"test-model-b8817a-kserve-self-signed-certs\"" Jul 10 00:50:23.093582 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:50:23.093543 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-b8817a\"/\"default-dockercfg-gbb6z\"" Jul 10 00:50:23.100374 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:50:23.100353 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-b8817a/test-model-b8817a-kserve-7fbd497899-2l8j6"] Jul 10 00:50:23.146786 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:50:23.146761 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b195937-d456-483a-ad5f-4c7cbff2f87e-dshm\") pod \"test-model-b8817a-kserve-7fbd497899-2l8j6\" (UID: \"5b195937-d456-483a-ad5f-4c7cbff2f87e\") " pod="ai-tenant-e2e-inf-a-b8817a/test-model-b8817a-kserve-7fbd497899-2l8j6" Jul 10 00:50:23.146883 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:50:23.146812 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhphg\" (UniqueName: \"kubernetes.io/projected/5b195937-d456-483a-ad5f-4c7cbff2f87e-kube-api-access-zhphg\") pod \"test-model-b8817a-kserve-7fbd497899-2l8j6\" (UID: \"5b195937-d456-483a-ad5f-4c7cbff2f87e\") " pod="ai-tenant-e2e-inf-a-b8817a/test-model-b8817a-kserve-7fbd497899-2l8j6" Jul 10 00:50:23.146883 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:50:23.146835 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b195937-d456-483a-ad5f-4c7cbff2f87e-home\") pod \"test-model-b8817a-kserve-7fbd497899-2l8j6\" (UID: \"5b195937-d456-483a-ad5f-4c7cbff2f87e\") " pod="ai-tenant-e2e-inf-a-b8817a/test-model-b8817a-kserve-7fbd497899-2l8j6" Jul 10 00:50:23.146883 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:50:23.146861 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b195937-d456-483a-ad5f-4c7cbff2f87e-model-cache\") pod \"test-model-b8817a-kserve-7fbd497899-2l8j6\" (UID: \"5b195937-d456-483a-ad5f-4c7cbff2f87e\") " pod="ai-tenant-e2e-inf-a-b8817a/test-model-b8817a-kserve-7fbd497899-2l8j6" Jul 10 00:50:23.146883 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:50:23.146877 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b195937-d456-483a-ad5f-4c7cbff2f87e-tls-certs\") pod \"test-model-b8817a-kserve-7fbd497899-2l8j6\" (UID: \"5b195937-d456-483a-ad5f-4c7cbff2f87e\") " pod="ai-tenant-e2e-inf-a-b8817a/test-model-b8817a-kserve-7fbd497899-2l8j6" Jul 10 00:50:23.147015 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:50:23.146960 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b195937-d456-483a-ad5f-4c7cbff2f87e-tmp-dir\") pod \"test-model-b8817a-kserve-7fbd497899-2l8j6\" (UID: \"5b195937-d456-483a-ad5f-4c7cbff2f87e\") " pod="ai-tenant-e2e-inf-a-b8817a/test-model-b8817a-kserve-7fbd497899-2l8j6" Jul 10 00:50:23.247999 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:50:23.247966 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b195937-d456-483a-ad5f-4c7cbff2f87e-tmp-dir\") pod \"test-model-b8817a-kserve-7fbd497899-2l8j6\" (UID: \"5b195937-d456-483a-ad5f-4c7cbff2f87e\") " pod="ai-tenant-e2e-inf-a-b8817a/test-model-b8817a-kserve-7fbd497899-2l8j6" Jul 10 00:50:23.248198 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:50:23.248025 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b195937-d456-483a-ad5f-4c7cbff2f87e-dshm\") pod \"test-model-b8817a-kserve-7fbd497899-2l8j6\" (UID: \"5b195937-d456-483a-ad5f-4c7cbff2f87e\") " pod="ai-tenant-e2e-inf-a-b8817a/test-model-b8817a-kserve-7fbd497899-2l8j6" Jul 10 00:50:23.248198 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:50:23.248117 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhphg\" (UniqueName: \"kubernetes.io/projected/5b195937-d456-483a-ad5f-4c7cbff2f87e-kube-api-access-zhphg\") pod \"test-model-b8817a-kserve-7fbd497899-2l8j6\" (UID: \"5b195937-d456-483a-ad5f-4c7cbff2f87e\") " pod="ai-tenant-e2e-inf-a-b8817a/test-model-b8817a-kserve-7fbd497899-2l8j6" Jul 10 00:50:23.248198 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:50:23.248140 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b195937-d456-483a-ad5f-4c7cbff2f87e-home\") pod \"test-model-b8817a-kserve-7fbd497899-2l8j6\" (UID: \"5b195937-d456-483a-ad5f-4c7cbff2f87e\") " pod="ai-tenant-e2e-inf-a-b8817a/test-model-b8817a-kserve-7fbd497899-2l8j6" Jul 10 00:50:23.248198 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:50:23.248164 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b195937-d456-483a-ad5f-4c7cbff2f87e-model-cache\") pod \"test-model-b8817a-kserve-7fbd497899-2l8j6\" (UID: \"5b195937-d456-483a-ad5f-4c7cbff2f87e\") " pod="ai-tenant-e2e-inf-a-b8817a/test-model-b8817a-kserve-7fbd497899-2l8j6" Jul 10 00:50:23.248198 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:50:23.248188 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b195937-d456-483a-ad5f-4c7cbff2f87e-tls-certs\") pod \"test-model-b8817a-kserve-7fbd497899-2l8j6\" (UID: \"5b195937-d456-483a-ad5f-4c7cbff2f87e\") " pod="ai-tenant-e2e-inf-a-b8817a/test-model-b8817a-kserve-7fbd497899-2l8j6" Jul 10 00:50:23.248462 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:50:23.248388 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b195937-d456-483a-ad5f-4c7cbff2f87e-tmp-dir\") pod \"test-model-b8817a-kserve-7fbd497899-2l8j6\" (UID: \"5b195937-d456-483a-ad5f-4c7cbff2f87e\") " pod="ai-tenant-e2e-inf-a-b8817a/test-model-b8817a-kserve-7fbd497899-2l8j6" Jul 10 00:50:23.248711 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:50:23.248488 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b195937-d456-483a-ad5f-4c7cbff2f87e-model-cache\") pod \"test-model-b8817a-kserve-7fbd497899-2l8j6\" (UID: \"5b195937-d456-483a-ad5f-4c7cbff2f87e\") " pod="ai-tenant-e2e-inf-a-b8817a/test-model-b8817a-kserve-7fbd497899-2l8j6" Jul 10 00:50:23.248711 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:50:23.248486 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b195937-d456-483a-ad5f-4c7cbff2f87e-home\") pod \"test-model-b8817a-kserve-7fbd497899-2l8j6\" (UID: \"5b195937-d456-483a-ad5f-4c7cbff2f87e\") " pod="ai-tenant-e2e-inf-a-b8817a/test-model-b8817a-kserve-7fbd497899-2l8j6" Jul 10 00:50:23.250355 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:50:23.250332 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b195937-d456-483a-ad5f-4c7cbff2f87e-dshm\") pod \"test-model-b8817a-kserve-7fbd497899-2l8j6\" (UID: \"5b195937-d456-483a-ad5f-4c7cbff2f87e\") " pod="ai-tenant-e2e-inf-a-b8817a/test-model-b8817a-kserve-7fbd497899-2l8j6" Jul 10 00:50:23.250567 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:50:23.250549 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b195937-d456-483a-ad5f-4c7cbff2f87e-tls-certs\") pod \"test-model-b8817a-kserve-7fbd497899-2l8j6\" (UID: \"5b195937-d456-483a-ad5f-4c7cbff2f87e\") " pod="ai-tenant-e2e-inf-a-b8817a/test-model-b8817a-kserve-7fbd497899-2l8j6" Jul 10 00:50:23.254699 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:50:23.254674 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhphg\" (UniqueName: \"kubernetes.io/projected/5b195937-d456-483a-ad5f-4c7cbff2f87e-kube-api-access-zhphg\") pod \"test-model-b8817a-kserve-7fbd497899-2l8j6\" (UID: \"5b195937-d456-483a-ad5f-4c7cbff2f87e\") " pod="ai-tenant-e2e-inf-a-b8817a/test-model-b8817a-kserve-7fbd497899-2l8j6" Jul 10 00:50:23.400546 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:50:23.400478 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b8817a/test-model-b8817a-kserve-7fbd497899-2l8j6" Jul 10 00:50:23.530886 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:50:23.530864 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-b8817a/test-model-b8817a-kserve-7fbd497899-2l8j6"] Jul 10 00:50:23.532454 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:50:23.532427 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b195937_d456_483a_ad5f_4c7cbff2f87e.slice/crio-b6bad3ef282f205d43ac5ad75c8c0a756c68cf259005c6ff2e6bf4e894dd26e8 WatchSource:0}: Error finding container b6bad3ef282f205d43ac5ad75c8c0a756c68cf259005c6ff2e6bf4e894dd26e8: Status 404 returned error can't find the container with id b6bad3ef282f205d43ac5ad75c8c0a756c68cf259005c6ff2e6bf4e894dd26e8 Jul 10 00:50:23.840096 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:50:23.840040 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b8817a/test-model-b8817a-kserve-7fbd497899-2l8j6" event={"ID":"5b195937-d456-483a-ad5f-4c7cbff2f87e","Type":"ContainerStarted","Data":"1f6f83fc27add2ffe48bc45341d14c741fc9f6c6c01a078ca7af65f8f390346c"} Jul 10 00:50:23.840300 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:50:23.840103 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b8817a/test-model-b8817a-kserve-7fbd497899-2l8j6" event={"ID":"5b195937-d456-483a-ad5f-4c7cbff2f87e","Type":"ContainerStarted","Data":"b6bad3ef282f205d43ac5ad75c8c0a756c68cf259005c6ff2e6bf4e894dd26e8"} Jul 10 00:50:23.857895 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:50:23.857856 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-b8817a/test-model-b8817a-kserve-7fbd497899-2l8j6" podStartSLOduration=0.857842437 podStartE2EDuration="857.842437ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:50:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:50:23.855610801 +0000 UTC m=+2090.673916296" watchObservedRunningTime="2026-07-10 00:50:23.857842437 +0000 UTC m=+2090.676147930" Jul 10 00:50:33.401035 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:50:33.400993 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-b8817a/test-model-b8817a-kserve-7fbd497899-2l8j6" Jul 10 00:50:33.401035 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:50:33.401045 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-b8817a/test-model-b8817a-kserve-7fbd497899-2l8j6" Jul 10 00:50:33.414552 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:50:33.414516 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-b8817a/test-model-b8817a-kserve-7fbd497899-2l8j6" Jul 10 00:50:33.814805 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:50:33.814779 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kjhg9_4651a90d-ef18-4974-9efd-a77fff3c9dfd/console-operator/1.log" Jul 10 00:50:33.818393 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:50:33.818367 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h5g9w_f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32/ovn-acl-logging/0.log" Jul 10 00:50:33.822926 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:50:33.822905 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kjhg9_4651a90d-ef18-4974-9efd-a77fff3c9dfd/console-operator/1.log" Jul 10 00:50:33.826263 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:50:33.826247 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h5g9w_f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32/ovn-acl-logging/0.log" Jul 10 00:50:33.905134 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:50:33.905105 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-b8817a/test-model-b8817a-kserve-7fbd497899-2l8j6" Jul 10 00:51:23.370528 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:51:23.370492 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-b8817a/test-model-b8817a-kserve-7fbd497899-2l8j6"] Jul 10 00:51:38.390749 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:51:38.390681 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-b8817a/test-model-b8817a-kserve-7fbd497899-2l8j6" podUID="5b195937-d456-483a-ad5f-4c7cbff2f87e" containerName="main" containerID="cri-o://1f6f83fc27add2ffe48bc45341d14c741fc9f6c6c01a078ca7af65f8f390346c" gracePeriod=45 Jul 10 00:51:38.634276 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:51:38.634254 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b8817a/test-model-b8817a-kserve-7fbd497899-2l8j6" Jul 10 00:51:38.790771 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:51:38.790688 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b195937-d456-483a-ad5f-4c7cbff2f87e-dshm\") pod \"5b195937-d456-483a-ad5f-4c7cbff2f87e\" (UID: \"5b195937-d456-483a-ad5f-4c7cbff2f87e\") " Jul 10 00:51:38.790771 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:51:38.790745 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zhphg\" (UniqueName: \"kubernetes.io/projected/5b195937-d456-483a-ad5f-4c7cbff2f87e-kube-api-access-zhphg\") pod \"5b195937-d456-483a-ad5f-4c7cbff2f87e\" (UID: \"5b195937-d456-483a-ad5f-4c7cbff2f87e\") " Jul 10 00:51:38.790771 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:51:38.790770 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b195937-d456-483a-ad5f-4c7cbff2f87e-tls-certs\") pod \"5b195937-d456-483a-ad5f-4c7cbff2f87e\" (UID: \"5b195937-d456-483a-ad5f-4c7cbff2f87e\") " Jul 10 00:51:38.791047 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:51:38.790823 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b195937-d456-483a-ad5f-4c7cbff2f87e-model-cache\") pod \"5b195937-d456-483a-ad5f-4c7cbff2f87e\" (UID: \"5b195937-d456-483a-ad5f-4c7cbff2f87e\") " Jul 10 00:51:38.791047 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:51:38.790883 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b195937-d456-483a-ad5f-4c7cbff2f87e-tmp-dir\") pod \"5b195937-d456-483a-ad5f-4c7cbff2f87e\" (UID: \"5b195937-d456-483a-ad5f-4c7cbff2f87e\") " Jul 10 00:51:38.791047 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:51:38.790981 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b195937-d456-483a-ad5f-4c7cbff2f87e-home\") pod \"5b195937-d456-483a-ad5f-4c7cbff2f87e\" (UID: \"5b195937-d456-483a-ad5f-4c7cbff2f87e\") " Jul 10 00:51:38.791309 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:51:38.791150 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b195937-d456-483a-ad5f-4c7cbff2f87e-model-cache" (OuterVolumeSpecName: "model-cache") pod "5b195937-d456-483a-ad5f-4c7cbff2f87e" (UID: "5b195937-d456-483a-ad5f-4c7cbff2f87e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:51:38.791309 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:51:38.791222 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b195937-d456-483a-ad5f-4c7cbff2f87e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5b195937-d456-483a-ad5f-4c7cbff2f87e" (UID: "5b195937-d456-483a-ad5f-4c7cbff2f87e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:51:38.791417 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:51:38.791332 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b195937-d456-483a-ad5f-4c7cbff2f87e-home" (OuterVolumeSpecName: "home") pod "5b195937-d456-483a-ad5f-4c7cbff2f87e" (UID: "5b195937-d456-483a-ad5f-4c7cbff2f87e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:51:38.791417 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:51:38.791393 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b195937-d456-483a-ad5f-4c7cbff2f87e-tmp-dir\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:51:38.791526 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:51:38.791415 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b195937-d456-483a-ad5f-4c7cbff2f87e-model-cache\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:51:38.793023 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:51:38.792990 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b195937-d456-483a-ad5f-4c7cbff2f87e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5b195937-d456-483a-ad5f-4c7cbff2f87e" (UID: "5b195937-d456-483a-ad5f-4c7cbff2f87e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:51:38.793178 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:51:38.793055 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b195937-d456-483a-ad5f-4c7cbff2f87e-dshm" (OuterVolumeSpecName: "dshm") pod "5b195937-d456-483a-ad5f-4c7cbff2f87e" (UID: "5b195937-d456-483a-ad5f-4c7cbff2f87e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:51:38.793250 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:51:38.793171 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b195937-d456-483a-ad5f-4c7cbff2f87e-kube-api-access-zhphg" (OuterVolumeSpecName: "kube-api-access-zhphg") pod "5b195937-d456-483a-ad5f-4c7cbff2f87e" (UID: "5b195937-d456-483a-ad5f-4c7cbff2f87e"). InnerVolumeSpecName "kube-api-access-zhphg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:51:38.892574 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:51:38.892542 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b195937-d456-483a-ad5f-4c7cbff2f87e-dshm\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:51:38.892574 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:51:38.892571 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhphg\" (UniqueName: \"kubernetes.io/projected/5b195937-d456-483a-ad5f-4c7cbff2f87e-kube-api-access-zhphg\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:51:38.892723 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:51:38.892588 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b195937-d456-483a-ad5f-4c7cbff2f87e-tls-certs\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:51:38.892723 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:51:38.892601 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b195937-d456-483a-ad5f-4c7cbff2f87e-home\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:51:39.216228 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:51:39.216198 2562 generic.go:356] "Generic (PLEG): container finished" podID="5b195937-d456-483a-ad5f-4c7cbff2f87e" containerID="1f6f83fc27add2ffe48bc45341d14c741fc9f6c6c01a078ca7af65f8f390346c" exitCode=0 Jul 10 00:51:39.216372 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:51:39.216240 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b8817a/test-model-b8817a-kserve-7fbd497899-2l8j6" Jul 10 00:51:39.216372 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:51:39.216280 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b8817a/test-model-b8817a-kserve-7fbd497899-2l8j6" event={"ID":"5b195937-d456-483a-ad5f-4c7cbff2f87e","Type":"ContainerDied","Data":"1f6f83fc27add2ffe48bc45341d14c741fc9f6c6c01a078ca7af65f8f390346c"} Jul 10 00:51:39.216372 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:51:39.216323 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b8817a/test-model-b8817a-kserve-7fbd497899-2l8j6" event={"ID":"5b195937-d456-483a-ad5f-4c7cbff2f87e","Type":"ContainerDied","Data":"b6bad3ef282f205d43ac5ad75c8c0a756c68cf259005c6ff2e6bf4e894dd26e8"} Jul 10 00:51:39.216372 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:51:39.216352 2562 scope.go:117] "RemoveContainer" containerID="1f6f83fc27add2ffe48bc45341d14c741fc9f6c6c01a078ca7af65f8f390346c" Jul 10 00:51:39.225131 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:51:39.225116 2562 scope.go:117] "RemoveContainer" containerID="1f6f83fc27add2ffe48bc45341d14c741fc9f6c6c01a078ca7af65f8f390346c" Jul 10 00:51:39.225387 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:51:39.225367 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f6f83fc27add2ffe48bc45341d14c741fc9f6c6c01a078ca7af65f8f390346c\": container with ID starting with 1f6f83fc27add2ffe48bc45341d14c741fc9f6c6c01a078ca7af65f8f390346c not found: ID does not exist" containerID="1f6f83fc27add2ffe48bc45341d14c741fc9f6c6c01a078ca7af65f8f390346c" Jul 10 00:51:39.225462 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:51:39.225398 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f6f83fc27add2ffe48bc45341d14c741fc9f6c6c01a078ca7af65f8f390346c"} err="failed to get container status \"1f6f83fc27add2ffe48bc45341d14c741fc9f6c6c01a078ca7af65f8f390346c\": rpc error: code = NotFound desc = could not find container \"1f6f83fc27add2ffe48bc45341d14c741fc9f6c6c01a078ca7af65f8f390346c\": container with ID starting with 1f6f83fc27add2ffe48bc45341d14c741fc9f6c6c01a078ca7af65f8f390346c not found: ID does not exist" Jul 10 00:51:39.236479 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:51:39.236458 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-b8817a/test-model-b8817a-kserve-7fbd497899-2l8j6"] Jul 10 00:51:39.242273 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:51:39.242244 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-b8817a/test-model-b8817a-kserve-7fbd497899-2l8j6"] Jul 10 00:51:39.735784 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:51:39.735757 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b195937-d456-483a-ad5f-4c7cbff2f87e" path="/var/lib/kubelet/pods/5b195937-d456-483a-ad5f-4c7cbff2f87e/volumes" Jul 10 00:53:48.397465 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:53:48.397431 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-f84011/auth-test-model-f84011-kserve-69fc966999-wpnlj"] Jul 10 00:53:48.397978 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:53:48.397959 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b195937-d456-483a-ad5f-4c7cbff2f87e" containerName="main" Jul 10 00:53:48.398025 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:53:48.397982 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b195937-d456-483a-ad5f-4c7cbff2f87e" containerName="main" Jul 10 00:53:48.398138 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:53:48.398113 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b195937-d456-483a-ad5f-4c7cbff2f87e" containerName="main" Jul 10 00:53:48.401334 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:53:48.401314 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f84011/auth-test-model-f84011-kserve-69fc966999-wpnlj" Jul 10 00:53:48.403732 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:53:48.403710 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-f84011\"/\"openshift-service-ca.crt\"" Jul 10 00:53:48.404429 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:53:48.404409 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-f84011\"/\"auth-test-model-f84011-kserve-self-signed-certs\"" Jul 10 00:53:48.404538 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:53:48.404458 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-f84011\"/\"default-dockercfg-8qj2f\"" Jul 10 00:53:48.404538 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:53:48.404432 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-f84011\"/\"kube-root-ca.crt\"" Jul 10 00:53:48.408767 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:53:48.408748 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-f84011/auth-test-model-f84011-kserve-69fc966999-wpnlj"] Jul 10 00:53:48.497160 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:53:48.497126 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06118c55-3bc2-490a-98a3-678bb492cb13-home\") pod \"auth-test-model-f84011-kserve-69fc966999-wpnlj\" (UID: \"06118c55-3bc2-490a-98a3-678bb492cb13\") " pod="ai-tenant-e2e-shared-b-f84011/auth-test-model-f84011-kserve-69fc966999-wpnlj" Jul 10 00:53:48.497336 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:53:48.497183 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06118c55-3bc2-490a-98a3-678bb492cb13-tmp-dir\") pod \"auth-test-model-f84011-kserve-69fc966999-wpnlj\" (UID: \"06118c55-3bc2-490a-98a3-678bb492cb13\") " pod="ai-tenant-e2e-shared-b-f84011/auth-test-model-f84011-kserve-69fc966999-wpnlj" Jul 10 00:53:48.497336 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:53:48.497218 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06118c55-3bc2-490a-98a3-678bb492cb13-tls-certs\") pod \"auth-test-model-f84011-kserve-69fc966999-wpnlj\" (UID: \"06118c55-3bc2-490a-98a3-678bb492cb13\") " pod="ai-tenant-e2e-shared-b-f84011/auth-test-model-f84011-kserve-69fc966999-wpnlj" Jul 10 00:53:48.497336 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:53:48.497263 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06118c55-3bc2-490a-98a3-678bb492cb13-model-cache\") pod \"auth-test-model-f84011-kserve-69fc966999-wpnlj\" (UID: \"06118c55-3bc2-490a-98a3-678bb492cb13\") " pod="ai-tenant-e2e-shared-b-f84011/auth-test-model-f84011-kserve-69fc966999-wpnlj" Jul 10 00:53:48.497336 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:53:48.497289 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf88t\" (UniqueName: \"kubernetes.io/projected/06118c55-3bc2-490a-98a3-678bb492cb13-kube-api-access-zf88t\") pod \"auth-test-model-f84011-kserve-69fc966999-wpnlj\" (UID: \"06118c55-3bc2-490a-98a3-678bb492cb13\") " pod="ai-tenant-e2e-shared-b-f84011/auth-test-model-f84011-kserve-69fc966999-wpnlj" Jul 10 00:53:48.497524 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:53:48.497371 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06118c55-3bc2-490a-98a3-678bb492cb13-dshm\") pod \"auth-test-model-f84011-kserve-69fc966999-wpnlj\" (UID: \"06118c55-3bc2-490a-98a3-678bb492cb13\") " pod="ai-tenant-e2e-shared-b-f84011/auth-test-model-f84011-kserve-69fc966999-wpnlj" Jul 10 00:53:48.598059 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:53:48.598025 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06118c55-3bc2-490a-98a3-678bb492cb13-home\") pod \"auth-test-model-f84011-kserve-69fc966999-wpnlj\" (UID: \"06118c55-3bc2-490a-98a3-678bb492cb13\") " pod="ai-tenant-e2e-shared-b-f84011/auth-test-model-f84011-kserve-69fc966999-wpnlj" Jul 10 00:53:48.598262 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:53:48.598096 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06118c55-3bc2-490a-98a3-678bb492cb13-tmp-dir\") pod \"auth-test-model-f84011-kserve-69fc966999-wpnlj\" (UID: \"06118c55-3bc2-490a-98a3-678bb492cb13\") " pod="ai-tenant-e2e-shared-b-f84011/auth-test-model-f84011-kserve-69fc966999-wpnlj" Jul 10 00:53:48.598262 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:53:48.598123 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06118c55-3bc2-490a-98a3-678bb492cb13-tls-certs\") pod \"auth-test-model-f84011-kserve-69fc966999-wpnlj\" (UID: \"06118c55-3bc2-490a-98a3-678bb492cb13\") " pod="ai-tenant-e2e-shared-b-f84011/auth-test-model-f84011-kserve-69fc966999-wpnlj" Jul 10 00:53:48.598262 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:53:48.598176 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06118c55-3bc2-490a-98a3-678bb492cb13-model-cache\") pod \"auth-test-model-f84011-kserve-69fc966999-wpnlj\" (UID: \"06118c55-3bc2-490a-98a3-678bb492cb13\") " pod="ai-tenant-e2e-shared-b-f84011/auth-test-model-f84011-kserve-69fc966999-wpnlj" Jul 10 00:53:48.598262 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:53:48.598206 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf88t\" (UniqueName: \"kubernetes.io/projected/06118c55-3bc2-490a-98a3-678bb492cb13-kube-api-access-zf88t\") pod \"auth-test-model-f84011-kserve-69fc966999-wpnlj\" (UID: \"06118c55-3bc2-490a-98a3-678bb492cb13\") " pod="ai-tenant-e2e-shared-b-f84011/auth-test-model-f84011-kserve-69fc966999-wpnlj" Jul 10 00:53:48.598503 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:53:48.598272 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06118c55-3bc2-490a-98a3-678bb492cb13-dshm\") pod \"auth-test-model-f84011-kserve-69fc966999-wpnlj\" (UID: \"06118c55-3bc2-490a-98a3-678bb492cb13\") " pod="ai-tenant-e2e-shared-b-f84011/auth-test-model-f84011-kserve-69fc966999-wpnlj" Jul 10 00:53:48.598594 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:53:48.598566 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06118c55-3bc2-490a-98a3-678bb492cb13-home\") pod \"auth-test-model-f84011-kserve-69fc966999-wpnlj\" (UID: \"06118c55-3bc2-490a-98a3-678bb492cb13\") " pod="ai-tenant-e2e-shared-b-f84011/auth-test-model-f84011-kserve-69fc966999-wpnlj" Jul 10 00:53:48.598811 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:53:48.598790 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06118c55-3bc2-490a-98a3-678bb492cb13-tmp-dir\") pod \"auth-test-model-f84011-kserve-69fc966999-wpnlj\" (UID: \"06118c55-3bc2-490a-98a3-678bb492cb13\") " pod="ai-tenant-e2e-shared-b-f84011/auth-test-model-f84011-kserve-69fc966999-wpnlj" Jul 10 00:53:48.598919 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:53:48.598883 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06118c55-3bc2-490a-98a3-678bb492cb13-model-cache\") pod \"auth-test-model-f84011-kserve-69fc966999-wpnlj\" (UID: \"06118c55-3bc2-490a-98a3-678bb492cb13\") " pod="ai-tenant-e2e-shared-b-f84011/auth-test-model-f84011-kserve-69fc966999-wpnlj" Jul 10 00:53:48.600618 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:53:48.600594 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06118c55-3bc2-490a-98a3-678bb492cb13-dshm\") pod \"auth-test-model-f84011-kserve-69fc966999-wpnlj\" (UID: \"06118c55-3bc2-490a-98a3-678bb492cb13\") " pod="ai-tenant-e2e-shared-b-f84011/auth-test-model-f84011-kserve-69fc966999-wpnlj" Jul 10 00:53:48.601678 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:53:48.601661 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06118c55-3bc2-490a-98a3-678bb492cb13-tls-certs\") pod \"auth-test-model-f84011-kserve-69fc966999-wpnlj\" (UID: \"06118c55-3bc2-490a-98a3-678bb492cb13\") " pod="ai-tenant-e2e-shared-b-f84011/auth-test-model-f84011-kserve-69fc966999-wpnlj" Jul 10 00:53:48.605404 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:53:48.605385 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf88t\" (UniqueName: \"kubernetes.io/projected/06118c55-3bc2-490a-98a3-678bb492cb13-kube-api-access-zf88t\") pod \"auth-test-model-f84011-kserve-69fc966999-wpnlj\" (UID: \"06118c55-3bc2-490a-98a3-678bb492cb13\") " pod="ai-tenant-e2e-shared-b-f84011/auth-test-model-f84011-kserve-69fc966999-wpnlj" Jul 10 00:53:48.711863 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:53:48.711773 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f84011/auth-test-model-f84011-kserve-69fc966999-wpnlj" Jul 10 00:53:48.849389 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:53:48.849363 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-f84011/auth-test-model-f84011-kserve-69fc966999-wpnlj"] Jul 10 00:53:48.851255 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:53:48.851218 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06118c55_3bc2_490a_98a3_678bb492cb13.slice/crio-cd3a5a50bb9f9bc8442ad78d789d8f01743ff5f6ec7170c0b953f05592e4c4cd WatchSource:0}: Error finding container cd3a5a50bb9f9bc8442ad78d789d8f01743ff5f6ec7170c0b953f05592e4c4cd: Status 404 returned error can't find the container with id cd3a5a50bb9f9bc8442ad78d789d8f01743ff5f6ec7170c0b953f05592e4c4cd Jul 10 00:53:48.869830 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:53:48.869804 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f84011/auth-test-model-f84011-kserve-69fc966999-wpnlj" event={"ID":"06118c55-3bc2-490a-98a3-678bb492cb13","Type":"ContainerStarted","Data":"cd3a5a50bb9f9bc8442ad78d789d8f01743ff5f6ec7170c0b953f05592e4c4cd"} Jul 10 00:53:49.876244 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:53:49.876200 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f84011/auth-test-model-f84011-kserve-69fc966999-wpnlj" event={"ID":"06118c55-3bc2-490a-98a3-678bb492cb13","Type":"ContainerStarted","Data":"61e9e059be2e7b77cafef7a12431fa1e28ce378783156fe946f7df3fbc6413c0"} Jul 10 00:53:49.892593 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:53:49.892545 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-f84011/auth-test-model-f84011-kserve-69fc966999-wpnlj" podStartSLOduration=1.892530619 podStartE2EDuration="1.892530619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:53:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:53:49.890001848 +0000 UTC m=+2296.708307343" watchObservedRunningTime="2026-07-10 00:53:49.892530619 +0000 UTC m=+2296.710836174" Jul 10 00:53:58.711922 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:53:58.711886 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-f84011/auth-test-model-f84011-kserve-69fc966999-wpnlj" Jul 10 00:53:58.711922 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:53:58.711934 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-f84011/auth-test-model-f84011-kserve-69fc966999-wpnlj" Jul 10 00:53:58.723948 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:53:58.723922 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-f84011/auth-test-model-f84011-kserve-69fc966999-wpnlj" Jul 10 00:53:58.930788 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:53:58.930758 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-f84011/auth-test-model-f84011-kserve-69fc966999-wpnlj" Jul 10 00:55:33.847060 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:55:33.847011 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kjhg9_4651a90d-ef18-4974-9efd-a77fff3c9dfd/console-operator/1.log" Jul 10 00:55:33.851061 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:55:33.851032 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h5g9w_f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32/ovn-acl-logging/0.log" Jul 10 00:55:33.856608 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:55:33.856588 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kjhg9_4651a90d-ef18-4974-9efd-a77fff3c9dfd/console-operator/1.log" Jul 10 00:55:33.860148 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:55:33.860128 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h5g9w_f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32/ovn-acl-logging/0.log" Jul 10 00:55:40.076794 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:55:40.076754 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-f84011/auth-test-model-f84011-kserve-69fc966999-wpnlj"] Jul 10 00:55:55.097267 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:55:55.097197 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-f84011/auth-test-model-f84011-kserve-69fc966999-wpnlj" podUID="06118c55-3bc2-490a-98a3-678bb492cb13" containerName="main" containerID="cri-o://61e9e059be2e7b77cafef7a12431fa1e28ce378783156fe946f7df3fbc6413c0" gracePeriod=45 Jul 10 00:55:55.354050 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:55:55.353987 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f84011/auth-test-model-f84011-kserve-69fc966999-wpnlj" Jul 10 00:55:55.421051 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:55:55.421018 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06118c55-3bc2-490a-98a3-678bb492cb13-dshm\") pod \"06118c55-3bc2-490a-98a3-678bb492cb13\" (UID: \"06118c55-3bc2-490a-98a3-678bb492cb13\") " Jul 10 00:55:55.421305 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:55:55.421067 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06118c55-3bc2-490a-98a3-678bb492cb13-home\") pod \"06118c55-3bc2-490a-98a3-678bb492cb13\" (UID: \"06118c55-3bc2-490a-98a3-678bb492cb13\") " Jul 10 00:55:55.421305 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:55:55.421124 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zf88t\" (UniqueName: \"kubernetes.io/projected/06118c55-3bc2-490a-98a3-678bb492cb13-kube-api-access-zf88t\") pod \"06118c55-3bc2-490a-98a3-678bb492cb13\" (UID: \"06118c55-3bc2-490a-98a3-678bb492cb13\") " Jul 10 00:55:55.421305 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:55:55.421169 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06118c55-3bc2-490a-98a3-678bb492cb13-tmp-dir\") pod \"06118c55-3bc2-490a-98a3-678bb492cb13\" (UID: \"06118c55-3bc2-490a-98a3-678bb492cb13\") " Jul 10 00:55:55.421305 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:55:55.421185 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06118c55-3bc2-490a-98a3-678bb492cb13-model-cache\") pod \"06118c55-3bc2-490a-98a3-678bb492cb13\" (UID: \"06118c55-3bc2-490a-98a3-678bb492cb13\") " Jul 10 00:55:55.421305 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:55:55.421216 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06118c55-3bc2-490a-98a3-678bb492cb13-tls-certs\") pod \"06118c55-3bc2-490a-98a3-678bb492cb13\" (UID: \"06118c55-3bc2-490a-98a3-678bb492cb13\") " Jul 10 00:55:55.421558 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:55:55.421460 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06118c55-3bc2-490a-98a3-678bb492cb13-home" (OuterVolumeSpecName: "home") pod "06118c55-3bc2-490a-98a3-678bb492cb13" (UID: "06118c55-3bc2-490a-98a3-678bb492cb13"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:55:55.421558 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:55:55.421480 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06118c55-3bc2-490a-98a3-678bb492cb13-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "06118c55-3bc2-490a-98a3-678bb492cb13" (UID: "06118c55-3bc2-490a-98a3-678bb492cb13"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:55:55.421558 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:55:55.421454 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06118c55-3bc2-490a-98a3-678bb492cb13-model-cache" (OuterVolumeSpecName: "model-cache") pod "06118c55-3bc2-490a-98a3-678bb492cb13" (UID: "06118c55-3bc2-490a-98a3-678bb492cb13"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:55:55.423439 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:55:55.423410 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06118c55-3bc2-490a-98a3-678bb492cb13-kube-api-access-zf88t" (OuterVolumeSpecName: "kube-api-access-zf88t") pod "06118c55-3bc2-490a-98a3-678bb492cb13" (UID: "06118c55-3bc2-490a-98a3-678bb492cb13"). InnerVolumeSpecName "kube-api-access-zf88t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:55:55.423547 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:55:55.423444 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06118c55-3bc2-490a-98a3-678bb492cb13-dshm" (OuterVolumeSpecName: "dshm") pod "06118c55-3bc2-490a-98a3-678bb492cb13" (UID: "06118c55-3bc2-490a-98a3-678bb492cb13"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:55:55.423547 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:55:55.423454 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06118c55-3bc2-490a-98a3-678bb492cb13-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "06118c55-3bc2-490a-98a3-678bb492cb13" (UID: "06118c55-3bc2-490a-98a3-678bb492cb13"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:55:55.507896 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:55:55.507859 2562 generic.go:356] "Generic (PLEG): container finished" podID="06118c55-3bc2-490a-98a3-678bb492cb13" containerID="61e9e059be2e7b77cafef7a12431fa1e28ce378783156fe946f7df3fbc6413c0" exitCode=0 Jul 10 00:55:55.507896 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:55:55.507898 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f84011/auth-test-model-f84011-kserve-69fc966999-wpnlj" Jul 10 00:55:55.508104 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:55:55.507941 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f84011/auth-test-model-f84011-kserve-69fc966999-wpnlj" event={"ID":"06118c55-3bc2-490a-98a3-678bb492cb13","Type":"ContainerDied","Data":"61e9e059be2e7b77cafef7a12431fa1e28ce378783156fe946f7df3fbc6413c0"} Jul 10 00:55:55.508104 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:55:55.507985 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f84011/auth-test-model-f84011-kserve-69fc966999-wpnlj" event={"ID":"06118c55-3bc2-490a-98a3-678bb492cb13","Type":"ContainerDied","Data":"cd3a5a50bb9f9bc8442ad78d789d8f01743ff5f6ec7170c0b953f05592e4c4cd"} Jul 10 00:55:55.508104 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:55:55.508003 2562 scope.go:117] "RemoveContainer" containerID="61e9e059be2e7b77cafef7a12431fa1e28ce378783156fe946f7df3fbc6413c0" Jul 10 00:55:55.517093 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:55:55.517055 2562 scope.go:117] "RemoveContainer" containerID="61e9e059be2e7b77cafef7a12431fa1e28ce378783156fe946f7df3fbc6413c0" Jul 10 00:55:55.517354 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:55:55.517337 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61e9e059be2e7b77cafef7a12431fa1e28ce378783156fe946f7df3fbc6413c0\": container with ID starting with 61e9e059be2e7b77cafef7a12431fa1e28ce378783156fe946f7df3fbc6413c0 not found: ID does not exist" containerID="61e9e059be2e7b77cafef7a12431fa1e28ce378783156fe946f7df3fbc6413c0" Jul 10 00:55:55.517429 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:55:55.517358 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61e9e059be2e7b77cafef7a12431fa1e28ce378783156fe946f7df3fbc6413c0"} err="failed to get container status \"61e9e059be2e7b77cafef7a12431fa1e28ce378783156fe946f7df3fbc6413c0\": rpc error: code = NotFound desc = could not find container \"61e9e059be2e7b77cafef7a12431fa1e28ce378783156fe946f7df3fbc6413c0\": container with ID starting with 61e9e059be2e7b77cafef7a12431fa1e28ce378783156fe946f7df3fbc6413c0 not found: ID does not exist" Jul 10 00:55:55.522490 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:55:55.522476 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06118c55-3bc2-490a-98a3-678bb492cb13-dshm\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:55:55.522545 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:55:55.522492 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06118c55-3bc2-490a-98a3-678bb492cb13-home\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:55:55.522545 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:55:55.522502 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zf88t\" (UniqueName: \"kubernetes.io/projected/06118c55-3bc2-490a-98a3-678bb492cb13-kube-api-access-zf88t\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:55:55.522545 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:55:55.522511 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06118c55-3bc2-490a-98a3-678bb492cb13-tmp-dir\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:55:55.522545 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:55:55.522519 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06118c55-3bc2-490a-98a3-678bb492cb13-model-cache\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:55:55.522545 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:55:55.522527 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06118c55-3bc2-490a-98a3-678bb492cb13-tls-certs\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:55:55.527775 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:55:55.527755 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-f84011/auth-test-model-f84011-kserve-69fc966999-wpnlj"] Jul 10 00:55:55.529828 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:55:55.529810 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-f84011/auth-test-model-f84011-kserve-69fc966999-wpnlj"] Jul 10 00:55:55.736245 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:55:55.736166 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06118c55-3bc2-490a-98a3-678bb492cb13" path="/var/lib/kubelet/pods/06118c55-3bc2-490a-98a3-678bb492cb13/volumes" Jul 10 00:56:10.796407 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:56:10.796374 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-f84011/sub-test-model-f84011-kserve-fd9d444f9-lwxbz"] Jul 10 00:56:10.796882 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:56:10.796692 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06118c55-3bc2-490a-98a3-678bb492cb13" containerName="main" Jul 10 00:56:10.796882 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:56:10.796704 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="06118c55-3bc2-490a-98a3-678bb492cb13" containerName="main" Jul 10 00:56:10.796882 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:56:10.796768 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="06118c55-3bc2-490a-98a3-678bb492cb13" containerName="main" Jul 10 00:56:10.799482 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:56:10.799465 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f84011/sub-test-model-f84011-kserve-fd9d444f9-lwxbz" Jul 10 00:56:10.802003 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:56:10.801972 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-f84011\"/\"openshift-service-ca.crt\"" Jul 10 00:56:10.802003 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:56:10.801985 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-f84011\"/\"default-dockercfg-8qj2f\"" Jul 10 00:56:10.802213 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:56:10.802032 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-f84011\"/\"kube-root-ca.crt\"" Jul 10 00:56:10.802213 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:56:10.802052 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-f84011\"/\"sub-test-model-f84011-kserve-self-signed-certs\"" Jul 10 00:56:10.805883 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:56:10.805861 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-f84011/sub-test-model-f84011-kserve-fd9d444f9-lwxbz"] Jul 10 00:56:10.963659 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:56:10.963616 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b5e6174-7853-4fc5-913e-7bbbecef37d4-tls-certs\") pod \"sub-test-model-f84011-kserve-fd9d444f9-lwxbz\" (UID: \"9b5e6174-7853-4fc5-913e-7bbbecef37d4\") " pod="ai-tenant-e2e-shared-b-f84011/sub-test-model-f84011-kserve-fd9d444f9-lwxbz" Jul 10 00:56:10.963828 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:56:10.963674 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b5e6174-7853-4fc5-913e-7bbbecef37d4-home\") pod \"sub-test-model-f84011-kserve-fd9d444f9-lwxbz\" (UID: \"9b5e6174-7853-4fc5-913e-7bbbecef37d4\") " pod="ai-tenant-e2e-shared-b-f84011/sub-test-model-f84011-kserve-fd9d444f9-lwxbz" Jul 10 00:56:10.963828 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:56:10.963730 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b5e6174-7853-4fc5-913e-7bbbecef37d4-dshm\") pod \"sub-test-model-f84011-kserve-fd9d444f9-lwxbz\" (UID: \"9b5e6174-7853-4fc5-913e-7bbbecef37d4\") " pod="ai-tenant-e2e-shared-b-f84011/sub-test-model-f84011-kserve-fd9d444f9-lwxbz" Jul 10 00:56:10.963828 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:56:10.963818 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b5e6174-7853-4fc5-913e-7bbbecef37d4-tmp-dir\") pod \"sub-test-model-f84011-kserve-fd9d444f9-lwxbz\" (UID: \"9b5e6174-7853-4fc5-913e-7bbbecef37d4\") " pod="ai-tenant-e2e-shared-b-f84011/sub-test-model-f84011-kserve-fd9d444f9-lwxbz" Jul 10 00:56:10.963966 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:56:10.963873 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kwbx\" (UniqueName: \"kubernetes.io/projected/9b5e6174-7853-4fc5-913e-7bbbecef37d4-kube-api-access-6kwbx\") pod \"sub-test-model-f84011-kserve-fd9d444f9-lwxbz\" (UID: \"9b5e6174-7853-4fc5-913e-7bbbecef37d4\") " pod="ai-tenant-e2e-shared-b-f84011/sub-test-model-f84011-kserve-fd9d444f9-lwxbz" Jul 10 00:56:10.963966 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:56:10.963895 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b5e6174-7853-4fc5-913e-7bbbecef37d4-model-cache\") pod \"sub-test-model-f84011-kserve-fd9d444f9-lwxbz\" (UID: \"9b5e6174-7853-4fc5-913e-7bbbecef37d4\") " pod="ai-tenant-e2e-shared-b-f84011/sub-test-model-f84011-kserve-fd9d444f9-lwxbz" Jul 10 00:56:11.065213 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:56:11.065186 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b5e6174-7853-4fc5-913e-7bbbecef37d4-home\") pod \"sub-test-model-f84011-kserve-fd9d444f9-lwxbz\" (UID: \"9b5e6174-7853-4fc5-913e-7bbbecef37d4\") " pod="ai-tenant-e2e-shared-b-f84011/sub-test-model-f84011-kserve-fd9d444f9-lwxbz" Jul 10 00:56:11.065213 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:56:11.065216 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b5e6174-7853-4fc5-913e-7bbbecef37d4-dshm\") pod \"sub-test-model-f84011-kserve-fd9d444f9-lwxbz\" (UID: \"9b5e6174-7853-4fc5-913e-7bbbecef37d4\") " pod="ai-tenant-e2e-shared-b-f84011/sub-test-model-f84011-kserve-fd9d444f9-lwxbz" Jul 10 00:56:11.065444 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:56:11.065259 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b5e6174-7853-4fc5-913e-7bbbecef37d4-tmp-dir\") pod \"sub-test-model-f84011-kserve-fd9d444f9-lwxbz\" (UID: \"9b5e6174-7853-4fc5-913e-7bbbecef37d4\") " pod="ai-tenant-e2e-shared-b-f84011/sub-test-model-f84011-kserve-fd9d444f9-lwxbz" Jul 10 00:56:11.065444 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:56:11.065296 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kwbx\" (UniqueName: \"kubernetes.io/projected/9b5e6174-7853-4fc5-913e-7bbbecef37d4-kube-api-access-6kwbx\") pod \"sub-test-model-f84011-kserve-fd9d444f9-lwxbz\" (UID: \"9b5e6174-7853-4fc5-913e-7bbbecef37d4\") " pod="ai-tenant-e2e-shared-b-f84011/sub-test-model-f84011-kserve-fd9d444f9-lwxbz" Jul 10 00:56:11.065444 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:56:11.065315 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b5e6174-7853-4fc5-913e-7bbbecef37d4-model-cache\") pod \"sub-test-model-f84011-kserve-fd9d444f9-lwxbz\" (UID: \"9b5e6174-7853-4fc5-913e-7bbbecef37d4\") " pod="ai-tenant-e2e-shared-b-f84011/sub-test-model-f84011-kserve-fd9d444f9-lwxbz" Jul 10 00:56:11.065444 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:56:11.065336 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b5e6174-7853-4fc5-913e-7bbbecef37d4-tls-certs\") pod \"sub-test-model-f84011-kserve-fd9d444f9-lwxbz\" (UID: \"9b5e6174-7853-4fc5-913e-7bbbecef37d4\") " pod="ai-tenant-e2e-shared-b-f84011/sub-test-model-f84011-kserve-fd9d444f9-lwxbz" Jul 10 00:56:11.065673 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:56:11.065650 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b5e6174-7853-4fc5-913e-7bbbecef37d4-home\") pod \"sub-test-model-f84011-kserve-fd9d444f9-lwxbz\" (UID: \"9b5e6174-7853-4fc5-913e-7bbbecef37d4\") " pod="ai-tenant-e2e-shared-b-f84011/sub-test-model-f84011-kserve-fd9d444f9-lwxbz" Jul 10 00:56:11.065727 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:56:11.065702 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b5e6174-7853-4fc5-913e-7bbbecef37d4-tmp-dir\") pod \"sub-test-model-f84011-kserve-fd9d444f9-lwxbz\" (UID: \"9b5e6174-7853-4fc5-913e-7bbbecef37d4\") " pod="ai-tenant-e2e-shared-b-f84011/sub-test-model-f84011-kserve-fd9d444f9-lwxbz" Jul 10 00:56:11.065780 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:56:11.065765 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b5e6174-7853-4fc5-913e-7bbbecef37d4-model-cache\") pod \"sub-test-model-f84011-kserve-fd9d444f9-lwxbz\" (UID: \"9b5e6174-7853-4fc5-913e-7bbbecef37d4\") " pod="ai-tenant-e2e-shared-b-f84011/sub-test-model-f84011-kserve-fd9d444f9-lwxbz" Jul 10 00:56:11.067465 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:56:11.067446 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b5e6174-7853-4fc5-913e-7bbbecef37d4-dshm\") pod \"sub-test-model-f84011-kserve-fd9d444f9-lwxbz\" (UID: \"9b5e6174-7853-4fc5-913e-7bbbecef37d4\") " pod="ai-tenant-e2e-shared-b-f84011/sub-test-model-f84011-kserve-fd9d444f9-lwxbz" Jul 10 00:56:11.067790 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:56:11.067775 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b5e6174-7853-4fc5-913e-7bbbecef37d4-tls-certs\") pod \"sub-test-model-f84011-kserve-fd9d444f9-lwxbz\" (UID: \"9b5e6174-7853-4fc5-913e-7bbbecef37d4\") " pod="ai-tenant-e2e-shared-b-f84011/sub-test-model-f84011-kserve-fd9d444f9-lwxbz" Jul 10 00:56:11.072834 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:56:11.072818 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kwbx\" (UniqueName: \"kubernetes.io/projected/9b5e6174-7853-4fc5-913e-7bbbecef37d4-kube-api-access-6kwbx\") pod \"sub-test-model-f84011-kserve-fd9d444f9-lwxbz\" (UID: \"9b5e6174-7853-4fc5-913e-7bbbecef37d4\") " pod="ai-tenant-e2e-shared-b-f84011/sub-test-model-f84011-kserve-fd9d444f9-lwxbz" Jul 10 00:56:11.109847 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:56:11.109820 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f84011/sub-test-model-f84011-kserve-fd9d444f9-lwxbz" Jul 10 00:56:11.242071 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:56:11.242044 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-f84011/sub-test-model-f84011-kserve-fd9d444f9-lwxbz"] Jul 10 00:56:11.243534 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:56:11.243497 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9b5e6174_7853_4fc5_913e_7bbbecef37d4.slice/crio-5c9acf6487e4108905c89efb35fc9fd2576222092db0c3b12b4ec1b2aa34f6b4 WatchSource:0}: Error finding container 5c9acf6487e4108905c89efb35fc9fd2576222092db0c3b12b4ec1b2aa34f6b4: Status 404 returned error can't find the container with id 5c9acf6487e4108905c89efb35fc9fd2576222092db0c3b12b4ec1b2aa34f6b4 Jul 10 00:56:11.587147 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:56:11.587111 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f84011/sub-test-model-f84011-kserve-fd9d444f9-lwxbz" event={"ID":"9b5e6174-7853-4fc5-913e-7bbbecef37d4","Type":"ContainerStarted","Data":"fc35d74737bd42556b5af669decd5da79f834eb4d29a3b2842023630424028a2"} Jul 10 00:56:11.587316 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:56:11.587154 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f84011/sub-test-model-f84011-kserve-fd9d444f9-lwxbz" event={"ID":"9b5e6174-7853-4fc5-913e-7bbbecef37d4","Type":"ContainerStarted","Data":"5c9acf6487e4108905c89efb35fc9fd2576222092db0c3b12b4ec1b2aa34f6b4"} Jul 10 00:56:11.603371 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:56:11.603324 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-f84011/sub-test-model-f84011-kserve-fd9d444f9-lwxbz" podStartSLOduration=1.603309313 podStartE2EDuration="1.603309313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:56:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:56:11.600725789 +0000 UTC m=+2438.419031284" watchObservedRunningTime="2026-07-10 00:56:11.603309313 +0000 UTC m=+2438.421614867" Jul 10 00:56:21.110453 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:56:21.110414 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-f84011/sub-test-model-f84011-kserve-fd9d444f9-lwxbz" Jul 10 00:56:21.110953 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:56:21.110715 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-f84011/sub-test-model-f84011-kserve-fd9d444f9-lwxbz" Jul 10 00:56:21.122525 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:56:21.122497 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-f84011/sub-test-model-f84011-kserve-fd9d444f9-lwxbz" Jul 10 00:56:21.651000 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:56:21.650972 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-f84011/sub-test-model-f84011-kserve-fd9d444f9-lwxbz" Jul 10 00:56:46.567847 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:56:46.567813 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-f84011/sub-test-model-f84011-kserve-fd9d444f9-lwxbz"] Jul 10 00:57:01.589759 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:01.589695 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-f84011/sub-test-model-f84011-kserve-fd9d444f9-lwxbz" podUID="9b5e6174-7853-4fc5-913e-7bbbecef37d4" containerName="main" containerID="cri-o://fc35d74737bd42556b5af669decd5da79f834eb4d29a3b2842023630424028a2" gracePeriod=45 Jul 10 00:57:01.641310 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:01.641270 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-f84011/sub-test-model-f84011-kserve-fd9d444f9-lwxbz" podUID="9b5e6174-7853-4fc5-913e-7bbbecef37d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": dial tcp 10.132.0.45:8000: connect: connection refused" Jul 10 00:57:01.843046 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:01.842984 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f84011/sub-test-model-f84011-kserve-fd9d444f9-lwxbz" Jul 10 00:57:01.848704 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:01.848669 2562 generic.go:356] "Generic (PLEG): container finished" podID="9b5e6174-7853-4fc5-913e-7bbbecef37d4" containerID="fc35d74737bd42556b5af669decd5da79f834eb4d29a3b2842023630424028a2" exitCode=0 Jul 10 00:57:01.848855 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:01.848705 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f84011/sub-test-model-f84011-kserve-fd9d444f9-lwxbz" Jul 10 00:57:01.848855 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:01.848741 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f84011/sub-test-model-f84011-kserve-fd9d444f9-lwxbz" event={"ID":"9b5e6174-7853-4fc5-913e-7bbbecef37d4","Type":"ContainerDied","Data":"fc35d74737bd42556b5af669decd5da79f834eb4d29a3b2842023630424028a2"} Jul 10 00:57:01.848855 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:01.848766 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f84011/sub-test-model-f84011-kserve-fd9d444f9-lwxbz" event={"ID":"9b5e6174-7853-4fc5-913e-7bbbecef37d4","Type":"ContainerDied","Data":"5c9acf6487e4108905c89efb35fc9fd2576222092db0c3b12b4ec1b2aa34f6b4"} Jul 10 00:57:01.848855 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:01.848782 2562 scope.go:117] "RemoveContainer" containerID="fc35d74737bd42556b5af669decd5da79f834eb4d29a3b2842023630424028a2" Jul 10 00:57:01.857500 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:01.857470 2562 scope.go:117] "RemoveContainer" containerID="fc35d74737bd42556b5af669decd5da79f834eb4d29a3b2842023630424028a2" Jul 10 00:57:01.857775 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:57:01.857756 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc35d74737bd42556b5af669decd5da79f834eb4d29a3b2842023630424028a2\": container with ID starting with fc35d74737bd42556b5af669decd5da79f834eb4d29a3b2842023630424028a2 not found: ID does not exist" containerID="fc35d74737bd42556b5af669decd5da79f834eb4d29a3b2842023630424028a2" Jul 10 00:57:01.857878 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:01.857783 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc35d74737bd42556b5af669decd5da79f834eb4d29a3b2842023630424028a2"} err="failed to get container status \"fc35d74737bd42556b5af669decd5da79f834eb4d29a3b2842023630424028a2\": rpc error: code = NotFound desc = could not find container \"fc35d74737bd42556b5af669decd5da79f834eb4d29a3b2842023630424028a2\": container with ID starting with fc35d74737bd42556b5af669decd5da79f834eb4d29a3b2842023630424028a2 not found: ID does not exist" Jul 10 00:57:02.018867 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:02.018823 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b5e6174-7853-4fc5-913e-7bbbecef37d4-tmp-dir\") pod \"9b5e6174-7853-4fc5-913e-7bbbecef37d4\" (UID: \"9b5e6174-7853-4fc5-913e-7bbbecef37d4\") " Jul 10 00:57:02.019067 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:02.018938 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b5e6174-7853-4fc5-913e-7bbbecef37d4-model-cache\") pod \"9b5e6174-7853-4fc5-913e-7bbbecef37d4\" (UID: \"9b5e6174-7853-4fc5-913e-7bbbecef37d4\") " Jul 10 00:57:02.019067 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:02.018985 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6kwbx\" (UniqueName: \"kubernetes.io/projected/9b5e6174-7853-4fc5-913e-7bbbecef37d4-kube-api-access-6kwbx\") pod \"9b5e6174-7853-4fc5-913e-7bbbecef37d4\" (UID: \"9b5e6174-7853-4fc5-913e-7bbbecef37d4\") " Jul 10 00:57:02.019214 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:02.019069 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b5e6174-7853-4fc5-913e-7bbbecef37d4-tls-certs\") pod \"9b5e6174-7853-4fc5-913e-7bbbecef37d4\" (UID: \"9b5e6174-7853-4fc5-913e-7bbbecef37d4\") " Jul 10 00:57:02.019214 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:02.019113 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b5e6174-7853-4fc5-913e-7bbbecef37d4-home\") pod \"9b5e6174-7853-4fc5-913e-7bbbecef37d4\" (UID: \"9b5e6174-7853-4fc5-913e-7bbbecef37d4\") " Jul 10 00:57:02.019214 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:02.019139 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b5e6174-7853-4fc5-913e-7bbbecef37d4-dshm\") pod \"9b5e6174-7853-4fc5-913e-7bbbecef37d4\" (UID: \"9b5e6174-7853-4fc5-913e-7bbbecef37d4\") " Jul 10 00:57:02.020283 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:02.020090 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b5e6174-7853-4fc5-913e-7bbbecef37d4-home" (OuterVolumeSpecName: "home") pod "9b5e6174-7853-4fc5-913e-7bbbecef37d4" (UID: "9b5e6174-7853-4fc5-913e-7bbbecef37d4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:57:02.020283 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:02.020148 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b5e6174-7853-4fc5-913e-7bbbecef37d4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9b5e6174-7853-4fc5-913e-7bbbecef37d4" (UID: "9b5e6174-7853-4fc5-913e-7bbbecef37d4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:57:02.020283 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:02.020243 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b5e6174-7853-4fc5-913e-7bbbecef37d4-model-cache" (OuterVolumeSpecName: "model-cache") pod "9b5e6174-7853-4fc5-913e-7bbbecef37d4" (UID: "9b5e6174-7853-4fc5-913e-7bbbecef37d4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:57:02.022382 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:02.022321 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b5e6174-7853-4fc5-913e-7bbbecef37d4-dshm" (OuterVolumeSpecName: "dshm") pod "9b5e6174-7853-4fc5-913e-7bbbecef37d4" (UID: "9b5e6174-7853-4fc5-913e-7bbbecef37d4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:57:02.030214 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:02.030185 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b5e6174-7853-4fc5-913e-7bbbecef37d4-kube-api-access-6kwbx" (OuterVolumeSpecName: "kube-api-access-6kwbx") pod "9b5e6174-7853-4fc5-913e-7bbbecef37d4" (UID: "9b5e6174-7853-4fc5-913e-7bbbecef37d4"). InnerVolumeSpecName "kube-api-access-6kwbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:57:02.030959 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:02.030934 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b5e6174-7853-4fc5-913e-7bbbecef37d4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9b5e6174-7853-4fc5-913e-7bbbecef37d4" (UID: "9b5e6174-7853-4fc5-913e-7bbbecef37d4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:57:02.119945 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:02.119837 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b5e6174-7853-4fc5-913e-7bbbecef37d4-tls-certs\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:57:02.119945 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:02.119867 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b5e6174-7853-4fc5-913e-7bbbecef37d4-home\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:57:02.119945 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:02.119875 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b5e6174-7853-4fc5-913e-7bbbecef37d4-dshm\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:57:02.119945 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:02.119882 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b5e6174-7853-4fc5-913e-7bbbecef37d4-tmp-dir\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:57:02.119945 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:02.119891 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b5e6174-7853-4fc5-913e-7bbbecef37d4-model-cache\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:57:02.119945 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:02.119899 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kwbx\" (UniqueName: \"kubernetes.io/projected/9b5e6174-7853-4fc5-913e-7bbbecef37d4-kube-api-access-6kwbx\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:57:02.174483 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:02.174452 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-f84011/sub-test-model-f84011-kserve-fd9d444f9-lwxbz"] Jul 10 00:57:02.177975 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:02.177949 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-f84011/sub-test-model-f84011-kserve-fd9d444f9-lwxbz"] Jul 10 00:57:03.735861 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:03.735826 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b5e6174-7853-4fc5-913e-7bbbecef37d4" path="/var/lib/kubelet/pods/9b5e6174-7853-4fc5-913e-7bbbecef37d4/volumes" Jul 10 00:57:19.094427 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:19.094397 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-f84011/rate-test-model-f84011-kserve-7bdb9dddf9-9b97j"] Jul 10 00:57:19.094891 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:19.094738 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b5e6174-7853-4fc5-913e-7bbbecef37d4" containerName="main" Jul 10 00:57:19.094891 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:19.094749 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b5e6174-7853-4fc5-913e-7bbbecef37d4" containerName="main" Jul 10 00:57:19.094891 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:19.094805 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b5e6174-7853-4fc5-913e-7bbbecef37d4" containerName="main" Jul 10 00:57:19.097472 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:19.097456 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f84011/rate-test-model-f84011-kserve-7bdb9dddf9-9b97j" Jul 10 00:57:19.100011 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:19.099988 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-f84011\"/\"kube-root-ca.crt\"" Jul 10 00:57:19.100011 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:19.100000 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-f84011\"/\"rate-test-model-f84011-kserve-self-signed-certs\"" Jul 10 00:57:19.100217 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:19.100042 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-f84011\"/\"default-dockercfg-8qj2f\"" Jul 10 00:57:19.100217 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:19.100138 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-f84011\"/\"openshift-service-ca.crt\"" Jul 10 00:57:19.105613 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:19.105591 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-f84011/rate-test-model-f84011-kserve-7bdb9dddf9-9b97j"] Jul 10 00:57:19.164793 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:19.164762 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25b3b776-d45c-45ef-9315-ef8ab25c8cb4-model-cache\") pod \"rate-test-model-f84011-kserve-7bdb9dddf9-9b97j\" (UID: \"25b3b776-d45c-45ef-9315-ef8ab25c8cb4\") " pod="ai-tenant-e2e-shared-b-f84011/rate-test-model-f84011-kserve-7bdb9dddf9-9b97j" Jul 10 00:57:19.164938 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:19.164812 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25b3b776-d45c-45ef-9315-ef8ab25c8cb4-home\") pod \"rate-test-model-f84011-kserve-7bdb9dddf9-9b97j\" (UID: \"25b3b776-d45c-45ef-9315-ef8ab25c8cb4\") " pod="ai-tenant-e2e-shared-b-f84011/rate-test-model-f84011-kserve-7bdb9dddf9-9b97j" Jul 10 00:57:19.164938 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:19.164829 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25b3b776-d45c-45ef-9315-ef8ab25c8cb4-tmp-dir\") pod \"rate-test-model-f84011-kserve-7bdb9dddf9-9b97j\" (UID: \"25b3b776-d45c-45ef-9315-ef8ab25c8cb4\") " pod="ai-tenant-e2e-shared-b-f84011/rate-test-model-f84011-kserve-7bdb9dddf9-9b97j" Jul 10 00:57:19.165031 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:19.164978 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25b3b776-d45c-45ef-9315-ef8ab25c8cb4-dshm\") pod \"rate-test-model-f84011-kserve-7bdb9dddf9-9b97j\" (UID: \"25b3b776-d45c-45ef-9315-ef8ab25c8cb4\") " pod="ai-tenant-e2e-shared-b-f84011/rate-test-model-f84011-kserve-7bdb9dddf9-9b97j" Jul 10 00:57:19.165031 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:19.165006 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25b3b776-d45c-45ef-9315-ef8ab25c8cb4-tls-certs\") pod \"rate-test-model-f84011-kserve-7bdb9dddf9-9b97j\" (UID: \"25b3b776-d45c-45ef-9315-ef8ab25c8cb4\") " pod="ai-tenant-e2e-shared-b-f84011/rate-test-model-f84011-kserve-7bdb9dddf9-9b97j" Jul 10 00:57:19.165124 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:19.165028 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxcrq\" (UniqueName: \"kubernetes.io/projected/25b3b776-d45c-45ef-9315-ef8ab25c8cb4-kube-api-access-bxcrq\") pod \"rate-test-model-f84011-kserve-7bdb9dddf9-9b97j\" (UID: \"25b3b776-d45c-45ef-9315-ef8ab25c8cb4\") " pod="ai-tenant-e2e-shared-b-f84011/rate-test-model-f84011-kserve-7bdb9dddf9-9b97j" Jul 10 00:57:19.265730 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:19.265698 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25b3b776-d45c-45ef-9315-ef8ab25c8cb4-dshm\") pod \"rate-test-model-f84011-kserve-7bdb9dddf9-9b97j\" (UID: \"25b3b776-d45c-45ef-9315-ef8ab25c8cb4\") " pod="ai-tenant-e2e-shared-b-f84011/rate-test-model-f84011-kserve-7bdb9dddf9-9b97j" Jul 10 00:57:19.265730 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:19.265733 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25b3b776-d45c-45ef-9315-ef8ab25c8cb4-tls-certs\") pod \"rate-test-model-f84011-kserve-7bdb9dddf9-9b97j\" (UID: \"25b3b776-d45c-45ef-9315-ef8ab25c8cb4\") " pod="ai-tenant-e2e-shared-b-f84011/rate-test-model-f84011-kserve-7bdb9dddf9-9b97j" Jul 10 00:57:19.266002 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:19.265751 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxcrq\" (UniqueName: \"kubernetes.io/projected/25b3b776-d45c-45ef-9315-ef8ab25c8cb4-kube-api-access-bxcrq\") pod \"rate-test-model-f84011-kserve-7bdb9dddf9-9b97j\" (UID: \"25b3b776-d45c-45ef-9315-ef8ab25c8cb4\") " pod="ai-tenant-e2e-shared-b-f84011/rate-test-model-f84011-kserve-7bdb9dddf9-9b97j" Jul 10 00:57:19.266002 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:19.265785 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25b3b776-d45c-45ef-9315-ef8ab25c8cb4-model-cache\") pod \"rate-test-model-f84011-kserve-7bdb9dddf9-9b97j\" (UID: \"25b3b776-d45c-45ef-9315-ef8ab25c8cb4\") " pod="ai-tenant-e2e-shared-b-f84011/rate-test-model-f84011-kserve-7bdb9dddf9-9b97j" Jul 10 00:57:19.266002 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:19.265831 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25b3b776-d45c-45ef-9315-ef8ab25c8cb4-home\") pod \"rate-test-model-f84011-kserve-7bdb9dddf9-9b97j\" (UID: \"25b3b776-d45c-45ef-9315-ef8ab25c8cb4\") " pod="ai-tenant-e2e-shared-b-f84011/rate-test-model-f84011-kserve-7bdb9dddf9-9b97j" Jul 10 00:57:19.266002 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:19.265856 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25b3b776-d45c-45ef-9315-ef8ab25c8cb4-tmp-dir\") pod \"rate-test-model-f84011-kserve-7bdb9dddf9-9b97j\" (UID: \"25b3b776-d45c-45ef-9315-ef8ab25c8cb4\") " pod="ai-tenant-e2e-shared-b-f84011/rate-test-model-f84011-kserve-7bdb9dddf9-9b97j" Jul 10 00:57:19.266352 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:19.266326 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25b3b776-d45c-45ef-9315-ef8ab25c8cb4-tmp-dir\") pod \"rate-test-model-f84011-kserve-7bdb9dddf9-9b97j\" (UID: \"25b3b776-d45c-45ef-9315-ef8ab25c8cb4\") " pod="ai-tenant-e2e-shared-b-f84011/rate-test-model-f84011-kserve-7bdb9dddf9-9b97j" Jul 10 00:57:19.266537 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:19.266486 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25b3b776-d45c-45ef-9315-ef8ab25c8cb4-model-cache\") pod \"rate-test-model-f84011-kserve-7bdb9dddf9-9b97j\" (UID: \"25b3b776-d45c-45ef-9315-ef8ab25c8cb4\") " pod="ai-tenant-e2e-shared-b-f84011/rate-test-model-f84011-kserve-7bdb9dddf9-9b97j" Jul 10 00:57:19.266628 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:19.266599 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25b3b776-d45c-45ef-9315-ef8ab25c8cb4-home\") pod \"rate-test-model-f84011-kserve-7bdb9dddf9-9b97j\" (UID: \"25b3b776-d45c-45ef-9315-ef8ab25c8cb4\") " pod="ai-tenant-e2e-shared-b-f84011/rate-test-model-f84011-kserve-7bdb9dddf9-9b97j" Jul 10 00:57:19.268817 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:19.268793 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25b3b776-d45c-45ef-9315-ef8ab25c8cb4-dshm\") pod \"rate-test-model-f84011-kserve-7bdb9dddf9-9b97j\" (UID: \"25b3b776-d45c-45ef-9315-ef8ab25c8cb4\") " pod="ai-tenant-e2e-shared-b-f84011/rate-test-model-f84011-kserve-7bdb9dddf9-9b97j" Jul 10 00:57:19.269143 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:19.269112 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25b3b776-d45c-45ef-9315-ef8ab25c8cb4-tls-certs\") pod \"rate-test-model-f84011-kserve-7bdb9dddf9-9b97j\" (UID: \"25b3b776-d45c-45ef-9315-ef8ab25c8cb4\") " pod="ai-tenant-e2e-shared-b-f84011/rate-test-model-f84011-kserve-7bdb9dddf9-9b97j" Jul 10 00:57:19.273737 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:19.273711 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxcrq\" (UniqueName: \"kubernetes.io/projected/25b3b776-d45c-45ef-9315-ef8ab25c8cb4-kube-api-access-bxcrq\") pod \"rate-test-model-f84011-kserve-7bdb9dddf9-9b97j\" (UID: \"25b3b776-d45c-45ef-9315-ef8ab25c8cb4\") " pod="ai-tenant-e2e-shared-b-f84011/rate-test-model-f84011-kserve-7bdb9dddf9-9b97j" Jul 10 00:57:19.408250 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:19.408147 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f84011/rate-test-model-f84011-kserve-7bdb9dddf9-9b97j" Jul 10 00:57:19.539470 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:19.539441 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-f84011/rate-test-model-f84011-kserve-7bdb9dddf9-9b97j"] Jul 10 00:57:19.541667 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:57:19.541636 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod25b3b776_d45c_45ef_9315_ef8ab25c8cb4.slice/crio-6de844f9c2bb46cd044ce3f1bc8e3d2f78f362164f66478dd20ea0f75c46df36 WatchSource:0}: Error finding container 6de844f9c2bb46cd044ce3f1bc8e3d2f78f362164f66478dd20ea0f75c46df36: Status 404 returned error can't find the container with id 6de844f9c2bb46cd044ce3f1bc8e3d2f78f362164f66478dd20ea0f75c46df36 Jul 10 00:57:19.945311 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:19.945277 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f84011/rate-test-model-f84011-kserve-7bdb9dddf9-9b97j" event={"ID":"25b3b776-d45c-45ef-9315-ef8ab25c8cb4","Type":"ContainerStarted","Data":"24bac16c8eee0dd23a24b676e0b0e8cf3faa7dab96cb86bc1fbc0a8c1b2e2026"} Jul 10 00:57:19.945311 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:19.945317 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f84011/rate-test-model-f84011-kserve-7bdb9dddf9-9b97j" event={"ID":"25b3b776-d45c-45ef-9315-ef8ab25c8cb4","Type":"ContainerStarted","Data":"6de844f9c2bb46cd044ce3f1bc8e3d2f78f362164f66478dd20ea0f75c46df36"} Jul 10 00:57:19.961114 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:19.961056 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-f84011/rate-test-model-f84011-kserve-7bdb9dddf9-9b97j" podStartSLOduration=0.961042202 podStartE2EDuration="961.042202ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:57:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:57:19.960282402 +0000 UTC m=+2506.778587893" watchObservedRunningTime="2026-07-10 00:57:19.961042202 +0000 UTC m=+2506.779347696" Jul 10 00:57:29.408489 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:29.408404 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-f84011/rate-test-model-f84011-kserve-7bdb9dddf9-9b97j" Jul 10 00:57:29.408489 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:29.408454 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-f84011/rate-test-model-f84011-kserve-7bdb9dddf9-9b97j" Jul 10 00:57:29.420419 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:29.420394 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-f84011/rate-test-model-f84011-kserve-7bdb9dddf9-9b97j" Jul 10 00:57:30.005489 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:57:30.005461 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-f84011/rate-test-model-f84011-kserve-7bdb9dddf9-9b97j" Jul 10 00:58:08.978137 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:08.978101 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-f84011/rate-test-model-f84011-kserve-7bdb9dddf9-9b97j"] Jul 10 00:58:18.788773 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:18.788744 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7610-140103577319296"] Jul 10 00:58:18.792111 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:18.792093 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7610-140103577319296" Jul 10 00:58:18.794419 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:18.794121 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-v5fdm\"" Jul 10 00:58:18.797499 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:18.797478 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7610-140103577319296"] Jul 10 00:58:18.887766 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:18.887731 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8gn6\" (UniqueName: \"kubernetes.io/projected/13da1a3a-8431-4ef2-8de9-ac390ab6515b-kube-api-access-p8gn6\") pod \"test-curl-7610-140103577319296\" (UID: \"13da1a3a-8431-4ef2-8de9-ac390ab6515b\") " pod="openshift-ingress/test-curl-7610-140103577319296" Jul 10 00:58:18.989211 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:18.989177 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8gn6\" (UniqueName: \"kubernetes.io/projected/13da1a3a-8431-4ef2-8de9-ac390ab6515b-kube-api-access-p8gn6\") pod \"test-curl-7610-140103577319296\" (UID: \"13da1a3a-8431-4ef2-8de9-ac390ab6515b\") " pod="openshift-ingress/test-curl-7610-140103577319296" Jul 10 00:58:18.996420 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:18.996395 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8gn6\" (UniqueName: \"kubernetes.io/projected/13da1a3a-8431-4ef2-8de9-ac390ab6515b-kube-api-access-p8gn6\") pod \"test-curl-7610-140103577319296\" (UID: \"13da1a3a-8431-4ef2-8de9-ac390ab6515b\") " pod="openshift-ingress/test-curl-7610-140103577319296" Jul 10 00:58:19.104190 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:19.104148 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7610-140103577319296" Jul 10 00:58:19.233157 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:19.233130 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7610-140103577319296"] Jul 10 00:58:19.234754 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:58:19.234725 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod13da1a3a_8431_4ef2_8de9_ac390ab6515b.slice/crio-577a909e13f01737be838e465b065c12cec5f04c235d24f56c07850c7e3b177d WatchSource:0}: Error finding container 577a909e13f01737be838e465b065c12cec5f04c235d24f56c07850c7e3b177d: Status 404 returned error can't find the container with id 577a909e13f01737be838e465b065c12cec5f04c235d24f56c07850c7e3b177d Jul 10 00:58:19.236060 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:19.236042 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 00:58:19.249016 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:19.248988 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7610-140103577319296" event={"ID":"13da1a3a-8431-4ef2-8de9-ac390ab6515b","Type":"ContainerStarted","Data":"577a909e13f01737be838e465b065c12cec5f04c235d24f56c07850c7e3b177d"} Jul 10 00:58:20.256221 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:20.256136 2562 generic.go:356] "Generic (PLEG): container finished" podID="13da1a3a-8431-4ef2-8de9-ac390ab6515b" containerID="e5dc9ee266562ddee5308c6dbbcac23f86610c153244b98a44698c0576dde02f" exitCode=0 Jul 10 00:58:20.256221 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:20.256195 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7610-140103577319296" event={"ID":"13da1a3a-8431-4ef2-8de9-ac390ab6515b","Type":"ContainerDied","Data":"e5dc9ee266562ddee5308c6dbbcac23f86610c153244b98a44698c0576dde02f"} Jul 10 00:58:21.393001 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:21.392976 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7610-140103577319296" Jul 10 00:58:21.424735 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:21.424710 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7610-140103577319296_13da1a3a-8431-4ef2-8de9-ac390ab6515b/test-curl-7610-140103577319296/0.log" Jul 10 00:58:21.447400 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:21.447373 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7610-140103577319296"] Jul 10 00:58:21.450002 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:21.449981 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7610-140103577319296"] Jul 10 00:58:21.513294 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:21.513237 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p8gn6\" (UniqueName: \"kubernetes.io/projected/13da1a3a-8431-4ef2-8de9-ac390ab6515b-kube-api-access-p8gn6\") pod \"13da1a3a-8431-4ef2-8de9-ac390ab6515b\" (UID: \"13da1a3a-8431-4ef2-8de9-ac390ab6515b\") " Jul 10 00:58:21.515222 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:21.515198 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13da1a3a-8431-4ef2-8de9-ac390ab6515b-kube-api-access-p8gn6" (OuterVolumeSpecName: "kube-api-access-p8gn6") pod "13da1a3a-8431-4ef2-8de9-ac390ab6515b" (UID: "13da1a3a-8431-4ef2-8de9-ac390ab6515b"). InnerVolumeSpecName "kube-api-access-p8gn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:58:21.576462 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:21.576431 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7610-140103577320304"] Jul 10 00:58:21.576816 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:21.576798 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13da1a3a-8431-4ef2-8de9-ac390ab6515b" containerName="test-curl-7610-140103577319296" Jul 10 00:58:21.576903 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:21.576818 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="13da1a3a-8431-4ef2-8de9-ac390ab6515b" containerName="test-curl-7610-140103577319296" Jul 10 00:58:21.576961 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:21.576923 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="13da1a3a-8431-4ef2-8de9-ac390ab6515b" containerName="test-curl-7610-140103577319296" Jul 10 00:58:21.579897 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:21.579880 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7610-140103577320304" Jul 10 00:58:21.585945 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:21.585918 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7610-140103577320304"] Jul 10 00:58:21.614212 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:21.614188 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8gn6\" (UniqueName: \"kubernetes.io/projected/13da1a3a-8431-4ef2-8de9-ac390ab6515b-kube-api-access-p8gn6\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:58:21.715414 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:21.715390 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgkdw\" (UniqueName: \"kubernetes.io/projected/1065ba1d-8d1d-486f-b1e8-793e20dc4578-kube-api-access-vgkdw\") pod \"test-curl-7610-140103577320304\" (UID: \"1065ba1d-8d1d-486f-b1e8-793e20dc4578\") " pod="openshift-ingress/test-curl-7610-140103577320304" Jul 10 00:58:21.734972 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:21.734944 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13da1a3a-8431-4ef2-8de9-ac390ab6515b" path="/var/lib/kubelet/pods/13da1a3a-8431-4ef2-8de9-ac390ab6515b/volumes" Jul 10 00:58:21.815886 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:21.815859 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgkdw\" (UniqueName: \"kubernetes.io/projected/1065ba1d-8d1d-486f-b1e8-793e20dc4578-kube-api-access-vgkdw\") pod \"test-curl-7610-140103577320304\" (UID: \"1065ba1d-8d1d-486f-b1e8-793e20dc4578\") " pod="openshift-ingress/test-curl-7610-140103577320304" Jul 10 00:58:21.823030 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:21.823009 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgkdw\" (UniqueName: \"kubernetes.io/projected/1065ba1d-8d1d-486f-b1e8-793e20dc4578-kube-api-access-vgkdw\") pod \"test-curl-7610-140103577320304\" (UID: \"1065ba1d-8d1d-486f-b1e8-793e20dc4578\") " pod="openshift-ingress/test-curl-7610-140103577320304" Jul 10 00:58:21.893705 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:21.893677 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7610-140103577320304" Jul 10 00:58:22.019412 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:22.019380 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7610-140103577320304"] Jul 10 00:58:22.020515 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:58:22.020486 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1065ba1d_8d1d_486f_b1e8_793e20dc4578.slice/crio-cc799c3c99d6d89471a9b1b119b62fb5f370f405239284f0b31f25fea4de3991 WatchSource:0}: Error finding container cc799c3c99d6d89471a9b1b119b62fb5f370f405239284f0b31f25fea4de3991: Status 404 returned error can't find the container with id cc799c3c99d6d89471a9b1b119b62fb5f370f405239284f0b31f25fea4de3991 Jul 10 00:58:22.267278 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:22.267193 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7610-140103577319296" Jul 10 00:58:22.267278 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:22.267213 2562 scope.go:117] "RemoveContainer" containerID="e5dc9ee266562ddee5308c6dbbcac23f86610c153244b98a44698c0576dde02f" Jul 10 00:58:22.268548 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:22.268447 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7610-140103577320304" event={"ID":"1065ba1d-8d1d-486f-b1e8-793e20dc4578","Type":"ContainerStarted","Data":"cc799c3c99d6d89471a9b1b119b62fb5f370f405239284f0b31f25fea4de3991"} Jul 10 00:58:23.275580 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:23.275545 2562 generic.go:356] "Generic (PLEG): container finished" podID="1065ba1d-8d1d-486f-b1e8-793e20dc4578" containerID="f072f59749fa3ecd5d072ce68551c1f903d406cab5f543c086ba45dafa1950d8" exitCode=0 Jul 10 00:58:23.275965 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:23.275630 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7610-140103577320304" event={"ID":"1065ba1d-8d1d-486f-b1e8-793e20dc4578","Type":"ContainerDied","Data":"f072f59749fa3ecd5d072ce68551c1f903d406cab5f543c086ba45dafa1950d8"} Jul 10 00:58:23.997155 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:23.997115 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-f84011/rate-test-model-f84011-kserve-7bdb9dddf9-9b97j" podUID="25b3b776-d45c-45ef-9315-ef8ab25c8cb4" containerName="main" containerID="cri-o://24bac16c8eee0dd23a24b676e0b0e8cf3faa7dab96cb86bc1fbc0a8c1b2e2026" gracePeriod=45 Jul 10 00:58:24.242635 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:24.242608 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f84011/rate-test-model-f84011-kserve-7bdb9dddf9-9b97j" Jul 10 00:58:24.282577 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:24.282509 2562 generic.go:356] "Generic (PLEG): container finished" podID="25b3b776-d45c-45ef-9315-ef8ab25c8cb4" containerID="24bac16c8eee0dd23a24b676e0b0e8cf3faa7dab96cb86bc1fbc0a8c1b2e2026" exitCode=0 Jul 10 00:58:24.282577 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:24.282550 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f84011/rate-test-model-f84011-kserve-7bdb9dddf9-9b97j" Jul 10 00:58:24.282979 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:24.282580 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f84011/rate-test-model-f84011-kserve-7bdb9dddf9-9b97j" event={"ID":"25b3b776-d45c-45ef-9315-ef8ab25c8cb4","Type":"ContainerDied","Data":"24bac16c8eee0dd23a24b676e0b0e8cf3faa7dab96cb86bc1fbc0a8c1b2e2026"} Jul 10 00:58:24.282979 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:24.282617 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f84011/rate-test-model-f84011-kserve-7bdb9dddf9-9b97j" event={"ID":"25b3b776-d45c-45ef-9315-ef8ab25c8cb4","Type":"ContainerDied","Data":"6de844f9c2bb46cd044ce3f1bc8e3d2f78f362164f66478dd20ea0f75c46df36"} Jul 10 00:58:24.282979 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:24.282638 2562 scope.go:117] "RemoveContainer" containerID="24bac16c8eee0dd23a24b676e0b0e8cf3faa7dab96cb86bc1fbc0a8c1b2e2026" Jul 10 00:58:24.291156 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:24.291121 2562 scope.go:117] "RemoveContainer" containerID="24bac16c8eee0dd23a24b676e0b0e8cf3faa7dab96cb86bc1fbc0a8c1b2e2026" Jul 10 00:58:24.291373 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:58:24.291355 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24bac16c8eee0dd23a24b676e0b0e8cf3faa7dab96cb86bc1fbc0a8c1b2e2026\": container with ID starting with 24bac16c8eee0dd23a24b676e0b0e8cf3faa7dab96cb86bc1fbc0a8c1b2e2026 not found: ID does not exist" containerID="24bac16c8eee0dd23a24b676e0b0e8cf3faa7dab96cb86bc1fbc0a8c1b2e2026" Jul 10 00:58:24.291432 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:24.291381 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24bac16c8eee0dd23a24b676e0b0e8cf3faa7dab96cb86bc1fbc0a8c1b2e2026"} err="failed to get container status \"24bac16c8eee0dd23a24b676e0b0e8cf3faa7dab96cb86bc1fbc0a8c1b2e2026\": rpc error: code = NotFound desc = could not find container \"24bac16c8eee0dd23a24b676e0b0e8cf3faa7dab96cb86bc1fbc0a8c1b2e2026\": container with ID starting with 24bac16c8eee0dd23a24b676e0b0e8cf3faa7dab96cb86bc1fbc0a8c1b2e2026 not found: ID does not exist" Jul 10 00:58:24.336433 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:24.336408 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25b3b776-d45c-45ef-9315-ef8ab25c8cb4-model-cache\") pod \"25b3b776-d45c-45ef-9315-ef8ab25c8cb4\" (UID: \"25b3b776-d45c-45ef-9315-ef8ab25c8cb4\") " Jul 10 00:58:24.336579 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:24.336464 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25b3b776-d45c-45ef-9315-ef8ab25c8cb4-dshm\") pod \"25b3b776-d45c-45ef-9315-ef8ab25c8cb4\" (UID: \"25b3b776-d45c-45ef-9315-ef8ab25c8cb4\") " Jul 10 00:58:24.336579 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:24.336507 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25b3b776-d45c-45ef-9315-ef8ab25c8cb4-home\") pod \"25b3b776-d45c-45ef-9315-ef8ab25c8cb4\" (UID: \"25b3b776-d45c-45ef-9315-ef8ab25c8cb4\") " Jul 10 00:58:24.336579 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:24.336538 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25b3b776-d45c-45ef-9315-ef8ab25c8cb4-tls-certs\") pod \"25b3b776-d45c-45ef-9315-ef8ab25c8cb4\" (UID: \"25b3b776-d45c-45ef-9315-ef8ab25c8cb4\") " Jul 10 00:58:24.336579 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:24.336578 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25b3b776-d45c-45ef-9315-ef8ab25c8cb4-tmp-dir\") pod \"25b3b776-d45c-45ef-9315-ef8ab25c8cb4\" (UID: \"25b3b776-d45c-45ef-9315-ef8ab25c8cb4\") " Jul 10 00:58:24.336804 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:24.336623 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bxcrq\" (UniqueName: \"kubernetes.io/projected/25b3b776-d45c-45ef-9315-ef8ab25c8cb4-kube-api-access-bxcrq\") pod \"25b3b776-d45c-45ef-9315-ef8ab25c8cb4\" (UID: \"25b3b776-d45c-45ef-9315-ef8ab25c8cb4\") " Jul 10 00:58:24.336804 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:24.336657 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25b3b776-d45c-45ef-9315-ef8ab25c8cb4-model-cache" (OuterVolumeSpecName: "model-cache") pod "25b3b776-d45c-45ef-9315-ef8ab25c8cb4" (UID: "25b3b776-d45c-45ef-9315-ef8ab25c8cb4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:58:24.337157 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:24.336947 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25b3b776-d45c-45ef-9315-ef8ab25c8cb4-model-cache\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:58:24.337157 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:24.337104 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25b3b776-d45c-45ef-9315-ef8ab25c8cb4-home" (OuterVolumeSpecName: "home") pod "25b3b776-d45c-45ef-9315-ef8ab25c8cb4" (UID: "25b3b776-d45c-45ef-9315-ef8ab25c8cb4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:58:24.337338 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:24.337185 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25b3b776-d45c-45ef-9315-ef8ab25c8cb4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "25b3b776-d45c-45ef-9315-ef8ab25c8cb4" (UID: "25b3b776-d45c-45ef-9315-ef8ab25c8cb4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:58:24.339013 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:24.338987 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25b3b776-d45c-45ef-9315-ef8ab25c8cb4-kube-api-access-bxcrq" (OuterVolumeSpecName: "kube-api-access-bxcrq") pod "25b3b776-d45c-45ef-9315-ef8ab25c8cb4" (UID: "25b3b776-d45c-45ef-9315-ef8ab25c8cb4"). InnerVolumeSpecName "kube-api-access-bxcrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:58:24.340733 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:24.340711 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25b3b776-d45c-45ef-9315-ef8ab25c8cb4-dshm" (OuterVolumeSpecName: "dshm") pod "25b3b776-d45c-45ef-9315-ef8ab25c8cb4" (UID: "25b3b776-d45c-45ef-9315-ef8ab25c8cb4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:58:24.340828 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:24.340709 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25b3b776-d45c-45ef-9315-ef8ab25c8cb4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "25b3b776-d45c-45ef-9315-ef8ab25c8cb4" (UID: "25b3b776-d45c-45ef-9315-ef8ab25c8cb4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:58:24.417088 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:24.417048 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7610-140103577320304" Jul 10 00:58:24.434001 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:24.433971 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7610-140103577320304_1065ba1d-8d1d-486f-b1e8-793e20dc4578/test-curl-7610-140103577320304/0.log" Jul 10 00:58:24.438275 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:24.438248 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25b3b776-d45c-45ef-9315-ef8ab25c8cb4-dshm\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:58:24.438275 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:24.438270 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25b3b776-d45c-45ef-9315-ef8ab25c8cb4-home\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:58:24.438275 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:24.438278 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25b3b776-d45c-45ef-9315-ef8ab25c8cb4-tls-certs\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:58:24.438516 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:24.438289 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25b3b776-d45c-45ef-9315-ef8ab25c8cb4-tmp-dir\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:58:24.438516 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:24.438297 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxcrq\" (UniqueName: \"kubernetes.io/projected/25b3b776-d45c-45ef-9315-ef8ab25c8cb4-kube-api-access-bxcrq\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:58:24.455953 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:24.455925 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7610-140103577320304"] Jul 10 00:58:24.459902 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:24.459880 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7610-140103577320304"] Jul 10 00:58:24.538943 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:24.538871 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vgkdw\" (UniqueName: \"kubernetes.io/projected/1065ba1d-8d1d-486f-b1e8-793e20dc4578-kube-api-access-vgkdw\") pod \"1065ba1d-8d1d-486f-b1e8-793e20dc4578\" (UID: \"1065ba1d-8d1d-486f-b1e8-793e20dc4578\") " Jul 10 00:58:24.540870 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:24.540840 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1065ba1d-8d1d-486f-b1e8-793e20dc4578-kube-api-access-vgkdw" (OuterVolumeSpecName: "kube-api-access-vgkdw") pod "1065ba1d-8d1d-486f-b1e8-793e20dc4578" (UID: "1065ba1d-8d1d-486f-b1e8-793e20dc4578"). InnerVolumeSpecName "kube-api-access-vgkdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:58:24.602614 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:24.602582 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-f84011/rate-test-model-f84011-kserve-7bdb9dddf9-9b97j"] Jul 10 00:58:24.604764 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:24.604737 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-f84011/rate-test-model-f84011-kserve-7bdb9dddf9-9b97j"] Jul 10 00:58:24.639680 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:24.639655 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgkdw\" (UniqueName: \"kubernetes.io/projected/1065ba1d-8d1d-486f-b1e8-793e20dc4578-kube-api-access-vgkdw\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 10 00:58:25.288683 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:25.288600 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7610-140103577320304" Jul 10 00:58:25.288683 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:25.288640 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc799c3c99d6d89471a9b1b119b62fb5f370f405239284f0b31f25fea4de3991" Jul 10 00:58:25.297856 ip-10-0-135-101 kubenswrapper[2562]: E0710 00:58:25.297828 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7610-140103577320304\" is forbidden: User \"system:node:ip-10-0-135-101.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-135-101.ec2.internal' and this object" podUID="1065ba1d-8d1d-486f-b1e8-793e20dc4578" pod="openshift-ingress/test-curl-7610-140103577320304" Jul 10 00:58:25.741065 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:25.741030 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1065ba1d-8d1d-486f-b1e8-793e20dc4578" path="/var/lib/kubelet/pods/1065ba1d-8d1d-486f-b1e8-793e20dc4578/volumes" Jul 10 00:58:25.741463 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:25.741448 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25b3b776-d45c-45ef-9315-ef8ab25c8cb4" path="/var/lib/kubelet/pods/25b3b776-d45c-45ef-9315-ef8ab25c8cb4/volumes" Jul 10 00:58:41.046828 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:41.046786 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-9c4cbf5d7-n88zq_7be3328e-4d6e-4dee-a615-90d6ade4f629/authorino/0.log" Jul 10 00:58:46.767682 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:46.767655 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-559fc4999-lqpff_22d431a4-4965-4605-b603-6773ffd80c0d/copy-manifests/0.log" Jul 10 00:58:46.785539 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:46.785508 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-559fc4999-lqpff_22d431a4-4965-4605-b603-6773ffd80c0d/manager/0.log" Jul 10 00:58:48.179213 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:48.179182 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk85r_080517b7-abe3-4d7f-9937-f3839e9a2417/util/0.log" Jul 10 00:58:48.185453 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:48.185431 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk85r_080517b7-abe3-4d7f-9937-f3839e9a2417/pull/0.log" Jul 10 00:58:48.191600 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:48.191579 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk85r_080517b7-abe3-4d7f-9937-f3839e9a2417/extract/0.log" Jul 10 00:58:48.325249 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:48.325224 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e087669_27fbf742-b6f0-448f-a86a-e4f40ef8b985/util/0.log" Jul 10 00:58:48.331970 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:48.331950 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e087669_27fbf742-b6f0-448f-a86a-e4f40ef8b985/pull/0.log" Jul 10 00:58:48.338446 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:48.338424 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e087669_27fbf742-b6f0-448f-a86a-e4f40ef8b985/extract/0.log" Jul 10 00:58:48.458788 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:48.458718 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7362l79_815d1fa6-90ea-4254-af1a-4be28806be87/extract/0.log" Jul 10 00:58:48.465947 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:48.465927 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7362l79_815d1fa6-90ea-4254-af1a-4be28806be87/util/0.log" Jul 10 00:58:48.472182 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:48.472162 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7362l79_815d1fa6-90ea-4254-af1a-4be28806be87/pull/0.log" Jul 10 00:58:48.587106 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:48.587064 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwhx6_b8c42013-1e7f-4f1c-a84b-d9f05a557e4d/util/0.log" Jul 10 00:58:48.592926 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:48.592907 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwhx6_b8c42013-1e7f-4f1c-a84b-d9f05a557e4d/pull/0.log" Jul 10 00:58:48.598600 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:48.598581 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwhx6_b8c42013-1e7f-4f1c-a84b-d9f05a557e4d/extract/0.log" Jul 10 00:58:48.727136 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:48.727046 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-9c4cbf5d7-n88zq_7be3328e-4d6e-4dee-a615-90d6ade4f629/authorino/0.log" Jul 10 00:58:49.957286 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:49.957255 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf47czbk_d0ba5141-fd5f-4e01-8865-037d6e6743d6/istio-proxy/0.log" Jul 10 00:58:50.315580 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:50.315547 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b9f4c495f-zv55m_f066a2dc-a4c4-4f25-94db-472621e7fc95/kube-auth-proxy/0.log" Jul 10 00:58:50.449650 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:50.449612 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-lhjrd_92786366-f55e-4e7b-b107-8cd178d52054/istio-proxy/1.log" Jul 10 00:58:50.831722 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:50.831684 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-76d8bdc488-dw9ns_3d7b4fce-5123-477b-800c-bec0da7fd372/router/0.log" Jul 10 00:58:51.664090 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:51.664050 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-wrxqq_3227fbbd-9453-4705-96a2-5d62ff23379d/main/0.log" Jul 10 00:58:51.778583 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:51.778548 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-6k2xw_408be265-42a3-4565-abd5-f133b31f0aec/main/0.log" Jul 10 00:58:58.218854 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:58.218821 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-nkcmr_451130fe-547b-40c5-a7dc-412d57fb5db5/global-pull-secret-syncer/0.log" Jul 10 00:58:58.335203 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:58.335168 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-shx2p_dfc778a3-c60f-4f5e-8b6f-3e613228a7f9/konnectivity-agent/0.log" Jul 10 00:58:58.376767 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:58:58.376740 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-101.ec2.internal_29d40385cd1620022aeccb7b81ab99c1/haproxy/0.log" Jul 10 00:59:02.371938 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:02.371912 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk85r_080517b7-abe3-4d7f-9937-f3839e9a2417/extract/0.log" Jul 10 00:59:02.390604 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:02.390575 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk85r_080517b7-abe3-4d7f-9937-f3839e9a2417/util/0.log" Jul 10 00:59:02.409534 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:02.409510 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk85r_080517b7-abe3-4d7f-9937-f3839e9a2417/pull/0.log" Jul 10 00:59:02.455446 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:02.455359 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e087669_27fbf742-b6f0-448f-a86a-e4f40ef8b985/extract/0.log" Jul 10 00:59:02.477805 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:02.477777 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e087669_27fbf742-b6f0-448f-a86a-e4f40ef8b985/util/0.log" Jul 10 00:59:02.497195 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:02.497152 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e087669_27fbf742-b6f0-448f-a86a-e4f40ef8b985/pull/0.log" Jul 10 00:59:02.526035 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:02.526005 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7362l79_815d1fa6-90ea-4254-af1a-4be28806be87/extract/0.log" Jul 10 00:59:02.545571 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:02.545548 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7362l79_815d1fa6-90ea-4254-af1a-4be28806be87/util/0.log" Jul 10 00:59:02.564048 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:02.564026 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7362l79_815d1fa6-90ea-4254-af1a-4be28806be87/pull/0.log" Jul 10 00:59:02.592600 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:02.592572 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwhx6_b8c42013-1e7f-4f1c-a84b-d9f05a557e4d/extract/0.log" Jul 10 00:59:02.610627 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:02.610602 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwhx6_b8c42013-1e7f-4f1c-a84b-d9f05a557e4d/util/0.log" Jul 10 00:59:02.636209 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:02.636183 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwhx6_b8c42013-1e7f-4f1c-a84b-d9f05a557e4d/pull/0.log" Jul 10 00:59:03.131024 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:03.130983 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-9c4cbf5d7-n88zq_7be3328e-4d6e-4dee-a615-90d6ade4f629/authorino/0.log" Jul 10 00:59:05.350854 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:05.350823 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-594957f4c7-s69q2_ff3ada19-4c2d-492d-9a57-b3b7ffa46306/metrics-server/0.log" Jul 10 00:59:05.408153 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:05.408123 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-k76q6_38181004-3a43-475c-81e5-3cc031197de2/node-exporter/0.log" Jul 10 00:59:05.427888 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:05.427863 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-k76q6_38181004-3a43-475c-81e5-3cc031197de2/kube-rbac-proxy/0.log" Jul 10 00:59:05.453386 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:05.453360 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-k76q6_38181004-3a43-475c-81e5-3cc031197de2/init-textfile/0.log" Jul 10 00:59:05.636282 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:05.636196 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-5nckm_51841204-01e4-47f8-8bc0-83f66203bc07/kube-rbac-proxy-main/0.log" Jul 10 00:59:05.670438 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:05.670409 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-5nckm_51841204-01e4-47f8-8bc0-83f66203bc07/kube-rbac-proxy-self/0.log" Jul 10 00:59:05.690529 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:05.690499 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-5nckm_51841204-01e4-47f8-8bc0-83f66203bc07/openshift-state-metrics/0.log" Jul 10 00:59:06.050423 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:06.050394 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6955559f45-ftdzl_90f6720d-0564-473d-93d3-68905e69d4ca/thanos-query/0.log" Jul 10 00:59:06.068563 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:06.068536 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6955559f45-ftdzl_90f6720d-0564-473d-93d3-68905e69d4ca/kube-rbac-proxy-web/0.log" Jul 10 00:59:06.088858 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:06.088820 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6955559f45-ftdzl_90f6720d-0564-473d-93d3-68905e69d4ca/kube-rbac-proxy/0.log" Jul 10 00:59:06.105468 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:06.105440 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6955559f45-ftdzl_90f6720d-0564-473d-93d3-68905e69d4ca/prom-label-proxy/0.log" Jul 10 00:59:06.126926 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:06.126901 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6955559f45-ftdzl_90f6720d-0564-473d-93d3-68905e69d4ca/kube-rbac-proxy-rules/0.log" Jul 10 00:59:06.146911 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:06.146879 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6955559f45-ftdzl_90f6720d-0564-473d-93d3-68905e69d4ca/kube-rbac-proxy-metrics/0.log" Jul 10 00:59:08.165991 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:08.165963 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kjhg9_4651a90d-ef18-4974-9efd-a77fff3c9dfd/console-operator/1.log" Jul 10 00:59:08.174972 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:08.174948 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kjhg9_4651a90d-ef18-4974-9efd-a77fff3c9dfd/console-operator/2.log" Jul 10 00:59:09.398265 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:09.398061 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tcnxx/perf-node-gather-daemonset-cklj6"] Jul 10 00:59:09.398746 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:09.398535 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1065ba1d-8d1d-486f-b1e8-793e20dc4578" containerName="test-curl-7610-140103577320304" Jul 10 00:59:09.398746 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:09.398551 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1065ba1d-8d1d-486f-b1e8-793e20dc4578" containerName="test-curl-7610-140103577320304" Jul 10 00:59:09.398746 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:09.398577 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25b3b776-d45c-45ef-9315-ef8ab25c8cb4" containerName="main" Jul 10 00:59:09.398746 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:09.398583 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="25b3b776-d45c-45ef-9315-ef8ab25c8cb4" containerName="main" Jul 10 00:59:09.398746 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:09.398649 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="1065ba1d-8d1d-486f-b1e8-793e20dc4578" containerName="test-curl-7610-140103577320304" Jul 10 00:59:09.398746 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:09.398660 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="25b3b776-d45c-45ef-9315-ef8ab25c8cb4" containerName="main" Jul 10 00:59:09.402835 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:09.402812 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tcnxx/perf-node-gather-daemonset-cklj6" Jul 10 00:59:09.403549 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:09.403530 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tcnxx/perf-node-gather-daemonset-cklj6"] Jul 10 00:59:09.404875 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:09.404840 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-tcnxx\"/\"default-dockercfg-cv5kp\"" Jul 10 00:59:09.404875 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:09.404877 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tcnxx\"/\"openshift-service-ca.crt\"" Jul 10 00:59:09.405431 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:09.405413 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tcnxx\"/\"kube-root-ca.crt\"" Jul 10 00:59:09.416135 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:09.416112 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k4jl\" (UniqueName: \"kubernetes.io/projected/3a8719d8-e530-46d1-a9c6-2c48e251d1ae-kube-api-access-6k4jl\") pod \"perf-node-gather-daemonset-cklj6\" (UID: \"3a8719d8-e530-46d1-a9c6-2c48e251d1ae\") " pod="openshift-must-gather-tcnxx/perf-node-gather-daemonset-cklj6" Jul 10 00:59:09.416243 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:09.416163 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3a8719d8-e530-46d1-a9c6-2c48e251d1ae-lib-modules\") pod \"perf-node-gather-daemonset-cklj6\" (UID: \"3a8719d8-e530-46d1-a9c6-2c48e251d1ae\") " pod="openshift-must-gather-tcnxx/perf-node-gather-daemonset-cklj6" Jul 10 00:59:09.416243 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:09.416180 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3a8719d8-e530-46d1-a9c6-2c48e251d1ae-podres\") pod \"perf-node-gather-daemonset-cklj6\" (UID: \"3a8719d8-e530-46d1-a9c6-2c48e251d1ae\") " pod="openshift-must-gather-tcnxx/perf-node-gather-daemonset-cklj6" Jul 10 00:59:09.416382 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:09.416249 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a8719d8-e530-46d1-a9c6-2c48e251d1ae-sys\") pod \"perf-node-gather-daemonset-cklj6\" (UID: \"3a8719d8-e530-46d1-a9c6-2c48e251d1ae\") " pod="openshift-must-gather-tcnxx/perf-node-gather-daemonset-cklj6" Jul 10 00:59:09.416382 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:09.416320 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3a8719d8-e530-46d1-a9c6-2c48e251d1ae-proc\") pod \"perf-node-gather-daemonset-cklj6\" (UID: \"3a8719d8-e530-46d1-a9c6-2c48e251d1ae\") " pod="openshift-must-gather-tcnxx/perf-node-gather-daemonset-cklj6" Jul 10 00:59:09.517450 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:09.517417 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3a8719d8-e530-46d1-a9c6-2c48e251d1ae-proc\") pod \"perf-node-gather-daemonset-cklj6\" (UID: \"3a8719d8-e530-46d1-a9c6-2c48e251d1ae\") " pod="openshift-must-gather-tcnxx/perf-node-gather-daemonset-cklj6" Jul 10 00:59:09.517647 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:09.517474 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k4jl\" (UniqueName: \"kubernetes.io/projected/3a8719d8-e530-46d1-a9c6-2c48e251d1ae-kube-api-access-6k4jl\") pod \"perf-node-gather-daemonset-cklj6\" (UID: \"3a8719d8-e530-46d1-a9c6-2c48e251d1ae\") " pod="openshift-must-gather-tcnxx/perf-node-gather-daemonset-cklj6" Jul 10 00:59:09.517647 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:09.517556 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3a8719d8-e530-46d1-a9c6-2c48e251d1ae-proc\") pod \"perf-node-gather-daemonset-cklj6\" (UID: \"3a8719d8-e530-46d1-a9c6-2c48e251d1ae\") " pod="openshift-must-gather-tcnxx/perf-node-gather-daemonset-cklj6" Jul 10 00:59:09.517647 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:09.517610 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3a8719d8-e530-46d1-a9c6-2c48e251d1ae-lib-modules\") pod \"perf-node-gather-daemonset-cklj6\" (UID: \"3a8719d8-e530-46d1-a9c6-2c48e251d1ae\") " pod="openshift-must-gather-tcnxx/perf-node-gather-daemonset-cklj6" Jul 10 00:59:09.517647 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:09.517639 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3a8719d8-e530-46d1-a9c6-2c48e251d1ae-podres\") pod \"perf-node-gather-daemonset-cklj6\" (UID: \"3a8719d8-e530-46d1-a9c6-2c48e251d1ae\") " pod="openshift-must-gather-tcnxx/perf-node-gather-daemonset-cklj6" Jul 10 00:59:09.517810 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:09.517664 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a8719d8-e530-46d1-a9c6-2c48e251d1ae-sys\") pod \"perf-node-gather-daemonset-cklj6\" (UID: \"3a8719d8-e530-46d1-a9c6-2c48e251d1ae\") " pod="openshift-must-gather-tcnxx/perf-node-gather-daemonset-cklj6" Jul 10 00:59:09.517810 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:09.517756 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3a8719d8-e530-46d1-a9c6-2c48e251d1ae-lib-modules\") pod \"perf-node-gather-daemonset-cklj6\" (UID: \"3a8719d8-e530-46d1-a9c6-2c48e251d1ae\") " pod="openshift-must-gather-tcnxx/perf-node-gather-daemonset-cklj6" Jul 10 00:59:09.517810 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:09.517763 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3a8719d8-e530-46d1-a9c6-2c48e251d1ae-podres\") pod \"perf-node-gather-daemonset-cklj6\" (UID: \"3a8719d8-e530-46d1-a9c6-2c48e251d1ae\") " pod="openshift-must-gather-tcnxx/perf-node-gather-daemonset-cklj6" Jul 10 00:59:09.517810 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:09.517779 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a8719d8-e530-46d1-a9c6-2c48e251d1ae-sys\") pod \"perf-node-gather-daemonset-cklj6\" (UID: \"3a8719d8-e530-46d1-a9c6-2c48e251d1ae\") " pod="openshift-must-gather-tcnxx/perf-node-gather-daemonset-cklj6" Jul 10 00:59:09.524482 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:09.524457 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k4jl\" (UniqueName: \"kubernetes.io/projected/3a8719d8-e530-46d1-a9c6-2c48e251d1ae-kube-api-access-6k4jl\") pod \"perf-node-gather-daemonset-cklj6\" (UID: \"3a8719d8-e530-46d1-a9c6-2c48e251d1ae\") " pod="openshift-must-gather-tcnxx/perf-node-gather-daemonset-cklj6" Jul 10 00:59:09.713478 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:09.713384 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tcnxx/perf-node-gather-daemonset-cklj6" Jul 10 00:59:09.840579 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:09.840550 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tcnxx/perf-node-gather-daemonset-cklj6"] Jul 10 00:59:09.842532 ip-10-0-135-101 kubenswrapper[2562]: W0710 00:59:09.842505 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3a8719d8_e530_46d1_a9c6_2c48e251d1ae.slice/crio-b3736dce567c9b201873303941cc5ac397cec5f66385d1c5dd99e70bddac89b9 WatchSource:0}: Error finding container b3736dce567c9b201873303941cc5ac397cec5f66385d1c5dd99e70bddac89b9: Status 404 returned error can't find the container with id b3736dce567c9b201873303941cc5ac397cec5f66385d1c5dd99e70bddac89b9 Jul 10 00:59:10.493918 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:10.493892 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9mt8s_b56f3715-4283-4226-8188-d1b3869826db/dns/0.log" Jul 10 00:59:10.512260 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:10.512232 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9mt8s_b56f3715-4283-4226-8188-d1b3869826db/kube-rbac-proxy/0.log" Jul 10 00:59:10.533303 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:10.533267 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tcnxx/perf-node-gather-daemonset-cklj6" event={"ID":"3a8719d8-e530-46d1-a9c6-2c48e251d1ae","Type":"ContainerStarted","Data":"232d76280d3c24f6cff474784975c3682968c2ba6f6081e1988f758bbc04ec1b"} Jul 10 00:59:10.533467 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:10.533321 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tcnxx/perf-node-gather-daemonset-cklj6" event={"ID":"3a8719d8-e530-46d1-a9c6-2c48e251d1ae","Type":"ContainerStarted","Data":"b3736dce567c9b201873303941cc5ac397cec5f66385d1c5dd99e70bddac89b9"} Jul 10 00:59:10.533467 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:10.533405 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-tcnxx/perf-node-gather-daemonset-cklj6" Jul 10 00:59:10.548406 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:10.548353 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tcnxx/perf-node-gather-daemonset-cklj6" podStartSLOduration=1.548337567 podStartE2EDuration="1.548337567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 00:59:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:59:10.54669669 +0000 UTC m=+2617.365002184" watchObservedRunningTime="2026-07-10 00:59:10.548337567 +0000 UTC m=+2617.366643059" Jul 10 00:59:10.618355 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:10.618293 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2mpf9_0da6eed2-eb77-4e83-bb07-a6163eaca53a/dns-node-resolver/0.log" Jul 10 00:59:11.198686 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:11.198656 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gjk2r_03b7fa42-b493-40ce-a0d1-ce267379f3df/node-ca/0.log" Jul 10 00:59:12.164383 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:12.164295 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf47czbk_d0ba5141-fd5f-4e01-8865-037d6e6743d6/istio-proxy/0.log" Jul 10 00:59:12.307347 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:12.307311 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b9f4c495f-zv55m_f066a2dc-a4c4-4f25-94db-472621e7fc95/kube-auth-proxy/0.log" Jul 10 00:59:12.333452 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:12.333407 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-lhjrd_92786366-f55e-4e7b-b107-8cd178d52054/istio-proxy/0.log" Jul 10 00:59:12.390817 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:12.390784 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-lhjrd_92786366-f55e-4e7b-b107-8cd178d52054/istio-proxy/1.log" Jul 10 00:59:12.470591 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:12.470496 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-76d8bdc488-dw9ns_3d7b4fce-5123-477b-800c-bec0da7fd372/router/0.log" Jul 10 00:59:13.164635 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:13.164605 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tjh96_55dcffca-f335-4b01-a6b8-4b2356d2fe57/serve-healthcheck-canary/0.log" Jul 10 00:59:13.701681 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:13.701649 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tff8n_53c5a23d-3991-4e70-b665-a45e2d68121e/kube-rbac-proxy/0.log" Jul 10 00:59:13.723616 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:13.723590 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tff8n_53c5a23d-3991-4e70-b665-a45e2d68121e/exporter/0.log" Jul 10 00:59:13.748970 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:13.748940 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tff8n_53c5a23d-3991-4e70-b665-a45e2d68121e/extractor/0.log" Jul 10 00:59:16.549426 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:16.549394 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-tcnxx/perf-node-gather-daemonset-cklj6" Jul 10 00:59:16.556716 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:16.556695 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-559fc4999-lqpff_22d431a4-4965-4605-b603-6773ffd80c0d/manager/0.log" Jul 10 00:59:16.574063 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:16.574047 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-559fc4999-lqpff_22d431a4-4965-4605-b603-6773ffd80c0d/copy-manifests/0.log" Jul 10 00:59:17.738693 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:17.738666 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7b4d8999df-x6scl_0d056fdd-af45-4a37-bacb-9daa0b022eeb/manager/0.log" Jul 10 00:59:17.764756 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:17.764732 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-fll5x_a854a498-ae6d-4153-b9ee-64927e960a9b/openshift-lws-operator/0.log" Jul 10 00:59:19.458424 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:19.458393 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf47czbk_d0ba5141-fd5f-4e01-8865-037d6e6743d6/istio-proxy/0.log" Jul 10 00:59:19.588643 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:19.588620 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b9f4c495f-zv55m_f066a2dc-a4c4-4f25-94db-472621e7fc95/kube-auth-proxy/0.log" Jul 10 00:59:19.608689 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:19.608665 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-lhjrd_92786366-f55e-4e7b-b107-8cd178d52054/istio-proxy/0.log" Jul 10 00:59:19.665357 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:19.665319 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-lhjrd_92786366-f55e-4e7b-b107-8cd178d52054/istio-proxy/1.log" Jul 10 00:59:19.727634 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:19.727605 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-76d8bdc488-dw9ns_3d7b4fce-5123-477b-800c-bec0da7fd372/router/0.log" Jul 10 00:59:20.966900 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:20.966867 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf47czbk_d0ba5141-fd5f-4e01-8865-037d6e6743d6/istio-proxy/0.log" Jul 10 00:59:21.097388 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:21.097345 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b9f4c495f-zv55m_f066a2dc-a4c4-4f25-94db-472621e7fc95/kube-auth-proxy/0.log" Jul 10 00:59:21.120524 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:21.120493 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-lhjrd_92786366-f55e-4e7b-b107-8cd178d52054/istio-proxy/0.log" Jul 10 00:59:21.176836 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:21.176811 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-lhjrd_92786366-f55e-4e7b-b107-8cd178d52054/istio-proxy/1.log" Jul 10 00:59:21.246353 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:21.246278 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-76d8bdc488-dw9ns_3d7b4fce-5123-477b-800c-bec0da7fd372/router/0.log" Jul 10 00:59:23.161913 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:23.161866 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-9w2qg_0064d533-5099-4a49-afef-12125cba31ad/migrator/0.log" Jul 10 00:59:23.178412 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:23.178386 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-9w2qg_0064d533-5099-4a49-afef-12125cba31ad/graceful-termination/0.log" Jul 10 00:59:24.472484 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:24.472458 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-5zlgk_0318fc8c-a177-4995-a1b6-80ccf2cf11cd/kube-multus/0.log" Jul 10 00:59:24.529302 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:24.529277 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5kkqc_a88d645c-45e4-4592-b451-04eea24d3afb/kube-multus-additional-cni-plugins/0.log" Jul 10 00:59:24.547703 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:24.547675 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5kkqc_a88d645c-45e4-4592-b451-04eea24d3afb/egress-router-binary-copy/0.log" Jul 10 00:59:24.564965 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:24.564947 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5kkqc_a88d645c-45e4-4592-b451-04eea24d3afb/cni-plugins/0.log" Jul 10 00:59:24.583638 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:24.583619 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5kkqc_a88d645c-45e4-4592-b451-04eea24d3afb/bond-cni-plugin/0.log" Jul 10 00:59:24.602071 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:24.602042 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5kkqc_a88d645c-45e4-4592-b451-04eea24d3afb/routeoverride-cni/0.log" Jul 10 00:59:24.619639 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:24.619618 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5kkqc_a88d645c-45e4-4592-b451-04eea24d3afb/whereabouts-cni-bincopy/0.log" Jul 10 00:59:24.637397 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:24.637372 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5kkqc_a88d645c-45e4-4592-b451-04eea24d3afb/whereabouts-cni/0.log" Jul 10 00:59:24.973374 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:24.973335 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-x8v89_62933c8a-818d-4580-a6c9-6d26dbdfd50d/network-metrics-daemon/0.log" Jul 10 00:59:24.989126 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:24.989100 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-x8v89_62933c8a-818d-4580-a6c9-6d26dbdfd50d/kube-rbac-proxy/0.log" Jul 10 00:59:26.208912 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:26.208880 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h5g9w_f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32/ovn-controller/0.log" Jul 10 00:59:26.224205 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:26.224176 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h5g9w_f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32/ovn-acl-logging/1.log" Jul 10 00:59:26.224205 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:26.224189 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h5g9w_f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32/ovn-acl-logging/0.log" Jul 10 00:59:26.245455 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:26.245432 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h5g9w_f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32/kube-rbac-proxy-node/0.log" Jul 10 00:59:26.268281 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:26.268257 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h5g9w_f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32/kube-rbac-proxy-ovn-metrics/0.log" Jul 10 00:59:26.282604 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:26.282582 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h5g9w_f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32/northd/0.log" Jul 10 00:59:26.302219 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:26.302193 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h5g9w_f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32/nbdb/0.log" Jul 10 00:59:26.320164 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:26.320132 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h5g9w_f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32/sbdb/0.log" Jul 10 00:59:26.390520 ip-10-0-135-101 kubenswrapper[2562]: I0710 00:59:26.390485 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h5g9w_f0a68b3c-8fb7-4b9e-b1c2-fb1ad811fd32/ovnkube-controller/0.log"