Jul 09 07:02:20.833871 ip-10-0-138-0 systemd[1]: Starting Kubernetes Kubelet... Jul 09 07:02:21.226527 ip-10-0-138-0 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 07:02:21.226527 ip-10-0-138-0 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 09 07:02:21.226527 ip-10-0-138-0 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 07:02:21.226527 ip-10-0-138-0 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 09 07:02:21.226527 ip-10-0-138-0 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 07:02:21.228123 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.228042 2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 09 07:02:21.232448 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232433 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 07:02:21.232488 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232450 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 07:02:21.232488 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232456 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 07:02:21.232488 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232459 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 07:02:21.232488 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232463 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 07:02:21.232488 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232465 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 07:02:21.232488 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232469 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 07:02:21.232488 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232471 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 07:02:21.232488 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232474 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 07:02:21.232488 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232477 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 07:02:21.232488 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232479 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 07:02:21.232488 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232482 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 07:02:21.232488 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232484 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 07:02:21.232488 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232487 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 07:02:21.232488 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232490 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 07:02:21.232488 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232492 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 07:02:21.232488 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232495 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 07:02:21.232488 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232498 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 07:02:21.232920 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232501 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 07:02:21.232920 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232504 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 07:02:21.232920 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232507 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 07:02:21.232920 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232533 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 07:02:21.232920 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232537 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 07:02:21.232920 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232539 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 07:02:21.232920 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232542 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 07:02:21.232920 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232544 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 07:02:21.232920 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232547 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 07:02:21.232920 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232550 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 07:02:21.232920 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232553 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 07:02:21.232920 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232555 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 07:02:21.232920 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232558 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 07:02:21.232920 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232566 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 07:02:21.232920 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232569 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 07:02:21.232920 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232571 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 07:02:21.232920 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232574 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 07:02:21.232920 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232576 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 07:02:21.232920 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232579 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 07:02:21.232920 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232582 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 07:02:21.233406 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232586 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 07:02:21.233406 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232589 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 07:02:21.233406 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232591 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 07:02:21.233406 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232594 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 07:02:21.233406 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232597 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 07:02:21.233406 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232600 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 07:02:21.233406 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232602 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 07:02:21.233406 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232605 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 07:02:21.233406 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232607 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 07:02:21.233406 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232610 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 07:02:21.233406 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232612 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 07:02:21.233406 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232615 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 07:02:21.233406 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232617 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 07:02:21.233406 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232621 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 07:02:21.233406 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232623 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 07:02:21.233406 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232626 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 07:02:21.233406 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232634 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 07:02:21.233406 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232637 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 07:02:21.233406 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232639 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 07:02:21.233876 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232642 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 07:02:21.233876 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232645 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 07:02:21.233876 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232647 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 07:02:21.233876 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232650 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 07:02:21.233876 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232652 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 07:02:21.233876 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232654 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 07:02:21.233876 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232657 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 07:02:21.233876 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232659 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 07:02:21.233876 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232662 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 07:02:21.233876 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232664 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 07:02:21.233876 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232666 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 07:02:21.233876 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232669 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 07:02:21.233876 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232671 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 07:02:21.233876 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232674 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 07:02:21.233876 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232676 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 07:02:21.233876 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232679 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 07:02:21.233876 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232681 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 07:02:21.233876 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232684 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 07:02:21.233876 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232686 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 07:02:21.233876 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232689 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 07:02:21.234360 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232691 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 07:02:21.234360 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232695 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 07:02:21.234360 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232698 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 07:02:21.234360 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232700 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 07:02:21.234360 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232703 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 07:02:21.234360 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232705 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 07:02:21.234360 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232709 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 07:02:21.234360 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232712 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 07:02:21.234360 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232714 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 07:02:21.234360 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232717 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 07:02:21.234360 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232725 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 07:02:21.234360 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232728 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 07:02:21.234360 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232730 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 09 07:02:21.234360 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232733 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 07:02:21.234360 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232736 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 07:02:21.234360 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232738 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 07:02:21.234360 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232740 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 07:02:21.234360 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232745 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 07:02:21.234360 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232748 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 07:02:21.234811 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232751 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 07:02:21.234811 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232753 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 07:02:21.234811 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.232756 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 07:02:21.234811 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233508 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 07:02:21.234811 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233515 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 07:02:21.234811 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233518 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 07:02:21.234811 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233521 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 07:02:21.234811 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233523 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 07:02:21.234811 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233526 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 07:02:21.234811 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233529 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 07:02:21.234811 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233531 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 07:02:21.234811 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233534 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 07:02:21.234811 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233536 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 07:02:21.234811 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233539 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 07:02:21.234811 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233541 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 07:02:21.234811 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233544 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 07:02:21.234811 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233546 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 07:02:21.234811 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233549 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 07:02:21.234811 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233551 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 07:02:21.234811 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233555 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 07:02:21.235347 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233558 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 07:02:21.235347 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233561 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 07:02:21.235347 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233564 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 07:02:21.235347 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233566 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 07:02:21.235347 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233569 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 07:02:21.235347 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233572 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 09 07:02:21.235347 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233574 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 07:02:21.235347 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233577 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 07:02:21.235347 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233579 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 07:02:21.235347 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233582 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 07:02:21.235347 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233585 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 07:02:21.235347 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233588 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 07:02:21.235347 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233590 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 07:02:21.235347 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233593 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 07:02:21.235347 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233595 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 07:02:21.235347 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233598 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 07:02:21.235347 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233600 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 07:02:21.235347 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233604 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 07:02:21.235347 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233608 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 07:02:21.235347 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233611 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 07:02:21.235861 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233614 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 07:02:21.235861 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233617 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 07:02:21.235861 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233620 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 07:02:21.235861 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233623 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 07:02:21.235861 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233625 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 07:02:21.235861 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233628 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 07:02:21.235861 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233630 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 07:02:21.235861 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233632 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 07:02:21.235861 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233635 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 07:02:21.235861 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233638 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 07:02:21.235861 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233640 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 07:02:21.235861 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233643 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 07:02:21.235861 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233646 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 07:02:21.235861 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233648 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 07:02:21.235861 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233651 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 07:02:21.235861 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233653 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 07:02:21.235861 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233656 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 07:02:21.235861 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233659 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 07:02:21.235861 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233661 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 07:02:21.235861 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233664 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 07:02:21.236344 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233666 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 07:02:21.236344 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233669 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 07:02:21.236344 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233672 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 07:02:21.236344 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233676 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 07:02:21.236344 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233679 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 07:02:21.236344 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233683 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 07:02:21.236344 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233686 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 07:02:21.236344 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233688 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 07:02:21.236344 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233691 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 07:02:21.236344 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233694 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 07:02:21.236344 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233696 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 07:02:21.236344 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233699 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 07:02:21.236344 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233701 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 07:02:21.236344 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233704 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 07:02:21.236344 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233706 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 07:02:21.236344 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233709 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 07:02:21.236344 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233711 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 07:02:21.236344 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233714 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 07:02:21.236344 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233716 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 07:02:21.236793 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233718 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 07:02:21.236793 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233721 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 07:02:21.236793 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233726 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 07:02:21.236793 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233729 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 07:02:21.236793 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233732 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 07:02:21.236793 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233735 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 07:02:21.236793 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233738 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 07:02:21.236793 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233740 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 07:02:21.236793 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233743 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 07:02:21.236793 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233746 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 07:02:21.236793 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233749 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 07:02:21.236793 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233752 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 07:02:21.236793 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233754 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 07:02:21.236793 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233757 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 07:02:21.236793 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233759 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 07:02:21.236793 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233762 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 07:02:21.236793 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233764 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 07:02:21.236793 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233767 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 07:02:21.236793 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233769 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 07:02:21.237446 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233772 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 07:02:21.237446 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233775 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 07:02:21.237446 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233778 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 07:02:21.237446 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.233780 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 07:02:21.237446 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234129 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 07:02:21.237446 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234134 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 07:02:21.237446 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234137 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 07:02:21.237446 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234139 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 07:02:21.237446 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234142 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 07:02:21.237446 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234144 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 07:02:21.237446 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234147 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 07:02:21.237446 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234150 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 07:02:21.237446 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234152 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 07:02:21.237446 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234155 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 07:02:21.237446 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234157 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 07:02:21.237446 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234160 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 07:02:21.237446 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234162 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 09 07:02:21.237446 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234166 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 07:02:21.237446 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234168 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 07:02:21.237446 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234171 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 07:02:21.237446 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234173 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 07:02:21.238280 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234176 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 07:02:21.238280 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234179 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 07:02:21.238280 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234181 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 07:02:21.238280 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234184 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 07:02:21.238280 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234187 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 07:02:21.238280 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234190 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 07:02:21.238280 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234192 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 07:02:21.238280 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234195 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 07:02:21.238280 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234198 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 07:02:21.238280 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234202 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 07:02:21.238280 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234205 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 07:02:21.238280 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234207 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 07:02:21.238280 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234210 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 07:02:21.238280 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234212 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 07:02:21.238280 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234215 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 07:02:21.238280 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234217 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 07:02:21.238280 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234221 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 07:02:21.238280 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234225 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 07:02:21.238280 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234229 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 07:02:21.238785 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234231 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 07:02:21.238785 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234234 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 07:02:21.238785 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234238 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 07:02:21.238785 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234241 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 07:02:21.238785 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234244 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 07:02:21.238785 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234247 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 07:02:21.238785 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234250 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 07:02:21.238785 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234253 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 07:02:21.238785 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234255 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 07:02:21.238785 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234258 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 07:02:21.238785 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234275 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 07:02:21.238785 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234278 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 07:02:21.238785 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234281 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 07:02:21.238785 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234286 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 07:02:21.238785 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234289 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 07:02:21.238785 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234293 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 07:02:21.238785 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234296 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 07:02:21.238785 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234299 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 07:02:21.238785 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234302 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 07:02:21.239225 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234304 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 07:02:21.239225 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234307 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 07:02:21.239225 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234309 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 07:02:21.239225 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234313 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 07:02:21.239225 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234316 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 07:02:21.239225 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234318 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 07:02:21.239225 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234321 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 07:02:21.239225 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234323 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 07:02:21.239225 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234326 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 07:02:21.239225 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234328 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 07:02:21.239225 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234331 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 07:02:21.239225 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234333 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 07:02:21.239225 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234336 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 07:02:21.239225 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234338 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 07:02:21.239225 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234340 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 07:02:21.239225 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234343 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 07:02:21.239225 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234346 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 07:02:21.239225 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234348 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 07:02:21.239225 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234351 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 07:02:21.239708 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234353 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 07:02:21.239708 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234355 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 07:02:21.239708 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234358 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 07:02:21.239708 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234360 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 07:02:21.239708 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234363 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 07:02:21.239708 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234366 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 07:02:21.239708 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234369 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 07:02:21.239708 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234372 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 07:02:21.239708 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234376 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 07:02:21.239708 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234378 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 07:02:21.239708 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234381 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 07:02:21.239708 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234383 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 07:02:21.239708 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234387 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 07:02:21.239708 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234389 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 07:02:21.239708 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234391 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 07:02:21.239708 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234394 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 07:02:21.239708 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234397 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 07:02:21.239708 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234400 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 07:02:21.239708 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234402 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 07:02:21.239708 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234405 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 07:02:21.240189 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234407 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 07:02:21.240189 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234410 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 07:02:21.240189 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234413 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 07:02:21.240189 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234415 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 07:02:21.240189 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.234417 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 07:02:21.240189 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234482 2564 flags.go:64] FLAG: --address="0.0.0.0" Jul 09 07:02:21.240189 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234488 2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 09 07:02:21.240189 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234494 2564 flags.go:64] FLAG: --anonymous-auth="true" Jul 09 07:02:21.240189 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234498 2564 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 09 07:02:21.240189 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234502 2564 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 09 07:02:21.240189 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234505 2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 09 07:02:21.240189 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234509 2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 09 07:02:21.240189 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234513 2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 09 07:02:21.240189 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234516 2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 09 07:02:21.240189 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234519 2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 09 07:02:21.240189 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234522 2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 09 07:02:21.240189 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234525 2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 09 07:02:21.240189 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234528 2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 09 07:02:21.240189 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234532 2564 flags.go:64] FLAG: --cgroup-root="" Jul 09 07:02:21.240189 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234534 2564 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 09 07:02:21.240189 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234537 2564 flags.go:64] FLAG: --client-ca-file="" Jul 09 07:02:21.240189 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234540 2564 flags.go:64] FLAG: --cloud-provider="external" Jul 09 07:02:21.240189 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234543 2564 flags.go:64] FLAG: --cluster-dns="[]" Jul 09 07:02:21.240189 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234548 2564 flags.go:64] FLAG: --cluster-domain="" Jul 09 07:02:21.240780 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234550 2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 09 07:02:21.240780 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234554 2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 09 07:02:21.240780 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234557 2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 09 07:02:21.240780 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234560 2564 flags.go:64] FLAG: --container-log-max-files="5" Jul 09 07:02:21.240780 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234564 2564 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 09 07:02:21.240780 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234567 2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 09 07:02:21.240780 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234570 2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 09 07:02:21.240780 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234574 2564 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 09 07:02:21.240780 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234577 2564 flags.go:64] FLAG: --contention-profiling="false" Jul 09 07:02:21.240780 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234580 2564 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 09 07:02:21.240780 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234582 2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 09 07:02:21.240780 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234586 2564 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 09 07:02:21.240780 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234589 2564 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 09 07:02:21.240780 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234593 2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 09 07:02:21.240780 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234596 2564 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 09 07:02:21.240780 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234599 2564 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 09 07:02:21.240780 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234601 2564 flags.go:64] FLAG: --enable-load-reader="false" Jul 09 07:02:21.240780 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234604 2564 flags.go:64] FLAG: --enable-server="true" Jul 09 07:02:21.240780 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234607 2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 09 07:02:21.240780 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234611 2564 flags.go:64] FLAG: --event-burst="100" Jul 09 07:02:21.240780 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234614 2564 flags.go:64] FLAG: --event-qps="50" Jul 09 07:02:21.240780 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234617 2564 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 09 07:02:21.240780 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234620 2564 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 09 07:02:21.240780 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234623 2564 flags.go:64] FLAG: --eviction-hard="" Jul 09 07:02:21.241373 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234626 2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 09 07:02:21.241373 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234629 2564 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 09 07:02:21.241373 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234632 2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 09 07:02:21.241373 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234635 2564 flags.go:64] FLAG: --eviction-soft="" Jul 09 07:02:21.241373 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234638 2564 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 09 07:02:21.241373 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234640 2564 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 09 07:02:21.241373 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234643 2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 09 07:02:21.241373 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234646 2564 flags.go:64] FLAG: --experimental-mounter-path="" Jul 09 07:02:21.241373 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234649 2564 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 09 07:02:21.241373 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234652 2564 flags.go:64] FLAG: --fail-swap-on="true" Jul 09 07:02:21.241373 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234655 2564 flags.go:64] FLAG: --feature-gates="" Jul 09 07:02:21.241373 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234659 2564 flags.go:64] FLAG: --file-check-frequency="20s" Jul 09 07:02:21.241373 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234662 2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 09 07:02:21.241373 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234665 2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 09 07:02:21.241373 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234668 2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 09 07:02:21.241373 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234671 2564 flags.go:64] FLAG: --healthz-port="10248" Jul 09 07:02:21.241373 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234674 2564 flags.go:64] FLAG: --help="false" Jul 09 07:02:21.241373 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234677 2564 flags.go:64] FLAG: --hostname-override="ip-10-0-138-0.ec2.internal" Jul 09 07:02:21.241373 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234680 2564 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 09 07:02:21.241373 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234684 2564 flags.go:64] FLAG: --http-check-frequency="20s" Jul 09 07:02:21.241373 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234687 2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 09 07:02:21.241373 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234690 2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 09 07:02:21.241373 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234694 2564 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 09 07:02:21.241974 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234696 2564 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 09 07:02:21.241974 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234699 2564 flags.go:64] FLAG: --image-service-endpoint="" Jul 09 07:02:21.241974 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234702 2564 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 09 07:02:21.241974 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234705 2564 flags.go:64] FLAG: --kube-api-burst="100" Jul 09 07:02:21.241974 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234708 2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 09 07:02:21.241974 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234711 2564 flags.go:64] FLAG: --kube-api-qps="50" Jul 09 07:02:21.241974 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234714 2564 flags.go:64] FLAG: --kube-reserved="" Jul 09 07:02:21.241974 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234717 2564 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 09 07:02:21.241974 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234719 2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 09 07:02:21.241974 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234722 2564 flags.go:64] FLAG: --kubelet-cgroups="" Jul 09 07:02:21.241974 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234725 2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 09 07:02:21.241974 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234728 2564 flags.go:64] FLAG: --lock-file="" Jul 09 07:02:21.241974 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234731 2564 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 09 07:02:21.241974 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234733 2564 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 09 07:02:21.241974 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234736 2564 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 09 07:02:21.241974 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234741 2564 flags.go:64] FLAG: --log-json-split-stream="false" Jul 09 07:02:21.241974 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234744 2564 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 09 07:02:21.241974 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234747 2564 flags.go:64] FLAG: --log-text-split-stream="false" Jul 09 07:02:21.241974 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234749 2564 flags.go:64] FLAG: --logging-format="text" Jul 09 07:02:21.241974 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234752 2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 09 07:02:21.241974 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234756 2564 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 09 07:02:21.241974 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234758 2564 flags.go:64] FLAG: --manifest-url="" Jul 09 07:02:21.241974 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234761 2564 flags.go:64] FLAG: --manifest-url-header="" Jul 09 07:02:21.241974 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234768 2564 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 09 07:02:21.241974 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234771 2564 flags.go:64] FLAG: --max-open-files="1000000" Jul 09 07:02:21.242599 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234774 2564 flags.go:64] FLAG: --max-pods="110" Jul 09 07:02:21.242599 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234777 2564 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 09 07:02:21.242599 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234780 2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 09 07:02:21.242599 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234784 2564 flags.go:64] FLAG: --memory-manager-policy="None" Jul 09 07:02:21.242599 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234787 2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 09 07:02:21.242599 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234791 2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 09 07:02:21.242599 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234794 2564 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 09 07:02:21.242599 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234797 2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 09 07:02:21.242599 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234803 2564 flags.go:64] FLAG: --node-status-max-images="50" Jul 09 07:02:21.242599 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234806 2564 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 09 07:02:21.242599 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234809 2564 flags.go:64] FLAG: --oom-score-adj="-999" Jul 09 07:02:21.242599 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234811 2564 flags.go:64] FLAG: --pod-cidr="" Jul 09 07:02:21.242599 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234814 2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 09 07:02:21.242599 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234820 2564 flags.go:64] FLAG: --pod-manifest-path="" Jul 09 07:02:21.242599 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234823 2564 flags.go:64] FLAG: --pod-max-pids="-1" Jul 09 07:02:21.242599 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234826 2564 flags.go:64] FLAG: --pods-per-core="0" Jul 09 07:02:21.242599 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234829 2564 flags.go:64] FLAG: --port="10250" Jul 09 07:02:21.242599 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234832 2564 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 09 07:02:21.242599 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234835 2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-007388c38af430820" Jul 09 07:02:21.242599 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234838 2564 flags.go:64] FLAG: --qos-reserved="" Jul 09 07:02:21.242599 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234840 2564 flags.go:64] FLAG: --read-only-port="10255" Jul 09 07:02:21.242599 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234843 2564 flags.go:64] FLAG: --register-node="true" Jul 09 07:02:21.242599 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234846 2564 flags.go:64] FLAG: --register-with-taints="" Jul 09 07:02:21.242599 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234850 2564 flags.go:64] FLAG: --registry-burst="10" Jul 09 07:02:21.243174 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234852 2564 flags.go:64] FLAG: --registry-qps="5" Jul 09 07:02:21.243174 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234855 2564 flags.go:64] FLAG: --reserved-cpus="" Jul 09 07:02:21.243174 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234858 2564 flags.go:64] FLAG: --reserved-memory="" Jul 09 07:02:21.243174 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234863 2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 09 07:02:21.243174 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234866 2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 09 07:02:21.243174 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234869 2564 flags.go:64] FLAG: --rotate-certificates="false" Jul 09 07:02:21.243174 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234872 2564 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 09 07:02:21.243174 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234875 2564 flags.go:64] FLAG: --runonce="false" Jul 09 07:02:21.243174 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234878 2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 09 07:02:21.243174 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234881 2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 09 07:02:21.243174 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234884 2564 flags.go:64] FLAG: --seccomp-default="false" Jul 09 07:02:21.243174 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234887 2564 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 09 07:02:21.243174 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234891 2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 09 07:02:21.243174 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234894 2564 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 09 07:02:21.243174 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234897 2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 09 07:02:21.243174 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234900 2564 flags.go:64] FLAG: --storage-driver-password="root" Jul 09 07:02:21.243174 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234903 2564 flags.go:64] FLAG: --storage-driver-secure="false" Jul 09 07:02:21.243174 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234906 2564 flags.go:64] FLAG: --storage-driver-table="stats" Jul 09 07:02:21.243174 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234908 2564 flags.go:64] FLAG: --storage-driver-user="root" Jul 09 07:02:21.243174 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234911 2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 09 07:02:21.243174 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234914 2564 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 09 07:02:21.243174 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234917 2564 flags.go:64] FLAG: --system-cgroups="" Jul 09 07:02:21.243174 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234920 2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 09 07:02:21.243174 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234925 2564 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 09 07:02:21.243174 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234928 2564 flags.go:64] FLAG: --tls-cert-file="" Jul 09 07:02:21.243954 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234931 2564 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 09 07:02:21.243954 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234935 2564 flags.go:64] FLAG: --tls-min-version="" Jul 09 07:02:21.243954 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234938 2564 flags.go:64] FLAG: --tls-private-key-file="" Jul 09 07:02:21.243954 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234941 2564 flags.go:64] FLAG: --topology-manager-policy="none" Jul 09 07:02:21.243954 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234944 2564 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 09 07:02:21.243954 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234946 2564 flags.go:64] FLAG: --topology-manager-scope="container" Jul 09 07:02:21.243954 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234949 2564 flags.go:64] FLAG: --v="2" Jul 09 07:02:21.243954 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234952 2564 flags.go:64] FLAG: --version="false" Jul 09 07:02:21.243954 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234956 2564 flags.go:64] FLAG: --vmodule="" Jul 09 07:02:21.243954 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234960 2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 09 07:02:21.243954 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.234964 2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 09 07:02:21.243954 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235073 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 07:02:21.243954 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235077 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 07:02:21.243954 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235080 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 07:02:21.243954 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235083 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 07:02:21.243954 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235086 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 07:02:21.243954 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235090 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 07:02:21.243954 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235094 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 07:02:21.243954 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235097 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 07:02:21.243954 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235101 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 07:02:21.243954 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235104 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 07:02:21.243954 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235107 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 07:02:21.244505 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235110 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 07:02:21.244505 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235113 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 07:02:21.244505 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235116 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 07:02:21.244505 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235119 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 07:02:21.244505 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235122 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 07:02:21.244505 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235125 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 07:02:21.244505 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235127 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 07:02:21.244505 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235130 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 07:02:21.244505 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235133 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 07:02:21.244505 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235136 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 07:02:21.244505 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235138 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 07:02:21.244505 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235141 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 07:02:21.244505 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235143 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 07:02:21.244505 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235146 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 07:02:21.244505 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235148 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 07:02:21.244505 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235152 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 07:02:21.244505 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235155 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 07:02:21.244505 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235158 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 07:02:21.244505 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235160 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 07:02:21.244961 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235163 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 07:02:21.244961 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235170 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 07:02:21.244961 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235172 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 07:02:21.244961 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235175 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 07:02:21.244961 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235178 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 07:02:21.244961 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235183 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 07:02:21.244961 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235186 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 07:02:21.244961 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235189 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 07:02:21.244961 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235191 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 07:02:21.244961 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235194 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 07:02:21.244961 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235198 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 07:02:21.244961 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235201 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 07:02:21.244961 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235203 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 07:02:21.244961 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235206 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 07:02:21.244961 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235208 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 07:02:21.244961 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235211 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 07:02:21.244961 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235214 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 07:02:21.244961 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235216 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 07:02:21.244961 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235219 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 07:02:21.244961 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235221 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 07:02:21.245487 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235224 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 07:02:21.245487 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235227 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 07:02:21.245487 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235229 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 07:02:21.245487 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235232 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 07:02:21.245487 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235234 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 07:02:21.245487 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235237 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 07:02:21.245487 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235239 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 07:02:21.245487 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235242 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 07:02:21.245487 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235245 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 07:02:21.245487 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235247 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 07:02:21.245487 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235250 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 07:02:21.245487 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235252 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 09 07:02:21.245487 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235254 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 07:02:21.245487 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235258 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 07:02:21.245487 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235274 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 07:02:21.245487 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235276 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 07:02:21.245487 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235279 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 07:02:21.245487 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235282 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 07:02:21.245487 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235286 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 07:02:21.245487 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235289 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 07:02:21.245962 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235292 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 07:02:21.245962 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235295 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 07:02:21.245962 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235299 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 07:02:21.245962 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235302 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 07:02:21.245962 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235305 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 07:02:21.245962 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235308 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 07:02:21.245962 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235310 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 07:02:21.245962 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235313 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 07:02:21.245962 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235315 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 07:02:21.245962 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235318 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 07:02:21.245962 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235320 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 07:02:21.245962 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235323 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 07:02:21.245962 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235326 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 07:02:21.245962 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235329 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 07:02:21.245962 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235331 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 07:02:21.245962 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235334 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 07:02:21.245962 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235336 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 07:02:21.245962 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235339 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 07:02:21.245962 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235342 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 07:02:21.245962 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235344 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 07:02:21.246465 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235347 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 07:02:21.246465 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235349 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 07:02:21.246465 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235352 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 07:02:21.246465 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235355 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 07:02:21.246465 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235358 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 07:02:21.246465 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235362 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 07:02:21.246465 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235365 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 07:02:21.246465 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235368 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 07:02:21.246465 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.235370 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 07:02:21.246465 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.236851 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 07:02:21.246465 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.243829 2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 09 07:02:21.246465 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.243843 2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 09 07:02:21.246465 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.243862 2564 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 09 07:02:21.246465 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.243870 2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 09 07:02:21.246465 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.243940 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 07:02:21.246838 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.243945 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 07:02:21.246838 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.243948 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 07:02:21.246838 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.243951 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 07:02:21.246838 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.243955 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 07:02:21.246838 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.243958 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 07:02:21.246838 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.243961 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 07:02:21.246838 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.243964 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 07:02:21.246838 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.243966 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 07:02:21.246838 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.243969 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 07:02:21.246838 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.243972 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 07:02:21.246838 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.243975 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 07:02:21.246838 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.243978 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 07:02:21.246838 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.243981 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 07:02:21.246838 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.243983 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 07:02:21.246838 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.243986 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 07:02:21.246838 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.243989 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 09 07:02:21.246838 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.243992 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 07:02:21.246838 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.243994 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 07:02:21.246838 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.243997 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 07:02:21.246838 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244000 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 07:02:21.247342 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244003 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 07:02:21.247342 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244005 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 07:02:21.247342 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244008 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 07:02:21.247342 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244011 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 07:02:21.247342 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244014 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 07:02:21.247342 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244016 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 07:02:21.247342 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244019 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 07:02:21.247342 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244021 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 07:02:21.247342 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244024 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 07:02:21.247342 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244027 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 07:02:21.247342 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244030 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 07:02:21.247342 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244035 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 07:02:21.247342 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244038 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 07:02:21.247342 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244041 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 07:02:21.247342 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244044 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 07:02:21.247342 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244047 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 07:02:21.247342 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244050 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 07:02:21.247342 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244053 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 07:02:21.247342 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244056 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 07:02:21.247801 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244058 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 07:02:21.247801 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244061 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 07:02:21.247801 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244064 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 07:02:21.247801 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244066 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 07:02:21.247801 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244070 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 07:02:21.247801 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244074 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 07:02:21.247801 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244077 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 07:02:21.247801 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244080 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 07:02:21.247801 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244083 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 07:02:21.247801 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244086 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 07:02:21.247801 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244088 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 07:02:21.247801 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244091 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 07:02:21.247801 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244093 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 07:02:21.247801 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244096 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 07:02:21.247801 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244098 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 07:02:21.247801 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244101 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 07:02:21.247801 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244103 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 07:02:21.247801 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244106 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 07:02:21.247801 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244108 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 07:02:21.247801 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244111 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 07:02:21.248416 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244113 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 07:02:21.248416 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244116 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 07:02:21.248416 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244119 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 07:02:21.248416 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244122 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 07:02:21.248416 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244125 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 07:02:21.248416 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244127 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 07:02:21.248416 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244130 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 07:02:21.248416 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244133 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 07:02:21.248416 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244135 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 07:02:21.248416 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244138 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 07:02:21.248416 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244141 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 07:02:21.248416 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244143 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 07:02:21.248416 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244146 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 07:02:21.248416 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244148 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 07:02:21.248416 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244151 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 07:02:21.248416 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244154 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 07:02:21.248416 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244156 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 07:02:21.248416 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244159 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 07:02:21.248416 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244167 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 07:02:21.248416 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244169 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 07:02:21.248892 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244172 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 07:02:21.248892 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244175 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 07:02:21.248892 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244177 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 07:02:21.248892 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244180 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 07:02:21.248892 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244184 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 07:02:21.248892 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244187 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 07:02:21.248892 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244189 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 07:02:21.248892 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244192 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 07:02:21.248892 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244195 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 07:02:21.248892 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244198 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 07:02:21.248892 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244201 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 07:02:21.248892 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244203 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 07:02:21.248892 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244206 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 07:02:21.248892 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244210 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 07:02:21.248892 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244212 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 07:02:21.248892 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244215 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 07:02:21.248892 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244217 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 07:02:21.248892 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244220 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 07:02:21.248892 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244222 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 07:02:21.249373 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.244227 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 07:02:21.249373 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244356 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 07:02:21.249373 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244362 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 07:02:21.249373 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244366 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 07:02:21.249373 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244370 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 07:02:21.249373 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244372 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 07:02:21.249373 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244375 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 07:02:21.249373 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244378 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 07:02:21.249373 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244380 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 07:02:21.249373 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244382 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 07:02:21.249373 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244385 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 07:02:21.249373 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244388 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 07:02:21.249373 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244391 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 07:02:21.249373 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244393 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 07:02:21.249709 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244396 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 07:02:21.249709 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244398 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 07:02:21.249709 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244401 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 07:02:21.249709 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244404 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 07:02:21.249709 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244406 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 07:02:21.249709 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244409 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 07:02:21.249709 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244412 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 07:02:21.249709 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244415 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 07:02:21.249709 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244417 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 07:02:21.249709 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244420 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 07:02:21.249709 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244422 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 07:02:21.249709 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244425 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 07:02:21.249709 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244427 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 07:02:21.249709 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244430 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 07:02:21.249709 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244433 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 07:02:21.249709 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244435 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 07:02:21.249709 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244438 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 07:02:21.249709 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244440 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 07:02:21.249709 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244443 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 07:02:21.249709 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244445 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 07:02:21.250193 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244448 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 07:02:21.250193 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244450 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 07:02:21.250193 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244453 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 07:02:21.250193 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244455 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 07:02:21.250193 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244458 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 07:02:21.250193 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244461 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 07:02:21.250193 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244463 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 07:02:21.250193 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244466 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 07:02:21.250193 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244468 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 07:02:21.250193 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244471 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 07:02:21.250193 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244475 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 07:02:21.250193 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244479 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 07:02:21.250193 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244482 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 07:02:21.250193 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244485 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 07:02:21.250193 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244488 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 07:02:21.250193 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244491 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 07:02:21.250193 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244493 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 07:02:21.250193 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244496 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 07:02:21.250193 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244498 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 07:02:21.250193 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244501 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 07:02:21.250666 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244503 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 07:02:21.250666 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244506 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 07:02:21.250666 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244508 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 07:02:21.250666 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244511 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 07:02:21.250666 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244514 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 07:02:21.250666 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244517 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 07:02:21.250666 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244520 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 07:02:21.250666 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244522 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 07:02:21.250666 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244525 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 07:02:21.250666 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244528 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 07:02:21.250666 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244530 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 09 07:02:21.250666 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244533 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 07:02:21.250666 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244535 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 07:02:21.250666 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244538 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 07:02:21.250666 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244540 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 07:02:21.250666 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244543 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 07:02:21.250666 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244547 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 07:02:21.250666 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244550 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 07:02:21.250666 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244553 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 07:02:21.251120 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244557 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 07:02:21.251120 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244559 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 07:02:21.251120 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244562 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 07:02:21.251120 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244565 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 07:02:21.251120 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244567 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 07:02:21.251120 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244570 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 07:02:21.251120 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244573 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 07:02:21.251120 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244575 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 07:02:21.251120 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244578 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 07:02:21.251120 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244580 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 07:02:21.251120 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244583 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 07:02:21.251120 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244585 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 07:02:21.251120 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244588 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 07:02:21.251120 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244590 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 07:02:21.251120 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244593 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 07:02:21.251120 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244595 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 07:02:21.251120 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244597 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 07:02:21.251120 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244600 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 07:02:21.251120 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244603 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 07:02:21.251584 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244605 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 07:02:21.251584 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244608 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 07:02:21.251584 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244611 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 07:02:21.251584 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244613 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 07:02:21.251584 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244616 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 07:02:21.251584 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244618 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 07:02:21.251584 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244620 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 07:02:21.251584 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:21.244627 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 07:02:21.251584 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.244632 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 07:02:21.251584 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.244730 2564 server.go:962] "Client rotation is on, will bootstrap in background" Jul 09 07:02:21.251584 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.246717 2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 09 07:02:21.251584 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.248286 2564 server.go:1019] "Starting client certificate rotation" Jul 09 07:02:21.251584 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.248390 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 09 07:02:21.251584 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.248424 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 09 07:02:21.251931 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.251061 2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 09 07:02:21.253942 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.253927 2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 09 07:02:21.265113 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.265096 2564 log.go:25] "Validated CRI v1 runtime API" Jul 09 07:02:21.267578 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.267566 2564 log.go:25] "Validated CRI v1 image API" Jul 09 07:02:21.268395 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.268381 2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 09 07:02:21.271124 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.271105 2564 fs.go:135] Filesystem UUIDs: map[2875f189-86bd-4180-9a47-6c1d4465ca93:/dev/nvme0n1p3 5e457442-2000-4777-a7aa-a686837e794d:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Jul 09 07:02:21.271203 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.271121 2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 09 07:02:21.276935 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.276673 2564 manager.go:216] Machine: {Timestamp:2026-07-09 07:02:21.275167038 +0000 UTC m=+0.326244261 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:2499998 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27ff23c4776c5f638a961f8cd15098 SystemUUID:ec27ff23-c477-6c5f-638a-961f8cd15098 BootID:35493098-8ff6-49bb-bc55-6c06de71bc6e Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:45:a7:fc:0d:c1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:45:a7:fc:0d:c1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7e:18:bd:b3:5d:7a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 09 07:02:21.276935 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.276927 2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 09 07:02:21.277066 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.277021 2564 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 09 07:02:21.277727 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.277712 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 09 07:02:21.279901 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.279870 2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 09 07:02:21.280028 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.279905 2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-0.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 09 07:02:21.280918 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.280908 2564 topology_manager.go:138] "Creating topology manager with none policy" Jul 09 07:02:21.280957 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.280920 2564 container_manager_linux.go:309] "Creating device plugin manager" Jul 09 07:02:21.280957 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.280928 2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 09 07:02:21.281667 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.281657 2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 09 07:02:21.282933 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.282923 2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 09 07:02:21.285775 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.285764 2564 state_mem.go:36] "Initialized new in-memory state store" Jul 09 07:02:21.285894 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.285885 2564 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 09 07:02:21.285923 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.285914 2564 kubelet.go:486] "Attempting to sync node with API server" Jul 09 07:02:21.285956 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.285923 2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 09 07:02:21.285956 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.285938 2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 09 07:02:21.285956 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.285948 2564 kubelet.go:398] "Adding apiserver pod source" Jul 09 07:02:21.285956 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.285957 2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 09 07:02:21.286340 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.286329 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 09 07:02:21.286391 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.286347 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 09 07:02:21.288455 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.288439 2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 09 07:02:21.290208 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.290195 2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 09 07:02:21.290274 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.290225 2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 09 07:02:21.291253 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.291241 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 09 07:02:21.291305 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.291257 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 09 07:02:21.291305 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.291277 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 09 07:02:21.291305 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.291283 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 09 07:02:21.291305 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.291291 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 09 07:02:21.291305 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.291297 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 09 07:02:21.291305 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.291305 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 09 07:02:21.291470 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.291310 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 09 07:02:21.291470 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.291316 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 09 07:02:21.291470 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.291321 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 09 07:02:21.291470 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.291331 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 09 07:02:21.291470 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.291336 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 09 07:02:21.291470 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.291359 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 09 07:02:21.291470 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.291364 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 09 07:02:21.294437 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.294424 2564 server.go:1268] "Started kubelet" Jul 09 07:02:21.294522 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.294502 2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 09 07:02:21.294646 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.294596 2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 09 07:02:21.294738 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.294664 2564 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 09 07:02:21.295434 ip-10-0-138-0 systemd[1]: Started Kubernetes Kubelet. Jul 09 07:02:21.296449 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:21.296413 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 09 07:02:21.296449 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:21.296428 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-0.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 09 07:02:21.296566 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.296541 2564 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-0.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 09 07:02:21.296756 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.296743 2564 server.go:310] "Adding debug handlers to kubelet server" Jul 09 07:02:21.299346 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.299328 2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 09 07:02:21.300083 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:21.299168 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-0.ec2.internal.18c08d0d7026ce6d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-0.ec2.internal,UID:ip-10-0-138-0.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-0.ec2.internal,},FirstTimestamp:2026-07-09 07:02:21.294399085 +0000 UTC m=+0.345476304,LastTimestamp:2026-07-09 07:02:21.294399085 +0000 UTC m=+0.345476304,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-0.ec2.internal,}" Jul 09 07:02:21.300781 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:21.300766 2564 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 09 07:02:21.304671 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.304652 2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 09 07:02:21.304774 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.304679 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 09 07:02:21.305242 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:21.305222 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-0.ec2.internal\" not found" Jul 09 07:02:21.305380 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.305294 2564 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 09 07:02:21.305477 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.305461 2564 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 09 07:02:21.305477 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.305310 2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 09 07:02:21.305624 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.305491 2564 factory.go:55] Registering systemd factory Jul 09 07:02:21.305624 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.305506 2564 factory.go:223] Registration of the systemd container factory successfully Jul 09 07:02:21.305624 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.305556 2564 reconstruct.go:97] "Volume reconstruction finished" Jul 09 07:02:21.305624 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.305564 2564 reconciler.go:29] "Reconciler: start to sync state" Jul 09 07:02:21.305833 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.305818 2564 factory.go:153] Registering CRI-O factory Jul 09 07:02:21.305833 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.305833 2564 factory.go:223] Registration of the crio container factory successfully Jul 09 07:02:21.305918 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.305883 2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 09 07:02:21.305918 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.305905 2564 factory.go:103] Registering Raw factory Jul 09 07:02:21.305918 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.305918 2564 manager.go:1196] Started watching for new ooms in manager Jul 09 07:02:21.306442 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:21.306411 2564 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-0.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 09 07:02:21.306529 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.306497 2564 manager.go:349] Starting recovery of all containers Jul 09 07:02:21.316817 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.316619 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 09 07:02:21.317478 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.317459 2564 manager.go:354] Recovery completed Jul 09 07:02:21.319206 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:21.319181 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 09 07:02:21.321772 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.321758 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 07:02:21.323621 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.323605 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-0.ec2.internal" event="NodeHasSufficientMemory" Jul 09 07:02:21.323698 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.323636 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-0.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 07:02:21.323698 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.323648 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-0.ec2.internal" event="NodeHasSufficientPID" Jul 09 07:02:21.324170 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.324156 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 07:02:21.324170 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.324159 2564 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 09 07:02:21.324351 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.324181 2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 09 07:02:21.324351 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.324199 2564 state_mem.go:36] "Initialized new in-memory state store" Jul 09 07:02:21.324786 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.324773 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-0.ec2.internal" event="NodeHasSufficientMemory" Jul 09 07:02:21.324825 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.324797 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-0.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 07:02:21.324825 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.324807 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-0.ec2.internal" event="NodeHasSufficientPID" Jul 09 07:02:21.325479 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.325466 2564 policy_none.go:49] "None policy: Start" Jul 09 07:02:21.325545 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.325483 2564 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 09 07:02:21.325545 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.325497 2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 09 07:02:21.326451 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.326439 2564 policy_none.go:47] "Start" Jul 09 07:02:21.334322 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:21.334223 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-0.ec2.internal.18c08d0d71e4af35 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-0.ec2.internal,UID:ip-10-0-138-0.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-0.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-0.ec2.internal,},FirstTimestamp:2026-07-09 07:02:21.323620149 +0000 UTC m=+0.374697370,LastTimestamp:2026-07-09 07:02:21.323620149 +0000 UTC m=+0.374697370,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-0.ec2.internal,}" Jul 09 07:02:21.338446 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.338430 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xbcmv" Jul 09 07:02:21.344336 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:21.344256 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-0.ec2.internal.18c08d0d71e50791 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-0.ec2.internal,UID:ip-10-0-138-0.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-0.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-0.ec2.internal,},FirstTimestamp:2026-07-09 07:02:21.323642769 +0000 UTC m=+0.374719990,LastTimestamp:2026-07-09 07:02:21.323642769 +0000 UTC m=+0.374719990,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-0.ec2.internal,}" Jul 09 07:02:21.345752 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.345739 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xbcmv" Jul 09 07:02:21.366027 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.366012 2564 manager.go:337] "Starting Device Plugin manager" Jul 09 07:02:21.366111 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:21.366045 2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 09 07:02:21.366111 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.366056 2564 server.go:87] "Starting device plugin registration server" Jul 09 07:02:21.366331 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.366315 2564 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 09 07:02:21.366587 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.366331 2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 09 07:02:21.366587 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.366438 2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 09 07:02:21.366587 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.366543 2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 09 07:02:21.366587 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.366553 2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 09 07:02:21.367089 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:21.367059 2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 09 07:02:21.367180 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:21.367126 2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-0.ec2.internal\" not found" Jul 09 07:02:21.456551 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.456517 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 09 07:02:21.457697 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.456562 2564 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 09 07:02:21.457697 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.456584 2564 kubelet.go:2448] "Starting kubelet main sync loop" Jul 09 07:02:21.457697 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:21.456617 2564 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 09 07:02:21.459410 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.459389 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 07:02:21.467314 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.467299 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 07:02:21.468584 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.468568 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-0.ec2.internal" event="NodeHasSufficientMemory" Jul 09 07:02:21.468656 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.468598 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-0.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 07:02:21.468656 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.468611 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-0.ec2.internal" event="NodeHasSufficientPID" Jul 09 07:02:21.468656 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.468638 2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-0.ec2.internal" Jul 09 07:02:21.475556 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.475539 2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-0.ec2.internal" Jul 09 07:02:21.475605 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:21.475560 2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-0.ec2.internal\": node \"ip-10-0-138-0.ec2.internal\" not found" Jul 09 07:02:21.501899 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:21.501848 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-0.ec2.internal\" not found" Jul 09 07:02:21.559382 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.559347 2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-138-0.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-0.ec2.internal"] Jul 09 07:02:21.559488 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.559415 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 07:02:21.562196 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.562176 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-0.ec2.internal" event="NodeHasSufficientMemory" Jul 09 07:02:21.562300 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.562208 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-0.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 07:02:21.562300 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.562218 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-0.ec2.internal" event="NodeHasSufficientPID" Jul 09 07:02:21.563535 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.563522 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 07:02:21.563655 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.563639 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-0.ec2.internal" Jul 09 07:02:21.563698 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.563670 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 07:02:21.564350 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.564318 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-0.ec2.internal" event="NodeHasSufficientMemory" Jul 09 07:02:21.564350 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.564347 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-0.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 07:02:21.564485 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.564358 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-0.ec2.internal" event="NodeHasSufficientPID" Jul 09 07:02:21.564485 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.564318 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-0.ec2.internal" event="NodeHasSufficientMemory" Jul 09 07:02:21.564485 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.564432 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-0.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 07:02:21.564485 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.564450 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-0.ec2.internal" event="NodeHasSufficientPID" Jul 09 07:02:21.565701 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.565685 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-0.ec2.internal" Jul 09 07:02:21.565773 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.565710 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 07:02:21.566418 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.566393 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-0.ec2.internal" event="NodeHasSufficientMemory" Jul 09 07:02:21.566510 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.566421 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-0.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 07:02:21.566510 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.566438 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-0.ec2.internal" event="NodeHasSufficientPID" Jul 09 07:02:21.581909 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:21.581888 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-0.ec2.internal\" not found" node="ip-10-0-138-0.ec2.internal" Jul 09 07:02:21.585494 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:21.585478 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-0.ec2.internal\" not found" node="ip-10-0-138-0.ec2.internal" Jul 09 07:02:21.601975 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:21.601953 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-0.ec2.internal\" not found" Jul 09 07:02:21.703096 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:21.703067 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-0.ec2.internal\" not found" Jul 09 07:02:21.707719 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.707692 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a0854266c70379aa8ce321ab152e516e-config\") pod \"kube-apiserver-proxy-ip-10-0-138-0.ec2.internal\" (UID: \"a0854266c70379aa8ce321ab152e516e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-0.ec2.internal" Jul 09 07:02:21.707804 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.707730 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/19c650d7b90f9cfab0129bb7ed79b977-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-0.ec2.internal\" (UID: \"19c650d7b90f9cfab0129bb7ed79b977\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-0.ec2.internal" Jul 09 07:02:21.707804 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.707752 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/19c650d7b90f9cfab0129bb7ed79b977-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-0.ec2.internal\" (UID: \"19c650d7b90f9cfab0129bb7ed79b977\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-0.ec2.internal" Jul 09 07:02:21.803616 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:21.803527 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-0.ec2.internal\" not found" Jul 09 07:02:21.808167 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.808151 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/19c650d7b90f9cfab0129bb7ed79b977-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-0.ec2.internal\" (UID: \"19c650d7b90f9cfab0129bb7ed79b977\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-0.ec2.internal" Jul 09 07:02:21.808273 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.808176 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/19c650d7b90f9cfab0129bb7ed79b977-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-0.ec2.internal\" (UID: \"19c650d7b90f9cfab0129bb7ed79b977\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-0.ec2.internal" Jul 09 07:02:21.808273 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.808196 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a0854266c70379aa8ce321ab152e516e-config\") pod \"kube-apiserver-proxy-ip-10-0-138-0.ec2.internal\" (UID: \"a0854266c70379aa8ce321ab152e516e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-0.ec2.internal" Jul 09 07:02:21.808273 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.808243 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a0854266c70379aa8ce321ab152e516e-config\") pod \"kube-apiserver-proxy-ip-10-0-138-0.ec2.internal\" (UID: \"a0854266c70379aa8ce321ab152e516e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-0.ec2.internal" Jul 09 07:02:21.808397 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.808277 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/19c650d7b90f9cfab0129bb7ed79b977-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-0.ec2.internal\" (UID: \"19c650d7b90f9cfab0129bb7ed79b977\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-0.ec2.internal" Jul 09 07:02:21.808397 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.808293 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/19c650d7b90f9cfab0129bb7ed79b977-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-0.ec2.internal\" (UID: \"19c650d7b90f9cfab0129bb7ed79b977\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-0.ec2.internal" Jul 09 07:02:21.887085 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.887047 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-0.ec2.internal" Jul 09 07:02:21.889349 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:21.889327 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-0.ec2.internal" Jul 09 07:02:21.904362 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:21.904337 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-0.ec2.internal\" not found" Jul 09 07:02:22.004739 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:22.004689 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-0.ec2.internal\" not found" Jul 09 07:02:22.105651 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:22.105566 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-0.ec2.internal\" not found" Jul 09 07:02:22.166625 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:22.166597 2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 07:02:22.207079 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:22.207048 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-0.ec2.internal\" not found" Jul 09 07:02:22.247688 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:22.247664 2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 09 07:02:22.248186 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:22.247800 2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 09 07:02:22.248186 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:22.247854 2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 09 07:02:22.307781 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:22.307759 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-0.ec2.internal\" not found" Jul 09 07:02:22.347240 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:22.347207 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-08 06:57:22 +0000 UTC" deadline="2028-02-07 07:41:23.587775342 +0000 UTC" Jul 09 07:02:22.347240 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:22.347235 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13872h39m1.240542689s" Jul 09 07:02:22.408167 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:22.408139 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-0.ec2.internal\" not found" Jul 09 07:02:22.418152 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:22.418121 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 07:02:22.419129 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:22.419114 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-0.ec2.internal" event="NodeHasSufficientMemory" Jul 09 07:02:22.419205 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:22.419147 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-0.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 07:02:22.419205 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:22.419163 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-0.ec2.internal" event="NodeHasSufficientPID" Jul 09 07:02:22.509004 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:22.508971 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-0.ec2.internal\" not found" Jul 09 07:02:22.609838 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:22.609801 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-0.ec2.internal\" not found" Jul 09 07:02:22.662065 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:22.661986 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19c650d7b90f9cfab0129bb7ed79b977.slice/crio-dbc7727a229877d9749fb10db7b77befa6c33e77f041156c4fe274338ba6ffd8 WatchSource:0}: Error finding container dbc7727a229877d9749fb10db7b77befa6c33e77f041156c4fe274338ba6ffd8: Status 404 returned error can't find the container with id dbc7727a229877d9749fb10db7b77befa6c33e77f041156c4fe274338ba6ffd8 Jul 09 07:02:22.662321 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:22.662308 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0854266c70379aa8ce321ab152e516e.slice/crio-66538b7a53c7777582854c6488e41f38611609a3341ce6cd53db19bc6589c1a0 WatchSource:0}: Error finding container 66538b7a53c7777582854c6488e41f38611609a3341ce6cd53db19bc6589c1a0: Status 404 returned error can't find the container with id 66538b7a53c7777582854c6488e41f38611609a3341ce6cd53db19bc6589c1a0 Jul 09 07:02:22.667081 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:22.667066 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 07:02:22.710888 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:22.710865 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-0.ec2.internal\" not found" Jul 09 07:02:22.748019 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:22.747992 2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 07:02:22.812057 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:22.812033 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-0.ec2.internal\" not found" Jul 09 07:02:22.895239 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:22.895216 2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 07:02:22.905918 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:22.905899 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-0.ec2.internal" Jul 09 07:02:22.915954 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:22.915902 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 09 07:02:22.916847 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:22.916832 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-0.ec2.internal" Jul 09 07:02:22.927526 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:22.927511 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 09 07:02:23.224845 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.224767 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 07:02:23.288573 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.288540 2564 apiserver.go:52] "Watching apiserver" Jul 09 07:02:23.298515 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.298493 2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 09 07:02:23.298901 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.298879 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-138-0.ec2.internal","openshift-cluster-node-tuning-operator/tuned-pjzmc","openshift-image-registry/node-ca-gxqds","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-0.ec2.internal","openshift-network-diagnostics/network-check-target-4899b","openshift-network-operator/iptables-alerter-d77xt","kube-system/global-pull-secret-syncer-2kgs9","kube-system/konnectivity-agent-fcljc","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnd85","openshift-dns/node-resolver-tk76r","openshift-multus/multus-additional-cni-plugins-46w6z","openshift-multus/multus-lw46s","openshift-multus/network-metrics-daemon-9nm6n","openshift-ovn-kubernetes/ovnkube-node-8s5zf"] Jul 09 07:02:23.301507 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.301491 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tk76r" Jul 09 07:02:23.302650 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.302633 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9nm6n" Jul 09 07:02:23.302733 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:23.302695 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9nm6n" podUID="b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0" Jul 09 07:02:23.303937 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.303921 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gxqds" Jul 09 07:02:23.304434 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.304411 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 09 07:02:23.304538 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.304482 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-grvzv\"" Jul 09 07:02:23.304538 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.304524 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 09 07:02:23.305465 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.305447 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:02:23.305555 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.305517 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 09 07:02:23.306043 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.306022 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 09 07:02:23.306379 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.306341 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 09 07:02:23.306542 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.306525 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 09 07:02:23.306628 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.306578 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-qr77n\"" Jul 09 07:02:23.307752 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.307734 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 09 07:02:23.307752 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.307734 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 09 07:02:23.307892 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.307744 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cwxht\"" Jul 09 07:02:23.308103 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.308085 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 09 07:02:23.308198 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.308112 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 09 07:02:23.308198 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.308128 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4899b" Jul 09 07:02:23.308198 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:23.308190 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4899b" podUID="67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc" Jul 09 07:02:23.308379 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.308241 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 09 07:02:23.309067 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.309047 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 09 07:02:23.312141 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.312113 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-d77xt" Jul 09 07:02:23.313563 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.313542 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnd85" Jul 09 07:02:23.314255 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.314239 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-d4ns8\"" Jul 09 07:02:23.314439 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.314421 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 09 07:02:23.314662 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.314643 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 09 07:02:23.314729 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.314668 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 09 07:02:23.314861 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.314847 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kgs9" Jul 09 07:02:23.314920 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:23.314899 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kgs9" podUID="2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986" Jul 09 07:02:23.315795 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.315777 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rxb24\"" Jul 09 07:02:23.316002 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.315984 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 09 07:02:23.316103 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.316082 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/37954298-c5a4-4800-bf9f-8dcd45536e67-run-systemd\") pod \"ovnkube-node-8s5zf\" (UID: \"37954298-c5a4-4800-bf9f-8dcd45536e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:02:23.316175 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.316115 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/37954298-c5a4-4800-bf9f-8dcd45536e67-run-ovn\") pod \"ovnkube-node-8s5zf\" (UID: \"37954298-c5a4-4800-bf9f-8dcd45536e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:02:23.316175 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.316143 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/37954298-c5a4-4800-bf9f-8dcd45536e67-ovn-node-metrics-cert\") pod \"ovnkube-node-8s5zf\" (UID: \"37954298-c5a4-4800-bf9f-8dcd45536e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:02:23.316175 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.316168 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/37954298-c5a4-4800-bf9f-8dcd45536e67-ovnkube-config\") pod \"ovnkube-node-8s5zf\" (UID: \"37954298-c5a4-4800-bf9f-8dcd45536e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:02:23.316369 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.316186 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 09 07:02:23.316369 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.316193 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/37954298-c5a4-4800-bf9f-8dcd45536e67-env-overrides\") pod \"ovnkube-node-8s5zf\" (UID: \"37954298-c5a4-4800-bf9f-8dcd45536e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:02:23.316369 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.316218 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjwm8\" (UniqueName: \"kubernetes.io/projected/9a204676-b17e-4b4b-aeed-d3c3e611f9e3-kube-api-access-jjwm8\") pod \"node-resolver-tk76r\" (UID: \"9a204676-b17e-4b4b-aeed-d3c3e611f9e3\") " pod="openshift-dns/node-resolver-tk76r" Jul 09 07:02:23.316369 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.316245 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djfzj\" (UniqueName: \"kubernetes.io/projected/b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0-kube-api-access-djfzj\") pod \"network-metrics-daemon-9nm6n\" (UID: \"b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0\") " pod="openshift-multus/network-metrics-daemon-9nm6n" Jul 09 07:02:23.316369 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.316279 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pjzmc" Jul 09 07:02:23.316369 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.316269 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kmsr\" (UniqueName: \"kubernetes.io/projected/7bac4227-dd99-4984-aa2e-efeaa7a04755-kube-api-access-4kmsr\") pod \"node-ca-gxqds\" (UID: \"7bac4227-dd99-4984-aa2e-efeaa7a04755\") " pod="openshift-image-registry/node-ca-gxqds" Jul 09 07:02:23.316369 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.316329 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/37954298-c5a4-4800-bf9f-8dcd45536e67-log-socket\") pod \"ovnkube-node-8s5zf\" (UID: \"37954298-c5a4-4800-bf9f-8dcd45536e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:02:23.316369 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.316339 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 09 07:02:23.316369 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.316352 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/37954298-c5a4-4800-bf9f-8dcd45536e67-host-cni-bin\") pod \"ovnkube-node-8s5zf\" (UID: \"37954298-c5a4-4800-bf9f-8dcd45536e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:02:23.316861 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.316375 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7bac4227-dd99-4984-aa2e-efeaa7a04755-host\") pod \"node-ca-gxqds\" (UID: \"7bac4227-dd99-4984-aa2e-efeaa7a04755\") " pod="openshift-image-registry/node-ca-gxqds" Jul 09 07:02:23.316861 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.316409 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/37954298-c5a4-4800-bf9f-8dcd45536e67-host-run-netns\") pod \"ovnkube-node-8s5zf\" (UID: \"37954298-c5a4-4800-bf9f-8dcd45536e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:02:23.316861 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.316435 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37954298-c5a4-4800-bf9f-8dcd45536e67-host-run-ovn-kubernetes\") pod \"ovnkube-node-8s5zf\" (UID: \"37954298-c5a4-4800-bf9f-8dcd45536e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:02:23.316861 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.316461 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37954298-c5a4-4800-bf9f-8dcd45536e67-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8s5zf\" (UID: \"37954298-c5a4-4800-bf9f-8dcd45536e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:02:23.316861 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.316485 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/37954298-c5a4-4800-bf9f-8dcd45536e67-host-kubelet\") pod \"ovnkube-node-8s5zf\" (UID: \"37954298-c5a4-4800-bf9f-8dcd45536e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:02:23.316861 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.316528 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/37954298-c5a4-4800-bf9f-8dcd45536e67-etc-openvswitch\") pod \"ovnkube-node-8s5zf\" (UID: \"37954298-c5a4-4800-bf9f-8dcd45536e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:02:23.316861 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.316570 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/37954298-c5a4-4800-bf9f-8dcd45536e67-ovnkube-script-lib\") pod \"ovnkube-node-8s5zf\" (UID: \"37954298-c5a4-4800-bf9f-8dcd45536e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:02:23.316861 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.316602 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/37954298-c5a4-4800-bf9f-8dcd45536e67-host-slash\") pod \"ovnkube-node-8s5zf\" (UID: \"37954298-c5a4-4800-bf9f-8dcd45536e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:02:23.316861 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.316630 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/37954298-c5a4-4800-bf9f-8dcd45536e67-systemd-units\") pod \"ovnkube-node-8s5zf\" (UID: \"37954298-c5a4-4800-bf9f-8dcd45536e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:02:23.316861 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.316655 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/37954298-c5a4-4800-bf9f-8dcd45536e67-host-cni-netd\") pod \"ovnkube-node-8s5zf\" (UID: \"37954298-c5a4-4800-bf9f-8dcd45536e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:02:23.316861 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.316680 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a204676-b17e-4b4b-aeed-d3c3e611f9e3-tmp-dir\") pod \"node-resolver-tk76r\" (UID: \"9a204676-b17e-4b4b-aeed-d3c3e611f9e3\") " pod="openshift-dns/node-resolver-tk76r" Jul 09 07:02:23.316861 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.316705 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/37954298-c5a4-4800-bf9f-8dcd45536e67-node-log\") pod \"ovnkube-node-8s5zf\" (UID: \"37954298-c5a4-4800-bf9f-8dcd45536e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:02:23.316861 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.316730 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9a204676-b17e-4b4b-aeed-d3c3e611f9e3-hosts-file\") pod \"node-resolver-tk76r\" (UID: \"9a204676-b17e-4b4b-aeed-d3c3e611f9e3\") " pod="openshift-dns/node-resolver-tk76r" Jul 09 07:02:23.316861 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.316755 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/37954298-c5a4-4800-bf9f-8dcd45536e67-var-lib-openvswitch\") pod \"ovnkube-node-8s5zf\" (UID: \"37954298-c5a4-4800-bf9f-8dcd45536e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:02:23.316861 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.316781 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/37954298-c5a4-4800-bf9f-8dcd45536e67-run-openvswitch\") pod \"ovnkube-node-8s5zf\" (UID: \"37954298-c5a4-4800-bf9f-8dcd45536e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:02:23.316861 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.316806 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnxd8\" (UniqueName: \"kubernetes.io/projected/37954298-c5a4-4800-bf9f-8dcd45536e67-kube-api-access-wnxd8\") pod \"ovnkube-node-8s5zf\" (UID: \"37954298-c5a4-4800-bf9f-8dcd45536e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:02:23.317461 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.316832 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxpzb\" (UniqueName: \"kubernetes.io/projected/67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc-kube-api-access-lxpzb\") pod \"network-check-target-4899b\" (UID: \"67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc\") " pod="openshift-network-diagnostics/network-check-target-4899b" Jul 09 07:02:23.317461 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.316876 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0-metrics-certs\") pod \"network-metrics-daemon-9nm6n\" (UID: \"b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0\") " pod="openshift-multus/network-metrics-daemon-9nm6n" Jul 09 07:02:23.317461 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.316900 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7bac4227-dd99-4984-aa2e-efeaa7a04755-serviceca\") pod \"node-ca-gxqds\" (UID: \"7bac4227-dd99-4984-aa2e-efeaa7a04755\") " pod="openshift-image-registry/node-ca-gxqds" Jul 09 07:02:23.317757 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.317650 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 09 07:02:23.317862 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.317787 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fcljc" Jul 09 07:02:23.318711 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.318694 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-g7ckm\"" Jul 09 07:02:23.318914 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.318901 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 09 07:02:23.319150 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.319138 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 09 07:02:23.319623 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.319609 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-46w6z" Jul 09 07:02:23.320149 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.320127 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 09 07:02:23.320274 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.320257 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-nlmgs\"" Jul 09 07:02:23.320368 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.320285 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 09 07:02:23.321116 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.321097 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lw46s" Jul 09 07:02:23.321707 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.321693 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 09 07:02:23.322270 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.322252 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 09 07:02:23.322358 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.322314 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 09 07:02:23.322552 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.322537 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5nfxj\"" Jul 09 07:02:23.322999 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.322983 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 09 07:02:23.323113 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.323095 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 09 07:02:23.323814 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.323797 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-mzsms\"" Jul 09 07:02:23.323903 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.323856 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 09 07:02:23.344807 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.344789 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-l2727" Jul 09 07:02:23.352080 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.352061 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-l2727" Jul 09 07:02:23.407170 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.407150 2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 09 07:02:23.417968 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.417940 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kmsr\" (UniqueName: \"kubernetes.io/projected/7bac4227-dd99-4984-aa2e-efeaa7a04755-kube-api-access-4kmsr\") pod \"node-ca-gxqds\" (UID: \"7bac4227-dd99-4984-aa2e-efeaa7a04755\") " pod="openshift-image-registry/node-ca-gxqds" Jul 09 07:02:23.418075 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.417974 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/37954298-c5a4-4800-bf9f-8dcd45536e67-log-socket\") pod \"ovnkube-node-8s5zf\" (UID: \"37954298-c5a4-4800-bf9f-8dcd45536e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:02:23.418075 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.418002 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/efb461ce-198a-48dd-8a66-dfb37357ac52-sys\") pod \"tuned-pjzmc\" (UID: \"efb461ce-198a-48dd-8a66-dfb37357ac52\") " pod="openshift-cluster-node-tuning-operator/tuned-pjzmc" Jul 09 07:02:23.418075 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.418027 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/461328f2-e00a-4d14-ab04-7651d773f7ca-host-var-lib-cni-multus\") pod \"multus-lw46s\" (UID: \"461328f2-e00a-4d14-ab04-7651d773f7ca\") " pod="openshift-multus/multus-lw46s" Jul 09 07:02:23.418075 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.418055 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37954298-c5a4-4800-bf9f-8dcd45536e67-host-run-ovn-kubernetes\") pod \"ovnkube-node-8s5zf\" (UID: \"37954298-c5a4-4800-bf9f-8dcd45536e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:02:23.418255 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.418070 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/37954298-c5a4-4800-bf9f-8dcd45536e67-log-socket\") pod \"ovnkube-node-8s5zf\" (UID: \"37954298-c5a4-4800-bf9f-8dcd45536e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:02:23.418255 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.418083 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/14b4eb3e-db26-4cd1-84b1-97f13b474c2e-iptables-alerter-script\") pod \"iptables-alerter-d77xt\" (UID: \"14b4eb3e-db26-4cd1-84b1-97f13b474c2e\") " pod="openshift-network-operator/iptables-alerter-d77xt" Jul 09 07:02:23.418255 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.418102 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37954298-c5a4-4800-bf9f-8dcd45536e67-host-run-ovn-kubernetes\") pod \"ovnkube-node-8s5zf\" (UID: \"37954298-c5a4-4800-bf9f-8dcd45536e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:02:23.418255 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.418121 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/43cb1cf4-40f4-477d-bc2c-6ef608e242a0-registration-dir\") pod \"aws-ebs-csi-driver-node-jnd85\" (UID: \"43cb1cf4-40f4-477d-bc2c-6ef608e242a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnd85" Jul 09 07:02:23.418255 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.418147 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/efb461ce-198a-48dd-8a66-dfb37357ac52-var-lib-kubelet\") pod \"tuned-pjzmc\" (UID: \"efb461ce-198a-48dd-8a66-dfb37357ac52\") " pod="openshift-cluster-node-tuning-operator/tuned-pjzmc" Jul 09 07:02:23.418255 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.418191 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlqqs\" (UniqueName: \"kubernetes.io/projected/efb461ce-198a-48dd-8a66-dfb37357ac52-kube-api-access-nlqqs\") pod \"tuned-pjzmc\" (UID: \"efb461ce-198a-48dd-8a66-dfb37357ac52\") " pod="openshift-cluster-node-tuning-operator/tuned-pjzmc" Jul 09 07:02:23.418255 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.418226 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/461328f2-e00a-4d14-ab04-7651d773f7ca-cni-binary-copy\") pod \"multus-lw46s\" (UID: \"461328f2-e00a-4d14-ab04-7651d773f7ca\") " pod="openshift-multus/multus-lw46s" Jul 09 07:02:23.418545 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.418263 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/461328f2-e00a-4d14-ab04-7651d773f7ca-multus-socket-dir-parent\") pod \"multus-lw46s\" (UID: \"461328f2-e00a-4d14-ab04-7651d773f7ca\") " pod="openshift-multus/multus-lw46s" Jul 09 07:02:23.418545 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.418304 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/37954298-c5a4-4800-bf9f-8dcd45536e67-host-kubelet\") pod \"ovnkube-node-8s5zf\" (UID: \"37954298-c5a4-4800-bf9f-8dcd45536e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:02:23.418545 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.418330 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/37954298-c5a4-4800-bf9f-8dcd45536e67-etc-openvswitch\") pod \"ovnkube-node-8s5zf\" (UID: \"37954298-c5a4-4800-bf9f-8dcd45536e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:02:23.418545 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.418338 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/37954298-c5a4-4800-bf9f-8dcd45536e67-host-kubelet\") pod \"ovnkube-node-8s5zf\" (UID: \"37954298-c5a4-4800-bf9f-8dcd45536e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:02:23.418545 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.418346 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/461328f2-e00a-4d14-ab04-7651d773f7ca-host-run-netns\") pod \"multus-lw46s\" (UID: \"461328f2-e00a-4d14-ab04-7651d773f7ca\") " pod="openshift-multus/multus-lw46s" Jul 09 07:02:23.418545 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.418362 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbvqd\" (UniqueName: \"kubernetes.io/projected/14b4eb3e-db26-4cd1-84b1-97f13b474c2e-kube-api-access-kbvqd\") pod \"iptables-alerter-d77xt\" (UID: \"14b4eb3e-db26-4cd1-84b1-97f13b474c2e\") " pod="openshift-network-operator/iptables-alerter-d77xt" Jul 09 07:02:23.418545 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.418373 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/37954298-c5a4-4800-bf9f-8dcd45536e67-etc-openvswitch\") pod \"ovnkube-node-8s5zf\" (UID: \"37954298-c5a4-4800-bf9f-8dcd45536e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:02:23.418545 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.418378 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/43cb1cf4-40f4-477d-bc2c-6ef608e242a0-etc-selinux\") pod \"aws-ebs-csi-driver-node-jnd85\" (UID: \"43cb1cf4-40f4-477d-bc2c-6ef608e242a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnd85" Jul 09 07:02:23.418545 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.418411 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a318fe5c-c8c8-4fff-b429-a7f4aa33b8c0-konnectivity-ca\") pod \"konnectivity-agent-fcljc\" (UID: \"a318fe5c-c8c8-4fff-b429-a7f4aa33b8c0\") " pod="kube-system/konnectivity-agent-fcljc" Jul 09 07:02:23.418545 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.418428 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8e9a1d6e-b7b9-494e-90f7-367f51bcb2aa-os-release\") pod \"multus-additional-cni-plugins-46w6z\" (UID: \"8e9a1d6e-b7b9-494e-90f7-367f51bcb2aa\") " pod="openshift-multus/multus-additional-cni-plugins-46w6z" Jul 09 07:02:23.418545 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.418443 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/461328f2-e00a-4d14-ab04-7651d773f7ca-os-release\") pod \"multus-lw46s\" (UID: \"461328f2-e00a-4d14-ab04-7651d773f7ca\") " pod="openshift-multus/multus-lw46s" Jul 09 07:02:23.418545 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.418462 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/43cb1cf4-40f4-477d-bc2c-6ef608e242a0-device-dir\") pod \"aws-ebs-csi-driver-node-jnd85\" (UID: \"43cb1cf4-40f4-477d-bc2c-6ef608e242a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnd85" Jul 09 07:02:23.418545 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.418482 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a318fe5c-c8c8-4fff-b429-a7f4aa33b8c0-agent-certs\") pod \"konnectivity-agent-fcljc\" (UID: \"a318fe5c-c8c8-4fff-b429-a7f4aa33b8c0\") " pod="kube-system/konnectivity-agent-fcljc" Jul 09 07:02:23.418545 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.418499 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/efb461ce-198a-48dd-8a66-dfb37357ac52-etc-sysctl-conf\") pod \"tuned-pjzmc\" (UID: \"efb461ce-198a-48dd-8a66-dfb37357ac52\") " pod="openshift-cluster-node-tuning-operator/tuned-pjzmc" Jul 09 07:02:23.418545 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.418528 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/461328f2-e00a-4d14-ab04-7651d773f7ca-host-run-k8s-cni-cncf-io\") pod \"multus-lw46s\" (UID: \"461328f2-e00a-4d14-ab04-7651d773f7ca\") " pod="openshift-multus/multus-lw46s" Jul 09 07:02:23.418545 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.418550 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a204676-b17e-4b4b-aeed-d3c3e611f9e3-tmp-dir\") pod \"node-resolver-tk76r\" (UID: \"9a204676-b17e-4b4b-aeed-d3c3e611f9e3\") " pod="openshift-dns/node-resolver-tk76r" Jul 09 07:02:23.419394 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.418570 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/14b4eb3e-db26-4cd1-84b1-97f13b474c2e-host-slash\") pod \"iptables-alerter-d77xt\" (UID: \"14b4eb3e-db26-4cd1-84b1-97f13b474c2e\") " pod="openshift-network-operator/iptables-alerter-d77xt" Jul 09 07:02:23.419394 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.418590 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/efb461ce-198a-48dd-8a66-dfb37357ac52-etc-sysconfig\") pod \"tuned-pjzmc\" (UID: \"efb461ce-198a-48dd-8a66-dfb37357ac52\") " pod="openshift-cluster-node-tuning-operator/tuned-pjzmc" Jul 09 07:02:23.419394 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.418613 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/461328f2-e00a-4d14-ab04-7651d773f7ca-system-cni-dir\") pod \"multus-lw46s\" (UID: \"461328f2-e00a-4d14-ab04-7651d773f7ca\") " pod="openshift-multus/multus-lw46s" Jul 09 07:02:23.419394 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.418631 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/461328f2-e00a-4d14-ab04-7651d773f7ca-cnibin\") pod \"multus-lw46s\" (UID: \"461328f2-e00a-4d14-ab04-7651d773f7ca\") " pod="openshift-multus/multus-lw46s" Jul 09 07:02:23.419394 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.418651 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/461328f2-e00a-4d14-ab04-7651d773f7ca-host-run-multus-certs\") pod \"multus-lw46s\" (UID: \"461328f2-e00a-4d14-ab04-7651d773f7ca\") " pod="openshift-multus/multus-lw46s" Jul 09 07:02:23.419394 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.418680 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9a204676-b17e-4b4b-aeed-d3c3e611f9e3-hosts-file\") pod \"node-resolver-tk76r\" (UID: \"9a204676-b17e-4b4b-aeed-d3c3e611f9e3\") " pod="openshift-dns/node-resolver-tk76r" Jul 09 07:02:23.419394 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.418724 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9a204676-b17e-4b4b-aeed-d3c3e611f9e3-hosts-file\") pod \"node-resolver-tk76r\" (UID: \"9a204676-b17e-4b4b-aeed-d3c3e611f9e3\") " pod="openshift-dns/node-resolver-tk76r" Jul 09 07:02:23.419394 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.418773 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/37954298-c5a4-4800-bf9f-8dcd45536e67-var-lib-openvswitch\") pod \"ovnkube-node-8s5zf\" (UID: \"37954298-c5a4-4800-bf9f-8dcd45536e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:02:23.419394 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.418798 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxpzb\" (UniqueName: \"kubernetes.io/projected/67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc-kube-api-access-lxpzb\") pod \"network-check-target-4899b\" (UID: \"67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc\") " pod="openshift-network-diagnostics/network-check-target-4899b" Jul 09 07:02:23.419394 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.418819 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgzj2\" (UniqueName: \"kubernetes.io/projected/461328f2-e00a-4d14-ab04-7651d773f7ca-kube-api-access-vgzj2\") pod \"multus-lw46s\" (UID: \"461328f2-e00a-4d14-ab04-7651d773f7ca\") " pod="openshift-multus/multus-lw46s" Jul 09 07:02:23.419394 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.418846 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7bac4227-dd99-4984-aa2e-efeaa7a04755-serviceca\") pod \"node-ca-gxqds\" (UID: \"7bac4227-dd99-4984-aa2e-efeaa7a04755\") " pod="openshift-image-registry/node-ca-gxqds" Jul 09 07:02:23.419394 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.418861 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/37954298-c5a4-4800-bf9f-8dcd45536e67-var-lib-openvswitch\") pod \"ovnkube-node-8s5zf\" (UID: \"37954298-c5a4-4800-bf9f-8dcd45536e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:02:23.419394 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.418870 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a204676-b17e-4b4b-aeed-d3c3e611f9e3-tmp-dir\") pod \"node-resolver-tk76r\" (UID: \"9a204676-b17e-4b4b-aeed-d3c3e611f9e3\") " pod="openshift-dns/node-resolver-tk76r" Jul 09 07:02:23.419394 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.419051 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/37954298-c5a4-4800-bf9f-8dcd45536e67-run-systemd\") pod \"ovnkube-node-8s5zf\" (UID: \"37954298-c5a4-4800-bf9f-8dcd45536e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:02:23.419394 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.419081 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/37954298-c5a4-4800-bf9f-8dcd45536e67-ovn-node-metrics-cert\") pod \"ovnkube-node-8s5zf\" (UID: \"37954298-c5a4-4800-bf9f-8dcd45536e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:02:23.419394 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.419110 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8e9a1d6e-b7b9-494e-90f7-367f51bcb2aa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-46w6z\" (UID: \"8e9a1d6e-b7b9-494e-90f7-367f51bcb2aa\") " pod="openshift-multus/multus-additional-cni-plugins-46w6z" Jul 09 07:02:23.419394 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.419140 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/461328f2-e00a-4d14-ab04-7651d773f7ca-multus-conf-dir\") pod \"multus-lw46s\" (UID: \"461328f2-e00a-4d14-ab04-7651d773f7ca\") " pod="openshift-multus/multus-lw46s" Jul 09 07:02:23.420213 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.419140 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/37954298-c5a4-4800-bf9f-8dcd45536e67-run-systemd\") pod \"ovnkube-node-8s5zf\" (UID: \"37954298-c5a4-4800-bf9f-8dcd45536e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:02:23.420213 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.419175 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/37954298-c5a4-4800-bf9f-8dcd45536e67-ovnkube-config\") pod \"ovnkube-node-8s5zf\" (UID: \"37954298-c5a4-4800-bf9f-8dcd45536e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:02:23.420213 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.419291 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/37954298-c5a4-4800-bf9f-8dcd45536e67-host-cni-bin\") pod \"ovnkube-node-8s5zf\" (UID: \"37954298-c5a4-4800-bf9f-8dcd45536e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:02:23.420213 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.419327 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/efb461ce-198a-48dd-8a66-dfb37357ac52-host\") pod \"tuned-pjzmc\" (UID: \"efb461ce-198a-48dd-8a66-dfb37357ac52\") " pod="openshift-cluster-node-tuning-operator/tuned-pjzmc" Jul 09 07:02:23.420213 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.419352 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7bac4227-dd99-4984-aa2e-efeaa7a04755-host\") pod \"node-ca-gxqds\" (UID: \"7bac4227-dd99-4984-aa2e-efeaa7a04755\") " pod="openshift-image-registry/node-ca-gxqds" Jul 09 07:02:23.420213 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.419379 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/37954298-c5a4-4800-bf9f-8dcd45536e67-host-run-netns\") pod \"ovnkube-node-8s5zf\" (UID: \"37954298-c5a4-4800-bf9f-8dcd45536e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:02:23.420213 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.419395 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/37954298-c5a4-4800-bf9f-8dcd45536e67-host-cni-bin\") pod \"ovnkube-node-8s5zf\" (UID: \"37954298-c5a4-4800-bf9f-8dcd45536e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:02:23.420213 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.419408 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37954298-c5a4-4800-bf9f-8dcd45536e67-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8s5zf\" (UID: \"37954298-c5a4-4800-bf9f-8dcd45536e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:02:23.420213 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.419437 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/43cb1cf4-40f4-477d-bc2c-6ef608e242a0-socket-dir\") pod \"aws-ebs-csi-driver-node-jnd85\" (UID: \"43cb1cf4-40f4-477d-bc2c-6ef608e242a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnd85" Jul 09 07:02:23.420213 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.419445 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7bac4227-dd99-4984-aa2e-efeaa7a04755-host\") pod \"node-ca-gxqds\" (UID: \"7bac4227-dd99-4984-aa2e-efeaa7a04755\") " pod="openshift-image-registry/node-ca-gxqds" Jul 09 07:02:23.420213 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.419455 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7bac4227-dd99-4984-aa2e-efeaa7a04755-serviceca\") pod \"node-ca-gxqds\" (UID: \"7bac4227-dd99-4984-aa2e-efeaa7a04755\") " pod="openshift-image-registry/node-ca-gxqds" Jul 09 07:02:23.420213 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.419464 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/43cb1cf4-40f4-477d-bc2c-6ef608e242a0-sys-fs\") pod \"aws-ebs-csi-driver-node-jnd85\" (UID: \"43cb1cf4-40f4-477d-bc2c-6ef608e242a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnd85" Jul 09 07:02:23.420213 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.419489 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/efb461ce-198a-48dd-8a66-dfb37357ac52-tmp\") pod \"tuned-pjzmc\" (UID: \"efb461ce-198a-48dd-8a66-dfb37357ac52\") " pod="openshift-cluster-node-tuning-operator/tuned-pjzmc" Jul 09 07:02:23.420213 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.419503 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/37954298-c5a4-4800-bf9f-8dcd45536e67-host-run-netns\") pod \"ovnkube-node-8s5zf\" (UID: \"37954298-c5a4-4800-bf9f-8dcd45536e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:02:23.420213 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.419517 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8e9a1d6e-b7b9-494e-90f7-367f51bcb2aa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-46w6z\" (UID: \"8e9a1d6e-b7b9-494e-90f7-367f51bcb2aa\") " pod="openshift-multus/multus-additional-cni-plugins-46w6z" Jul 09 07:02:23.420213 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.419539 2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 09 07:02:23.420213 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.419550 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/37954298-c5a4-4800-bf9f-8dcd45536e67-ovnkube-script-lib\") pod \"ovnkube-node-8s5zf\" (UID: \"37954298-c5a4-4800-bf9f-8dcd45536e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:02:23.420801 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.419568 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986-original-pull-secret\") pod \"global-pull-secret-syncer-2kgs9\" (UID: \"2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986\") " pod="kube-system/global-pull-secret-syncer-2kgs9" Jul 09 07:02:23.420801 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.419584 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/461328f2-e00a-4d14-ab04-7651d773f7ca-etc-kubernetes\") pod \"multus-lw46s\" (UID: \"461328f2-e00a-4d14-ab04-7651d773f7ca\") " pod="openshift-multus/multus-lw46s" Jul 09 07:02:23.420801 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.419608 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/37954298-c5a4-4800-bf9f-8dcd45536e67-host-slash\") pod \"ovnkube-node-8s5zf\" (UID: \"37954298-c5a4-4800-bf9f-8dcd45536e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:02:23.420801 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.419632 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8e9a1d6e-b7b9-494e-90f7-367f51bcb2aa-cni-binary-copy\") pod \"multus-additional-cni-plugins-46w6z\" (UID: \"8e9a1d6e-b7b9-494e-90f7-367f51bcb2aa\") " pod="openshift-multus/multus-additional-cni-plugins-46w6z" Jul 09 07:02:23.420801 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.419657 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8e9a1d6e-b7b9-494e-90f7-367f51bcb2aa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-46w6z\" (UID: \"8e9a1d6e-b7b9-494e-90f7-367f51bcb2aa\") " pod="openshift-multus/multus-additional-cni-plugins-46w6z" Jul 09 07:02:23.420801 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.419681 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/461328f2-e00a-4d14-ab04-7651d773f7ca-multus-cni-dir\") pod \"multus-lw46s\" (UID: \"461328f2-e00a-4d14-ab04-7651d773f7ca\") " pod="openshift-multus/multus-lw46s" Jul 09 07:02:23.420801 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.419705 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/461328f2-e00a-4d14-ab04-7651d773f7ca-hostroot\") pod \"multus-lw46s\" (UID: \"461328f2-e00a-4d14-ab04-7651d773f7ca\") " pod="openshift-multus/multus-lw46s" Jul 09 07:02:23.420801 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.419543 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37954298-c5a4-4800-bf9f-8dcd45536e67-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8s5zf\" (UID: \"37954298-c5a4-4800-bf9f-8dcd45536e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:02:23.420801 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.419738 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/37954298-c5a4-4800-bf9f-8dcd45536e67-host-slash\") pod \"ovnkube-node-8s5zf\" (UID: \"37954298-c5a4-4800-bf9f-8dcd45536e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:02:23.420801 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.419751 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/37954298-c5a4-4800-bf9f-8dcd45536e67-systemd-units\") pod \"ovnkube-node-8s5zf\" (UID: \"37954298-c5a4-4800-bf9f-8dcd45536e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:02:23.420801 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.419863 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/37954298-c5a4-4800-bf9f-8dcd45536e67-host-cni-netd\") pod \"ovnkube-node-8s5zf\" (UID: \"37954298-c5a4-4800-bf9f-8dcd45536e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:02:23.420801 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.419876 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/37954298-c5a4-4800-bf9f-8dcd45536e67-systemd-units\") pod \"ovnkube-node-8s5zf\" (UID: \"37954298-c5a4-4800-bf9f-8dcd45536e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:02:23.420801 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.419904 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/43cb1cf4-40f4-477d-bc2c-6ef608e242a0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jnd85\" (UID: \"43cb1cf4-40f4-477d-bc2c-6ef608e242a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnd85" Jul 09 07:02:23.420801 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.419923 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/37954298-c5a4-4800-bf9f-8dcd45536e67-host-cni-netd\") pod \"ovnkube-node-8s5zf\" (UID: \"37954298-c5a4-4800-bf9f-8dcd45536e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:02:23.420801 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.419954 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzrnz\" (UniqueName: \"kubernetes.io/projected/43cb1cf4-40f4-477d-bc2c-6ef608e242a0-kube-api-access-tzrnz\") pod \"aws-ebs-csi-driver-node-jnd85\" (UID: \"43cb1cf4-40f4-477d-bc2c-6ef608e242a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnd85" Jul 09 07:02:23.420801 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.419982 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986-kubelet-config\") pod \"global-pull-secret-syncer-2kgs9\" (UID: \"2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986\") " pod="kube-system/global-pull-secret-syncer-2kgs9" Jul 09 07:02:23.420801 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.420007 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/efb461ce-198a-48dd-8a66-dfb37357ac52-etc-sysctl-d\") pod \"tuned-pjzmc\" (UID: \"efb461ce-198a-48dd-8a66-dfb37357ac52\") " pod="openshift-cluster-node-tuning-operator/tuned-pjzmc" Jul 09 07:02:23.421426 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.420039 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/efb461ce-198a-48dd-8a66-dfb37357ac52-lib-modules\") pod \"tuned-pjzmc\" (UID: \"efb461ce-198a-48dd-8a66-dfb37357ac52\") " pod="openshift-cluster-node-tuning-operator/tuned-pjzmc" Jul 09 07:02:23.421426 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.420066 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjcgs\" (UniqueName: \"kubernetes.io/projected/8e9a1d6e-b7b9-494e-90f7-367f51bcb2aa-kube-api-access-gjcgs\") pod \"multus-additional-cni-plugins-46w6z\" (UID: \"8e9a1d6e-b7b9-494e-90f7-367f51bcb2aa\") " pod="openshift-multus/multus-additional-cni-plugins-46w6z" Jul 09 07:02:23.421426 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.420075 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/37954298-c5a4-4800-bf9f-8dcd45536e67-ovnkube-config\") pod \"ovnkube-node-8s5zf\" (UID: \"37954298-c5a4-4800-bf9f-8dcd45536e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:02:23.421426 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.420086 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/37954298-c5a4-4800-bf9f-8dcd45536e67-ovnkube-script-lib\") pod \"ovnkube-node-8s5zf\" (UID: \"37954298-c5a4-4800-bf9f-8dcd45536e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:02:23.421426 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.420094 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/37954298-c5a4-4800-bf9f-8dcd45536e67-node-log\") pod \"ovnkube-node-8s5zf\" (UID: \"37954298-c5a4-4800-bf9f-8dcd45536e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:02:23.421426 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.420128 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986-dbus\") pod \"global-pull-secret-syncer-2kgs9\" (UID: \"2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986\") " pod="kube-system/global-pull-secret-syncer-2kgs9" Jul 09 07:02:23.421426 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.420139 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/37954298-c5a4-4800-bf9f-8dcd45536e67-node-log\") pod \"ovnkube-node-8s5zf\" (UID: \"37954298-c5a4-4800-bf9f-8dcd45536e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:02:23.421426 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.420155 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/efb461ce-198a-48dd-8a66-dfb37357ac52-etc-systemd\") pod \"tuned-pjzmc\" (UID: \"efb461ce-198a-48dd-8a66-dfb37357ac52\") " pod="openshift-cluster-node-tuning-operator/tuned-pjzmc" Jul 09 07:02:23.421426 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.420225 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e9a1d6e-b7b9-494e-90f7-367f51bcb2aa-system-cni-dir\") pod \"multus-additional-cni-plugins-46w6z\" (UID: \"8e9a1d6e-b7b9-494e-90f7-367f51bcb2aa\") " pod="openshift-multus/multus-additional-cni-plugins-46w6z" Jul 09 07:02:23.421426 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.420242 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/461328f2-e00a-4d14-ab04-7651d773f7ca-host-var-lib-kubelet\") pod \"multus-lw46s\" (UID: \"461328f2-e00a-4d14-ab04-7651d773f7ca\") " pod="openshift-multus/multus-lw46s" Jul 09 07:02:23.421426 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.420282 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/37954298-c5a4-4800-bf9f-8dcd45536e67-run-openvswitch\") pod \"ovnkube-node-8s5zf\" (UID: \"37954298-c5a4-4800-bf9f-8dcd45536e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:02:23.421426 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.420303 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnxd8\" (UniqueName: \"kubernetes.io/projected/37954298-c5a4-4800-bf9f-8dcd45536e67-kube-api-access-wnxd8\") pod \"ovnkube-node-8s5zf\" (UID: \"37954298-c5a4-4800-bf9f-8dcd45536e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:02:23.421426 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.420351 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/37954298-c5a4-4800-bf9f-8dcd45536e67-run-openvswitch\") pod \"ovnkube-node-8s5zf\" (UID: \"37954298-c5a4-4800-bf9f-8dcd45536e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:02:23.421426 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.420363 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/efb461ce-198a-48dd-8a66-dfb37357ac52-run\") pod \"tuned-pjzmc\" (UID: \"efb461ce-198a-48dd-8a66-dfb37357ac52\") " pod="openshift-cluster-node-tuning-operator/tuned-pjzmc" Jul 09 07:02:23.421426 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.420440 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/efb461ce-198a-48dd-8a66-dfb37357ac52-etc-tuned\") pod \"tuned-pjzmc\" (UID: \"efb461ce-198a-48dd-8a66-dfb37357ac52\") " pod="openshift-cluster-node-tuning-operator/tuned-pjzmc" Jul 09 07:02:23.421426 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.420466 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8e9a1d6e-b7b9-494e-90f7-367f51bcb2aa-cnibin\") pod \"multus-additional-cni-plugins-46w6z\" (UID: \"8e9a1d6e-b7b9-494e-90f7-367f51bcb2aa\") " pod="openshift-multus/multus-additional-cni-plugins-46w6z" Jul 09 07:02:23.421426 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.420493 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/461328f2-e00a-4d14-ab04-7651d773f7ca-host-var-lib-cni-bin\") pod \"multus-lw46s\" (UID: \"461328f2-e00a-4d14-ab04-7651d773f7ca\") " pod="openshift-multus/multus-lw46s" Jul 09 07:02:23.421930 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.420525 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0-metrics-certs\") pod \"network-metrics-daemon-9nm6n\" (UID: \"b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0\") " pod="openshift-multus/network-metrics-daemon-9nm6n" Jul 09 07:02:23.421930 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.420553 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/37954298-c5a4-4800-bf9f-8dcd45536e67-run-ovn\") pod \"ovnkube-node-8s5zf\" (UID: \"37954298-c5a4-4800-bf9f-8dcd45536e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:02:23.421930 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.420581 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/efb461ce-198a-48dd-8a66-dfb37357ac52-etc-modprobe-d\") pod \"tuned-pjzmc\" (UID: \"efb461ce-198a-48dd-8a66-dfb37357ac52\") " pod="openshift-cluster-node-tuning-operator/tuned-pjzmc" Jul 09 07:02:23.421930 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.420609 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/efb461ce-198a-48dd-8a66-dfb37357ac52-etc-kubernetes\") pod \"tuned-pjzmc\" (UID: \"efb461ce-198a-48dd-8a66-dfb37357ac52\") " pod="openshift-cluster-node-tuning-operator/tuned-pjzmc" Jul 09 07:02:23.421930 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.420619 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/37954298-c5a4-4800-bf9f-8dcd45536e67-run-ovn\") pod \"ovnkube-node-8s5zf\" (UID: \"37954298-c5a4-4800-bf9f-8dcd45536e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:02:23.421930 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.420668 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/461328f2-e00a-4d14-ab04-7651d773f7ca-multus-daemon-config\") pod \"multus-lw46s\" (UID: \"461328f2-e00a-4d14-ab04-7651d773f7ca\") " pod="openshift-multus/multus-lw46s" Jul 09 07:02:23.421930 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:23.420675 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 07:02:23.421930 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.420704 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/37954298-c5a4-4800-bf9f-8dcd45536e67-env-overrides\") pod \"ovnkube-node-8s5zf\" (UID: \"37954298-c5a4-4800-bf9f-8dcd45536e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:02:23.421930 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:23.420745 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0-metrics-certs podName:b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0 nodeName:}" failed. No retries permitted until 2026-07-09 07:02:23.920708842 +0000 UTC m=+2.971786070 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0-metrics-certs") pod "network-metrics-daemon-9nm6n" (UID: "b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 07:02:23.421930 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.420771 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjwm8\" (UniqueName: \"kubernetes.io/projected/9a204676-b17e-4b4b-aeed-d3c3e611f9e3-kube-api-access-jjwm8\") pod \"node-resolver-tk76r\" (UID: \"9a204676-b17e-4b4b-aeed-d3c3e611f9e3\") " pod="openshift-dns/node-resolver-tk76r" Jul 09 07:02:23.421930 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.420788 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djfzj\" (UniqueName: \"kubernetes.io/projected/b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0-kube-api-access-djfzj\") pod \"network-metrics-daemon-9nm6n\" (UID: \"b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0\") " pod="openshift-multus/network-metrics-daemon-9nm6n" Jul 09 07:02:23.421930 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.421034 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/37954298-c5a4-4800-bf9f-8dcd45536e67-env-overrides\") pod \"ovnkube-node-8s5zf\" (UID: \"37954298-c5a4-4800-bf9f-8dcd45536e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:02:23.422910 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.422892 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/37954298-c5a4-4800-bf9f-8dcd45536e67-ovn-node-metrics-cert\") pod \"ovnkube-node-8s5zf\" (UID: \"37954298-c5a4-4800-bf9f-8dcd45536e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:02:23.424539 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:23.424525 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 07:02:23.424585 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:23.424543 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 07:02:23.424585 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:23.424553 2564 projected.go:196] Error preparing data for projected volume kube-api-access-lxpzb for pod openshift-network-diagnostics/network-check-target-4899b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 07:02:23.424662 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:23.424609 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc-kube-api-access-lxpzb podName:67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc nodeName:}" failed. No retries permitted until 2026-07-09 07:02:23.924596821 +0000 UTC m=+2.975674034 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-lxpzb" (UniqueName: "kubernetes.io/projected/67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc-kube-api-access-lxpzb") pod "network-check-target-4899b" (UID: "67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 07:02:23.426304 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.426285 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kmsr\" (UniqueName: \"kubernetes.io/projected/7bac4227-dd99-4984-aa2e-efeaa7a04755-kube-api-access-4kmsr\") pod \"node-ca-gxqds\" (UID: \"7bac4227-dd99-4984-aa2e-efeaa7a04755\") " pod="openshift-image-registry/node-ca-gxqds" Jul 09 07:02:23.428883 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.428863 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djfzj\" (UniqueName: \"kubernetes.io/projected/b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0-kube-api-access-djfzj\") pod \"network-metrics-daemon-9nm6n\" (UID: \"b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0\") " pod="openshift-multus/network-metrics-daemon-9nm6n" Jul 09 07:02:23.429603 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.429585 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjwm8\" (UniqueName: \"kubernetes.io/projected/9a204676-b17e-4b4b-aeed-d3c3e611f9e3-kube-api-access-jjwm8\") pod \"node-resolver-tk76r\" (UID: \"9a204676-b17e-4b4b-aeed-d3c3e611f9e3\") " pod="openshift-dns/node-resolver-tk76r" Jul 09 07:02:23.429891 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.429875 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnxd8\" (UniqueName: \"kubernetes.io/projected/37954298-c5a4-4800-bf9f-8dcd45536e67-kube-api-access-wnxd8\") pod \"ovnkube-node-8s5zf\" (UID: \"37954298-c5a4-4800-bf9f-8dcd45536e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:02:23.462197 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.462142 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-0.ec2.internal" event={"ID":"19c650d7b90f9cfab0129bb7ed79b977","Type":"ContainerStarted","Data":"dbc7727a229877d9749fb10db7b77befa6c33e77f041156c4fe274338ba6ffd8"} Jul 09 07:02:23.463359 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.463334 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-0.ec2.internal" event={"ID":"a0854266c70379aa8ce321ab152e516e","Type":"ContainerStarted","Data":"66538b7a53c7777582854c6488e41f38611609a3341ce6cd53db19bc6589c1a0"} Jul 09 07:02:23.521176 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.521086 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/461328f2-e00a-4d14-ab04-7651d773f7ca-host-run-netns\") pod \"multus-lw46s\" (UID: \"461328f2-e00a-4d14-ab04-7651d773f7ca\") " pod="openshift-multus/multus-lw46s" Jul 09 07:02:23.521176 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.521139 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbvqd\" (UniqueName: \"kubernetes.io/projected/14b4eb3e-db26-4cd1-84b1-97f13b474c2e-kube-api-access-kbvqd\") pod \"iptables-alerter-d77xt\" (UID: \"14b4eb3e-db26-4cd1-84b1-97f13b474c2e\") " pod="openshift-network-operator/iptables-alerter-d77xt" Jul 09 07:02:23.521176 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.521157 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/461328f2-e00a-4d14-ab04-7651d773f7ca-host-run-netns\") pod \"multus-lw46s\" (UID: \"461328f2-e00a-4d14-ab04-7651d773f7ca\") " pod="openshift-multus/multus-lw46s" Jul 09 07:02:23.521514 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.521167 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/43cb1cf4-40f4-477d-bc2c-6ef608e242a0-etc-selinux\") pod \"aws-ebs-csi-driver-node-jnd85\" (UID: \"43cb1cf4-40f4-477d-bc2c-6ef608e242a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnd85" Jul 09 07:02:23.521514 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.521223 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a318fe5c-c8c8-4fff-b429-a7f4aa33b8c0-konnectivity-ca\") pod \"konnectivity-agent-fcljc\" (UID: \"a318fe5c-c8c8-4fff-b429-a7f4aa33b8c0\") " pod="kube-system/konnectivity-agent-fcljc" Jul 09 07:02:23.521514 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.521244 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8e9a1d6e-b7b9-494e-90f7-367f51bcb2aa-os-release\") pod \"multus-additional-cni-plugins-46w6z\" (UID: \"8e9a1d6e-b7b9-494e-90f7-367f51bcb2aa\") " pod="openshift-multus/multus-additional-cni-plugins-46w6z" Jul 09 07:02:23.521514 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.521270 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/461328f2-e00a-4d14-ab04-7651d773f7ca-os-release\") pod \"multus-lw46s\" (UID: \"461328f2-e00a-4d14-ab04-7651d773f7ca\") " pod="openshift-multus/multus-lw46s" Jul 09 07:02:23.521514 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.521289 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/43cb1cf4-40f4-477d-bc2c-6ef608e242a0-etc-selinux\") pod \"aws-ebs-csi-driver-node-jnd85\" (UID: \"43cb1cf4-40f4-477d-bc2c-6ef608e242a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnd85" Jul 09 07:02:23.521514 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.521312 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/43cb1cf4-40f4-477d-bc2c-6ef608e242a0-device-dir\") pod \"aws-ebs-csi-driver-node-jnd85\" (UID: \"43cb1cf4-40f4-477d-bc2c-6ef608e242a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnd85" Jul 09 07:02:23.521514 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.521337 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a318fe5c-c8c8-4fff-b429-a7f4aa33b8c0-agent-certs\") pod \"konnectivity-agent-fcljc\" (UID: \"a318fe5c-c8c8-4fff-b429-a7f4aa33b8c0\") " pod="kube-system/konnectivity-agent-fcljc" Jul 09 07:02:23.521514 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.521365 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/efb461ce-198a-48dd-8a66-dfb37357ac52-etc-sysctl-conf\") pod \"tuned-pjzmc\" (UID: \"efb461ce-198a-48dd-8a66-dfb37357ac52\") " pod="openshift-cluster-node-tuning-operator/tuned-pjzmc" Jul 09 07:02:23.521514 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.521372 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/461328f2-e00a-4d14-ab04-7651d773f7ca-os-release\") pod \"multus-lw46s\" (UID: \"461328f2-e00a-4d14-ab04-7651d773f7ca\") " pod="openshift-multus/multus-lw46s" Jul 09 07:02:23.521514 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.521388 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/461328f2-e00a-4d14-ab04-7651d773f7ca-host-run-k8s-cni-cncf-io\") pod \"multus-lw46s\" (UID: \"461328f2-e00a-4d14-ab04-7651d773f7ca\") " pod="openshift-multus/multus-lw46s" Jul 09 07:02:23.521514 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.521390 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8e9a1d6e-b7b9-494e-90f7-367f51bcb2aa-os-release\") pod \"multus-additional-cni-plugins-46w6z\" (UID: \"8e9a1d6e-b7b9-494e-90f7-367f51bcb2aa\") " pod="openshift-multus/multus-additional-cni-plugins-46w6z" Jul 09 07:02:23.521514 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.521422 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/14b4eb3e-db26-4cd1-84b1-97f13b474c2e-host-slash\") pod \"iptables-alerter-d77xt\" (UID: \"14b4eb3e-db26-4cd1-84b1-97f13b474c2e\") " pod="openshift-network-operator/iptables-alerter-d77xt" Jul 09 07:02:23.521514 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.521429 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/461328f2-e00a-4d14-ab04-7651d773f7ca-host-run-k8s-cni-cncf-io\") pod \"multus-lw46s\" (UID: \"461328f2-e00a-4d14-ab04-7651d773f7ca\") " pod="openshift-multus/multus-lw46s" Jul 09 07:02:23.521514 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.521450 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/efb461ce-198a-48dd-8a66-dfb37357ac52-etc-sysconfig\") pod \"tuned-pjzmc\" (UID: \"efb461ce-198a-48dd-8a66-dfb37357ac52\") " pod="openshift-cluster-node-tuning-operator/tuned-pjzmc" Jul 09 07:02:23.521514 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.521476 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/461328f2-e00a-4d14-ab04-7651d773f7ca-system-cni-dir\") pod \"multus-lw46s\" (UID: \"461328f2-e00a-4d14-ab04-7651d773f7ca\") " pod="openshift-multus/multus-lw46s" Jul 09 07:02:23.521514 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.521498 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/461328f2-e00a-4d14-ab04-7651d773f7ca-cnibin\") pod \"multus-lw46s\" (UID: \"461328f2-e00a-4d14-ab04-7651d773f7ca\") " pod="openshift-multus/multus-lw46s" Jul 09 07:02:23.521514 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.521500 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/14b4eb3e-db26-4cd1-84b1-97f13b474c2e-host-slash\") pod \"iptables-alerter-d77xt\" (UID: \"14b4eb3e-db26-4cd1-84b1-97f13b474c2e\") " pod="openshift-network-operator/iptables-alerter-d77xt" Jul 09 07:02:23.522443 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.521547 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/461328f2-e00a-4d14-ab04-7651d773f7ca-host-run-multus-certs\") pod \"multus-lw46s\" (UID: \"461328f2-e00a-4d14-ab04-7651d773f7ca\") " pod="openshift-multus/multus-lw46s" Jul 09 07:02:23.522443 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.521570 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/efb461ce-198a-48dd-8a66-dfb37357ac52-etc-sysctl-conf\") pod \"tuned-pjzmc\" (UID: \"efb461ce-198a-48dd-8a66-dfb37357ac52\") " pod="openshift-cluster-node-tuning-operator/tuned-pjzmc" Jul 09 07:02:23.522443 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.521572 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/461328f2-e00a-4d14-ab04-7651d773f7ca-system-cni-dir\") pod \"multus-lw46s\" (UID: \"461328f2-e00a-4d14-ab04-7651d773f7ca\") " pod="openshift-multus/multus-lw46s" Jul 09 07:02:23.522443 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.521593 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgzj2\" (UniqueName: \"kubernetes.io/projected/461328f2-e00a-4d14-ab04-7651d773f7ca-kube-api-access-vgzj2\") pod \"multus-lw46s\" (UID: \"461328f2-e00a-4d14-ab04-7651d773f7ca\") " pod="openshift-multus/multus-lw46s" Jul 09 07:02:23.522443 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.521628 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/43cb1cf4-40f4-477d-bc2c-6ef608e242a0-device-dir\") pod \"aws-ebs-csi-driver-node-jnd85\" (UID: \"43cb1cf4-40f4-477d-bc2c-6ef608e242a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnd85" Jul 09 07:02:23.522443 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.521635 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8e9a1d6e-b7b9-494e-90f7-367f51bcb2aa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-46w6z\" (UID: \"8e9a1d6e-b7b9-494e-90f7-367f51bcb2aa\") " pod="openshift-multus/multus-additional-cni-plugins-46w6z" Jul 09 07:02:23.522443 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.521666 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/461328f2-e00a-4d14-ab04-7651d773f7ca-multus-conf-dir\") pod \"multus-lw46s\" (UID: \"461328f2-e00a-4d14-ab04-7651d773f7ca\") " pod="openshift-multus/multus-lw46s" Jul 09 07:02:23.522443 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.521678 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/efb461ce-198a-48dd-8a66-dfb37357ac52-etc-sysconfig\") pod \"tuned-pjzmc\" (UID: \"efb461ce-198a-48dd-8a66-dfb37357ac52\") " pod="openshift-cluster-node-tuning-operator/tuned-pjzmc" Jul 09 07:02:23.522443 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.521693 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/efb461ce-198a-48dd-8a66-dfb37357ac52-host\") pod \"tuned-pjzmc\" (UID: \"efb461ce-198a-48dd-8a66-dfb37357ac52\") " pod="openshift-cluster-node-tuning-operator/tuned-pjzmc" Jul 09 07:02:23.522443 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.521724 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/43cb1cf4-40f4-477d-bc2c-6ef608e242a0-socket-dir\") pod \"aws-ebs-csi-driver-node-jnd85\" (UID: \"43cb1cf4-40f4-477d-bc2c-6ef608e242a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnd85" Jul 09 07:02:23.522443 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.521730 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/461328f2-e00a-4d14-ab04-7651d773f7ca-cnibin\") pod \"multus-lw46s\" (UID: \"461328f2-e00a-4d14-ab04-7651d773f7ca\") " pod="openshift-multus/multus-lw46s" Jul 09 07:02:23.522443 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.521761 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/43cb1cf4-40f4-477d-bc2c-6ef608e242a0-sys-fs\") pod \"aws-ebs-csi-driver-node-jnd85\" (UID: \"43cb1cf4-40f4-477d-bc2c-6ef608e242a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnd85" Jul 09 07:02:23.522443 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.521792 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/efb461ce-198a-48dd-8a66-dfb37357ac52-tmp\") pod \"tuned-pjzmc\" (UID: \"efb461ce-198a-48dd-8a66-dfb37357ac52\") " pod="openshift-cluster-node-tuning-operator/tuned-pjzmc" Jul 09 07:02:23.522443 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.521803 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/461328f2-e00a-4d14-ab04-7651d773f7ca-multus-conf-dir\") pod \"multus-lw46s\" (UID: \"461328f2-e00a-4d14-ab04-7651d773f7ca\") " pod="openshift-multus/multus-lw46s" Jul 09 07:02:23.522443 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.521820 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8e9a1d6e-b7b9-494e-90f7-367f51bcb2aa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-46w6z\" (UID: \"8e9a1d6e-b7b9-494e-90f7-367f51bcb2aa\") " pod="openshift-multus/multus-additional-cni-plugins-46w6z" Jul 09 07:02:23.522443 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.521840 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/43cb1cf4-40f4-477d-bc2c-6ef608e242a0-sys-fs\") pod \"aws-ebs-csi-driver-node-jnd85\" (UID: \"43cb1cf4-40f4-477d-bc2c-6ef608e242a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnd85" Jul 09 07:02:23.522443 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.521843 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/43cb1cf4-40f4-477d-bc2c-6ef608e242a0-socket-dir\") pod \"aws-ebs-csi-driver-node-jnd85\" (UID: \"43cb1cf4-40f4-477d-bc2c-6ef608e242a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnd85" Jul 09 07:02:23.523301 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.521793 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/461328f2-e00a-4d14-ab04-7651d773f7ca-host-run-multus-certs\") pod \"multus-lw46s\" (UID: \"461328f2-e00a-4d14-ab04-7651d773f7ca\") " pod="openshift-multus/multus-lw46s" Jul 09 07:02:23.523301 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.521889 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/efb461ce-198a-48dd-8a66-dfb37357ac52-host\") pod \"tuned-pjzmc\" (UID: \"efb461ce-198a-48dd-8a66-dfb37357ac52\") " pod="openshift-cluster-node-tuning-operator/tuned-pjzmc" Jul 09 07:02:23.523301 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.521887 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986-original-pull-secret\") pod \"global-pull-secret-syncer-2kgs9\" (UID: \"2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986\") " pod="kube-system/global-pull-secret-syncer-2kgs9" Jul 09 07:02:23.523301 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:23.521968 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 07:02:23.523301 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.521983 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/461328f2-e00a-4d14-ab04-7651d773f7ca-etc-kubernetes\") pod \"multus-lw46s\" (UID: \"461328f2-e00a-4d14-ab04-7651d773f7ca\") " pod="openshift-multus/multus-lw46s" Jul 09 07:02:23.523301 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.521987 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a318fe5c-c8c8-4fff-b429-a7f4aa33b8c0-konnectivity-ca\") pod \"konnectivity-agent-fcljc\" (UID: \"a318fe5c-c8c8-4fff-b429-a7f4aa33b8c0\") " pod="kube-system/konnectivity-agent-fcljc" Jul 09 07:02:23.523301 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:23.522021 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986-original-pull-secret podName:2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986 nodeName:}" failed. No retries permitted until 2026-07-09 07:02:24.022003664 +0000 UTC m=+3.073080889 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986-original-pull-secret") pod "global-pull-secret-syncer-2kgs9" (UID: "2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986") : object "kube-system"/"original-pull-secret" not registered Jul 09 07:02:23.523301 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.522041 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8e9a1d6e-b7b9-494e-90f7-367f51bcb2aa-cni-binary-copy\") pod \"multus-additional-cni-plugins-46w6z\" (UID: \"8e9a1d6e-b7b9-494e-90f7-367f51bcb2aa\") " pod="openshift-multus/multus-additional-cni-plugins-46w6z" Jul 09 07:02:23.523301 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.522064 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/461328f2-e00a-4d14-ab04-7651d773f7ca-etc-kubernetes\") pod \"multus-lw46s\" (UID: \"461328f2-e00a-4d14-ab04-7651d773f7ca\") " pod="openshift-multus/multus-lw46s" Jul 09 07:02:23.523301 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.522069 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8e9a1d6e-b7b9-494e-90f7-367f51bcb2aa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-46w6z\" (UID: \"8e9a1d6e-b7b9-494e-90f7-367f51bcb2aa\") " pod="openshift-multus/multus-additional-cni-plugins-46w6z" Jul 09 07:02:23.523301 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.522226 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8e9a1d6e-b7b9-494e-90f7-367f51bcb2aa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-46w6z\" (UID: \"8e9a1d6e-b7b9-494e-90f7-367f51bcb2aa\") " pod="openshift-multus/multus-additional-cni-plugins-46w6z" Jul 09 07:02:23.523301 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.522259 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/461328f2-e00a-4d14-ab04-7651d773f7ca-multus-cni-dir\") pod \"multus-lw46s\" (UID: \"461328f2-e00a-4d14-ab04-7651d773f7ca\") " pod="openshift-multus/multus-lw46s" Jul 09 07:02:23.523301 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.522309 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/461328f2-e00a-4d14-ab04-7651d773f7ca-hostroot\") pod \"multus-lw46s\" (UID: \"461328f2-e00a-4d14-ab04-7651d773f7ca\") " pod="openshift-multus/multus-lw46s" Jul 09 07:02:23.523301 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.522339 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/43cb1cf4-40f4-477d-bc2c-6ef608e242a0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jnd85\" (UID: \"43cb1cf4-40f4-477d-bc2c-6ef608e242a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnd85" Jul 09 07:02:23.523301 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.522366 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzrnz\" (UniqueName: \"kubernetes.io/projected/43cb1cf4-40f4-477d-bc2c-6ef608e242a0-kube-api-access-tzrnz\") pod \"aws-ebs-csi-driver-node-jnd85\" (UID: \"43cb1cf4-40f4-477d-bc2c-6ef608e242a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnd85" Jul 09 07:02:23.523301 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.522395 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986-kubelet-config\") pod \"global-pull-secret-syncer-2kgs9\" (UID: \"2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986\") " pod="kube-system/global-pull-secret-syncer-2kgs9" Jul 09 07:02:23.523301 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.522421 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/efb461ce-198a-48dd-8a66-dfb37357ac52-etc-sysctl-d\") pod \"tuned-pjzmc\" (UID: \"efb461ce-198a-48dd-8a66-dfb37357ac52\") " pod="openshift-cluster-node-tuning-operator/tuned-pjzmc" Jul 09 07:02:23.524192 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.522443 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/461328f2-e00a-4d14-ab04-7651d773f7ca-multus-cni-dir\") pod \"multus-lw46s\" (UID: \"461328f2-e00a-4d14-ab04-7651d773f7ca\") " pod="openshift-multus/multus-lw46s" Jul 09 07:02:23.524192 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.522446 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/efb461ce-198a-48dd-8a66-dfb37357ac52-lib-modules\") pod \"tuned-pjzmc\" (UID: \"efb461ce-198a-48dd-8a66-dfb37357ac52\") " pod="openshift-cluster-node-tuning-operator/tuned-pjzmc" Jul 09 07:02:23.524192 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.522490 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjcgs\" (UniqueName: \"kubernetes.io/projected/8e9a1d6e-b7b9-494e-90f7-367f51bcb2aa-kube-api-access-gjcgs\") pod \"multus-additional-cni-plugins-46w6z\" (UID: \"8e9a1d6e-b7b9-494e-90f7-367f51bcb2aa\") " pod="openshift-multus/multus-additional-cni-plugins-46w6z" Jul 09 07:02:23.524192 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.522497 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8e9a1d6e-b7b9-494e-90f7-367f51bcb2aa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-46w6z\" (UID: \"8e9a1d6e-b7b9-494e-90f7-367f51bcb2aa\") " pod="openshift-multus/multus-additional-cni-plugins-46w6z" Jul 09 07:02:23.524192 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.522519 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986-dbus\") pod \"global-pull-secret-syncer-2kgs9\" (UID: \"2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986\") " pod="kube-system/global-pull-secret-syncer-2kgs9" Jul 09 07:02:23.524192 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.522549 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/efb461ce-198a-48dd-8a66-dfb37357ac52-etc-systemd\") pod \"tuned-pjzmc\" (UID: \"efb461ce-198a-48dd-8a66-dfb37357ac52\") " pod="openshift-cluster-node-tuning-operator/tuned-pjzmc" Jul 09 07:02:23.524192 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.522556 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8e9a1d6e-b7b9-494e-90f7-367f51bcb2aa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-46w6z\" (UID: \"8e9a1d6e-b7b9-494e-90f7-367f51bcb2aa\") " pod="openshift-multus/multus-additional-cni-plugins-46w6z" Jul 09 07:02:23.524192 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.522575 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e9a1d6e-b7b9-494e-90f7-367f51bcb2aa-system-cni-dir\") pod \"multus-additional-cni-plugins-46w6z\" (UID: \"8e9a1d6e-b7b9-494e-90f7-367f51bcb2aa\") " pod="openshift-multus/multus-additional-cni-plugins-46w6z" Jul 09 07:02:23.524192 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.522563 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/efb461ce-198a-48dd-8a66-dfb37357ac52-lib-modules\") pod \"tuned-pjzmc\" (UID: \"efb461ce-198a-48dd-8a66-dfb37357ac52\") " pod="openshift-cluster-node-tuning-operator/tuned-pjzmc" Jul 09 07:02:23.524192 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.522615 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986-kubelet-config\") pod \"global-pull-secret-syncer-2kgs9\" (UID: \"2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986\") " pod="kube-system/global-pull-secret-syncer-2kgs9" Jul 09 07:02:23.524192 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.522621 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/461328f2-e00a-4d14-ab04-7651d773f7ca-host-var-lib-kubelet\") pod \"multus-lw46s\" (UID: \"461328f2-e00a-4d14-ab04-7651d773f7ca\") " pod="openshift-multus/multus-lw46s" Jul 09 07:02:23.524192 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.522637 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/efb461ce-198a-48dd-8a66-dfb37357ac52-etc-systemd\") pod \"tuned-pjzmc\" (UID: \"efb461ce-198a-48dd-8a66-dfb37357ac52\") " pod="openshift-cluster-node-tuning-operator/tuned-pjzmc" Jul 09 07:02:23.524192 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.522658 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/efb461ce-198a-48dd-8a66-dfb37357ac52-run\") pod \"tuned-pjzmc\" (UID: \"efb461ce-198a-48dd-8a66-dfb37357ac52\") " pod="openshift-cluster-node-tuning-operator/tuned-pjzmc" Jul 09 07:02:23.524192 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.522668 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/461328f2-e00a-4d14-ab04-7651d773f7ca-host-var-lib-kubelet\") pod \"multus-lw46s\" (UID: \"461328f2-e00a-4d14-ab04-7651d773f7ca\") " pod="openshift-multus/multus-lw46s" Jul 09 07:02:23.524192 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.522673 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e9a1d6e-b7b9-494e-90f7-367f51bcb2aa-system-cni-dir\") pod \"multus-additional-cni-plugins-46w6z\" (UID: \"8e9a1d6e-b7b9-494e-90f7-367f51bcb2aa\") " pod="openshift-multus/multus-additional-cni-plugins-46w6z" Jul 09 07:02:23.524192 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.522684 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/efb461ce-198a-48dd-8a66-dfb37357ac52-etc-tuned\") pod \"tuned-pjzmc\" (UID: \"efb461ce-198a-48dd-8a66-dfb37357ac52\") " pod="openshift-cluster-node-tuning-operator/tuned-pjzmc" Jul 09 07:02:23.524192 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.522562 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8e9a1d6e-b7b9-494e-90f7-367f51bcb2aa-cni-binary-copy\") pod \"multus-additional-cni-plugins-46w6z\" (UID: \"8e9a1d6e-b7b9-494e-90f7-367f51bcb2aa\") " pod="openshift-multus/multus-additional-cni-plugins-46w6z" Jul 09 07:02:23.525001 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.522737 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/efb461ce-198a-48dd-8a66-dfb37357ac52-run\") pod \"tuned-pjzmc\" (UID: \"efb461ce-198a-48dd-8a66-dfb37357ac52\") " pod="openshift-cluster-node-tuning-operator/tuned-pjzmc" Jul 09 07:02:23.525001 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.522767 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8e9a1d6e-b7b9-494e-90f7-367f51bcb2aa-cnibin\") pod \"multus-additional-cni-plugins-46w6z\" (UID: \"8e9a1d6e-b7b9-494e-90f7-367f51bcb2aa\") " pod="openshift-multus/multus-additional-cni-plugins-46w6z" Jul 09 07:02:23.525001 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.522785 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/461328f2-e00a-4d14-ab04-7651d773f7ca-hostroot\") pod \"multus-lw46s\" (UID: \"461328f2-e00a-4d14-ab04-7651d773f7ca\") " pod="openshift-multus/multus-lw46s" Jul 09 07:02:23.525001 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.522798 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/efb461ce-198a-48dd-8a66-dfb37357ac52-etc-sysctl-d\") pod \"tuned-pjzmc\" (UID: \"efb461ce-198a-48dd-8a66-dfb37357ac52\") " pod="openshift-cluster-node-tuning-operator/tuned-pjzmc" Jul 09 07:02:23.525001 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.522816 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/461328f2-e00a-4d14-ab04-7651d773f7ca-host-var-lib-cni-bin\") pod \"multus-lw46s\" (UID: \"461328f2-e00a-4d14-ab04-7651d773f7ca\") " pod="openshift-multus/multus-lw46s" Jul 09 07:02:23.525001 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.522851 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8e9a1d6e-b7b9-494e-90f7-367f51bcb2aa-cnibin\") pod \"multus-additional-cni-plugins-46w6z\" (UID: \"8e9a1d6e-b7b9-494e-90f7-367f51bcb2aa\") " pod="openshift-multus/multus-additional-cni-plugins-46w6z" Jul 09 07:02:23.525001 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.522860 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/efb461ce-198a-48dd-8a66-dfb37357ac52-etc-modprobe-d\") pod \"tuned-pjzmc\" (UID: \"efb461ce-198a-48dd-8a66-dfb37357ac52\") " pod="openshift-cluster-node-tuning-operator/tuned-pjzmc" Jul 09 07:02:23.525001 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.522850 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/43cb1cf4-40f4-477d-bc2c-6ef608e242a0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jnd85\" (UID: \"43cb1cf4-40f4-477d-bc2c-6ef608e242a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnd85" Jul 09 07:02:23.525001 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.522890 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/efb461ce-198a-48dd-8a66-dfb37357ac52-etc-kubernetes\") pod \"tuned-pjzmc\" (UID: \"efb461ce-198a-48dd-8a66-dfb37357ac52\") " pod="openshift-cluster-node-tuning-operator/tuned-pjzmc" Jul 09 07:02:23.525001 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.522897 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/461328f2-e00a-4d14-ab04-7651d773f7ca-host-var-lib-cni-bin\") pod \"multus-lw46s\" (UID: \"461328f2-e00a-4d14-ab04-7651d773f7ca\") " pod="openshift-multus/multus-lw46s" Jul 09 07:02:23.525001 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.522924 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/461328f2-e00a-4d14-ab04-7651d773f7ca-multus-daemon-config\") pod \"multus-lw46s\" (UID: \"461328f2-e00a-4d14-ab04-7651d773f7ca\") " pod="openshift-multus/multus-lw46s" Jul 09 07:02:23.525001 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.522938 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/efb461ce-198a-48dd-8a66-dfb37357ac52-etc-kubernetes\") pod \"tuned-pjzmc\" (UID: \"efb461ce-198a-48dd-8a66-dfb37357ac52\") " pod="openshift-cluster-node-tuning-operator/tuned-pjzmc" Jul 09 07:02:23.525001 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.522956 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/efb461ce-198a-48dd-8a66-dfb37357ac52-sys\") pod \"tuned-pjzmc\" (UID: \"efb461ce-198a-48dd-8a66-dfb37357ac52\") " pod="openshift-cluster-node-tuning-operator/tuned-pjzmc" Jul 09 07:02:23.525001 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.522962 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986-dbus\") pod \"global-pull-secret-syncer-2kgs9\" (UID: \"2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986\") " pod="kube-system/global-pull-secret-syncer-2kgs9" Jul 09 07:02:23.525001 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.522983 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/461328f2-e00a-4d14-ab04-7651d773f7ca-host-var-lib-cni-multus\") pod \"multus-lw46s\" (UID: \"461328f2-e00a-4d14-ab04-7651d773f7ca\") " pod="openshift-multus/multus-lw46s" Jul 09 07:02:23.525001 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.522994 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/efb461ce-198a-48dd-8a66-dfb37357ac52-etc-modprobe-d\") pod \"tuned-pjzmc\" (UID: \"efb461ce-198a-48dd-8a66-dfb37357ac52\") " pod="openshift-cluster-node-tuning-operator/tuned-pjzmc" Jul 09 07:02:23.525001 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.523011 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/14b4eb3e-db26-4cd1-84b1-97f13b474c2e-iptables-alerter-script\") pod \"iptables-alerter-d77xt\" (UID: \"14b4eb3e-db26-4cd1-84b1-97f13b474c2e\") " pod="openshift-network-operator/iptables-alerter-d77xt" Jul 09 07:02:23.525001 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.523018 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/efb461ce-198a-48dd-8a66-dfb37357ac52-sys\") pod \"tuned-pjzmc\" (UID: \"efb461ce-198a-48dd-8a66-dfb37357ac52\") " pod="openshift-cluster-node-tuning-operator/tuned-pjzmc" Jul 09 07:02:23.525719 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.523051 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/43cb1cf4-40f4-477d-bc2c-6ef608e242a0-registration-dir\") pod \"aws-ebs-csi-driver-node-jnd85\" (UID: \"43cb1cf4-40f4-477d-bc2c-6ef608e242a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnd85" Jul 09 07:02:23.525719 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.523055 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/461328f2-e00a-4d14-ab04-7651d773f7ca-host-var-lib-cni-multus\") pod \"multus-lw46s\" (UID: \"461328f2-e00a-4d14-ab04-7651d773f7ca\") " pod="openshift-multus/multus-lw46s" Jul 09 07:02:23.525719 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.523082 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/efb461ce-198a-48dd-8a66-dfb37357ac52-var-lib-kubelet\") pod \"tuned-pjzmc\" (UID: \"efb461ce-198a-48dd-8a66-dfb37357ac52\") " pod="openshift-cluster-node-tuning-operator/tuned-pjzmc" Jul 09 07:02:23.525719 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.523108 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlqqs\" (UniqueName: \"kubernetes.io/projected/efb461ce-198a-48dd-8a66-dfb37357ac52-kube-api-access-nlqqs\") pod \"tuned-pjzmc\" (UID: \"efb461ce-198a-48dd-8a66-dfb37357ac52\") " pod="openshift-cluster-node-tuning-operator/tuned-pjzmc" Jul 09 07:02:23.525719 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.523133 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/461328f2-e00a-4d14-ab04-7651d773f7ca-cni-binary-copy\") pod \"multus-lw46s\" (UID: \"461328f2-e00a-4d14-ab04-7651d773f7ca\") " pod="openshift-multus/multus-lw46s" Jul 09 07:02:23.525719 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.523159 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/461328f2-e00a-4d14-ab04-7651d773f7ca-multus-socket-dir-parent\") pod \"multus-lw46s\" (UID: \"461328f2-e00a-4d14-ab04-7651d773f7ca\") " pod="openshift-multus/multus-lw46s" Jul 09 07:02:23.525719 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.523241 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/461328f2-e00a-4d14-ab04-7651d773f7ca-multus-socket-dir-parent\") pod \"multus-lw46s\" (UID: \"461328f2-e00a-4d14-ab04-7651d773f7ca\") " pod="openshift-multus/multus-lw46s" Jul 09 07:02:23.525719 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.523231 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/efb461ce-198a-48dd-8a66-dfb37357ac52-var-lib-kubelet\") pod \"tuned-pjzmc\" (UID: \"efb461ce-198a-48dd-8a66-dfb37357ac52\") " pod="openshift-cluster-node-tuning-operator/tuned-pjzmc" Jul 09 07:02:23.525719 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.523311 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/43cb1cf4-40f4-477d-bc2c-6ef608e242a0-registration-dir\") pod \"aws-ebs-csi-driver-node-jnd85\" (UID: \"43cb1cf4-40f4-477d-bc2c-6ef608e242a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnd85" Jul 09 07:02:23.525719 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.523538 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/461328f2-e00a-4d14-ab04-7651d773f7ca-multus-daemon-config\") pod \"multus-lw46s\" (UID: \"461328f2-e00a-4d14-ab04-7651d773f7ca\") " pod="openshift-multus/multus-lw46s" Jul 09 07:02:23.525719 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.523599 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/14b4eb3e-db26-4cd1-84b1-97f13b474c2e-iptables-alerter-script\") pod \"iptables-alerter-d77xt\" (UID: \"14b4eb3e-db26-4cd1-84b1-97f13b474c2e\") " pod="openshift-network-operator/iptables-alerter-d77xt" Jul 09 07:02:23.525719 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.523750 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/461328f2-e00a-4d14-ab04-7651d773f7ca-cni-binary-copy\") pod \"multus-lw46s\" (UID: \"461328f2-e00a-4d14-ab04-7651d773f7ca\") " pod="openshift-multus/multus-lw46s" Jul 09 07:02:23.525719 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.524482 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/efb461ce-198a-48dd-8a66-dfb37357ac52-tmp\") pod \"tuned-pjzmc\" (UID: \"efb461ce-198a-48dd-8a66-dfb37357ac52\") " pod="openshift-cluster-node-tuning-operator/tuned-pjzmc" Jul 09 07:02:23.525719 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.524594 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a318fe5c-c8c8-4fff-b429-a7f4aa33b8c0-agent-certs\") pod \"konnectivity-agent-fcljc\" (UID: \"a318fe5c-c8c8-4fff-b429-a7f4aa33b8c0\") " pod="kube-system/konnectivity-agent-fcljc" Jul 09 07:02:23.525719 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.525442 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/efb461ce-198a-48dd-8a66-dfb37357ac52-etc-tuned\") pod \"tuned-pjzmc\" (UID: \"efb461ce-198a-48dd-8a66-dfb37357ac52\") " pod="openshift-cluster-node-tuning-operator/tuned-pjzmc" Jul 09 07:02:23.537332 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.537306 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbvqd\" (UniqueName: \"kubernetes.io/projected/14b4eb3e-db26-4cd1-84b1-97f13b474c2e-kube-api-access-kbvqd\") pod \"iptables-alerter-d77xt\" (UID: \"14b4eb3e-db26-4cd1-84b1-97f13b474c2e\") " pod="openshift-network-operator/iptables-alerter-d77xt" Jul 09 07:02:23.538973 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.538943 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlqqs\" (UniqueName: \"kubernetes.io/projected/efb461ce-198a-48dd-8a66-dfb37357ac52-kube-api-access-nlqqs\") pod \"tuned-pjzmc\" (UID: \"efb461ce-198a-48dd-8a66-dfb37357ac52\") " pod="openshift-cluster-node-tuning-operator/tuned-pjzmc" Jul 09 07:02:23.540213 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.539707 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgzj2\" (UniqueName: \"kubernetes.io/projected/461328f2-e00a-4d14-ab04-7651d773f7ca-kube-api-access-vgzj2\") pod \"multus-lw46s\" (UID: \"461328f2-e00a-4d14-ab04-7651d773f7ca\") " pod="openshift-multus/multus-lw46s" Jul 09 07:02:23.541223 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.541187 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzrnz\" (UniqueName: \"kubernetes.io/projected/43cb1cf4-40f4-477d-bc2c-6ef608e242a0-kube-api-access-tzrnz\") pod \"aws-ebs-csi-driver-node-jnd85\" (UID: \"43cb1cf4-40f4-477d-bc2c-6ef608e242a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnd85" Jul 09 07:02:23.546685 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.546666 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjcgs\" (UniqueName: \"kubernetes.io/projected/8e9a1d6e-b7b9-494e-90f7-367f51bcb2aa-kube-api-access-gjcgs\") pod \"multus-additional-cni-plugins-46w6z\" (UID: \"8e9a1d6e-b7b9-494e-90f7-367f51bcb2aa\") " pod="openshift-multus/multus-additional-cni-plugins-46w6z" Jul 09 07:02:23.613360 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.613328 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tk76r" Jul 09 07:02:23.621532 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.621505 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gxqds" Jul 09 07:02:23.623481 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:23.623420 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a204676_b17e_4b4b_aeed_d3c3e611f9e3.slice/crio-9258cb89e7b7760b173e6b5f0e701090c4cbd95a8b9ecab6d32f79159a281f98 WatchSource:0}: Error finding container 9258cb89e7b7760b173e6b5f0e701090c4cbd95a8b9ecab6d32f79159a281f98: Status 404 returned error can't find the container with id 9258cb89e7b7760b173e6b5f0e701090c4cbd95a8b9ecab6d32f79159a281f98 Jul 09 07:02:23.628714 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.628541 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:02:23.631831 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:23.631808 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bac4227_dd99_4984_aa2e_efeaa7a04755.slice/crio-27f077e7c4a4e9f9dd92eaf7ef4ba6390e35247de69c77b3a66ee6829552bfe1 WatchSource:0}: Error finding container 27f077e7c4a4e9f9dd92eaf7ef4ba6390e35247de69c77b3a66ee6829552bfe1: Status 404 returned error can't find the container with id 27f077e7c4a4e9f9dd92eaf7ef4ba6390e35247de69c77b3a66ee6829552bfe1 Jul 09 07:02:23.633573 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.633549 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-d77xt" Jul 09 07:02:23.638199 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:23.638171 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37954298_c5a4_4800_bf9f_8dcd45536e67.slice/crio-9f773d26ca0130b08bf81ae07e9468d9d57185bd749597ea6d7d1e6881fb998f WatchSource:0}: Error finding container 9f773d26ca0130b08bf81ae07e9468d9d57185bd749597ea6d7d1e6881fb998f: Status 404 returned error can't find the container with id 9f773d26ca0130b08bf81ae07e9468d9d57185bd749597ea6d7d1e6881fb998f Jul 09 07:02:23.639232 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.639210 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnd85" Jul 09 07:02:23.643848 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:23.643826 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14b4eb3e_db26_4cd1_84b1_97f13b474c2e.slice/crio-1457552773e676906dcff4b5f2f6f5bb457193441623d8f2cc2cfcfe3e79a690 WatchSource:0}: Error finding container 1457552773e676906dcff4b5f2f6f5bb457193441623d8f2cc2cfcfe3e79a690: Status 404 returned error can't find the container with id 1457552773e676906dcff4b5f2f6f5bb457193441623d8f2cc2cfcfe3e79a690 Jul 09 07:02:23.645312 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.645291 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pjzmc" Jul 09 07:02:23.649725 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:23.649700 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43cb1cf4_40f4_477d_bc2c_6ef608e242a0.slice/crio-b4ddb2e38671f7148766a5b419423a371d0ed56d4b459b65c6c2b9929c7dc7fd WatchSource:0}: Error finding container b4ddb2e38671f7148766a5b419423a371d0ed56d4b459b65c6c2b9929c7dc7fd: Status 404 returned error can't find the container with id b4ddb2e38671f7148766a5b419423a371d0ed56d4b459b65c6c2b9929c7dc7fd Jul 09 07:02:23.651769 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.651618 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fcljc" Jul 09 07:02:23.654733 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:23.654713 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefb461ce_198a_48dd_8a66_dfb37357ac52.slice/crio-c1d1060abf7678f8e4b698fa15e61f5c320b3a67b64bb4a58bb35bb4b619ded0 WatchSource:0}: Error finding container c1d1060abf7678f8e4b698fa15e61f5c320b3a67b64bb4a58bb35bb4b619ded0: Status 404 returned error can't find the container with id c1d1060abf7678f8e4b698fa15e61f5c320b3a67b64bb4a58bb35bb4b619ded0 Jul 09 07:02:23.657179 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.657153 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-46w6z" Jul 09 07:02:23.660266 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:23.660241 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda318fe5c_c8c8_4fff_b429_a7f4aa33b8c0.slice/crio-87479d37323a0fda2dbfd6e55eb8ca3406257f664396eae8875da67aea8a801d WatchSource:0}: Error finding container 87479d37323a0fda2dbfd6e55eb8ca3406257f664396eae8875da67aea8a801d: Status 404 returned error can't find the container with id 87479d37323a0fda2dbfd6e55eb8ca3406257f664396eae8875da67aea8a801d Jul 09 07:02:23.662238 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.662208 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lw46s" Jul 09 07:02:23.666139 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:23.666113 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e9a1d6e_b7b9_494e_90f7_367f51bcb2aa.slice/crio-971c151c739fc8874bee192fa955311c78bace1daecd5ce039337b6ccfd684d0 WatchSource:0}: Error finding container 971c151c739fc8874bee192fa955311c78bace1daecd5ce039337b6ccfd684d0: Status 404 returned error can't find the container with id 971c151c739fc8874bee192fa955311c78bace1daecd5ce039337b6ccfd684d0 Jul 09 07:02:23.672114 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:02:23.672092 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod461328f2_e00a_4d14_ab04_7651d773f7ca.slice/crio-dce79d539de89250fd00cd8afd5ef906cb0da29495c06a39c2706316779c9179 WatchSource:0}: Error finding container dce79d539de89250fd00cd8afd5ef906cb0da29495c06a39c2706316779c9179: Status 404 returned error can't find the container with id dce79d539de89250fd00cd8afd5ef906cb0da29495c06a39c2706316779c9179 Jul 09 07:02:23.926474 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.926432 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxpzb\" (UniqueName: \"kubernetes.io/projected/67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc-kube-api-access-lxpzb\") pod \"network-check-target-4899b\" (UID: \"67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc\") " pod="openshift-network-diagnostics/network-check-target-4899b" Jul 09 07:02:23.926653 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:23.926513 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0-metrics-certs\") pod \"network-metrics-daemon-9nm6n\" (UID: \"b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0\") " pod="openshift-multus/network-metrics-daemon-9nm6n" Jul 09 07:02:23.926653 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:23.926629 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 07:02:23.926653 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:23.926635 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 07:02:23.926881 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:23.926660 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 07:02:23.926881 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:23.926675 2564 projected.go:196] Error preparing data for projected volume kube-api-access-lxpzb for pod openshift-network-diagnostics/network-check-target-4899b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 07:02:23.926881 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:23.926700 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0-metrics-certs podName:b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0 nodeName:}" failed. No retries permitted until 2026-07-09 07:02:24.926680741 +0000 UTC m=+3.977757971 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0-metrics-certs") pod "network-metrics-daemon-9nm6n" (UID: "b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 07:02:23.926881 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:23.926730 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc-kube-api-access-lxpzb podName:67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc nodeName:}" failed. No retries permitted until 2026-07-09 07:02:24.926712322 +0000 UTC m=+3.977789532 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-lxpzb" (UniqueName: "kubernetes.io/projected/67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc-kube-api-access-lxpzb") pod "network-check-target-4899b" (UID: "67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 07:02:24.027637 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:24.027574 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986-original-pull-secret\") pod \"global-pull-secret-syncer-2kgs9\" (UID: \"2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986\") " pod="kube-system/global-pull-secret-syncer-2kgs9" Jul 09 07:02:24.027820 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:24.027765 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 07:02:24.027877 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:24.027848 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986-original-pull-secret podName:2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986 nodeName:}" failed. No retries permitted until 2026-07-09 07:02:25.027826393 +0000 UTC m=+4.078903617 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986-original-pull-secret") pod "global-pull-secret-syncer-2kgs9" (UID: "2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986") : object "kube-system"/"original-pull-secret" not registered Jul 09 07:02:24.355273 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:24.355169 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 06:57:23 +0000 UTC" deadline="2028-02-16 10:17:16.571881491 +0000 UTC" Jul 09 07:02:24.355273 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:24.355223 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14091h14m52.216662137s" Jul 09 07:02:24.465975 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:24.465933 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lw46s" event={"ID":"461328f2-e00a-4d14-ab04-7651d773f7ca","Type":"ContainerStarted","Data":"dce79d539de89250fd00cd8afd5ef906cb0da29495c06a39c2706316779c9179"} Jul 09 07:02:24.467255 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:24.467222 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-46w6z" event={"ID":"8e9a1d6e-b7b9-494e-90f7-367f51bcb2aa","Type":"ContainerStarted","Data":"971c151c739fc8874bee192fa955311c78bace1daecd5ce039337b6ccfd684d0"} Jul 09 07:02:24.468568 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:24.468537 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fcljc" event={"ID":"a318fe5c-c8c8-4fff-b429-a7f4aa33b8c0","Type":"ContainerStarted","Data":"87479d37323a0fda2dbfd6e55eb8ca3406257f664396eae8875da67aea8a801d"} Jul 09 07:02:24.469862 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:24.469838 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" event={"ID":"37954298-c5a4-4800-bf9f-8dcd45536e67","Type":"ContainerStarted","Data":"9f773d26ca0130b08bf81ae07e9468d9d57185bd749597ea6d7d1e6881fb998f"} Jul 09 07:02:24.471064 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:24.471036 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pjzmc" event={"ID":"efb461ce-198a-48dd-8a66-dfb37357ac52","Type":"ContainerStarted","Data":"c1d1060abf7678f8e4b698fa15e61f5c320b3a67b64bb4a58bb35bb4b619ded0"} Jul 09 07:02:24.472045 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:24.472016 2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 07:02:24.472256 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:24.472222 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnd85" event={"ID":"43cb1cf4-40f4-477d-bc2c-6ef608e242a0","Type":"ContainerStarted","Data":"b4ddb2e38671f7148766a5b419423a371d0ed56d4b459b65c6c2b9929c7dc7fd"} Jul 09 07:02:24.473513 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:24.473485 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-d77xt" event={"ID":"14b4eb3e-db26-4cd1-84b1-97f13b474c2e","Type":"ContainerStarted","Data":"1457552773e676906dcff4b5f2f6f5bb457193441623d8f2cc2cfcfe3e79a690"} Jul 09 07:02:24.474646 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:24.474623 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gxqds" event={"ID":"7bac4227-dd99-4984-aa2e-efeaa7a04755","Type":"ContainerStarted","Data":"27f077e7c4a4e9f9dd92eaf7ef4ba6390e35247de69c77b3a66ee6829552bfe1"} Jul 09 07:02:24.475756 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:24.475738 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tk76r" event={"ID":"9a204676-b17e-4b4b-aeed-d3c3e611f9e3","Type":"ContainerStarted","Data":"9258cb89e7b7760b173e6b5f0e701090c4cbd95a8b9ecab6d32f79159a281f98"} Jul 09 07:02:24.935270 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:24.935203 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0-metrics-certs\") pod \"network-metrics-daemon-9nm6n\" (UID: \"b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0\") " pod="openshift-multus/network-metrics-daemon-9nm6n" Jul 09 07:02:24.935469 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:24.935280 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxpzb\" (UniqueName: \"kubernetes.io/projected/67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc-kube-api-access-lxpzb\") pod \"network-check-target-4899b\" (UID: \"67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc\") " pod="openshift-network-diagnostics/network-check-target-4899b" Jul 09 07:02:24.935563 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:24.935489 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 07:02:24.935563 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:24.935509 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 07:02:24.935563 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:24.935523 2564 projected.go:196] Error preparing data for projected volume kube-api-access-lxpzb for pod openshift-network-diagnostics/network-check-target-4899b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 07:02:24.935729 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:24.935587 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc-kube-api-access-lxpzb podName:67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc nodeName:}" failed. No retries permitted until 2026-07-09 07:02:26.935566449 +0000 UTC m=+5.986643674 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-lxpzb" (UniqueName: "kubernetes.io/projected/67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc-kube-api-access-lxpzb") pod "network-check-target-4899b" (UID: "67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 07:02:24.936034 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:24.936015 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 07:02:24.936109 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:24.936074 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0-metrics-certs podName:b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0 nodeName:}" failed. No retries permitted until 2026-07-09 07:02:26.936057508 +0000 UTC m=+5.987134718 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0-metrics-certs") pod "network-metrics-daemon-9nm6n" (UID: "b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 07:02:25.036622 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:25.036587 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986-original-pull-secret\") pod \"global-pull-secret-syncer-2kgs9\" (UID: \"2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986\") " pod="kube-system/global-pull-secret-syncer-2kgs9" Jul 09 07:02:25.036808 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:25.036788 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 07:02:25.036888 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:25.036866 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986-original-pull-secret podName:2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986 nodeName:}" failed. No retries permitted until 2026-07-09 07:02:27.036843069 +0000 UTC m=+6.087920295 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986-original-pull-secret") pod "global-pull-secret-syncer-2kgs9" (UID: "2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986") : object "kube-system"/"original-pull-secret" not registered Jul 09 07:02:25.356202 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:25.356098 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 06:57:23 +0000 UTC" deadline="2028-01-18 11:32:21.338332059 +0000 UTC" Jul 09 07:02:25.356202 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:25.356144 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13396h29m55.982193552s" Jul 09 07:02:25.459684 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:25.459649 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9nm6n" Jul 09 07:02:25.459865 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:25.459803 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9nm6n" podUID="b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0" Jul 09 07:02:25.460257 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:25.460233 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4899b" Jul 09 07:02:25.460389 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:25.460364 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4899b" podUID="67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc" Jul 09 07:02:25.460458 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:25.460451 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kgs9" Jul 09 07:02:25.460547 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:25.460528 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kgs9" podUID="2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986" Jul 09 07:02:26.497404 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:26.496972 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-0.ec2.internal" event={"ID":"a0854266c70379aa8ce321ab152e516e","Type":"ContainerStarted","Data":"980baf529a1e67e12701ecef0c55f9e3fa50dc6cce8a7f26ffbd7659a5de0767"} Jul 09 07:02:26.517974 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:26.517911 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-0.ec2.internal" podStartSLOduration=3.517888576 podStartE2EDuration="3.517888576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 07:02:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 07:02:26.51712096 +0000 UTC m=+5.568198194" watchObservedRunningTime="2026-07-09 07:02:26.517888576 +0000 UTC m=+5.568965808" Jul 09 07:02:26.959148 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:26.959117 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxpzb\" (UniqueName: \"kubernetes.io/projected/67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc-kube-api-access-lxpzb\") pod \"network-check-target-4899b\" (UID: \"67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc\") " pod="openshift-network-diagnostics/network-check-target-4899b" Jul 09 07:02:26.959284 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:26.959219 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0-metrics-certs\") pod \"network-metrics-daemon-9nm6n\" (UID: \"b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0\") " pod="openshift-multus/network-metrics-daemon-9nm6n" Jul 09 07:02:26.959412 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:26.959394 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 07:02:26.959480 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:26.959470 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0-metrics-certs podName:b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0 nodeName:}" failed. No retries permitted until 2026-07-09 07:02:30.959448943 +0000 UTC m=+10.010526167 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0-metrics-certs") pod "network-metrics-daemon-9nm6n" (UID: "b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 07:02:26.959943 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:26.959918 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 07:02:26.960057 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:26.959948 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 07:02:26.960057 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:26.959961 2564 projected.go:196] Error preparing data for projected volume kube-api-access-lxpzb for pod openshift-network-diagnostics/network-check-target-4899b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 07:02:26.960057 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:26.960008 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc-kube-api-access-lxpzb podName:67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc nodeName:}" failed. No retries permitted until 2026-07-09 07:02:30.95999251 +0000 UTC m=+10.011069721 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-lxpzb" (UniqueName: "kubernetes.io/projected/67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc-kube-api-access-lxpzb") pod "network-check-target-4899b" (UID: "67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 07:02:27.061106 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:27.060573 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986-original-pull-secret\") pod \"global-pull-secret-syncer-2kgs9\" (UID: \"2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986\") " pod="kube-system/global-pull-secret-syncer-2kgs9" Jul 09 07:02:27.061106 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:27.060713 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 07:02:27.061106 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:27.060777 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986-original-pull-secret podName:2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986 nodeName:}" failed. No retries permitted until 2026-07-09 07:02:31.060758144 +0000 UTC m=+10.111835358 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986-original-pull-secret") pod "global-pull-secret-syncer-2kgs9" (UID: "2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986") : object "kube-system"/"original-pull-secret" not registered Jul 09 07:02:27.457444 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:27.457412 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9nm6n" Jul 09 07:02:27.457651 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:27.457567 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9nm6n" podUID="b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0" Jul 09 07:02:27.458358 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:27.457991 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4899b" Jul 09 07:02:27.458358 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:27.458089 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4899b" podUID="67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc" Jul 09 07:02:27.458358 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:27.458154 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kgs9" Jul 09 07:02:27.458358 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:27.458228 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kgs9" podUID="2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986" Jul 09 07:02:27.505085 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:27.505013 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-0.ec2.internal" event={"ID":"19c650d7b90f9cfab0129bb7ed79b977","Type":"ContainerStarted","Data":"258d86c306205f0c0b70db6a587cd13b62c86619731f6b6708b1167baa9ebd2f"} Jul 09 07:02:29.457709 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:29.457673 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9nm6n" Jul 09 07:02:29.458179 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:29.457684 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4899b" Jul 09 07:02:29.458179 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:29.457827 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9nm6n" podUID="b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0" Jul 09 07:02:29.458179 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:29.457916 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4899b" podUID="67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc" Jul 09 07:02:29.458179 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:29.457976 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kgs9" Jul 09 07:02:29.458179 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:29.458075 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kgs9" podUID="2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986" Jul 09 07:02:30.998792 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:30.998747 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0-metrics-certs\") pod \"network-metrics-daemon-9nm6n\" (UID: \"b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0\") " pod="openshift-multus/network-metrics-daemon-9nm6n" Jul 09 07:02:30.999298 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:30.998828 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxpzb\" (UniqueName: \"kubernetes.io/projected/67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc-kube-api-access-lxpzb\") pod \"network-check-target-4899b\" (UID: \"67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc\") " pod="openshift-network-diagnostics/network-check-target-4899b" Jul 09 07:02:30.999298 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:30.998943 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 07:02:30.999298 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:30.999010 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 07:02:30.999298 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:30.999025 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0-metrics-certs podName:b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0 nodeName:}" failed. No retries permitted until 2026-07-09 07:02:38.999002981 +0000 UTC m=+18.050080202 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0-metrics-certs") pod "network-metrics-daemon-9nm6n" (UID: "b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 07:02:30.999298 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:30.999031 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 07:02:30.999298 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:30.999045 2564 projected.go:196] Error preparing data for projected volume kube-api-access-lxpzb for pod openshift-network-diagnostics/network-check-target-4899b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 07:02:30.999298 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:30.999097 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc-kube-api-access-lxpzb podName:67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc nodeName:}" failed. No retries permitted until 2026-07-09 07:02:38.999078686 +0000 UTC m=+18.050155913 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-lxpzb" (UniqueName: "kubernetes.io/projected/67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc-kube-api-access-lxpzb") pod "network-check-target-4899b" (UID: "67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 07:02:31.100253 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:31.100212 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986-original-pull-secret\") pod \"global-pull-secret-syncer-2kgs9\" (UID: \"2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986\") " pod="kube-system/global-pull-secret-syncer-2kgs9" Jul 09 07:02:31.100458 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:31.100405 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 07:02:31.100520 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:31.100485 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986-original-pull-secret podName:2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986 nodeName:}" failed. No retries permitted until 2026-07-09 07:02:39.100462197 +0000 UTC m=+18.151539419 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986-original-pull-secret") pod "global-pull-secret-syncer-2kgs9" (UID: "2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986") : object "kube-system"/"original-pull-secret" not registered Jul 09 07:02:31.457756 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:31.457718 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9nm6n" Jul 09 07:02:31.457940 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:31.457854 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4899b" Jul 09 07:02:31.457940 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:31.457865 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9nm6n" podUID="b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0" Jul 09 07:02:31.458056 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:31.457950 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4899b" podUID="67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc" Jul 09 07:02:31.458056 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:31.457985 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kgs9" Jul 09 07:02:31.458155 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:31.458119 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kgs9" podUID="2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986" Jul 09 07:02:32.516336 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:32.516298 2564 generic.go:356] "Generic (PLEG): container finished" podID="19c650d7b90f9cfab0129bb7ed79b977" containerID="258d86c306205f0c0b70db6a587cd13b62c86619731f6b6708b1167baa9ebd2f" exitCode=0 Jul 09 07:02:32.516336 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:32.516334 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-0.ec2.internal" event={"ID":"19c650d7b90f9cfab0129bb7ed79b977","Type":"ContainerDied","Data":"258d86c306205f0c0b70db6a587cd13b62c86619731f6b6708b1167baa9ebd2f"} Jul 09 07:02:33.457402 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:33.457321 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9nm6n" Jul 09 07:02:33.457402 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:33.457370 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4899b" Jul 09 07:02:33.457631 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:33.457456 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kgs9" Jul 09 07:02:33.457631 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:33.457473 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9nm6n" podUID="b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0" Jul 09 07:02:33.457631 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:33.457575 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kgs9" podUID="2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986" Jul 09 07:02:33.457790 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:33.457647 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4899b" podUID="67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc" Jul 09 07:02:35.457114 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:35.456916 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9nm6n" Jul 09 07:02:35.457590 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:35.456916 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4899b" Jul 09 07:02:35.457590 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:35.457207 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9nm6n" podUID="b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0" Jul 09 07:02:35.457590 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:35.456993 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kgs9" Jul 09 07:02:35.457590 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:35.457277 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4899b" podUID="67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc" Jul 09 07:02:35.457590 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:35.457347 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kgs9" podUID="2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986" Jul 09 07:02:35.523204 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:35.523166 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fcljc" event={"ID":"a318fe5c-c8c8-4fff-b429-a7f4aa33b8c0","Type":"ContainerStarted","Data":"c86e6d3ebaee15b0b1eda954a83e3e8dbaecc410fb704e75ee6b7d3c60e65e39"} Jul 09 07:02:35.524816 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:35.524778 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pjzmc" event={"ID":"efb461ce-198a-48dd-8a66-dfb37357ac52","Type":"ContainerStarted","Data":"304fac0ce292df311f12639e682e2e1ad59566956e05d2dc47de761adfc36212"} Jul 09 07:02:35.526619 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:35.526586 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnd85" event={"ID":"43cb1cf4-40f4-477d-bc2c-6ef608e242a0","Type":"ContainerStarted","Data":"40b9844529c356bc7c7d964bc1951bce3098fdd533af19212736facadc502770"} Jul 09 07:02:35.528177 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:35.528147 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gxqds" event={"ID":"7bac4227-dd99-4984-aa2e-efeaa7a04755","Type":"ContainerStarted","Data":"0fd98f9838488a4d743d37bb4d4da0ef168f8c03487631989727bbe007daac7d"} Jul 09 07:02:35.530116 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:35.530098 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tk76r" event={"ID":"9a204676-b17e-4b4b-aeed-d3c3e611f9e3","Type":"ContainerStarted","Data":"ffb94d4d646ec360799a52c56417f1ad405613747617445c37c3214a91d37422"} Jul 09 07:02:35.532019 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:35.531995 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-0.ec2.internal" event={"ID":"19c650d7b90f9cfab0129bb7ed79b977","Type":"ContainerStarted","Data":"ef82486199a59349f248e3bad2eb7bf017a2e3b2f3e11f8998d80f0dcafaa822"} Jul 09 07:02:35.533783 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:35.533760 2564 generic.go:356] "Generic (PLEG): container finished" podID="8e9a1d6e-b7b9-494e-90f7-367f51bcb2aa" containerID="b14fdf426a3479676671211442e365770da1e6ba2a4aa730a31e1c2dcb612ebc" exitCode=0 Jul 09 07:02:35.533868 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:35.533818 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-46w6z" event={"ID":"8e9a1d6e-b7b9-494e-90f7-367f51bcb2aa","Type":"ContainerDied","Data":"b14fdf426a3479676671211442e365770da1e6ba2a4aa730a31e1c2dcb612ebc"} Jul 09 07:02:35.536251 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:35.536218 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-fcljc" podStartSLOduration=2.564832488 podStartE2EDuration="13.536207691s" totalImagesPullingTime="10.971375203s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 07:02:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 07:02:35.535859884 +0000 UTC m=+14.586937111" watchObservedRunningTime="2026-07-09 07:02:35.536207691 +0000 UTC m=+14.587284919" Jul 09 07:02:35.568446 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:35.568403 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-0.ec2.internal" podStartSLOduration=12.568389712 podStartE2EDuration="12.568389712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 07:02:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 07:02:35.568066645 +0000 UTC m=+14.619143873" watchObservedRunningTime="2026-07-09 07:02:35.568389712 +0000 UTC m=+14.619466939" Jul 09 07:02:35.577494 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:35.577472 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-fcljc" Jul 09 07:02:35.580609 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:35.580581 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-tk76r" podStartSLOduration=2.574086427 podStartE2EDuration="13.580571043s" totalImagesPullingTime="11.006484616s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 07:02:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 07:02:35.580209817 +0000 UTC m=+14.631287045" watchObservedRunningTime="2026-07-09 07:02:35.580571043 +0000 UTC m=+14.631648270" Jul 09 07:02:35.592276 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:35.592232 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-gxqds" podStartSLOduration=2.592653633 podStartE2EDuration="13.59222084s" totalImagesPullingTime="10.999567207s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 07:02:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 07:02:35.592124406 +0000 UTC m=+14.643201635" watchObservedRunningTime="2026-07-09 07:02:35.59222084 +0000 UTC m=+14.643298068" Jul 09 07:02:35.605479 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:35.605445 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-pjzmc" podStartSLOduration=2.628389216 podStartE2EDuration="13.605433872s" totalImagesPullingTime="10.977044656s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 07:02:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 07:02:35.605119862 +0000 UTC m=+14.656197092" watchObservedRunningTime="2026-07-09 07:02:35.605433872 +0000 UTC m=+14.656511099" Jul 09 07:02:36.538457 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:36.538380 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-d77xt" event={"ID":"14b4eb3e-db26-4cd1-84b1-97f13b474c2e","Type":"ContainerStarted","Data":"14e00a2a0d940f14db6c145a82aceee25a95d2289599f770f744ac7cf49ad01f"} Jul 09 07:02:36.552889 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:36.552649 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-d77xt" podStartSLOduration=3.414797994 podStartE2EDuration="14.552628583s" totalImagesPullingTime="11.137830589s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 07:02:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 07:02:36.551757038 +0000 UTC m=+15.602834266" watchObservedRunningTime="2026-07-09 07:02:36.552628583 +0000 UTC m=+15.603705813" Jul 09 07:02:37.457611 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:37.457577 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9nm6n" Jul 09 07:02:37.457789 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:37.457577 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4899b" Jul 09 07:02:37.457789 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:37.457727 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kgs9" Jul 09 07:02:37.457900 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:37.457706 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9nm6n" podUID="b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0" Jul 09 07:02:37.457900 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:37.457816 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4899b" podUID="67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc" Jul 09 07:02:37.458008 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:37.457941 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kgs9" podUID="2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986" Jul 09 07:02:39.061151 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:39.061114 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0-metrics-certs\") pod \"network-metrics-daemon-9nm6n\" (UID: \"b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0\") " pod="openshift-multus/network-metrics-daemon-9nm6n" Jul 09 07:02:39.061957 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:39.061184 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxpzb\" (UniqueName: \"kubernetes.io/projected/67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc-kube-api-access-lxpzb\") pod \"network-check-target-4899b\" (UID: \"67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc\") " pod="openshift-network-diagnostics/network-check-target-4899b" Jul 09 07:02:39.061957 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:39.061375 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 07:02:39.061957 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:39.061396 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 07:02:39.061957 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:39.061409 2564 projected.go:196] Error preparing data for projected volume kube-api-access-lxpzb for pod openshift-network-diagnostics/network-check-target-4899b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 07:02:39.061957 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:39.061413 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 07:02:39.061957 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:39.061475 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc-kube-api-access-lxpzb podName:67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc nodeName:}" failed. No retries permitted until 2026-07-09 07:02:55.061454589 +0000 UTC m=+34.112531805 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-lxpzb" (UniqueName: "kubernetes.io/projected/67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc-kube-api-access-lxpzb") pod "network-check-target-4899b" (UID: "67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 07:02:39.061957 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:39.061498 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0-metrics-certs podName:b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0 nodeName:}" failed. No retries permitted until 2026-07-09 07:02:55.06148557 +0000 UTC m=+34.112562808 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0-metrics-certs") pod "network-metrics-daemon-9nm6n" (UID: "b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 07:02:39.161570 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:39.161529 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986-original-pull-secret\") pod \"global-pull-secret-syncer-2kgs9\" (UID: \"2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986\") " pod="kube-system/global-pull-secret-syncer-2kgs9" Jul 09 07:02:39.161745 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:39.161687 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 07:02:39.161806 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:39.161770 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986-original-pull-secret podName:2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986 nodeName:}" failed. No retries permitted until 2026-07-09 07:02:55.161747842 +0000 UTC m=+34.212825066 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986-original-pull-secret") pod "global-pull-secret-syncer-2kgs9" (UID: "2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986") : object "kube-system"/"original-pull-secret" not registered Jul 09 07:02:39.457781 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:39.457712 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9nm6n" Jul 09 07:02:39.457781 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:39.457751 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kgs9" Jul 09 07:02:39.458025 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:39.457714 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4899b" Jul 09 07:02:39.458025 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:39.457851 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9nm6n" podUID="b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0" Jul 09 07:02:39.458025 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:39.457968 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4899b" podUID="67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc" Jul 09 07:02:39.458166 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:39.458055 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kgs9" podUID="2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986" Jul 09 07:02:39.594080 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:39.594050 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-fcljc" Jul 09 07:02:39.594870 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:39.594845 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-fcljc" Jul 09 07:02:40.548981 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:40.548952 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-fcljc" Jul 09 07:02:41.458567 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:41.458344 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9nm6n" Jul 09 07:02:41.458721 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:41.458423 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kgs9" Jul 09 07:02:41.458721 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:41.458650 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9nm6n" podUID="b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0" Jul 09 07:02:41.458839 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:41.458460 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4899b" Jul 09 07:02:41.458839 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:41.458748 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kgs9" podUID="2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986" Jul 09 07:02:41.458912 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:41.458852 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4899b" podUID="67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc" Jul 09 07:02:43.457502 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:43.457471 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kgs9" Jul 09 07:02:43.458046 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:43.457471 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9nm6n" Jul 09 07:02:43.458046 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:43.457568 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kgs9" podUID="2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986" Jul 09 07:02:43.458046 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:43.457471 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4899b" Jul 09 07:02:43.458046 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:43.457686 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9nm6n" podUID="b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0" Jul 09 07:02:43.458046 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:43.457713 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4899b" podUID="67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc" Jul 09 07:02:45.457558 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:45.457519 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9nm6n" Jul 09 07:02:45.458058 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:45.457519 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4899b" Jul 09 07:02:45.458058 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:45.457660 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9nm6n" podUID="b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0" Jul 09 07:02:45.458058 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:45.457531 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kgs9" Jul 09 07:02:45.458058 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:45.457738 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4899b" podUID="67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc" Jul 09 07:02:45.458058 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:45.457784 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kgs9" podUID="2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986" Jul 09 07:02:47.162238 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:47.162201 2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 09 07:02:47.384197 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:47.383975 2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-09T07:02:47.162221111Z","UUID":"55d61abc-742f-4ea3-b75a-33cb84eb5a1c","Handler":null,"Name":"","Endpoint":""} Jul 09 07:02:47.385327 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:47.385308 2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 09 07:02:47.385413 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:47.385348 2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 09 07:02:47.456791 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:47.456726 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4899b" Jul 09 07:02:47.456791 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:47.456732 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9nm6n" Jul 09 07:02:47.456916 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:47.456819 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4899b" podUID="67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc" Jul 09 07:02:47.456916 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:47.456894 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9nm6n" podUID="b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0" Jul 09 07:02:47.457014 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:47.456926 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kgs9" Jul 09 07:02:47.457066 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:47.457012 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kgs9" podUID="2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986" Jul 09 07:02:47.559011 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:47.558985 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lw46s" event={"ID":"461328f2-e00a-4d14-ab04-7651d773f7ca","Type":"ContainerStarted","Data":"575ce685636c7ddb33ca25e7f46acdb25a6ed94431e8050ca767206ac0e2d374"} Jul 09 07:02:47.560793 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:47.560769 2564 generic.go:356] "Generic (PLEG): container finished" podID="8e9a1d6e-b7b9-494e-90f7-367f51bcb2aa" containerID="36d98a2fe3ffe37cbdd21e316b1bc48ae34d3acee9aa6173a9816d0e38b0a446" exitCode=0 Jul 09 07:02:47.560879 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:47.560851 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-46w6z" event={"ID":"8e9a1d6e-b7b9-494e-90f7-367f51bcb2aa","Type":"ContainerDied","Data":"36d98a2fe3ffe37cbdd21e316b1bc48ae34d3acee9aa6173a9816d0e38b0a446"} Jul 09 07:02:47.563705 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:47.563685 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" event={"ID":"37954298-c5a4-4800-bf9f-8dcd45536e67","Type":"ContainerStarted","Data":"51bd4ec3380333799b0e809f967d226de711d0e88effc76eb10a69107dbbb436"} Jul 09 07:02:47.563798 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:47.563766 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" event={"ID":"37954298-c5a4-4800-bf9f-8dcd45536e67","Type":"ContainerStarted","Data":"afe4338888369b3a607c7e74a62dd130a3b2e56c3545f283c88f1938b808a5b1"} Jul 09 07:02:47.563863 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:47.563808 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" event={"ID":"37954298-c5a4-4800-bf9f-8dcd45536e67","Type":"ContainerStarted","Data":"ab5e13bede56e0a914ea3e5e482c82afa9ee4c827d3a7ce8a5fe3e91c7afa993"} Jul 09 07:02:47.563863 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:47.563843 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" event={"ID":"37954298-c5a4-4800-bf9f-8dcd45536e67","Type":"ContainerStarted","Data":"10d89e75498ce213bf975c906a8cc6d0d7284c690fcb258219931a212c6d330c"} Jul 09 07:02:47.563960 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:47.563870 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" event={"ID":"37954298-c5a4-4800-bf9f-8dcd45536e67","Type":"ContainerStarted","Data":"b6a67190d8b8f84ae690d4d9e21c045b0af8638142fa9a32c02c95ebb5e2d010"} Jul 09 07:02:47.566020 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:47.565998 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnd85" event={"ID":"43cb1cf4-40f4-477d-bc2c-6ef608e242a0","Type":"ContainerStarted","Data":"ca404aa23a9514321920e2317220730cd169dd8ceb4a3e47c7770dbddf4235bf"} Jul 09 07:02:47.574744 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:47.574710 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-lw46s" podStartSLOduration=2.23127978 podStartE2EDuration="25.574698693s" totalImagesPullingTime="23.343418913s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 07:02:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 07:02:47.574439372 +0000 UTC m=+26.625516602" watchObservedRunningTime="2026-07-09 07:02:47.574698693 +0000 UTC m=+26.625775920" Jul 09 07:02:48.571654 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:48.571610 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" event={"ID":"37954298-c5a4-4800-bf9f-8dcd45536e67","Type":"ContainerStarted","Data":"f93ae255eb717651da25c6db1d4529dd34752bfe33f1f2545c02de51bdaaf8b2"} Jul 09 07:02:49.457697 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:49.457672 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9nm6n" Jul 09 07:02:49.457994 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:49.457672 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4899b" Jul 09 07:02:49.457994 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:49.457777 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9nm6n" podUID="b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0" Jul 09 07:02:49.457994 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:49.457678 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kgs9" Jul 09 07:02:49.457994 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:49.457856 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4899b" podUID="67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc" Jul 09 07:02:49.458189 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:49.457901 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kgs9" podUID="2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986" Jul 09 07:02:49.577442 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:49.577411 2564 generic.go:356] "Generic (PLEG): container finished" podID="8e9a1d6e-b7b9-494e-90f7-367f51bcb2aa" containerID="2ad79133e4a3d7213019dda85671bf10e48c8890928bc31ec2ab5a843c6ff40b" exitCode=0 Jul 09 07:02:49.577951 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:49.577505 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-46w6z" event={"ID":"8e9a1d6e-b7b9-494e-90f7-367f51bcb2aa","Type":"ContainerDied","Data":"2ad79133e4a3d7213019dda85671bf10e48c8890928bc31ec2ab5a843c6ff40b"} Jul 09 07:02:49.579926 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:49.579905 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnd85" event={"ID":"43cb1cf4-40f4-477d-bc2c-6ef608e242a0","Type":"ContainerStarted","Data":"62c48db2630b560dcb678f3e12f145ce845182a7725aa9e7361be88daab0fc54"} Jul 09 07:02:49.616444 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:49.616406 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jnd85" podStartSLOduration=2.646128779 podStartE2EDuration="27.616393454s" totalImagesPullingTime="24.970264675s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 07:02:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 07:02:49.616301497 +0000 UTC m=+28.667378726" watchObservedRunningTime="2026-07-09 07:02:49.616393454 +0000 UTC m=+28.667470681" Jul 09 07:02:50.584626 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:50.584595 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" event={"ID":"37954298-c5a4-4800-bf9f-8dcd45536e67","Type":"ContainerStarted","Data":"739e0d43e4e1814cbd7e921f8b92260f3278daaa4639ba60186676d769cdee9f"} Jul 09 07:02:51.457638 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:51.457614 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9nm6n" Jul 09 07:02:51.457783 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:51.457701 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9nm6n" podUID="b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0" Jul 09 07:02:51.457783 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:51.457735 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kgs9" Jul 09 07:02:51.457783 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:51.457755 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4899b" Jul 09 07:02:51.457887 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:51.457821 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kgs9" podUID="2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986" Jul 09 07:02:51.457920 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:51.457890 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4899b" podUID="67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc" Jul 09 07:02:51.591520 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:51.591486 2564 generic.go:356] "Generic (PLEG): container finished" podID="8e9a1d6e-b7b9-494e-90f7-367f51bcb2aa" containerID="259084c23e87b805df1040cf3cf6de5a0fc9eaa49ebd0324c7ac011f1253c0fb" exitCode=0 Jul 09 07:02:51.591970 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:51.591534 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-46w6z" event={"ID":"8e9a1d6e-b7b9-494e-90f7-367f51bcb2aa","Type":"ContainerDied","Data":"259084c23e87b805df1040cf3cf6de5a0fc9eaa49ebd0324c7ac011f1253c0fb"} Jul 09 07:02:52.597469 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:52.597272 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" event={"ID":"37954298-c5a4-4800-bf9f-8dcd45536e67","Type":"ContainerStarted","Data":"7be6d94ce532d81f3d3b808c03676c05e5a6bac8dd7f2b3ffc37c3597eb8b34a"} Jul 09 07:02:52.597958 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:52.597510 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:02:52.597958 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:52.597550 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:02:52.597958 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:52.597582 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:02:52.612362 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:52.612340 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:02:52.612486 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:52.612438 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:02:52.623490 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:52.623455 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" podStartSLOduration=7.198801314 podStartE2EDuration="30.623440244s" totalImagesPullingTime="23.42463893s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 07:02:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 07:02:52.622351354 +0000 UTC m=+31.673428581" watchObservedRunningTime="2026-07-09 07:02:52.623440244 +0000 UTC m=+31.674517463" Jul 09 07:02:53.456971 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:53.456944 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4899b" Jul 09 07:02:53.456971 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:53.456960 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kgs9" Jul 09 07:02:53.457182 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:53.456950 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9nm6n" Jul 09 07:02:53.457182 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:53.457069 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4899b" podUID="67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc" Jul 09 07:02:53.457303 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:53.457214 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9nm6n" podUID="b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0" Jul 09 07:02:53.457352 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:53.457324 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kgs9" podUID="2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986" Jul 09 07:02:53.928551 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:53.928516 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4899b"] Jul 09 07:02:53.929023 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:53.928707 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4899b" Jul 09 07:02:53.929023 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:53.928818 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4899b" podUID="67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc" Jul 09 07:02:53.932048 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:53.931866 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9nm6n"] Jul 09 07:02:53.932048 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:53.931990 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9nm6n" Jul 09 07:02:53.932221 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:53.932102 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9nm6n" podUID="b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0" Jul 09 07:02:53.932538 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:53.932504 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2kgs9"] Jul 09 07:02:53.932622 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:53.932605 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kgs9" Jul 09 07:02:53.932715 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:53.932691 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kgs9" podUID="2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986" Jul 09 07:02:55.077051 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:55.077017 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxpzb\" (UniqueName: \"kubernetes.io/projected/67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc-kube-api-access-lxpzb\") pod \"network-check-target-4899b\" (UID: \"67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc\") " pod="openshift-network-diagnostics/network-check-target-4899b" Jul 09 07:02:55.077534 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:55.077091 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0-metrics-certs\") pod \"network-metrics-daemon-9nm6n\" (UID: \"b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0\") " pod="openshift-multus/network-metrics-daemon-9nm6n" Jul 09 07:02:55.077534 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:55.077190 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 07:02:55.077534 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:55.077197 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 07:02:55.077534 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:55.077222 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 07:02:55.077534 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:55.077234 2564 projected.go:196] Error preparing data for projected volume kube-api-access-lxpzb for pod openshift-network-diagnostics/network-check-target-4899b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 07:02:55.077534 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:55.077242 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0-metrics-certs podName:b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0 nodeName:}" failed. No retries permitted until 2026-07-09 07:03:27.077229379 +0000 UTC m=+66.128306585 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0-metrics-certs") pod "network-metrics-daemon-9nm6n" (UID: "b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 07:02:55.077534 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:55.077296 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc-kube-api-access-lxpzb podName:67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc nodeName:}" failed. No retries permitted until 2026-07-09 07:03:27.077282116 +0000 UTC m=+66.128359339 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-lxpzb" (UniqueName: "kubernetes.io/projected/67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc-kube-api-access-lxpzb") pod "network-check-target-4899b" (UID: "67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 07:02:55.177981 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:55.177948 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986-original-pull-secret\") pod \"global-pull-secret-syncer-2kgs9\" (UID: \"2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986\") " pod="kube-system/global-pull-secret-syncer-2kgs9" Jul 09 07:02:55.178118 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:55.178045 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 07:02:55.178118 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:55.178091 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986-original-pull-secret podName:2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986 nodeName:}" failed. No retries permitted until 2026-07-09 07:03:27.178079242 +0000 UTC m=+66.229156448 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986-original-pull-secret") pod "global-pull-secret-syncer-2kgs9" (UID: "2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986") : object "kube-system"/"original-pull-secret" not registered Jul 09 07:02:55.457365 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:55.457338 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4899b" Jul 09 07:02:55.457520 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:55.457338 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kgs9" Jul 09 07:02:55.457520 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:55.457446 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4899b" podUID="67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc" Jul 09 07:02:55.457594 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:55.457539 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kgs9" podUID="2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986" Jul 09 07:02:55.457594 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:55.457337 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9nm6n" Jul 09 07:02:55.457675 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:55.457644 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9nm6n" podUID="b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0" Jul 09 07:02:57.457505 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:57.457470 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4899b" Jul 09 07:02:57.457920 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:57.457601 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4899b" podUID="67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc" Jul 09 07:02:57.457920 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:57.457474 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9nm6n" Jul 09 07:02:57.457920 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:57.457470 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kgs9" Jul 09 07:02:57.457920 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:57.457709 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9nm6n" podUID="b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0" Jul 09 07:02:57.457920 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:57.457789 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kgs9" podUID="2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986" Jul 09 07:02:58.615615 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:58.615368 2564 generic.go:356] "Generic (PLEG): container finished" podID="8e9a1d6e-b7b9-494e-90f7-367f51bcb2aa" containerID="ae43dc5bbc4b19be7e4f9aa102aec322f3ae7977a0198ac9585b4a7b5142ad02" exitCode=0 Jul 09 07:02:58.615953 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:58.615448 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-46w6z" event={"ID":"8e9a1d6e-b7b9-494e-90f7-367f51bcb2aa","Type":"ContainerDied","Data":"ae43dc5bbc4b19be7e4f9aa102aec322f3ae7977a0198ac9585b4a7b5142ad02"} Jul 09 07:02:59.457259 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:59.457230 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4899b" Jul 09 07:02:59.457427 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:59.457230 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kgs9" Jul 09 07:02:59.457427 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:59.457351 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4899b" podUID="67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc" Jul 09 07:02:59.457427 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:59.457407 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kgs9" podUID="2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986" Jul 09 07:02:59.457427 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:59.457230 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9nm6n" Jul 09 07:02:59.457652 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:02:59.457488 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9nm6n" podUID="b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0" Jul 09 07:02:59.619774 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:59.619746 2564 generic.go:356] "Generic (PLEG): container finished" podID="8e9a1d6e-b7b9-494e-90f7-367f51bcb2aa" containerID="63944643ecdf9935f6ef1bfb830f0e2c5d5becb3ed14d679a8b72db654c9f9c9" exitCode=0 Jul 09 07:02:59.620109 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:02:59.619797 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-46w6z" event={"ID":"8e9a1d6e-b7b9-494e-90f7-367f51bcb2aa","Type":"ContainerDied","Data":"63944643ecdf9935f6ef1bfb830f0e2c5d5becb3ed14d679a8b72db654c9f9c9"} Jul 09 07:03:00.624460 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:00.624426 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-46w6z" event={"ID":"8e9a1d6e-b7b9-494e-90f7-367f51bcb2aa","Type":"ContainerStarted","Data":"0528d5cdfc06b3906a019ff92f5e99805a01cee33333270ceba5174804c94637"} Jul 09 07:03:00.646620 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:00.646579 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-46w6z" podStartSLOduration=7.013486957 podStartE2EDuration="38.646564317s" totalImagesPullingTime="31.63307736s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 07:02:22 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 07:03:00.644968779 +0000 UTC m=+39.696046009" watchObservedRunningTime="2026-07-09 07:03:00.646564317 +0000 UTC m=+39.697641546" Jul 09 07:03:00.810794 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:00.810718 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-0.ec2.internal" event="NodeReady" Jul 09 07:03:00.810962 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:00.810842 2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 09 07:03:00.858377 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:00.858353 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-sd74f"] Jul 09 07:03:00.872933 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:00.872911 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8wqzv"] Jul 09 07:03:00.873067 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:00.873045 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sd74f" Jul 09 07:03:00.875601 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:00.875578 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 09 07:03:00.875715 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:00.875609 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 09 07:03:00.876058 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:00.875935 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-k8zmf\"" Jul 09 07:03:00.885030 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:00.885013 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8wqzv"] Jul 09 07:03:00.885111 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:00.885038 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sd74f"] Jul 09 07:03:00.885146 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:00.885109 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8wqzv" Jul 09 07:03:00.887467 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:00.887447 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-kvbwn\"" Jul 09 07:03:00.887575 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:00.887503 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 09 07:03:00.887575 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:00.887522 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 09 07:03:00.887575 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:00.887523 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 09 07:03:01.023513 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:01.023490 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn6l7\" (UniqueName: \"kubernetes.io/projected/99def812-f7ca-4ac1-b0b9-227eb153b1bb-kube-api-access-gn6l7\") pod \"dns-default-sd74f\" (UID: \"99def812-f7ca-4ac1-b0b9-227eb153b1bb\") " pod="openshift-dns/dns-default-sd74f" Jul 09 07:03:01.023513 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:01.023521 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/99def812-f7ca-4ac1-b0b9-227eb153b1bb-config-volume\") pod \"dns-default-sd74f\" (UID: \"99def812-f7ca-4ac1-b0b9-227eb153b1bb\") " pod="openshift-dns/dns-default-sd74f" Jul 09 07:03:01.023716 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:01.023541 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b28zg\" (UniqueName: \"kubernetes.io/projected/e5d54ab9-973e-4ed3-b78d-d90373e6264a-kube-api-access-b28zg\") pod \"ingress-canary-8wqzv\" (UID: \"e5d54ab9-973e-4ed3-b78d-d90373e6264a\") " pod="openshift-ingress-canary/ingress-canary-8wqzv" Jul 09 07:03:01.023716 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:01.023602 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99def812-f7ca-4ac1-b0b9-227eb153b1bb-tmp-dir\") pod \"dns-default-sd74f\" (UID: \"99def812-f7ca-4ac1-b0b9-227eb153b1bb\") " pod="openshift-dns/dns-default-sd74f" Jul 09 07:03:01.023716 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:01.023651 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/99def812-f7ca-4ac1-b0b9-227eb153b1bb-metrics-tls\") pod \"dns-default-sd74f\" (UID: \"99def812-f7ca-4ac1-b0b9-227eb153b1bb\") " pod="openshift-dns/dns-default-sd74f" Jul 09 07:03:01.023716 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:01.023671 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e5d54ab9-973e-4ed3-b78d-d90373e6264a-cert\") pod \"ingress-canary-8wqzv\" (UID: \"e5d54ab9-973e-4ed3-b78d-d90373e6264a\") " pod="openshift-ingress-canary/ingress-canary-8wqzv" Jul 09 07:03:01.124176 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:01.124122 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn6l7\" (UniqueName: \"kubernetes.io/projected/99def812-f7ca-4ac1-b0b9-227eb153b1bb-kube-api-access-gn6l7\") pod \"dns-default-sd74f\" (UID: \"99def812-f7ca-4ac1-b0b9-227eb153b1bb\") " pod="openshift-dns/dns-default-sd74f" Jul 09 07:03:01.124176 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:01.124158 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/99def812-f7ca-4ac1-b0b9-227eb153b1bb-config-volume\") pod \"dns-default-sd74f\" (UID: \"99def812-f7ca-4ac1-b0b9-227eb153b1bb\") " pod="openshift-dns/dns-default-sd74f" Jul 09 07:03:01.124336 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:01.124183 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b28zg\" (UniqueName: \"kubernetes.io/projected/e5d54ab9-973e-4ed3-b78d-d90373e6264a-kube-api-access-b28zg\") pod \"ingress-canary-8wqzv\" (UID: \"e5d54ab9-973e-4ed3-b78d-d90373e6264a\") " pod="openshift-ingress-canary/ingress-canary-8wqzv" Jul 09 07:03:01.124336 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:01.124209 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99def812-f7ca-4ac1-b0b9-227eb153b1bb-tmp-dir\") pod \"dns-default-sd74f\" (UID: \"99def812-f7ca-4ac1-b0b9-227eb153b1bb\") " pod="openshift-dns/dns-default-sd74f" Jul 09 07:03:01.124336 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:01.124229 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/99def812-f7ca-4ac1-b0b9-227eb153b1bb-metrics-tls\") pod \"dns-default-sd74f\" (UID: \"99def812-f7ca-4ac1-b0b9-227eb153b1bb\") " pod="openshift-dns/dns-default-sd74f" Jul 09 07:03:01.124336 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:01.124245 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e5d54ab9-973e-4ed3-b78d-d90373e6264a-cert\") pod \"ingress-canary-8wqzv\" (UID: \"e5d54ab9-973e-4ed3-b78d-d90373e6264a\") " pod="openshift-ingress-canary/ingress-canary-8wqzv" Jul 09 07:03:01.124518 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:03:01.124351 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 07:03:01.124518 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:03:01.124416 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99def812-f7ca-4ac1-b0b9-227eb153b1bb-metrics-tls podName:99def812-f7ca-4ac1-b0b9-227eb153b1bb nodeName:}" failed. No retries permitted until 2026-07-09 07:03:01.624396268 +0000 UTC m=+40.675473494 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/99def812-f7ca-4ac1-b0b9-227eb153b1bb-metrics-tls") pod "dns-default-sd74f" (UID: "99def812-f7ca-4ac1-b0b9-227eb153b1bb") : secret "dns-default-metrics-tls" not found Jul 09 07:03:01.128404 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:01.128384 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e5d54ab9-973e-4ed3-b78d-d90373e6264a-cert\") pod \"ingress-canary-8wqzv\" (UID: \"e5d54ab9-973e-4ed3-b78d-d90373e6264a\") " pod="openshift-ingress-canary/ingress-canary-8wqzv" Jul 09 07:03:01.134799 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:01.134780 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b28zg\" (UniqueName: \"kubernetes.io/projected/e5d54ab9-973e-4ed3-b78d-d90373e6264a-kube-api-access-b28zg\") pod \"ingress-canary-8wqzv\" (UID: \"e5d54ab9-973e-4ed3-b78d-d90373e6264a\") " pod="openshift-ingress-canary/ingress-canary-8wqzv" Jul 09 07:03:01.135005 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:01.134993 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99def812-f7ca-4ac1-b0b9-227eb153b1bb-tmp-dir\") pod \"dns-default-sd74f\" (UID: \"99def812-f7ca-4ac1-b0b9-227eb153b1bb\") " pod="openshift-dns/dns-default-sd74f" Jul 09 07:03:01.135211 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:01.135196 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn6l7\" (UniqueName: \"kubernetes.io/projected/99def812-f7ca-4ac1-b0b9-227eb153b1bb-kube-api-access-gn6l7\") pod \"dns-default-sd74f\" (UID: \"99def812-f7ca-4ac1-b0b9-227eb153b1bb\") " pod="openshift-dns/dns-default-sd74f" Jul 09 07:03:01.135417 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:01.135396 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/99def812-f7ca-4ac1-b0b9-227eb153b1bb-config-volume\") pod \"dns-default-sd74f\" (UID: \"99def812-f7ca-4ac1-b0b9-227eb153b1bb\") " pod="openshift-dns/dns-default-sd74f" Jul 09 07:03:01.192780 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:01.192749 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8wqzv" Jul 09 07:03:01.368220 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:01.368043 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8wqzv"] Jul 09 07:03:01.371811 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:03:01.371781 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5d54ab9_973e_4ed3_b78d_d90373e6264a.slice/crio-795e9613b215edbe13457dddd191fcb69368d9b6ff1d70361e88349d3cac4646 WatchSource:0}: Error finding container 795e9613b215edbe13457dddd191fcb69368d9b6ff1d70361e88349d3cac4646: Status 404 returned error can't find the container with id 795e9613b215edbe13457dddd191fcb69368d9b6ff1d70361e88349d3cac4646 Jul 09 07:03:01.460324 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:01.458763 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4899b" Jul 09 07:03:01.460324 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:01.459221 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kgs9" Jul 09 07:03:01.460324 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:01.460150 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9nm6n" Jul 09 07:03:01.461865 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:01.461848 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 09 07:03:01.462690 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:01.462671 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6ktp4\"" Jul 09 07:03:01.462744 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:01.462679 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 09 07:03:01.462744 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:01.462715 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 09 07:03:01.462744 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:01.462733 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 09 07:03:01.462855 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:01.462793 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-h62pq\"" Jul 09 07:03:01.626895 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:01.626826 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/99def812-f7ca-4ac1-b0b9-227eb153b1bb-metrics-tls\") pod \"dns-default-sd74f\" (UID: \"99def812-f7ca-4ac1-b0b9-227eb153b1bb\") " pod="openshift-dns/dns-default-sd74f" Jul 09 07:03:01.627539 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:03:01.626941 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 07:03:01.627539 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:03:01.627001 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99def812-f7ca-4ac1-b0b9-227eb153b1bb-metrics-tls podName:99def812-f7ca-4ac1-b0b9-227eb153b1bb nodeName:}" failed. No retries permitted until 2026-07-09 07:03:02.62698456 +0000 UTC m=+41.678061779 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/99def812-f7ca-4ac1-b0b9-227eb153b1bb-metrics-tls") pod "dns-default-sd74f" (UID: "99def812-f7ca-4ac1-b0b9-227eb153b1bb") : secret "dns-default-metrics-tls" not found Jul 09 07:03:01.628237 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:01.628210 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8wqzv" event={"ID":"e5d54ab9-973e-4ed3-b78d-d90373e6264a","Type":"ContainerStarted","Data":"795e9613b215edbe13457dddd191fcb69368d9b6ff1d70361e88349d3cac4646"} Jul 09 07:03:02.632916 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:02.632885 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/99def812-f7ca-4ac1-b0b9-227eb153b1bb-metrics-tls\") pod \"dns-default-sd74f\" (UID: \"99def812-f7ca-4ac1-b0b9-227eb153b1bb\") " pod="openshift-dns/dns-default-sd74f" Jul 09 07:03:02.633282 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:03:02.632994 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 07:03:02.633282 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:03:02.633063 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99def812-f7ca-4ac1-b0b9-227eb153b1bb-metrics-tls podName:99def812-f7ca-4ac1-b0b9-227eb153b1bb nodeName:}" failed. No retries permitted until 2026-07-09 07:03:04.633043933 +0000 UTC m=+43.684121143 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/99def812-f7ca-4ac1-b0b9-227eb153b1bb-metrics-tls") pod "dns-default-sd74f" (UID: "99def812-f7ca-4ac1-b0b9-227eb153b1bb") : secret "dns-default-metrics-tls" not found Jul 09 07:03:04.635711 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:04.635677 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8wqzv" event={"ID":"e5d54ab9-973e-4ed3-b78d-d90373e6264a","Type":"ContainerStarted","Data":"6e9174df41dad86194d4f2777beea2128508f2fe97225bc0dea35efd90ecbdaf"} Jul 09 07:03:04.648483 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:04.648463 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/99def812-f7ca-4ac1-b0b9-227eb153b1bb-metrics-tls\") pod \"dns-default-sd74f\" (UID: \"99def812-f7ca-4ac1-b0b9-227eb153b1bb\") " pod="openshift-dns/dns-default-sd74f" Jul 09 07:03:04.648587 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:03:04.648571 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 07:03:04.648631 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:03:04.648620 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99def812-f7ca-4ac1-b0b9-227eb153b1bb-metrics-tls podName:99def812-f7ca-4ac1-b0b9-227eb153b1bb nodeName:}" failed. No retries permitted until 2026-07-09 07:03:08.648605849 +0000 UTC m=+47.699683055 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/99def812-f7ca-4ac1-b0b9-227eb153b1bb-metrics-tls") pod "dns-default-sd74f" (UID: "99def812-f7ca-4ac1-b0b9-227eb153b1bb") : secret "dns-default-metrics-tls" not found Jul 09 07:03:04.651423 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:04.651297 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8wqzv" podStartSLOduration=1.7916454179999999 podStartE2EDuration="4.651282173s" totalImagesPullingTime="2.859636755s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 07:03:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 07:03:04.651012883 +0000 UTC m=+43.702090111" watchObservedRunningTime="2026-07-09 07:03:04.651282173 +0000 UTC m=+43.702359392" Jul 09 07:03:08.676128 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:08.676095 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/99def812-f7ca-4ac1-b0b9-227eb153b1bb-metrics-tls\") pod \"dns-default-sd74f\" (UID: \"99def812-f7ca-4ac1-b0b9-227eb153b1bb\") " pod="openshift-dns/dns-default-sd74f" Jul 09 07:03:08.676500 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:03:08.676205 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 07:03:08.676500 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:03:08.676285 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99def812-f7ca-4ac1-b0b9-227eb153b1bb-metrics-tls podName:99def812-f7ca-4ac1-b0b9-227eb153b1bb nodeName:}" failed. No retries permitted until 2026-07-09 07:03:16.676251415 +0000 UTC m=+55.727328621 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/99def812-f7ca-4ac1-b0b9-227eb153b1bb-metrics-tls") pod "dns-default-sd74f" (UID: "99def812-f7ca-4ac1-b0b9-227eb153b1bb") : secret "dns-default-metrics-tls" not found Jul 09 07:03:12.393478 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:12.393441 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-7knwb"] Jul 09 07:03:12.404580 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:12.404557 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-7knwb" Jul 09 07:03:12.405586 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:12.405565 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-7knwb"] Jul 09 07:03:12.407356 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:12.407335 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 09 07:03:12.407449 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:12.407371 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 09 07:03:12.408081 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:12.408062 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-nmncg\"" Jul 09 07:03:12.601296 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:12.601237 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjncc\" (UniqueName: \"kubernetes.io/projected/5e203da9-5aba-4c20-a42c-45b5cf360ab3-kube-api-access-fjncc\") pod \"migrator-d57d58cc9-7knwb\" (UID: \"5e203da9-5aba-4c20-a42c-45b5cf360ab3\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-7knwb" Jul 09 07:03:12.701810 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:12.701739 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjncc\" (UniqueName: \"kubernetes.io/projected/5e203da9-5aba-4c20-a42c-45b5cf360ab3-kube-api-access-fjncc\") pod \"migrator-d57d58cc9-7knwb\" (UID: \"5e203da9-5aba-4c20-a42c-45b5cf360ab3\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-7knwb" Jul 09 07:03:12.710270 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:12.710245 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjncc\" (UniqueName: \"kubernetes.io/projected/5e203da9-5aba-4c20-a42c-45b5cf360ab3-kube-api-access-fjncc\") pod \"migrator-d57d58cc9-7knwb\" (UID: \"5e203da9-5aba-4c20-a42c-45b5cf360ab3\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-7knwb" Jul 09 07:03:12.714052 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:12.714033 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-7knwb" Jul 09 07:03:12.852847 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:12.852821 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-7knwb"] Jul 09 07:03:13.116745 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:13.116674 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-w9slp"] Jul 09 07:03:13.141040 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:13.141010 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-w9slp"] Jul 09 07:03:13.141164 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:13.141131 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-w9slp" Jul 09 07:03:13.143484 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:13.143459 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 09 07:03:13.144510 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:13.144489 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 09 07:03:13.144643 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:13.144517 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 09 07:03:13.144643 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:13.144523 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 09 07:03:13.144643 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:13.144621 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-rfq8r\"" Jul 09 07:03:13.306177 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:13.306139 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7d6b67d6-76c8-4abc-a51b-3d7e326051d7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w9slp\" (UID: \"7d6b67d6-76c8-4abc-a51b-3d7e326051d7\") " pod="openshift-insights/insights-runtime-extractor-w9slp" Jul 09 07:03:13.306177 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:13.306174 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbpck\" (UniqueName: \"kubernetes.io/projected/7d6b67d6-76c8-4abc-a51b-3d7e326051d7-kube-api-access-qbpck\") pod \"insights-runtime-extractor-w9slp\" (UID: \"7d6b67d6-76c8-4abc-a51b-3d7e326051d7\") " pod="openshift-insights/insights-runtime-extractor-w9slp" Jul 09 07:03:13.306392 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:13.306199 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7d6b67d6-76c8-4abc-a51b-3d7e326051d7-crio-socket\") pod \"insights-runtime-extractor-w9slp\" (UID: \"7d6b67d6-76c8-4abc-a51b-3d7e326051d7\") " pod="openshift-insights/insights-runtime-extractor-w9slp" Jul 09 07:03:13.306392 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:13.306341 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7d6b67d6-76c8-4abc-a51b-3d7e326051d7-data-volume\") pod \"insights-runtime-extractor-w9slp\" (UID: \"7d6b67d6-76c8-4abc-a51b-3d7e326051d7\") " pod="openshift-insights/insights-runtime-extractor-w9slp" Jul 09 07:03:13.306392 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:13.306370 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7d6b67d6-76c8-4abc-a51b-3d7e326051d7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w9slp\" (UID: \"7d6b67d6-76c8-4abc-a51b-3d7e326051d7\") " pod="openshift-insights/insights-runtime-extractor-w9slp" Jul 09 07:03:13.406680 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:13.406651 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7d6b67d6-76c8-4abc-a51b-3d7e326051d7-data-volume\") pod \"insights-runtime-extractor-w9slp\" (UID: \"7d6b67d6-76c8-4abc-a51b-3d7e326051d7\") " pod="openshift-insights/insights-runtime-extractor-w9slp" Jul 09 07:03:13.407101 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:13.406686 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7d6b67d6-76c8-4abc-a51b-3d7e326051d7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w9slp\" (UID: \"7d6b67d6-76c8-4abc-a51b-3d7e326051d7\") " pod="openshift-insights/insights-runtime-extractor-w9slp" Jul 09 07:03:13.407101 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:03:13.406758 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 07:03:13.407101 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:03:13.406824 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7d6b67d6-76c8-4abc-a51b-3d7e326051d7-kube-rbac-proxy-cm podName:7d6b67d6-76c8-4abc-a51b-3d7e326051d7 nodeName:}" failed. No retries permitted until 2026-07-09 07:03:13.906809196 +0000 UTC m=+52.957886401 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7d6b67d6-76c8-4abc-a51b-3d7e326051d7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w9slp" (UID: "7d6b67d6-76c8-4abc-a51b-3d7e326051d7") : configmap "kube-rbac-proxy" not found Jul 09 07:03:13.407101 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:13.406847 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7d6b67d6-76c8-4abc-a51b-3d7e326051d7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w9slp\" (UID: \"7d6b67d6-76c8-4abc-a51b-3d7e326051d7\") " pod="openshift-insights/insights-runtime-extractor-w9slp" Jul 09 07:03:13.407101 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:13.406878 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbpck\" (UniqueName: \"kubernetes.io/projected/7d6b67d6-76c8-4abc-a51b-3d7e326051d7-kube-api-access-qbpck\") pod \"insights-runtime-extractor-w9slp\" (UID: \"7d6b67d6-76c8-4abc-a51b-3d7e326051d7\") " pod="openshift-insights/insights-runtime-extractor-w9slp" Jul 09 07:03:13.407101 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:13.406907 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7d6b67d6-76c8-4abc-a51b-3d7e326051d7-crio-socket\") pod \"insights-runtime-extractor-w9slp\" (UID: \"7d6b67d6-76c8-4abc-a51b-3d7e326051d7\") " pod="openshift-insights/insights-runtime-extractor-w9slp" Jul 09 07:03:13.407101 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:13.406960 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7d6b67d6-76c8-4abc-a51b-3d7e326051d7-data-volume\") pod \"insights-runtime-extractor-w9slp\" (UID: \"7d6b67d6-76c8-4abc-a51b-3d7e326051d7\") " pod="openshift-insights/insights-runtime-extractor-w9slp" Jul 09 07:03:13.407101 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:13.406990 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7d6b67d6-76c8-4abc-a51b-3d7e326051d7-crio-socket\") pod \"insights-runtime-extractor-w9slp\" (UID: \"7d6b67d6-76c8-4abc-a51b-3d7e326051d7\") " pod="openshift-insights/insights-runtime-extractor-w9slp" Jul 09 07:03:13.407101 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:03:13.407013 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 07:03:13.407101 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:03:13.407081 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d6b67d6-76c8-4abc-a51b-3d7e326051d7-insights-runtime-extractor-tls podName:7d6b67d6-76c8-4abc-a51b-3d7e326051d7 nodeName:}" failed. No retries permitted until 2026-07-09 07:03:13.907065378 +0000 UTC m=+52.958142600 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7d6b67d6-76c8-4abc-a51b-3d7e326051d7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w9slp" (UID: "7d6b67d6-76c8-4abc-a51b-3d7e326051d7") : secret "insights-runtime-extractor-tls" not found Jul 09 07:03:13.414749 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:13.414724 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbpck\" (UniqueName: \"kubernetes.io/projected/7d6b67d6-76c8-4abc-a51b-3d7e326051d7-kube-api-access-qbpck\") pod \"insights-runtime-extractor-w9slp\" (UID: \"7d6b67d6-76c8-4abc-a51b-3d7e326051d7\") " pod="openshift-insights/insights-runtime-extractor-w9slp" Jul 09 07:03:13.657555 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:13.657473 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-7knwb" event={"ID":"5e203da9-5aba-4c20-a42c-45b5cf360ab3","Type":"ContainerStarted","Data":"df3c04bfa37d0e2a55428b44c0a92e89c967dbeb7ff3ddb5c067287829b1f473"} Jul 09 07:03:13.912514 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:13.912435 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7d6b67d6-76c8-4abc-a51b-3d7e326051d7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w9slp\" (UID: \"7d6b67d6-76c8-4abc-a51b-3d7e326051d7\") " pod="openshift-insights/insights-runtime-extractor-w9slp" Jul 09 07:03:13.912514 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:13.912498 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7d6b67d6-76c8-4abc-a51b-3d7e326051d7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w9slp\" (UID: \"7d6b67d6-76c8-4abc-a51b-3d7e326051d7\") " pod="openshift-insights/insights-runtime-extractor-w9slp" Jul 09 07:03:13.912732 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:03:13.912565 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 07:03:13.912732 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:03:13.912596 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 07:03:13.912732 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:03:13.912644 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7d6b67d6-76c8-4abc-a51b-3d7e326051d7-kube-rbac-proxy-cm podName:7d6b67d6-76c8-4abc-a51b-3d7e326051d7 nodeName:}" failed. No retries permitted until 2026-07-09 07:03:14.912624016 +0000 UTC m=+53.963701238 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7d6b67d6-76c8-4abc-a51b-3d7e326051d7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w9slp" (UID: "7d6b67d6-76c8-4abc-a51b-3d7e326051d7") : configmap "kube-rbac-proxy" not found Jul 09 07:03:13.912732 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:03:13.912661 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d6b67d6-76c8-4abc-a51b-3d7e326051d7-insights-runtime-extractor-tls podName:7d6b67d6-76c8-4abc-a51b-3d7e326051d7 nodeName:}" failed. No retries permitted until 2026-07-09 07:03:14.912653545 +0000 UTC m=+53.963730776 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7d6b67d6-76c8-4abc-a51b-3d7e326051d7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w9slp" (UID: "7d6b67d6-76c8-4abc-a51b-3d7e326051d7") : secret "insights-runtime-extractor-tls" not found Jul 09 07:03:14.261602 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:14.261503 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-sh4zk"] Jul 09 07:03:14.283398 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:14.283371 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-sh4zk"] Jul 09 07:03:14.283545 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:14.283465 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-sh4zk" Jul 09 07:03:14.286351 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:14.286325 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 09 07:03:14.286485 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:14.286450 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 09 07:03:14.286485 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:14.286452 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-5thdv\"" Jul 09 07:03:14.287451 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:14.287426 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 09 07:03:14.287451 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:14.287440 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 09 07:03:14.314043 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:14.314019 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59mm5\" (UniqueName: \"kubernetes.io/projected/8b2cf476-9da0-4b97-83b0-b4e673b90264-kube-api-access-59mm5\") pod \"service-ca-758f599c5-sh4zk\" (UID: \"8b2cf476-9da0-4b97-83b0-b4e673b90264\") " pod="openshift-service-ca/service-ca-758f599c5-sh4zk" Jul 09 07:03:14.314135 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:14.314064 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8b2cf476-9da0-4b97-83b0-b4e673b90264-signing-cabundle\") pod \"service-ca-758f599c5-sh4zk\" (UID: \"8b2cf476-9da0-4b97-83b0-b4e673b90264\") " pod="openshift-service-ca/service-ca-758f599c5-sh4zk" Jul 09 07:03:14.314135 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:14.314085 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8b2cf476-9da0-4b97-83b0-b4e673b90264-signing-key\") pod \"service-ca-758f599c5-sh4zk\" (UID: \"8b2cf476-9da0-4b97-83b0-b4e673b90264\") " pod="openshift-service-ca/service-ca-758f599c5-sh4zk" Jul 09 07:03:14.415055 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:14.415017 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59mm5\" (UniqueName: \"kubernetes.io/projected/8b2cf476-9da0-4b97-83b0-b4e673b90264-kube-api-access-59mm5\") pod \"service-ca-758f599c5-sh4zk\" (UID: \"8b2cf476-9da0-4b97-83b0-b4e673b90264\") " pod="openshift-service-ca/service-ca-758f599c5-sh4zk" Jul 09 07:03:14.415482 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:14.415169 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8b2cf476-9da0-4b97-83b0-b4e673b90264-signing-cabundle\") pod \"service-ca-758f599c5-sh4zk\" (UID: \"8b2cf476-9da0-4b97-83b0-b4e673b90264\") " pod="openshift-service-ca/service-ca-758f599c5-sh4zk" Jul 09 07:03:14.415482 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:14.415207 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8b2cf476-9da0-4b97-83b0-b4e673b90264-signing-key\") pod \"service-ca-758f599c5-sh4zk\" (UID: \"8b2cf476-9da0-4b97-83b0-b4e673b90264\") " pod="openshift-service-ca/service-ca-758f599c5-sh4zk" Jul 09 07:03:14.416002 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:14.415977 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8b2cf476-9da0-4b97-83b0-b4e673b90264-signing-cabundle\") pod \"service-ca-758f599c5-sh4zk\" (UID: \"8b2cf476-9da0-4b97-83b0-b4e673b90264\") " pod="openshift-service-ca/service-ca-758f599c5-sh4zk" Jul 09 07:03:14.417911 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:14.417891 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8b2cf476-9da0-4b97-83b0-b4e673b90264-signing-key\") pod \"service-ca-758f599c5-sh4zk\" (UID: \"8b2cf476-9da0-4b97-83b0-b4e673b90264\") " pod="openshift-service-ca/service-ca-758f599c5-sh4zk" Jul 09 07:03:14.422430 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:14.422414 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59mm5\" (UniqueName: \"kubernetes.io/projected/8b2cf476-9da0-4b97-83b0-b4e673b90264-kube-api-access-59mm5\") pod \"service-ca-758f599c5-sh4zk\" (UID: \"8b2cf476-9da0-4b97-83b0-b4e673b90264\") " pod="openshift-service-ca/service-ca-758f599c5-sh4zk" Jul 09 07:03:14.593130 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:14.593049 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-sh4zk" Jul 09 07:03:14.891945 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:14.891916 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-sh4zk"] Jul 09 07:03:14.895398 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:03:14.895359 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b2cf476_9da0_4b97_83b0_b4e673b90264.slice/crio-1bf266ec442ff443a42a367d1442e7fa41c0d78d89931fa01debeecb63672109 WatchSource:0}: Error finding container 1bf266ec442ff443a42a367d1442e7fa41c0d78d89931fa01debeecb63672109: Status 404 returned error can't find the container with id 1bf266ec442ff443a42a367d1442e7fa41c0d78d89931fa01debeecb63672109 Jul 09 07:03:14.918625 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:14.918599 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7d6b67d6-76c8-4abc-a51b-3d7e326051d7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w9slp\" (UID: \"7d6b67d6-76c8-4abc-a51b-3d7e326051d7\") " pod="openshift-insights/insights-runtime-extractor-w9slp" Jul 09 07:03:14.918707 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:14.918665 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7d6b67d6-76c8-4abc-a51b-3d7e326051d7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w9slp\" (UID: \"7d6b67d6-76c8-4abc-a51b-3d7e326051d7\") " pod="openshift-insights/insights-runtime-extractor-w9slp" Jul 09 07:03:14.918745 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:03:14.918703 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 07:03:14.918777 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:03:14.918759 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7d6b67d6-76c8-4abc-a51b-3d7e326051d7-kube-rbac-proxy-cm podName:7d6b67d6-76c8-4abc-a51b-3d7e326051d7 nodeName:}" failed. No retries permitted until 2026-07-09 07:03:16.918744807 +0000 UTC m=+55.969822016 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7d6b67d6-76c8-4abc-a51b-3d7e326051d7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w9slp" (UID: "7d6b67d6-76c8-4abc-a51b-3d7e326051d7") : configmap "kube-rbac-proxy" not found Jul 09 07:03:14.918816 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:03:14.918784 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 07:03:14.918851 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:03:14.918825 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d6b67d6-76c8-4abc-a51b-3d7e326051d7-insights-runtime-extractor-tls podName:7d6b67d6-76c8-4abc-a51b-3d7e326051d7 nodeName:}" failed. No retries permitted until 2026-07-09 07:03:16.918811397 +0000 UTC m=+55.969888622 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7d6b67d6-76c8-4abc-a51b-3d7e326051d7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w9slp" (UID: "7d6b67d6-76c8-4abc-a51b-3d7e326051d7") : secret "insights-runtime-extractor-tls" not found Jul 09 07:03:15.664255 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:15.664214 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-7knwb" event={"ID":"5e203da9-5aba-4c20-a42c-45b5cf360ab3","Type":"ContainerStarted","Data":"ccaa241a99cb07580c6b15f497c31261a86e1e07e3074ebdad733b1edb0e9115"} Jul 09 07:03:15.664809 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:15.664303 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-7knwb" event={"ID":"5e203da9-5aba-4c20-a42c-45b5cf360ab3","Type":"ContainerStarted","Data":"7143851d84dd09794983c52bb65cb671f0a56f5e0987575ffb4b4602c61feed8"} Jul 09 07:03:15.665416 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:15.665391 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-sh4zk" event={"ID":"8b2cf476-9da0-4b97-83b0-b4e673b90264","Type":"ContainerStarted","Data":"1bf266ec442ff443a42a367d1442e7fa41c0d78d89931fa01debeecb63672109"} Jul 09 07:03:15.681176 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:15.681135 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-7knwb" podStartSLOduration=1.508064848 podStartE2EDuration="3.681121312s" totalImagesPullingTime="2.173056464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 07:03:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 07:03:15.680008426 +0000 UTC m=+54.731085655" watchObservedRunningTime="2026-07-09 07:03:15.681121312 +0000 UTC m=+54.732198540" Jul 09 07:03:15.909074 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:15.909049 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tk76r_9a204676-b17e-4b4b-aeed-d3c3e611f9e3/dns-node-resolver/0.log" Jul 09 07:03:16.732114 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:16.732083 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/99def812-f7ca-4ac1-b0b9-227eb153b1bb-metrics-tls\") pod \"dns-default-sd74f\" (UID: \"99def812-f7ca-4ac1-b0b9-227eb153b1bb\") " pod="openshift-dns/dns-default-sd74f" Jul 09 07:03:16.732645 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:03:16.732239 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 07:03:16.732645 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:03:16.732337 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99def812-f7ca-4ac1-b0b9-227eb153b1bb-metrics-tls podName:99def812-f7ca-4ac1-b0b9-227eb153b1bb nodeName:}" failed. No retries permitted until 2026-07-09 07:03:32.732308711 +0000 UTC m=+71.783385941 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/99def812-f7ca-4ac1-b0b9-227eb153b1bb-metrics-tls") pod "dns-default-sd74f" (UID: "99def812-f7ca-4ac1-b0b9-227eb153b1bb") : secret "dns-default-metrics-tls" not found Jul 09 07:03:16.909681 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:16.909658 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gxqds_7bac4227-dd99-4984-aa2e-efeaa7a04755/node-ca/0.log" Jul 09 07:03:16.933167 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:16.933144 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7d6b67d6-76c8-4abc-a51b-3d7e326051d7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w9slp\" (UID: \"7d6b67d6-76c8-4abc-a51b-3d7e326051d7\") " pod="openshift-insights/insights-runtime-extractor-w9slp" Jul 09 07:03:16.933253 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:03:16.933215 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 07:03:16.933253 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:16.933237 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7d6b67d6-76c8-4abc-a51b-3d7e326051d7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w9slp\" (UID: \"7d6b67d6-76c8-4abc-a51b-3d7e326051d7\") " pod="openshift-insights/insights-runtime-extractor-w9slp" Jul 09 07:03:16.933339 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:03:16.933282 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7d6b67d6-76c8-4abc-a51b-3d7e326051d7-kube-rbac-proxy-cm podName:7d6b67d6-76c8-4abc-a51b-3d7e326051d7 nodeName:}" failed. No retries permitted until 2026-07-09 07:03:20.933250828 +0000 UTC m=+59.984328046 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7d6b67d6-76c8-4abc-a51b-3d7e326051d7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w9slp" (UID: "7d6b67d6-76c8-4abc-a51b-3d7e326051d7") : configmap "kube-rbac-proxy" not found Jul 09 07:03:16.933339 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:03:16.933333 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 07:03:16.933412 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:03:16.933373 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d6b67d6-76c8-4abc-a51b-3d7e326051d7-insights-runtime-extractor-tls podName:7d6b67d6-76c8-4abc-a51b-3d7e326051d7 nodeName:}" failed. No retries permitted until 2026-07-09 07:03:20.933361951 +0000 UTC m=+59.984439157 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7d6b67d6-76c8-4abc-a51b-3d7e326051d7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w9slp" (UID: "7d6b67d6-76c8-4abc-a51b-3d7e326051d7") : secret "insights-runtime-extractor-tls" not found Jul 09 07:03:17.510072 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:17.510032 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8wqzv_e5d54ab9-973e-4ed3-b78d-d90373e6264a/serve-healthcheck-canary/0.log" Jul 09 07:03:17.672358 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:17.672321 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-sh4zk" event={"ID":"8b2cf476-9da0-4b97-83b0-b4e673b90264","Type":"ContainerStarted","Data":"78bc15b3ec2f8966dcbe182270cc5de8e3fa11f6dacd5f2a9c8336979e097fce"} Jul 09 07:03:17.689499 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:17.689433 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-sh4zk" podStartSLOduration=1.702036533 podStartE2EDuration="3.689416639s" totalImagesPullingTime="1.987380106s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 07:03:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 07:03:17.689102604 +0000 UTC m=+56.740179832" watchObservedRunningTime="2026-07-09 07:03:17.689416639 +0000 UTC m=+56.740493873" Jul 09 07:03:18.111103 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:18.111069 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-7knwb_5e203da9-5aba-4c20-a42c-45b5cf360ab3/migrator/0.log" Jul 09 07:03:18.319984 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:18.319953 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-7knwb_5e203da9-5aba-4c20-a42c-45b5cf360ab3/graceful-termination/0.log" Jul 09 07:03:20.964686 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:20.964650 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7d6b67d6-76c8-4abc-a51b-3d7e326051d7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w9slp\" (UID: \"7d6b67d6-76c8-4abc-a51b-3d7e326051d7\") " pod="openshift-insights/insights-runtime-extractor-w9slp" Jul 09 07:03:20.965122 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:20.964712 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7d6b67d6-76c8-4abc-a51b-3d7e326051d7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w9slp\" (UID: \"7d6b67d6-76c8-4abc-a51b-3d7e326051d7\") " pod="openshift-insights/insights-runtime-extractor-w9slp" Jul 09 07:03:20.965122 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:03:20.964771 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 07:03:20.965122 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:03:20.964818 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7d6b67d6-76c8-4abc-a51b-3d7e326051d7-kube-rbac-proxy-cm podName:7d6b67d6-76c8-4abc-a51b-3d7e326051d7 nodeName:}" failed. No retries permitted until 2026-07-09 07:03:28.964805651 +0000 UTC m=+68.015882856 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7d6b67d6-76c8-4abc-a51b-3d7e326051d7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w9slp" (UID: "7d6b67d6-76c8-4abc-a51b-3d7e326051d7") : configmap "kube-rbac-proxy" not found Jul 09 07:03:20.965122 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:03:20.964824 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 07:03:20.965122 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:03:20.964908 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d6b67d6-76c8-4abc-a51b-3d7e326051d7-insights-runtime-extractor-tls podName:7d6b67d6-76c8-4abc-a51b-3d7e326051d7 nodeName:}" failed. No retries permitted until 2026-07-09 07:03:28.964890772 +0000 UTC m=+68.015967981 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7d6b67d6-76c8-4abc-a51b-3d7e326051d7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w9slp" (UID: "7d6b67d6-76c8-4abc-a51b-3d7e326051d7") : secret "insights-runtime-extractor-tls" not found Jul 09 07:03:24.613398 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:24.613371 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8s5zf" Jul 09 07:03:27.112043 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:27.112008 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0-metrics-certs\") pod \"network-metrics-daemon-9nm6n\" (UID: \"b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0\") " pod="openshift-multus/network-metrics-daemon-9nm6n" Jul 09 07:03:27.112492 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:27.112103 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxpzb\" (UniqueName: \"kubernetes.io/projected/67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc-kube-api-access-lxpzb\") pod \"network-check-target-4899b\" (UID: \"67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc\") " pod="openshift-network-diagnostics/network-check-target-4899b" Jul 09 07:03:27.114290 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:27.114251 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 09 07:03:27.115149 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:27.115135 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 09 07:03:27.124856 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:27.124808 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 09 07:03:27.125023 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:27.125005 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0-metrics-certs\") pod \"network-metrics-daemon-9nm6n\" (UID: \"b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0\") " pod="openshift-multus/network-metrics-daemon-9nm6n" Jul 09 07:03:27.135687 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:27.135661 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxpzb\" (UniqueName: \"kubernetes.io/projected/67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc-kube-api-access-lxpzb\") pod \"network-check-target-4899b\" (UID: \"67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc\") " pod="openshift-network-diagnostics/network-check-target-4899b" Jul 09 07:03:27.213041 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:27.213005 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986-original-pull-secret\") pod \"global-pull-secret-syncer-2kgs9\" (UID: \"2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986\") " pod="kube-system/global-pull-secret-syncer-2kgs9" Jul 09 07:03:27.215306 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:27.215288 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 09 07:03:27.226132 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:27.226108 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986-original-pull-secret\") pod \"global-pull-secret-syncer-2kgs9\" (UID: \"2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986\") " pod="kube-system/global-pull-secret-syncer-2kgs9" Jul 09 07:03:27.273246 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:27.273223 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6ktp4\"" Jul 09 07:03:27.276073 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:27.276061 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kgs9" Jul 09 07:03:27.281486 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:27.281473 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4899b" Jul 09 07:03:27.282336 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:27.282320 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-h62pq\"" Jul 09 07:03:27.291252 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:27.291236 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9nm6n" Jul 09 07:03:27.414784 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:27.414751 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4899b"] Jul 09 07:03:27.417860 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:03:27.417835 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67d1a6fb_d0ac_4336_9fa7_f0b1f12260cc.slice/crio-940b4f19ddb41c7fa56f2867f20d36c0a201c5d062f084f9aaceea7ef00f4864 WatchSource:0}: Error finding container 940b4f19ddb41c7fa56f2867f20d36c0a201c5d062f084f9aaceea7ef00f4864: Status 404 returned error can't find the container with id 940b4f19ddb41c7fa56f2867f20d36c0a201c5d062f084f9aaceea7ef00f4864 Jul 09 07:03:27.637093 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:27.636978 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9nm6n"] Jul 09 07:03:27.639712 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:27.639687 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2kgs9"] Jul 09 07:03:27.639829 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:03:27.639763 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7c8ddce_98ed_4d90_be63_ae97fcd9c6f0.slice/crio-fe09b841b2c2e1744594b9bdc1277146c9881ba0db6444d0113250ff62cda59f WatchSource:0}: Error finding container fe09b841b2c2e1744594b9bdc1277146c9881ba0db6444d0113250ff62cda59f: Status 404 returned error can't find the container with id fe09b841b2c2e1744594b9bdc1277146c9881ba0db6444d0113250ff62cda59f Jul 09 07:03:27.642326 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:03:27.642304 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c7d2ce1_e4bb_4b4f_b4ad_fffb23b6f986.slice/crio-2d9cde13a8591fbb035c66aed5f4cb02b99e33d92de014ac58128332d598ab49 WatchSource:0}: Error finding container 2d9cde13a8591fbb035c66aed5f4cb02b99e33d92de014ac58128332d598ab49: Status 404 returned error can't find the container with id 2d9cde13a8591fbb035c66aed5f4cb02b99e33d92de014ac58128332d598ab49 Jul 09 07:03:27.695281 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:27.695229 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9nm6n" event={"ID":"b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0","Type":"ContainerStarted","Data":"fe09b841b2c2e1744594b9bdc1277146c9881ba0db6444d0113250ff62cda59f"} Jul 09 07:03:27.696385 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:27.696360 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4899b" event={"ID":"67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc","Type":"ContainerStarted","Data":"940b4f19ddb41c7fa56f2867f20d36c0a201c5d062f084f9aaceea7ef00f4864"} Jul 09 07:03:27.697475 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:27.697458 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2kgs9" event={"ID":"2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986","Type":"ContainerStarted","Data":"2d9cde13a8591fbb035c66aed5f4cb02b99e33d92de014ac58128332d598ab49"} Jul 09 07:03:29.025995 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:29.025927 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7d6b67d6-76c8-4abc-a51b-3d7e326051d7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w9slp\" (UID: \"7d6b67d6-76c8-4abc-a51b-3d7e326051d7\") " pod="openshift-insights/insights-runtime-extractor-w9slp" Jul 09 07:03:29.026496 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:29.026027 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7d6b67d6-76c8-4abc-a51b-3d7e326051d7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w9slp\" (UID: \"7d6b67d6-76c8-4abc-a51b-3d7e326051d7\") " pod="openshift-insights/insights-runtime-extractor-w9slp" Jul 09 07:03:29.026713 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:29.026665 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7d6b67d6-76c8-4abc-a51b-3d7e326051d7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w9slp\" (UID: \"7d6b67d6-76c8-4abc-a51b-3d7e326051d7\") " pod="openshift-insights/insights-runtime-extractor-w9slp" Jul 09 07:03:29.028734 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:29.028715 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7d6b67d6-76c8-4abc-a51b-3d7e326051d7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w9slp\" (UID: \"7d6b67d6-76c8-4abc-a51b-3d7e326051d7\") " pod="openshift-insights/insights-runtime-extractor-w9slp" Jul 09 07:03:29.052026 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:29.052002 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-rfq8r\"" Jul 09 07:03:29.060033 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:29.060010 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-w9slp" Jul 09 07:03:29.286382 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:29.286242 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-w9slp"] Jul 09 07:03:29.290056 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:03:29.290025 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d6b67d6_76c8_4abc_a51b_3d7e326051d7.slice/crio-000ff8e7fc4a5d972e2d1baa89fbeeafa1edb79ae63f3161ccc7fcc0f0bc5da7 WatchSource:0}: Error finding container 000ff8e7fc4a5d972e2d1baa89fbeeafa1edb79ae63f3161ccc7fcc0f0bc5da7: Status 404 returned error can't find the container with id 000ff8e7fc4a5d972e2d1baa89fbeeafa1edb79ae63f3161ccc7fcc0f0bc5da7 Jul 09 07:03:29.707811 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:29.707773 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9nm6n" event={"ID":"b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0","Type":"ContainerStarted","Data":"65aeb4b910b562126ed57f953d5256763e0ff477dc3bbce0968a65b32b647670"} Jul 09 07:03:29.709564 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:29.709536 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w9slp" event={"ID":"7d6b67d6-76c8-4abc-a51b-3d7e326051d7","Type":"ContainerStarted","Data":"000ff8e7fc4a5d972e2d1baa89fbeeafa1edb79ae63f3161ccc7fcc0f0bc5da7"} Jul 09 07:03:30.717039 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:30.716947 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9nm6n" event={"ID":"b7c8ddce-98ed-4d90-be63-ae97fcd9c6f0","Type":"ContainerStarted","Data":"532dc94ab06f342386be2993a501f6e6bb2093c9cd3fb170555256934eefbcf8"} Jul 09 07:03:30.732340 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:30.731866 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-9nm6n" podStartSLOduration=66.867570511 podStartE2EDuration="1m8.731845604s" totalImagesPullingTime="1.864275093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 07:02:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 07:03:30.731142808 +0000 UTC m=+69.782220038" watchObservedRunningTime="2026-07-09 07:03:30.731845604 +0000 UTC m=+69.782922833" Jul 09 07:03:32.754755 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:32.754723 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/99def812-f7ca-4ac1-b0b9-227eb153b1bb-metrics-tls\") pod \"dns-default-sd74f\" (UID: \"99def812-f7ca-4ac1-b0b9-227eb153b1bb\") " pod="openshift-dns/dns-default-sd74f" Jul 09 07:03:32.757119 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:32.757092 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/99def812-f7ca-4ac1-b0b9-227eb153b1bb-metrics-tls\") pod \"dns-default-sd74f\" (UID: \"99def812-f7ca-4ac1-b0b9-227eb153b1bb\") " pod="openshift-dns/dns-default-sd74f" Jul 09 07:03:32.984918 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:32.984891 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-k8zmf\"" Jul 09 07:03:32.993924 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:32.993892 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sd74f" Jul 09 07:03:34.765537 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:34.765502 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-849b848c86-db8th"] Jul 09 07:03:34.767902 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:34.767872 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-849b848c86-db8th" Jul 09 07:03:34.770412 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:34.770336 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 09 07:03:34.770412 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:34.770347 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 09 07:03:34.770412 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:34.770345 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 09 07:03:34.770652 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:34.770351 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 09 07:03:34.771322 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:34.771285 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 09 07:03:34.771441 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:34.771368 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 09 07:03:34.771441 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:34.771407 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 09 07:03:34.781832 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:34.781811 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-849b848c86-db8th"] Jul 09 07:03:34.869994 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:34.869630 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cb20232-28c3-47f2-821d-17b31a4daaba-additional-service-ca\") pod \"cluster-proxy-proxy-agent-849b848c86-db8th\" (UID: \"9cb20232-28c3-47f2-821d-17b31a4daaba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-849b848c86-db8th" Jul 09 07:03:34.869994 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:34.869678 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/9cb20232-28c3-47f2-821d-17b31a4daaba-ca\") pod \"cluster-proxy-proxy-agent-849b848c86-db8th\" (UID: \"9cb20232-28c3-47f2-821d-17b31a4daaba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-849b848c86-db8th" Jul 09 07:03:34.869994 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:34.869729 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/9cb20232-28c3-47f2-821d-17b31a4daaba-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-849b848c86-db8th\" (UID: \"9cb20232-28c3-47f2-821d-17b31a4daaba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-849b848c86-db8th" Jul 09 07:03:34.869994 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:34.869776 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/9cb20232-28c3-47f2-821d-17b31a4daaba-hub\") pod \"cluster-proxy-proxy-agent-849b848c86-db8th\" (UID: \"9cb20232-28c3-47f2-821d-17b31a4daaba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-849b848c86-db8th" Jul 09 07:03:34.869994 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:34.869802 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9cb20232-28c3-47f2-821d-17b31a4daaba-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-849b848c86-db8th\" (UID: \"9cb20232-28c3-47f2-821d-17b31a4daaba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-849b848c86-db8th" Jul 09 07:03:34.869994 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:34.869850 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s9zx\" (UniqueName: \"kubernetes.io/projected/9cb20232-28c3-47f2-821d-17b31a4daaba-kube-api-access-5s9zx\") pod \"cluster-proxy-proxy-agent-849b848c86-db8th\" (UID: \"9cb20232-28c3-47f2-821d-17b31a4daaba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-849b848c86-db8th" Jul 09 07:03:34.954471 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:34.954382 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sd74f"] Jul 09 07:03:34.957815 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:03:34.957784 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99def812_f7ca_4ac1_b0b9_227eb153b1bb.slice/crio-c66579fbc04140151be52cdc8214005439f9e26a1508669c10b0cf25ce5c8bf4 WatchSource:0}: Error finding container c66579fbc04140151be52cdc8214005439f9e26a1508669c10b0cf25ce5c8bf4: Status 404 returned error can't find the container with id c66579fbc04140151be52cdc8214005439f9e26a1508669c10b0cf25ce5c8bf4 Jul 09 07:03:34.970486 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:34.970459 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s9zx\" (UniqueName: \"kubernetes.io/projected/9cb20232-28c3-47f2-821d-17b31a4daaba-kube-api-access-5s9zx\") pod \"cluster-proxy-proxy-agent-849b848c86-db8th\" (UID: \"9cb20232-28c3-47f2-821d-17b31a4daaba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-849b848c86-db8th" Jul 09 07:03:34.970592 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:34.970492 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cb20232-28c3-47f2-821d-17b31a4daaba-additional-service-ca\") pod \"cluster-proxy-proxy-agent-849b848c86-db8th\" (UID: \"9cb20232-28c3-47f2-821d-17b31a4daaba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-849b848c86-db8th" Jul 09 07:03:34.970592 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:34.970510 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/9cb20232-28c3-47f2-821d-17b31a4daaba-ca\") pod \"cluster-proxy-proxy-agent-849b848c86-db8th\" (UID: \"9cb20232-28c3-47f2-821d-17b31a4daaba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-849b848c86-db8th" Jul 09 07:03:34.970592 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:34.970541 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/9cb20232-28c3-47f2-821d-17b31a4daaba-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-849b848c86-db8th\" (UID: \"9cb20232-28c3-47f2-821d-17b31a4daaba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-849b848c86-db8th" Jul 09 07:03:34.970592 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:34.970566 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/9cb20232-28c3-47f2-821d-17b31a4daaba-hub\") pod \"cluster-proxy-proxy-agent-849b848c86-db8th\" (UID: \"9cb20232-28c3-47f2-821d-17b31a4daaba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-849b848c86-db8th" Jul 09 07:03:34.970592 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:34.970582 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9cb20232-28c3-47f2-821d-17b31a4daaba-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-849b848c86-db8th\" (UID: \"9cb20232-28c3-47f2-821d-17b31a4daaba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-849b848c86-db8th" Jul 09 07:03:34.971913 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:34.971864 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cb20232-28c3-47f2-821d-17b31a4daaba-additional-service-ca\") pod \"cluster-proxy-proxy-agent-849b848c86-db8th\" (UID: \"9cb20232-28c3-47f2-821d-17b31a4daaba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-849b848c86-db8th" Jul 09 07:03:34.973383 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:34.973333 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/9cb20232-28c3-47f2-821d-17b31a4daaba-ca\") pod \"cluster-proxy-proxy-agent-849b848c86-db8th\" (UID: \"9cb20232-28c3-47f2-821d-17b31a4daaba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-849b848c86-db8th" Jul 09 07:03:34.973667 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:34.973645 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9cb20232-28c3-47f2-821d-17b31a4daaba-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-849b848c86-db8th\" (UID: \"9cb20232-28c3-47f2-821d-17b31a4daaba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-849b848c86-db8th" Jul 09 07:03:34.974364 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:34.974330 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/9cb20232-28c3-47f2-821d-17b31a4daaba-hub\") pod \"cluster-proxy-proxy-agent-849b848c86-db8th\" (UID: \"9cb20232-28c3-47f2-821d-17b31a4daaba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-849b848c86-db8th" Jul 09 07:03:34.974547 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:34.974531 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/9cb20232-28c3-47f2-821d-17b31a4daaba-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-849b848c86-db8th\" (UID: \"9cb20232-28c3-47f2-821d-17b31a4daaba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-849b848c86-db8th" Jul 09 07:03:34.981464 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:34.981441 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s9zx\" (UniqueName: \"kubernetes.io/projected/9cb20232-28c3-47f2-821d-17b31a4daaba-kube-api-access-5s9zx\") pod \"cluster-proxy-proxy-agent-849b848c86-db8th\" (UID: \"9cb20232-28c3-47f2-821d-17b31a4daaba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-849b848c86-db8th" Jul 09 07:03:35.089634 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:35.089548 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-849b848c86-db8th" Jul 09 07:03:35.210569 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:35.210532 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-849b848c86-db8th"] Jul 09 07:03:35.213102 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:03:35.213079 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9cb20232_28c3_47f2_821d_17b31a4daaba.slice/crio-7bf37e65a267da5e513804b306772faa69fd6d76f9a8da9769323a5e2656be0f WatchSource:0}: Error finding container 7bf37e65a267da5e513804b306772faa69fd6d76f9a8da9769323a5e2656be0f: Status 404 returned error can't find the container with id 7bf37e65a267da5e513804b306772faa69fd6d76f9a8da9769323a5e2656be0f Jul 09 07:03:35.736409 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:35.736321 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4899b" event={"ID":"67d1a6fb-d0ac-4336-9fa7-f0b1f12260cc","Type":"ContainerStarted","Data":"3fad8bf19355ad24964ed972eaa5467686e2ea3d1d2821732ad2c855c136ca28"} Jul 09 07:03:35.736569 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:35.736547 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-4899b" Jul 09 07:03:35.738429 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:35.738397 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w9slp" event={"ID":"7d6b67d6-76c8-4abc-a51b-3d7e326051d7","Type":"ContainerStarted","Data":"57a2c88391f1aeb7d577aa6978b3fb78b301ee8b6d03c328e88cd8ae7617a800"} Jul 09 07:03:35.738566 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:35.738451 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w9slp" event={"ID":"7d6b67d6-76c8-4abc-a51b-3d7e326051d7","Type":"ContainerStarted","Data":"c217c7da2dff3390c842841d0003235ba38ae6ab797f82e40b5c17116d6a7349"} Jul 09 07:03:35.739706 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:35.739679 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sd74f" event={"ID":"99def812-f7ca-4ac1-b0b9-227eb153b1bb","Type":"ContainerStarted","Data":"c66579fbc04140151be52cdc8214005439f9e26a1508669c10b0cf25ce5c8bf4"} Jul 09 07:03:35.740864 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:35.740841 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-849b848c86-db8th" event={"ID":"9cb20232-28c3-47f2-821d-17b31a4daaba","Type":"ContainerStarted","Data":"7bf37e65a267da5e513804b306772faa69fd6d76f9a8da9769323a5e2656be0f"} Jul 09 07:03:35.741951 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:35.741931 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2kgs9" event={"ID":"2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986","Type":"ContainerStarted","Data":"2f4c7fc1194d6c18165afb2567adb029377d2ae4a7a00f964d740d7f02d8fb05"} Jul 09 07:03:35.750776 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:35.750731 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-4899b" podStartSLOduration=66.38420776 podStartE2EDuration="1m13.750719137s" totalImagesPullingTime="7.366511377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 07:02:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 07:03:35.750389616 +0000 UTC m=+74.801466845" watchObservedRunningTime="2026-07-09 07:03:35.750719137 +0000 UTC m=+74.801796343" Jul 09 07:03:35.762906 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:35.762868 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-2kgs9" podStartSLOduration=66.619162005 podStartE2EDuration="1m13.762852634s" totalImagesPullingTime="7.143690629s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 07:02:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 07:03:35.762789 +0000 UTC m=+74.813866230" watchObservedRunningTime="2026-07-09 07:03:35.762852634 +0000 UTC m=+74.813929840" Jul 09 07:03:40.764118 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:40.764078 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w9slp" event={"ID":"7d6b67d6-76c8-4abc-a51b-3d7e326051d7","Type":"ContainerStarted","Data":"dcae65d5a7334bc434735e4286de3c712590b397322596d1d701c57e3225874f"} Jul 09 07:03:40.765974 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:40.765951 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sd74f" event={"ID":"99def812-f7ca-4ac1-b0b9-227eb153b1bb","Type":"ContainerStarted","Data":"cb5e330854c002feab9c16fd7b2377dc071d421f751cf247e71432ef2fb86300"} Jul 09 07:03:40.766078 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:40.765992 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sd74f" event={"ID":"99def812-f7ca-4ac1-b0b9-227eb153b1bb","Type":"ContainerStarted","Data":"d5e1e38723c663332fdf5fbc66c2020ee9c2b4ec49898773e9722015219042a1"} Jul 09 07:03:40.766078 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:40.766041 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-sd74f" Jul 09 07:03:40.768009 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:40.767988 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-849b848c86-db8th" event={"ID":"9cb20232-28c3-47f2-821d-17b31a4daaba","Type":"ContainerStarted","Data":"ad7d9b90f71deb4c0372eb9987da55bb0016b903f5aa47736889318c03e58e71"} Jul 09 07:03:40.768108 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:40.768041 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-849b848c86-db8th" event={"ID":"9cb20232-28c3-47f2-821d-17b31a4daaba","Type":"ContainerStarted","Data":"a4ba560dc0673474f22a54f17510596a9a6bfc3ef83afa7fd2834091b938a42f"} Jul 09 07:03:40.768108 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:40.768054 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-849b848c86-db8th" event={"ID":"9cb20232-28c3-47f2-821d-17b31a4daaba","Type":"ContainerStarted","Data":"cf3b8b3075b230953810ef1dd4a0ec917c27e632c41c56f32f38ca231c3549cf"} Jul 09 07:03:40.782814 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:40.782773 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-w9slp" podStartSLOduration=17.006794921 podStartE2EDuration="27.782761126s" totalImagesPullingTime="10.775966205s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 07:03:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 07:03:40.780987592 +0000 UTC m=+79.832064819" watchObservedRunningTime="2026-07-09 07:03:40.782761126 +0000 UTC m=+79.833838373" Jul 09 07:03:40.797708 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:40.797668 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-849b848c86-db8th" podStartSLOduration=1.8068163149999998 podStartE2EDuration="6.797656299s" totalImagesPullingTime="4.990839984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 07:03:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 07:03:40.796791742 +0000 UTC m=+79.847868969" watchObservedRunningTime="2026-07-09 07:03:40.797656299 +0000 UTC m=+79.848733534" Jul 09 07:03:50.773182 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:50.773058 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-sd74f" Jul 09 07:03:50.791239 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:50.791194 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-sd74f" podStartSLOduration=45.558113773 podStartE2EDuration="50.791181344s" totalImagesPullingTime="5.233067571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 07:03:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 07:03:40.812091801 +0000 UTC m=+79.863169029" watchObservedRunningTime="2026-07-09 07:03:50.791181344 +0000 UTC m=+89.842258586" Jul 09 07:03:52.381023 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.380991 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-7q8rc"] Jul 09 07:03:52.387464 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.387441 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-ql4f7"] Jul 09 07:03:52.387590 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.387560 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7q8rc" Jul 09 07:03:52.390209 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.390161 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-d5tmz\"" Jul 09 07:03:52.390467 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.390447 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 09 07:03:52.390544 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.390451 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 09 07:03:52.391222 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.391181 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 09 07:03:52.391222 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.391198 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 09 07:03:52.391405 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.391232 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 09 07:03:52.391831 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.391808 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ql4f7" Jul 09 07:03:52.393365 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.393348 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-pnpsf"] Jul 09 07:03:52.394196 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.394175 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 09 07:03:52.394417 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.394402 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-8kjsj\"" Jul 09 07:03:52.394474 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.394451 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 09 07:03:52.394535 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.394405 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 09 07:03:52.397251 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.397234 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-7q8rc"] Jul 09 07:03:52.397373 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.397358 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-pnpsf" Jul 09 07:03:52.400008 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.399991 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 09 07:03:52.400197 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.400181 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 09 07:03:52.400406 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.400390 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 09 07:03:52.400619 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.400604 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-l55nm\"" Jul 09 07:03:52.410099 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.410072 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-pnpsf"] Jul 09 07:03:52.491427 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.491396 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e5b9b919-a410-4786-acf2-8e5b052e0ff3-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-pnpsf\" (UID: \"e5b9b919-a410-4786-acf2-8e5b052e0ff3\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-pnpsf" Jul 09 07:03:52.491603 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.491432 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9b494327-fac4-4296-8418-e71c08fbac25-root\") pod \"node-exporter-ql4f7\" (UID: \"9b494327-fac4-4296-8418-e71c08fbac25\") " pod="openshift-monitoring/node-exporter-ql4f7" Jul 09 07:03:52.491603 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.491459 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9b494327-fac4-4296-8418-e71c08fbac25-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ql4f7\" (UID: \"9b494327-fac4-4296-8418-e71c08fbac25\") " pod="openshift-monitoring/node-exporter-ql4f7" Jul 09 07:03:52.491603 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.491498 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/43d4b98d-4299-469f-8fd3-90cafd9aa515-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-7q8rc\" (UID: \"43d4b98d-4299-469f-8fd3-90cafd9aa515\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7q8rc" Jul 09 07:03:52.491603 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.491522 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzjv9\" (UniqueName: \"kubernetes.io/projected/9b494327-fac4-4296-8418-e71c08fbac25-kube-api-access-bzjv9\") pod \"node-exporter-ql4f7\" (UID: \"9b494327-fac4-4296-8418-e71c08fbac25\") " pod="openshift-monitoring/node-exporter-ql4f7" Jul 09 07:03:52.491603 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.491554 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e5b9b919-a410-4786-acf2-8e5b052e0ff3-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-pnpsf\" (UID: \"e5b9b919-a410-4786-acf2-8e5b052e0ff3\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-pnpsf" Jul 09 07:03:52.491603 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.491579 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e5b9b919-a410-4786-acf2-8e5b052e0ff3-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-pnpsf\" (UID: \"e5b9b919-a410-4786-acf2-8e5b052e0ff3\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-pnpsf" Jul 09 07:03:52.491824 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.491610 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43d4b98d-4299-469f-8fd3-90cafd9aa515-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-7q8rc\" (UID: \"43d4b98d-4299-469f-8fd3-90cafd9aa515\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7q8rc" Jul 09 07:03:52.491824 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.491641 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqjxw\" (UniqueName: \"kubernetes.io/projected/e5b9b919-a410-4786-acf2-8e5b052e0ff3-kube-api-access-qqjxw\") pod \"kube-state-metrics-7c9dbb8b7d-pnpsf\" (UID: \"e5b9b919-a410-4786-acf2-8e5b052e0ff3\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-pnpsf" Jul 09 07:03:52.491824 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.491699 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9b494327-fac4-4296-8418-e71c08fbac25-node-exporter-textfile\") pod \"node-exporter-ql4f7\" (UID: \"9b494327-fac4-4296-8418-e71c08fbac25\") " pod="openshift-monitoring/node-exporter-ql4f7" Jul 09 07:03:52.491824 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.491762 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b494327-fac4-4296-8418-e71c08fbac25-metrics-client-ca\") pod \"node-exporter-ql4f7\" (UID: \"9b494327-fac4-4296-8418-e71c08fbac25\") " pod="openshift-monitoring/node-exporter-ql4f7" Jul 09 07:03:52.491824 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.491786 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9b494327-fac4-4296-8418-e71c08fbac25-node-exporter-tls\") pod \"node-exporter-ql4f7\" (UID: \"9b494327-fac4-4296-8418-e71c08fbac25\") " pod="openshift-monitoring/node-exporter-ql4f7" Jul 09 07:03:52.491824 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.491809 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6qdr\" (UniqueName: \"kubernetes.io/projected/43d4b98d-4299-469f-8fd3-90cafd9aa515-kube-api-access-f6qdr\") pod \"openshift-state-metrics-54cf96dfcf-7q8rc\" (UID: \"43d4b98d-4299-469f-8fd3-90cafd9aa515\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7q8rc" Jul 09 07:03:52.492066 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.491832 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9b494327-fac4-4296-8418-e71c08fbac25-sys\") pod \"node-exporter-ql4f7\" (UID: \"9b494327-fac4-4296-8418-e71c08fbac25\") " pod="openshift-monitoring/node-exporter-ql4f7" Jul 09 07:03:52.492066 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.491854 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/43d4b98d-4299-469f-8fd3-90cafd9aa515-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-7q8rc\" (UID: \"43d4b98d-4299-469f-8fd3-90cafd9aa515\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7q8rc" Jul 09 07:03:52.492066 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.491874 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e5b9b919-a410-4786-acf2-8e5b052e0ff3-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-pnpsf\" (UID: \"e5b9b919-a410-4786-acf2-8e5b052e0ff3\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-pnpsf" Jul 09 07:03:52.492066 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.491938 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e5b9b919-a410-4786-acf2-8e5b052e0ff3-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-pnpsf\" (UID: \"e5b9b919-a410-4786-acf2-8e5b052e0ff3\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-pnpsf" Jul 09 07:03:52.492066 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.491989 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9b494327-fac4-4296-8418-e71c08fbac25-node-exporter-accelerators-collector-config\") pod \"node-exporter-ql4f7\" (UID: \"9b494327-fac4-4296-8418-e71c08fbac25\") " pod="openshift-monitoring/node-exporter-ql4f7" Jul 09 07:03:52.492066 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.492012 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9b494327-fac4-4296-8418-e71c08fbac25-node-exporter-wtmp\") pod \"node-exporter-ql4f7\" (UID: \"9b494327-fac4-4296-8418-e71c08fbac25\") " pod="openshift-monitoring/node-exporter-ql4f7" Jul 09 07:03:52.592460 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.592433 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/43d4b98d-4299-469f-8fd3-90cafd9aa515-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-7q8rc\" (UID: \"43d4b98d-4299-469f-8fd3-90cafd9aa515\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7q8rc" Jul 09 07:03:52.592606 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.592465 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e5b9b919-a410-4786-acf2-8e5b052e0ff3-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-pnpsf\" (UID: \"e5b9b919-a410-4786-acf2-8e5b052e0ff3\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-pnpsf" Jul 09 07:03:52.592606 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.592488 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e5b9b919-a410-4786-acf2-8e5b052e0ff3-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-pnpsf\" (UID: \"e5b9b919-a410-4786-acf2-8e5b052e0ff3\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-pnpsf" Jul 09 07:03:52.592696 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.592675 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9b494327-fac4-4296-8418-e71c08fbac25-node-exporter-accelerators-collector-config\") pod \"node-exporter-ql4f7\" (UID: \"9b494327-fac4-4296-8418-e71c08fbac25\") " pod="openshift-monitoring/node-exporter-ql4f7" Jul 09 07:03:52.592743 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.592721 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9b494327-fac4-4296-8418-e71c08fbac25-node-exporter-wtmp\") pod \"node-exporter-ql4f7\" (UID: \"9b494327-fac4-4296-8418-e71c08fbac25\") " pod="openshift-monitoring/node-exporter-ql4f7" Jul 09 07:03:52.592790 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.592753 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e5b9b919-a410-4786-acf2-8e5b052e0ff3-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-pnpsf\" (UID: \"e5b9b919-a410-4786-acf2-8e5b052e0ff3\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-pnpsf" Jul 09 07:03:52.592850 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.592786 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9b494327-fac4-4296-8418-e71c08fbac25-root\") pod \"node-exporter-ql4f7\" (UID: \"9b494327-fac4-4296-8418-e71c08fbac25\") " pod="openshift-monitoring/node-exporter-ql4f7" Jul 09 07:03:52.592850 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.592812 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9b494327-fac4-4296-8418-e71c08fbac25-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ql4f7\" (UID: \"9b494327-fac4-4296-8418-e71c08fbac25\") " pod="openshift-monitoring/node-exporter-ql4f7" Jul 09 07:03:52.592950 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.592866 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e5b9b919-a410-4786-acf2-8e5b052e0ff3-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-pnpsf\" (UID: \"e5b9b919-a410-4786-acf2-8e5b052e0ff3\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-pnpsf" Jul 09 07:03:52.592950 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.592885 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9b494327-fac4-4296-8418-e71c08fbac25-root\") pod \"node-exporter-ql4f7\" (UID: \"9b494327-fac4-4296-8418-e71c08fbac25\") " pod="openshift-monitoring/node-exporter-ql4f7" Jul 09 07:03:52.593048 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.592950 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9b494327-fac4-4296-8418-e71c08fbac25-node-exporter-wtmp\") pod \"node-exporter-ql4f7\" (UID: \"9b494327-fac4-4296-8418-e71c08fbac25\") " pod="openshift-monitoring/node-exporter-ql4f7" Jul 09 07:03:52.593048 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.592980 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/43d4b98d-4299-469f-8fd3-90cafd9aa515-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-7q8rc\" (UID: \"43d4b98d-4299-469f-8fd3-90cafd9aa515\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7q8rc" Jul 09 07:03:52.593048 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.593022 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzjv9\" (UniqueName: \"kubernetes.io/projected/9b494327-fac4-4296-8418-e71c08fbac25-kube-api-access-bzjv9\") pod \"node-exporter-ql4f7\" (UID: \"9b494327-fac4-4296-8418-e71c08fbac25\") " pod="openshift-monitoring/node-exporter-ql4f7" Jul 09 07:03:52.593197 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.593066 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e5b9b919-a410-4786-acf2-8e5b052e0ff3-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-pnpsf\" (UID: \"e5b9b919-a410-4786-acf2-8e5b052e0ff3\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-pnpsf" Jul 09 07:03:52.593197 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.593094 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e5b9b919-a410-4786-acf2-8e5b052e0ff3-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-pnpsf\" (UID: \"e5b9b919-a410-4786-acf2-8e5b052e0ff3\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-pnpsf" Jul 09 07:03:52.593197 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.593131 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43d4b98d-4299-469f-8fd3-90cafd9aa515-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-7q8rc\" (UID: \"43d4b98d-4299-469f-8fd3-90cafd9aa515\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7q8rc" Jul 09 07:03:52.593197 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.593161 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqjxw\" (UniqueName: \"kubernetes.io/projected/e5b9b919-a410-4786-acf2-8e5b052e0ff3-kube-api-access-qqjxw\") pod \"kube-state-metrics-7c9dbb8b7d-pnpsf\" (UID: \"e5b9b919-a410-4786-acf2-8e5b052e0ff3\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-pnpsf" Jul 09 07:03:52.593197 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.593190 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9b494327-fac4-4296-8418-e71c08fbac25-node-exporter-textfile\") pod \"node-exporter-ql4f7\" (UID: \"9b494327-fac4-4296-8418-e71c08fbac25\") " pod="openshift-monitoring/node-exporter-ql4f7" Jul 09 07:03:52.593466 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.593221 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b494327-fac4-4296-8418-e71c08fbac25-metrics-client-ca\") pod \"node-exporter-ql4f7\" (UID: \"9b494327-fac4-4296-8418-e71c08fbac25\") " pod="openshift-monitoring/node-exporter-ql4f7" Jul 09 07:03:52.593466 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.593254 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9b494327-fac4-4296-8418-e71c08fbac25-node-exporter-tls\") pod \"node-exporter-ql4f7\" (UID: \"9b494327-fac4-4296-8418-e71c08fbac25\") " pod="openshift-monitoring/node-exporter-ql4f7" Jul 09 07:03:52.593466 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.593309 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6qdr\" (UniqueName: \"kubernetes.io/projected/43d4b98d-4299-469f-8fd3-90cafd9aa515-kube-api-access-f6qdr\") pod \"openshift-state-metrics-54cf96dfcf-7q8rc\" (UID: \"43d4b98d-4299-469f-8fd3-90cafd9aa515\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7q8rc" Jul 09 07:03:52.593466 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.593331 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9b494327-fac4-4296-8418-e71c08fbac25-sys\") pod \"node-exporter-ql4f7\" (UID: \"9b494327-fac4-4296-8418-e71c08fbac25\") " pod="openshift-monitoring/node-exporter-ql4f7" Jul 09 07:03:52.593466 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.593366 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9b494327-fac4-4296-8418-e71c08fbac25-node-exporter-accelerators-collector-config\") pod \"node-exporter-ql4f7\" (UID: \"9b494327-fac4-4296-8418-e71c08fbac25\") " pod="openshift-monitoring/node-exporter-ql4f7" Jul 09 07:03:52.593761 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.593563 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e5b9b919-a410-4786-acf2-8e5b052e0ff3-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-pnpsf\" (UID: \"e5b9b919-a410-4786-acf2-8e5b052e0ff3\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-pnpsf" Jul 09 07:03:52.593939 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.593917 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9b494327-fac4-4296-8418-e71c08fbac25-node-exporter-textfile\") pod \"node-exporter-ql4f7\" (UID: \"9b494327-fac4-4296-8418-e71c08fbac25\") " pod="openshift-monitoring/node-exporter-ql4f7" Jul 09 07:03:52.594017 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.593979 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9b494327-fac4-4296-8418-e71c08fbac25-sys\") pod \"node-exporter-ql4f7\" (UID: \"9b494327-fac4-4296-8418-e71c08fbac25\") " pod="openshift-monitoring/node-exporter-ql4f7" Jul 09 07:03:52.594077 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.594011 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e5b9b919-a410-4786-acf2-8e5b052e0ff3-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-pnpsf\" (UID: \"e5b9b919-a410-4786-acf2-8e5b052e0ff3\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-pnpsf" Jul 09 07:03:52.594174 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.594151 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b494327-fac4-4296-8418-e71c08fbac25-metrics-client-ca\") pod \"node-exporter-ql4f7\" (UID: \"9b494327-fac4-4296-8418-e71c08fbac25\") " pod="openshift-monitoring/node-exporter-ql4f7" Jul 09 07:03:52.594708 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.594682 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43d4b98d-4299-469f-8fd3-90cafd9aa515-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-7q8rc\" (UID: \"43d4b98d-4299-469f-8fd3-90cafd9aa515\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7q8rc" Jul 09 07:03:52.595474 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.595450 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9b494327-fac4-4296-8418-e71c08fbac25-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ql4f7\" (UID: \"9b494327-fac4-4296-8418-e71c08fbac25\") " pod="openshift-monitoring/node-exporter-ql4f7" Jul 09 07:03:52.595582 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.595541 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e5b9b919-a410-4786-acf2-8e5b052e0ff3-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-pnpsf\" (UID: \"e5b9b919-a410-4786-acf2-8e5b052e0ff3\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-pnpsf" Jul 09 07:03:52.595647 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.595612 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/43d4b98d-4299-469f-8fd3-90cafd9aa515-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-7q8rc\" (UID: \"43d4b98d-4299-469f-8fd3-90cafd9aa515\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7q8rc" Jul 09 07:03:52.596339 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.596319 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9b494327-fac4-4296-8418-e71c08fbac25-node-exporter-tls\") pod \"node-exporter-ql4f7\" (UID: \"9b494327-fac4-4296-8418-e71c08fbac25\") " pod="openshift-monitoring/node-exporter-ql4f7" Jul 09 07:03:52.596452 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.596436 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e5b9b919-a410-4786-acf2-8e5b052e0ff3-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-pnpsf\" (UID: \"e5b9b919-a410-4786-acf2-8e5b052e0ff3\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-pnpsf" Jul 09 07:03:52.596499 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.596432 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/43d4b98d-4299-469f-8fd3-90cafd9aa515-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-7q8rc\" (UID: \"43d4b98d-4299-469f-8fd3-90cafd9aa515\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7q8rc" Jul 09 07:03:52.602096 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.602071 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6qdr\" (UniqueName: \"kubernetes.io/projected/43d4b98d-4299-469f-8fd3-90cafd9aa515-kube-api-access-f6qdr\") pod \"openshift-state-metrics-54cf96dfcf-7q8rc\" (UID: \"43d4b98d-4299-469f-8fd3-90cafd9aa515\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7q8rc" Jul 09 07:03:52.602227 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.602205 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzjv9\" (UniqueName: \"kubernetes.io/projected/9b494327-fac4-4296-8418-e71c08fbac25-kube-api-access-bzjv9\") pod \"node-exporter-ql4f7\" (UID: \"9b494327-fac4-4296-8418-e71c08fbac25\") " pod="openshift-monitoring/node-exporter-ql4f7" Jul 09 07:03:52.602559 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.602541 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqjxw\" (UniqueName: \"kubernetes.io/projected/e5b9b919-a410-4786-acf2-8e5b052e0ff3-kube-api-access-qqjxw\") pod \"kube-state-metrics-7c9dbb8b7d-pnpsf\" (UID: \"e5b9b919-a410-4786-acf2-8e5b052e0ff3\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-pnpsf" Jul 09 07:03:52.704593 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.704510 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7q8rc" Jul 09 07:03:52.711737 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.711718 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ql4f7" Jul 09 07:03:52.719579 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.719546 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-pnpsf" Jul 09 07:03:52.722786 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:03:52.722760 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b494327_fac4_4296_8418_e71c08fbac25.slice/crio-a92bf7c7320c5c73b42ddad762bc2d82bcece20f10076b4e5a50d31f15d4194b WatchSource:0}: Error finding container a92bf7c7320c5c73b42ddad762bc2d82bcece20f10076b4e5a50d31f15d4194b: Status 404 returned error can't find the container with id a92bf7c7320c5c73b42ddad762bc2d82bcece20f10076b4e5a50d31f15d4194b Jul 09 07:03:52.815668 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.815633 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ql4f7" event={"ID":"9b494327-fac4-4296-8418-e71c08fbac25","Type":"ContainerStarted","Data":"a92bf7c7320c5c73b42ddad762bc2d82bcece20f10076b4e5a50d31f15d4194b"} Jul 09 07:03:52.867961 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.867882 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-7q8rc"] Jul 09 07:03:52.869854 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:03:52.869829 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43d4b98d_4299_469f_8fd3_90cafd9aa515.slice/crio-cc8d6f3ff3484dddbc7257aeb31e0e6cc00ec6c785f5de168b692e86bdf5966f WatchSource:0}: Error finding container cc8d6f3ff3484dddbc7257aeb31e0e6cc00ec6c785f5de168b692e86bdf5966f: Status 404 returned error can't find the container with id cc8d6f3ff3484dddbc7257aeb31e0e6cc00ec6c785f5de168b692e86bdf5966f Jul 09 07:03:52.881891 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:52.881870 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-pnpsf"] Jul 09 07:03:52.885610 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:03:52.885588 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5b9b919_a410_4786_acf2_8e5b052e0ff3.slice/crio-50bc5a9d8be610f680947a32e83f9780cafa45fad1d2c0ea32016b1a0bf06c47 WatchSource:0}: Error finding container 50bc5a9d8be610f680947a32e83f9780cafa45fad1d2c0ea32016b1a0bf06c47: Status 404 returned error can't find the container with id 50bc5a9d8be610f680947a32e83f9780cafa45fad1d2c0ea32016b1a0bf06c47 Jul 09 07:03:53.820113 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:53.820074 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-pnpsf" event={"ID":"e5b9b919-a410-4786-acf2-8e5b052e0ff3","Type":"ContainerStarted","Data":"50bc5a9d8be610f680947a32e83f9780cafa45fad1d2c0ea32016b1a0bf06c47"} Jul 09 07:03:53.821786 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:53.821761 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7q8rc" event={"ID":"43d4b98d-4299-469f-8fd3-90cafd9aa515","Type":"ContainerStarted","Data":"1b500f01d218fe62d7a484b9e023b778cb555dae6d7dcf18aa7ed267c82c5309"} Jul 09 07:03:53.821900 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:53.821810 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7q8rc" event={"ID":"43d4b98d-4299-469f-8fd3-90cafd9aa515","Type":"ContainerStarted","Data":"a5ccd8eedf25e360bc7ea4a992fc9e3a7d132f0b633aaed23ce0158e3fa9db01"} Jul 09 07:03:53.821900 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:53.821831 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7q8rc" event={"ID":"43d4b98d-4299-469f-8fd3-90cafd9aa515","Type":"ContainerStarted","Data":"cc8d6f3ff3484dddbc7257aeb31e0e6cc00ec6c785f5de168b692e86bdf5966f"} Jul 09 07:03:54.827512 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:54.827477 2564 generic.go:356] "Generic (PLEG): container finished" podID="9b494327-fac4-4296-8418-e71c08fbac25" containerID="1bce267fa0e43ccb34a0a506abb5b65d3931c4819ab7a0bba8b866c0a4ac5c35" exitCode=0 Jul 09 07:03:54.827863 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:54.827575 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ql4f7" event={"ID":"9b494327-fac4-4296-8418-e71c08fbac25","Type":"ContainerDied","Data":"1bce267fa0e43ccb34a0a506abb5b65d3931c4819ab7a0bba8b866c0a4ac5c35"} Jul 09 07:03:54.831463 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:54.831424 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-pnpsf" event={"ID":"e5b9b919-a410-4786-acf2-8e5b052e0ff3","Type":"ContainerStarted","Data":"4d61f3e9a8cccf37f1ad6ea20b656076eeb7c475eb08e2eede26eb3efe141efd"} Jul 09 07:03:54.833991 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:54.833964 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7q8rc" event={"ID":"43d4b98d-4299-469f-8fd3-90cafd9aa515","Type":"ContainerStarted","Data":"a2492ad4a99022b01566fdb4423aaf282ae52059a0b7adb82e231b6f09b968dc"} Jul 09 07:03:54.866052 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:54.865618 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7q8rc" podStartSLOduration=1.669207997 podStartE2EDuration="2.865596133s" totalImagesPullingTime="1.196388136s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 07:03:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 07:03:54.863654865 +0000 UTC m=+93.914732094" watchObservedRunningTime="2026-07-09 07:03:54.865596133 +0000 UTC m=+93.916673362" Jul 09 07:03:55.839900 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:55.839861 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ql4f7" event={"ID":"9b494327-fac4-4296-8418-e71c08fbac25","Type":"ContainerStarted","Data":"2c9a253200b4a7817b3bd3c0e820c55e795e21d67b5d56c6af771719b0740233"} Jul 09 07:03:55.840381 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:55.839921 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ql4f7" event={"ID":"9b494327-fac4-4296-8418-e71c08fbac25","Type":"ContainerStarted","Data":"20c3e2f6e84147e3b612dfb852c3be047799f379cfa636a6c093b6feb15d657a"} Jul 09 07:03:55.841989 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:55.841964 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-pnpsf" event={"ID":"e5b9b919-a410-4786-acf2-8e5b052e0ff3","Type":"ContainerStarted","Data":"e925eee24af95eb32cd05700f3ec314a7e922eceb670e36391a065da07bc2d85"} Jul 09 07:03:55.842058 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:55.842023 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-pnpsf" event={"ID":"e5b9b919-a410-4786-acf2-8e5b052e0ff3","Type":"ContainerStarted","Data":"50674d9a844e4f2a9e2dfe2d6b874be04d97c4bb928ba36a446c4f973a4c2f0e"} Jul 09 07:03:55.893630 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:55.893583 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-ql4f7" podStartSLOduration=2.678368617 podStartE2EDuration="3.893568712s" totalImagesPullingTime="1.144544575s" totalInitContainerRuntime="70.65552ms" isStatefulPod=true podCreationTimestamp="2026-07-09 07:03:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 07:03:55.892416097 +0000 UTC m=+94.943493326" watchObservedRunningTime="2026-07-09 07:03:55.893568712 +0000 UTC m=+94.944645940" Jul 09 07:03:55.909766 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:55.909728 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-pnpsf" podStartSLOduration=2.073758795 podStartE2EDuration="3.909714243s" totalImagesPullingTime="1.835955448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 07:03:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 07:03:55.908934911 +0000 UTC m=+94.960012142" watchObservedRunningTime="2026-07-09 07:03:55.909714243 +0000 UTC m=+94.960791470" Jul 09 07:03:57.178592 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:57.178558 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-frjxp"] Jul 09 07:03:57.181642 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:57.181622 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-frjxp" Jul 09 07:03:57.183950 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:57.183926 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-ckvwz\"" Jul 09 07:03:57.183950 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:57.183941 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 09 07:03:57.191905 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:57.191874 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-frjxp"] Jul 09 07:03:57.238987 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:57.238954 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6knz\" (UniqueName: \"kubernetes.io/projected/d713e588-85eb-4e22-93d5-414e9a28aa30-kube-api-access-c6knz\") pod \"monitoring-plugin-d5dcff7cf-frjxp\" (UID: \"d713e588-85eb-4e22-93d5-414e9a28aa30\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-frjxp" Jul 09 07:03:57.239139 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:57.238998 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d713e588-85eb-4e22-93d5-414e9a28aa30-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-frjxp\" (UID: \"d713e588-85eb-4e22-93d5-414e9a28aa30\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-frjxp" Jul 09 07:03:57.340447 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:57.340411 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6knz\" (UniqueName: \"kubernetes.io/projected/d713e588-85eb-4e22-93d5-414e9a28aa30-kube-api-access-c6knz\") pod \"monitoring-plugin-d5dcff7cf-frjxp\" (UID: \"d713e588-85eb-4e22-93d5-414e9a28aa30\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-frjxp" Jul 09 07:03:57.340609 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:57.340461 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d713e588-85eb-4e22-93d5-414e9a28aa30-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-frjxp\" (UID: \"d713e588-85eb-4e22-93d5-414e9a28aa30\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-frjxp" Jul 09 07:03:57.340609 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:03:57.340583 2564 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 09 07:03:57.340728 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:03:57.340644 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d713e588-85eb-4e22-93d5-414e9a28aa30-monitoring-plugin-cert podName:d713e588-85eb-4e22-93d5-414e9a28aa30 nodeName:}" failed. No retries permitted until 2026-07-09 07:03:57.840625575 +0000 UTC m=+96.891702795 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/d713e588-85eb-4e22-93d5-414e9a28aa30-monitoring-plugin-cert") pod "monitoring-plugin-d5dcff7cf-frjxp" (UID: "d713e588-85eb-4e22-93d5-414e9a28aa30") : secret "monitoring-plugin-cert" not found Jul 09 07:03:57.352165 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:57.352132 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6knz\" (UniqueName: \"kubernetes.io/projected/d713e588-85eb-4e22-93d5-414e9a28aa30-kube-api-access-c6knz\") pod \"monitoring-plugin-d5dcff7cf-frjxp\" (UID: \"d713e588-85eb-4e22-93d5-414e9a28aa30\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-frjxp" Jul 09 07:03:57.844920 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:57.844876 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d713e588-85eb-4e22-93d5-414e9a28aa30-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-frjxp\" (UID: \"d713e588-85eb-4e22-93d5-414e9a28aa30\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-frjxp" Jul 09 07:03:57.847796 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:57.847768 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d713e588-85eb-4e22-93d5-414e9a28aa30-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-frjxp\" (UID: \"d713e588-85eb-4e22-93d5-414e9a28aa30\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-frjxp" Jul 09 07:03:58.096745 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:58.096660 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-frjxp" Jul 09 07:03:58.216185 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:58.216150 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-frjxp"] Jul 09 07:03:58.218808 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:03:58.218780 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd713e588_85eb_4e22_93d5_414e9a28aa30.slice/crio-b142e2ca265e1fc7543a56ac5a98c85dba82d26774d49720bca215cc0012b001 WatchSource:0}: Error finding container b142e2ca265e1fc7543a56ac5a98c85dba82d26774d49720bca215cc0012b001: Status 404 returned error can't find the container with id b142e2ca265e1fc7543a56ac5a98c85dba82d26774d49720bca215cc0012b001 Jul 09 07:03:58.857104 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:58.857066 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-frjxp" event={"ID":"d713e588-85eb-4e22-93d5-414e9a28aa30","Type":"ContainerStarted","Data":"b142e2ca265e1fc7543a56ac5a98c85dba82d26774d49720bca215cc0012b001"} Jul 09 07:03:59.862714 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:59.862682 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-frjxp" event={"ID":"d713e588-85eb-4e22-93d5-414e9a28aa30","Type":"ContainerStarted","Data":"ea63454e2598d58c868dc17dcaf00cf184513af6d4d2d0d03110facdf63f24c4"} Jul 09 07:03:59.863072 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:59.862836 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-frjxp" Jul 09 07:03:59.864198 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:59.864171 2564 patch_prober.go:28] interesting pod/monitoring-plugin-d5dcff7cf-frjxp container/monitoring-plugin namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.132.0.14:9443/health\": dial tcp 10.132.0.14:9443: connect: connection refused" start-of-body= Jul 09 07:03:59.864317 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:59.864221 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-frjxp" podUID="d713e588-85eb-4e22-93d5-414e9a28aa30" containerName="monitoring-plugin" probeResult="failure" output="Get \"https://10.132.0.14:9443/health\": dial tcp 10.132.0.14:9443: connect: connection refused" Jul 09 07:03:59.877007 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:03:59.876966 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-frjxp" podStartSLOduration=1.307547411 podStartE2EDuration="2.876952501s" totalImagesPullingTime="1.56940509s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 07:03:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 07:03:59.875958502 +0000 UTC m=+98.927035732" watchObservedRunningTime="2026-07-09 07:03:59.876952501 +0000 UTC m=+98.928029728" Jul 09 07:04:00.869834 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:04:00.869809 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-frjxp" Jul 09 07:04:01.513280 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:04:01.513233 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6877d4dfd6-vsmbd"] Jul 09 07:04:01.516549 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:04:01.516530 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6877d4dfd6-vsmbd" Jul 09 07:04:01.519086 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:04:01.519069 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 09 07:04:01.520384 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:04:01.520358 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 09 07:04:01.520481 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:04:01.520424 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-9l2ts\"" Jul 09 07:04:01.520481 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:04:01.520424 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 09 07:04:01.520481 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:04:01.520460 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 09 07:04:01.520481 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:04:01.520464 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 09 07:04:01.520693 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:04:01.520481 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 09 07:04:01.520812 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:04:01.520797 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 09 07:04:01.525506 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:04:01.525486 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 09 07:04:01.526403 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:04:01.526387 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6877d4dfd6-vsmbd"] Jul 09 07:04:01.672021 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:04:01.671966 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bc386366-eeb5-4230-a67b-964891ca9ae1-console-oauth-config\") pod \"console-6877d4dfd6-vsmbd\" (UID: \"bc386366-eeb5-4230-a67b-964891ca9ae1\") " pod="openshift-console/console-6877d4dfd6-vsmbd" Jul 09 07:04:01.672021 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:04:01.672019 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bc386366-eeb5-4230-a67b-964891ca9ae1-service-ca\") pod \"console-6877d4dfd6-vsmbd\" (UID: \"bc386366-eeb5-4230-a67b-964891ca9ae1\") " pod="openshift-console/console-6877d4dfd6-vsmbd" Jul 09 07:04:01.672021 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:04:01.672038 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bc386366-eeb5-4230-a67b-964891ca9ae1-oauth-serving-cert\") pod \"console-6877d4dfd6-vsmbd\" (UID: \"bc386366-eeb5-4230-a67b-964891ca9ae1\") " pod="openshift-console/console-6877d4dfd6-vsmbd" Jul 09 07:04:01.672373 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:04:01.672177 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb4s5\" (UniqueName: \"kubernetes.io/projected/bc386366-eeb5-4230-a67b-964891ca9ae1-kube-api-access-gb4s5\") pod \"console-6877d4dfd6-vsmbd\" (UID: \"bc386366-eeb5-4230-a67b-964891ca9ae1\") " pod="openshift-console/console-6877d4dfd6-vsmbd" Jul 09 07:04:01.672373 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:04:01.672227 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc386366-eeb5-4230-a67b-964891ca9ae1-trusted-ca-bundle\") pod \"console-6877d4dfd6-vsmbd\" (UID: \"bc386366-eeb5-4230-a67b-964891ca9ae1\") " pod="openshift-console/console-6877d4dfd6-vsmbd" Jul 09 07:04:01.672373 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:04:01.672256 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc386366-eeb5-4230-a67b-964891ca9ae1-console-serving-cert\") pod \"console-6877d4dfd6-vsmbd\" (UID: \"bc386366-eeb5-4230-a67b-964891ca9ae1\") " pod="openshift-console/console-6877d4dfd6-vsmbd" Jul 09 07:04:01.672373 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:04:01.672348 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bc386366-eeb5-4230-a67b-964891ca9ae1-console-config\") pod \"console-6877d4dfd6-vsmbd\" (UID: \"bc386366-eeb5-4230-a67b-964891ca9ae1\") " pod="openshift-console/console-6877d4dfd6-vsmbd" Jul 09 07:04:01.773477 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:04:01.773371 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bc386366-eeb5-4230-a67b-964891ca9ae1-console-oauth-config\") pod \"console-6877d4dfd6-vsmbd\" (UID: \"bc386366-eeb5-4230-a67b-964891ca9ae1\") " pod="openshift-console/console-6877d4dfd6-vsmbd" Jul 09 07:04:01.773477 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:04:01.773420 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bc386366-eeb5-4230-a67b-964891ca9ae1-service-ca\") pod \"console-6877d4dfd6-vsmbd\" (UID: \"bc386366-eeb5-4230-a67b-964891ca9ae1\") " pod="openshift-console/console-6877d4dfd6-vsmbd" Jul 09 07:04:01.773477 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:04:01.773439 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bc386366-eeb5-4230-a67b-964891ca9ae1-oauth-serving-cert\") pod \"console-6877d4dfd6-vsmbd\" (UID: \"bc386366-eeb5-4230-a67b-964891ca9ae1\") " pod="openshift-console/console-6877d4dfd6-vsmbd" Jul 09 07:04:01.773477 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:04:01.773477 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb4s5\" (UniqueName: \"kubernetes.io/projected/bc386366-eeb5-4230-a67b-964891ca9ae1-kube-api-access-gb4s5\") pod \"console-6877d4dfd6-vsmbd\" (UID: \"bc386366-eeb5-4230-a67b-964891ca9ae1\") " pod="openshift-console/console-6877d4dfd6-vsmbd" Jul 09 07:04:01.773785 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:04:01.773500 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc386366-eeb5-4230-a67b-964891ca9ae1-trusted-ca-bundle\") pod \"console-6877d4dfd6-vsmbd\" (UID: \"bc386366-eeb5-4230-a67b-964891ca9ae1\") " pod="openshift-console/console-6877d4dfd6-vsmbd" Jul 09 07:04:01.773785 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:04:01.773517 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc386366-eeb5-4230-a67b-964891ca9ae1-console-serving-cert\") pod \"console-6877d4dfd6-vsmbd\" (UID: \"bc386366-eeb5-4230-a67b-964891ca9ae1\") " pod="openshift-console/console-6877d4dfd6-vsmbd" Jul 09 07:04:01.773785 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:04:01.773542 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bc386366-eeb5-4230-a67b-964891ca9ae1-console-config\") pod \"console-6877d4dfd6-vsmbd\" (UID: \"bc386366-eeb5-4230-a67b-964891ca9ae1\") " pod="openshift-console/console-6877d4dfd6-vsmbd" Jul 09 07:04:01.774198 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:04:01.774172 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bc386366-eeb5-4230-a67b-964891ca9ae1-service-ca\") pod \"console-6877d4dfd6-vsmbd\" (UID: \"bc386366-eeb5-4230-a67b-964891ca9ae1\") " pod="openshift-console/console-6877d4dfd6-vsmbd" Jul 09 07:04:01.774329 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:04:01.774257 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bc386366-eeb5-4230-a67b-964891ca9ae1-console-config\") pod \"console-6877d4dfd6-vsmbd\" (UID: \"bc386366-eeb5-4230-a67b-964891ca9ae1\") " pod="openshift-console/console-6877d4dfd6-vsmbd" Jul 09 07:04:01.774386 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:04:01.774339 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bc386366-eeb5-4230-a67b-964891ca9ae1-oauth-serving-cert\") pod \"console-6877d4dfd6-vsmbd\" (UID: \"bc386366-eeb5-4230-a67b-964891ca9ae1\") " pod="openshift-console/console-6877d4dfd6-vsmbd" Jul 09 07:04:01.774505 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:04:01.774488 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc386366-eeb5-4230-a67b-964891ca9ae1-trusted-ca-bundle\") pod \"console-6877d4dfd6-vsmbd\" (UID: \"bc386366-eeb5-4230-a67b-964891ca9ae1\") " pod="openshift-console/console-6877d4dfd6-vsmbd" Jul 09 07:04:01.776416 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:04:01.776392 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bc386366-eeb5-4230-a67b-964891ca9ae1-console-oauth-config\") pod \"console-6877d4dfd6-vsmbd\" (UID: \"bc386366-eeb5-4230-a67b-964891ca9ae1\") " pod="openshift-console/console-6877d4dfd6-vsmbd" Jul 09 07:04:01.776520 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:04:01.776504 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc386366-eeb5-4230-a67b-964891ca9ae1-console-serving-cert\") pod \"console-6877d4dfd6-vsmbd\" (UID: \"bc386366-eeb5-4230-a67b-964891ca9ae1\") " pod="openshift-console/console-6877d4dfd6-vsmbd" Jul 09 07:04:01.781482 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:04:01.781463 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb4s5\" (UniqueName: \"kubernetes.io/projected/bc386366-eeb5-4230-a67b-964891ca9ae1-kube-api-access-gb4s5\") pod \"console-6877d4dfd6-vsmbd\" (UID: \"bc386366-eeb5-4230-a67b-964891ca9ae1\") " pod="openshift-console/console-6877d4dfd6-vsmbd" Jul 09 07:04:01.826615 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:04:01.826580 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6877d4dfd6-vsmbd" Jul 09 07:04:01.954317 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:04:01.954140 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6877d4dfd6-vsmbd"] Jul 09 07:04:01.956964 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:04:01.956932 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc386366_eeb5_4230_a67b_964891ca9ae1.slice/crio-d7b1650bc10d0fdef6cbcd52addb4fac2d010e781a74cb94364955c6e7304fd8 WatchSource:0}: Error finding container d7b1650bc10d0fdef6cbcd52addb4fac2d010e781a74cb94364955c6e7304fd8: Status 404 returned error can't find the container with id d7b1650bc10d0fdef6cbcd52addb4fac2d010e781a74cb94364955c6e7304fd8 Jul 09 07:04:02.873916 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:04:02.873885 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6877d4dfd6-vsmbd" event={"ID":"bc386366-eeb5-4230-a67b-964891ca9ae1","Type":"ContainerStarted","Data":"d7b1650bc10d0fdef6cbcd52addb4fac2d010e781a74cb94364955c6e7304fd8"} Jul 09 07:04:05.886293 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:04:05.886245 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6877d4dfd6-vsmbd" event={"ID":"bc386366-eeb5-4230-a67b-964891ca9ae1","Type":"ContainerStarted","Data":"84db10670959505e2dfd2b908af3c27ddadf09c05612701ebbb4ef4460d68220"} Jul 09 07:04:05.903137 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:04:05.903090 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6877d4dfd6-vsmbd" podStartSLOduration=1.527902547 podStartE2EDuration="4.903076378s" totalImagesPullingTime="3.375173831s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 07:04:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 07:04:05.902625842 +0000 UTC m=+104.953703071" watchObservedRunningTime="2026-07-09 07:04:05.903076378 +0000 UTC m=+104.954153606" Jul 09 07:04:06.749088 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:04:06.749053 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-4899b" Jul 09 07:04:11.827714 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:04:11.827677 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6877d4dfd6-vsmbd" Jul 09 07:04:11.828092 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:04:11.827764 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6877d4dfd6-vsmbd" Jul 09 07:04:11.832604 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:04:11.832583 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6877d4dfd6-vsmbd" Jul 09 07:04:11.910015 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:04:11.909989 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6877d4dfd6-vsmbd" Jul 09 07:05:31.710823 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:05:31.710787 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6877d4dfd6-vsmbd"] Jul 09 07:05:56.731078 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:05:56.731018 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-6877d4dfd6-vsmbd" podUID="bc386366-eeb5-4230-a67b-964891ca9ae1" containerName="console" containerID="cri-o://84db10670959505e2dfd2b908af3c27ddadf09c05612701ebbb4ef4460d68220" gracePeriod=15 Jul 09 07:05:56.970504 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:05:56.970484 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6877d4dfd6-vsmbd" Jul 09 07:05:57.003528 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:05:57.003458 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bc386366-eeb5-4230-a67b-964891ca9ae1-service-ca\") pod \"bc386366-eeb5-4230-a67b-964891ca9ae1\" (UID: \"bc386366-eeb5-4230-a67b-964891ca9ae1\") " Jul 09 07:05:57.003528 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:05:57.003513 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bc386366-eeb5-4230-a67b-964891ca9ae1-oauth-serving-cert\") pod \"bc386366-eeb5-4230-a67b-964891ca9ae1\" (UID: \"bc386366-eeb5-4230-a67b-964891ca9ae1\") " Jul 09 07:05:57.003705 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:05:57.003541 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc386366-eeb5-4230-a67b-964891ca9ae1-trusted-ca-bundle\") pod \"bc386366-eeb5-4230-a67b-964891ca9ae1\" (UID: \"bc386366-eeb5-4230-a67b-964891ca9ae1\") " Jul 09 07:05:57.003705 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:05:57.003589 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bc386366-eeb5-4230-a67b-964891ca9ae1-console-oauth-config\") pod \"bc386366-eeb5-4230-a67b-964891ca9ae1\" (UID: \"bc386366-eeb5-4230-a67b-964891ca9ae1\") " Jul 09 07:05:57.003705 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:05:57.003633 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc386366-eeb5-4230-a67b-964891ca9ae1-console-serving-cert\") pod \"bc386366-eeb5-4230-a67b-964891ca9ae1\" (UID: \"bc386366-eeb5-4230-a67b-964891ca9ae1\") " Jul 09 07:05:57.003705 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:05:57.003668 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gb4s5\" (UniqueName: \"kubernetes.io/projected/bc386366-eeb5-4230-a67b-964891ca9ae1-kube-api-access-gb4s5\") pod \"bc386366-eeb5-4230-a67b-964891ca9ae1\" (UID: \"bc386366-eeb5-4230-a67b-964891ca9ae1\") " Jul 09 07:05:57.003887 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:05:57.003706 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bc386366-eeb5-4230-a67b-964891ca9ae1-console-config\") pod \"bc386366-eeb5-4230-a67b-964891ca9ae1\" (UID: \"bc386366-eeb5-4230-a67b-964891ca9ae1\") " Jul 09 07:05:57.003957 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:05:57.003932 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc386366-eeb5-4230-a67b-964891ca9ae1-service-ca" (OuterVolumeSpecName: "service-ca") pod "bc386366-eeb5-4230-a67b-964891ca9ae1" (UID: "bc386366-eeb5-4230-a67b-964891ca9ae1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 07:05:57.004096 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:05:57.004068 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc386366-eeb5-4230-a67b-964891ca9ae1-console-config" (OuterVolumeSpecName: "console-config") pod "bc386366-eeb5-4230-a67b-964891ca9ae1" (UID: "bc386366-eeb5-4230-a67b-964891ca9ae1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 07:05:57.004141 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:05:57.004081 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc386366-eeb5-4230-a67b-964891ca9ae1-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "bc386366-eeb5-4230-a67b-964891ca9ae1" (UID: "bc386366-eeb5-4230-a67b-964891ca9ae1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 07:05:57.004141 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:05:57.004120 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc386366-eeb5-4230-a67b-964891ca9ae1-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "bc386366-eeb5-4230-a67b-964891ca9ae1" (UID: "bc386366-eeb5-4230-a67b-964891ca9ae1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 07:05:57.005956 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:05:57.005925 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc386366-eeb5-4230-a67b-964891ca9ae1-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "bc386366-eeb5-4230-a67b-964891ca9ae1" (UID: "bc386366-eeb5-4230-a67b-964891ca9ae1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 07:05:57.006040 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:05:57.005982 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc386366-eeb5-4230-a67b-964891ca9ae1-kube-api-access-gb4s5" (OuterVolumeSpecName: "kube-api-access-gb4s5") pod "bc386366-eeb5-4230-a67b-964891ca9ae1" (UID: "bc386366-eeb5-4230-a67b-964891ca9ae1"). InnerVolumeSpecName "kube-api-access-gb4s5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 07:05:57.006040 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:05:57.005980 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc386366-eeb5-4230-a67b-964891ca9ae1-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "bc386366-eeb5-4230-a67b-964891ca9ae1" (UID: "bc386366-eeb5-4230-a67b-964891ca9ae1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 07:05:57.105057 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:05:57.105031 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc386366-eeb5-4230-a67b-964891ca9ae1-console-serving-cert\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:05:57.105057 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:05:57.105054 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb4s5\" (UniqueName: \"kubernetes.io/projected/bc386366-eeb5-4230-a67b-964891ca9ae1-kube-api-access-gb4s5\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:05:57.105201 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:05:57.105064 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bc386366-eeb5-4230-a67b-964891ca9ae1-console-config\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:05:57.105201 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:05:57.105073 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bc386366-eeb5-4230-a67b-964891ca9ae1-service-ca\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:05:57.105201 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:05:57.105081 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bc386366-eeb5-4230-a67b-964891ca9ae1-oauth-serving-cert\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:05:57.105201 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:05:57.105089 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc386366-eeb5-4230-a67b-964891ca9ae1-trusted-ca-bundle\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:05:57.105201 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:05:57.105097 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bc386366-eeb5-4230-a67b-964891ca9ae1-console-oauth-config\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:05:57.269819 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:05:57.269730 2564 generic.go:356] "Generic (PLEG): container finished" podID="bc386366-eeb5-4230-a67b-964891ca9ae1" containerID="84db10670959505e2dfd2b908af3c27ddadf09c05612701ebbb4ef4460d68220" exitCode=0 Jul 09 07:05:57.269819 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:05:57.269774 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6877d4dfd6-vsmbd" Jul 09 07:05:57.270006 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:05:57.269820 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6877d4dfd6-vsmbd" event={"ID":"bc386366-eeb5-4230-a67b-964891ca9ae1","Type":"ContainerDied","Data":"84db10670959505e2dfd2b908af3c27ddadf09c05612701ebbb4ef4460d68220"} Jul 09 07:05:57.270006 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:05:57.269865 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6877d4dfd6-vsmbd" event={"ID":"bc386366-eeb5-4230-a67b-964891ca9ae1","Type":"ContainerDied","Data":"d7b1650bc10d0fdef6cbcd52addb4fac2d010e781a74cb94364955c6e7304fd8"} Jul 09 07:05:57.270006 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:05:57.269885 2564 scope.go:117] "RemoveContainer" containerID="84db10670959505e2dfd2b908af3c27ddadf09c05612701ebbb4ef4460d68220" Jul 09 07:05:57.277906 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:05:57.277890 2564 scope.go:117] "RemoveContainer" containerID="84db10670959505e2dfd2b908af3c27ddadf09c05612701ebbb4ef4460d68220" Jul 09 07:05:57.278150 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:05:57.278131 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84db10670959505e2dfd2b908af3c27ddadf09c05612701ebbb4ef4460d68220\": container with ID starting with 84db10670959505e2dfd2b908af3c27ddadf09c05612701ebbb4ef4460d68220 not found: ID does not exist" containerID="84db10670959505e2dfd2b908af3c27ddadf09c05612701ebbb4ef4460d68220" Jul 09 07:05:57.278199 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:05:57.278157 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84db10670959505e2dfd2b908af3c27ddadf09c05612701ebbb4ef4460d68220"} err="failed to get container status \"84db10670959505e2dfd2b908af3c27ddadf09c05612701ebbb4ef4460d68220\": rpc error: code = NotFound desc = could not find container \"84db10670959505e2dfd2b908af3c27ddadf09c05612701ebbb4ef4460d68220\": container with ID starting with 84db10670959505e2dfd2b908af3c27ddadf09c05612701ebbb4ef4460d68220 not found: ID does not exist" Jul 09 07:05:57.289743 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:05:57.289724 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6877d4dfd6-vsmbd"] Jul 09 07:05:57.292877 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:05:57.292859 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6877d4dfd6-vsmbd"] Jul 09 07:05:57.462014 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:05:57.461982 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc386366-eeb5-4230-a67b-964891ca9ae1" path="/var/lib/kubelet/pods/bc386366-eeb5-4230-a67b-964891ca9ae1/volumes" Jul 09 07:07:21.322287 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:07:21.322235 2564 kubelet.go:1648] "Image garbage collection succeeded" Jul 09 07:08:36.787594 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:08:36.787562 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-n78gp"] Jul 09 07:08:36.787997 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:08:36.787805 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc386366-eeb5-4230-a67b-964891ca9ae1" containerName="console" Jul 09 07:08:36.787997 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:08:36.787815 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc386366-eeb5-4230-a67b-964891ca9ae1" containerName="console" Jul 09 07:08:36.787997 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:08:36.787859 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc386366-eeb5-4230-a67b-964891ca9ae1" containerName="console" Jul 09 07:08:36.790534 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:08:36.790518 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-n78gp" Jul 09 07:08:36.792653 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:08:36.792632 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 09 07:08:36.793626 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:08:36.793609 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 09 07:08:36.793708 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:08:36.793632 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-7v2gw\"" Jul 09 07:08:36.799472 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:08:36.799452 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-n78gp"] Jul 09 07:08:36.892159 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:08:36.892131 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fff87a4a-7758-4135-8ca3-1b9eec238383-bound-sa-token\") pod \"cert-manager-55b4d954d5-n78gp\" (UID: \"fff87a4a-7758-4135-8ca3-1b9eec238383\") " pod="cert-manager/cert-manager-55b4d954d5-n78gp" Jul 09 07:08:36.892338 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:08:36.892170 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd82k\" (UniqueName: \"kubernetes.io/projected/fff87a4a-7758-4135-8ca3-1b9eec238383-kube-api-access-wd82k\") pod \"cert-manager-55b4d954d5-n78gp\" (UID: \"fff87a4a-7758-4135-8ca3-1b9eec238383\") " pod="cert-manager/cert-manager-55b4d954d5-n78gp" Jul 09 07:08:36.992757 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:08:36.992725 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fff87a4a-7758-4135-8ca3-1b9eec238383-bound-sa-token\") pod \"cert-manager-55b4d954d5-n78gp\" (UID: \"fff87a4a-7758-4135-8ca3-1b9eec238383\") " pod="cert-manager/cert-manager-55b4d954d5-n78gp" Jul 09 07:08:36.992888 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:08:36.992765 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd82k\" (UniqueName: \"kubernetes.io/projected/fff87a4a-7758-4135-8ca3-1b9eec238383-kube-api-access-wd82k\") pod \"cert-manager-55b4d954d5-n78gp\" (UID: \"fff87a4a-7758-4135-8ca3-1b9eec238383\") " pod="cert-manager/cert-manager-55b4d954d5-n78gp" Jul 09 07:08:37.000572 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:08:37.000543 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fff87a4a-7758-4135-8ca3-1b9eec238383-bound-sa-token\") pod \"cert-manager-55b4d954d5-n78gp\" (UID: \"fff87a4a-7758-4135-8ca3-1b9eec238383\") " pod="cert-manager/cert-manager-55b4d954d5-n78gp" Jul 09 07:08:37.000725 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:08:37.000706 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd82k\" (UniqueName: \"kubernetes.io/projected/fff87a4a-7758-4135-8ca3-1b9eec238383-kube-api-access-wd82k\") pod \"cert-manager-55b4d954d5-n78gp\" (UID: \"fff87a4a-7758-4135-8ca3-1b9eec238383\") " pod="cert-manager/cert-manager-55b4d954d5-n78gp" Jul 09 07:08:37.099623 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:08:37.099533 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-n78gp" Jul 09 07:08:37.223760 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:08:37.223732 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-n78gp"] Jul 09 07:08:37.226293 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:08:37.226249 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfff87a4a_7758_4135_8ca3_1b9eec238383.slice/crio-1a22e68b0602bf9c1aa95bddb4345d7f8f24308c0362ec4a525ac55d7b3aa5a3 WatchSource:0}: Error finding container 1a22e68b0602bf9c1aa95bddb4345d7f8f24308c0362ec4a525ac55d7b3aa5a3: Status 404 returned error can't find the container with id 1a22e68b0602bf9c1aa95bddb4345d7f8f24308c0362ec4a525ac55d7b3aa5a3 Jul 09 07:08:37.228049 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:08:37.228034 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 07:08:37.791428 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:08:37.791389 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-n78gp" event={"ID":"fff87a4a-7758-4135-8ca3-1b9eec238383","Type":"ContainerStarted","Data":"1a22e68b0602bf9c1aa95bddb4345d7f8f24308c0362ec4a525ac55d7b3aa5a3"} Jul 09 07:08:40.803819 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:08:40.803786 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-n78gp" event={"ID":"fff87a4a-7758-4135-8ca3-1b9eec238383","Type":"ContainerStarted","Data":"b4005989911b3410700b9e04174665787ad692105c0fc5af912cdceff17661f5"} Jul 09 07:08:40.819818 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:08:40.819770 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-n78gp" podStartSLOduration=2.017092913 podStartE2EDuration="4.819755741s" totalImagesPullingTime="2.802662828s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 07:08:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 07:08:40.819077202 +0000 UTC m=+379.870154431" watchObservedRunningTime="2026-07-09 07:08:40.819755741 +0000 UTC m=+379.870832969" Jul 09 07:08:53.492494 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:08:53.492463 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hjbpw"] Jul 09 07:08:53.498281 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:08:53.498245 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hjbpw" Jul 09 07:08:53.500656 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:08:53.500631 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-lmk4r\"" Jul 09 07:08:53.500774 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:08:53.500666 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 09 07:08:53.501507 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:08:53.501493 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 09 07:08:53.508950 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:08:53.508928 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hjbpw"] Jul 09 07:08:53.618496 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:08:53.618464 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/04588e70-1de7-4a10-8c25-a47e1484b076-tmp\") pod \"openshift-lws-operator-fd8ccff4c-hjbpw\" (UID: \"04588e70-1de7-4a10-8c25-a47e1484b076\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hjbpw" Jul 09 07:08:53.618671 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:08:53.618514 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t57d2\" (UniqueName: \"kubernetes.io/projected/04588e70-1de7-4a10-8c25-a47e1484b076-kube-api-access-t57d2\") pod \"openshift-lws-operator-fd8ccff4c-hjbpw\" (UID: \"04588e70-1de7-4a10-8c25-a47e1484b076\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hjbpw" Jul 09 07:08:53.719745 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:08:53.719708 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/04588e70-1de7-4a10-8c25-a47e1484b076-tmp\") pod \"openshift-lws-operator-fd8ccff4c-hjbpw\" (UID: \"04588e70-1de7-4a10-8c25-a47e1484b076\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hjbpw" Jul 09 07:08:53.719884 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:08:53.719753 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t57d2\" (UniqueName: \"kubernetes.io/projected/04588e70-1de7-4a10-8c25-a47e1484b076-kube-api-access-t57d2\") pod \"openshift-lws-operator-fd8ccff4c-hjbpw\" (UID: \"04588e70-1de7-4a10-8c25-a47e1484b076\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hjbpw" Jul 09 07:08:53.720158 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:08:53.720140 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/04588e70-1de7-4a10-8c25-a47e1484b076-tmp\") pod \"openshift-lws-operator-fd8ccff4c-hjbpw\" (UID: \"04588e70-1de7-4a10-8c25-a47e1484b076\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hjbpw" Jul 09 07:08:53.729065 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:08:53.729045 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t57d2\" (UniqueName: \"kubernetes.io/projected/04588e70-1de7-4a10-8c25-a47e1484b076-kube-api-access-t57d2\") pod \"openshift-lws-operator-fd8ccff4c-hjbpw\" (UID: \"04588e70-1de7-4a10-8c25-a47e1484b076\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hjbpw" Jul 09 07:08:53.810544 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:08:53.810475 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hjbpw" Jul 09 07:08:53.935301 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:08:53.935254 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hjbpw"] Jul 09 07:08:53.938364 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:08:53.938335 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod04588e70_1de7_4a10_8c25_a47e1484b076.slice/crio-82bda1a76ce8c886dea704d7d6d04db2aa28d8aec217747d4211bf8118367c79 WatchSource:0}: Error finding container 82bda1a76ce8c886dea704d7d6d04db2aa28d8aec217747d4211bf8118367c79: Status 404 returned error can't find the container with id 82bda1a76ce8c886dea704d7d6d04db2aa28d8aec217747d4211bf8118367c79 Jul 09 07:08:54.852251 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:08:54.852215 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hjbpw" event={"ID":"04588e70-1de7-4a10-8c25-a47e1484b076","Type":"ContainerStarted","Data":"82bda1a76ce8c886dea704d7d6d04db2aa28d8aec217747d4211bf8118367c79"} Jul 09 07:08:56.863860 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:08:56.863820 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hjbpw" event={"ID":"04588e70-1de7-4a10-8c25-a47e1484b076","Type":"ContainerStarted","Data":"a9705a216b0bc871b03dea9d9e41104d05422e3019859d8734a9b75ee1eb8ca3"} Jul 09 07:08:56.880549 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:08:56.880494 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hjbpw" podStartSLOduration=1.147023363 podStartE2EDuration="3.880479139s" totalImagesPullingTime="2.733455776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 07:08:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 07:08:56.879388128 +0000 UTC m=+395.930465356" watchObservedRunningTime="2026-07-09 07:08:56.880479139 +0000 UTC m=+395.931556367" Jul 09 07:09:16.403848 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:16.403818 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b96c58b47-ws4xb"] Jul 09 07:09:16.406971 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:16.406952 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-b96c58b47-ws4xb" Jul 09 07:09:16.410200 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:16.410181 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 09 07:09:16.410387 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:16.410365 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 09 07:09:16.410387 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:16.410379 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 09 07:09:16.410649 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:16.410631 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 09 07:09:16.410746 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:16.410699 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-l47m2\"" Jul 09 07:09:16.422480 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:16.422462 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b96c58b47-ws4xb"] Jul 09 07:09:16.583084 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:16.583050 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/74cb03fe-bacc-4d19-971d-db6542495050-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b96c58b47-ws4xb\" (UID: \"74cb03fe-bacc-4d19-971d-db6542495050\") " pod="opendatahub/opendatahub-operator-controller-manager-b96c58b47-ws4xb" Jul 09 07:09:16.583084 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:16.583089 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6lwm\" (UniqueName: \"kubernetes.io/projected/74cb03fe-bacc-4d19-971d-db6542495050-kube-api-access-j6lwm\") pod \"opendatahub-operator-controller-manager-b96c58b47-ws4xb\" (UID: \"74cb03fe-bacc-4d19-971d-db6542495050\") " pod="opendatahub/opendatahub-operator-controller-manager-b96c58b47-ws4xb" Jul 09 07:09:16.583360 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:16.583113 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/74cb03fe-bacc-4d19-971d-db6542495050-manifests\") pod \"opendatahub-operator-controller-manager-b96c58b47-ws4xb\" (UID: \"74cb03fe-bacc-4d19-971d-db6542495050\") " pod="opendatahub/opendatahub-operator-controller-manager-b96c58b47-ws4xb" Jul 09 07:09:16.583360 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:16.583198 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/74cb03fe-bacc-4d19-971d-db6542495050-webhook-cert\") pod \"opendatahub-operator-controller-manager-b96c58b47-ws4xb\" (UID: \"74cb03fe-bacc-4d19-971d-db6542495050\") " pod="opendatahub/opendatahub-operator-controller-manager-b96c58b47-ws4xb" Jul 09 07:09:16.583360 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:16.583243 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/74cb03fe-bacc-4d19-971d-db6542495050-tmp\") pod \"opendatahub-operator-controller-manager-b96c58b47-ws4xb\" (UID: \"74cb03fe-bacc-4d19-971d-db6542495050\") " pod="opendatahub/opendatahub-operator-controller-manager-b96c58b47-ws4xb" Jul 09 07:09:16.684160 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:16.684077 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/74cb03fe-bacc-4d19-971d-db6542495050-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b96c58b47-ws4xb\" (UID: \"74cb03fe-bacc-4d19-971d-db6542495050\") " pod="opendatahub/opendatahub-operator-controller-manager-b96c58b47-ws4xb" Jul 09 07:09:16.684160 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:16.684113 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6lwm\" (UniqueName: \"kubernetes.io/projected/74cb03fe-bacc-4d19-971d-db6542495050-kube-api-access-j6lwm\") pod \"opendatahub-operator-controller-manager-b96c58b47-ws4xb\" (UID: \"74cb03fe-bacc-4d19-971d-db6542495050\") " pod="opendatahub/opendatahub-operator-controller-manager-b96c58b47-ws4xb" Jul 09 07:09:16.684160 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:16.684135 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/74cb03fe-bacc-4d19-971d-db6542495050-manifests\") pod \"opendatahub-operator-controller-manager-b96c58b47-ws4xb\" (UID: \"74cb03fe-bacc-4d19-971d-db6542495050\") " pod="opendatahub/opendatahub-operator-controller-manager-b96c58b47-ws4xb" Jul 09 07:09:16.684451 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:16.684252 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/74cb03fe-bacc-4d19-971d-db6542495050-webhook-cert\") pod \"opendatahub-operator-controller-manager-b96c58b47-ws4xb\" (UID: \"74cb03fe-bacc-4d19-971d-db6542495050\") " pod="opendatahub/opendatahub-operator-controller-manager-b96c58b47-ws4xb" Jul 09 07:09:16.684451 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:16.684325 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/74cb03fe-bacc-4d19-971d-db6542495050-tmp\") pod \"opendatahub-operator-controller-manager-b96c58b47-ws4xb\" (UID: \"74cb03fe-bacc-4d19-971d-db6542495050\") " pod="opendatahub/opendatahub-operator-controller-manager-b96c58b47-ws4xb" Jul 09 07:09:16.684615 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:16.684588 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/74cb03fe-bacc-4d19-971d-db6542495050-manifests\") pod \"opendatahub-operator-controller-manager-b96c58b47-ws4xb\" (UID: \"74cb03fe-bacc-4d19-971d-db6542495050\") " pod="opendatahub/opendatahub-operator-controller-manager-b96c58b47-ws4xb" Jul 09 07:09:16.686527 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:16.686506 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/74cb03fe-bacc-4d19-971d-db6542495050-tmp\") pod \"opendatahub-operator-controller-manager-b96c58b47-ws4xb\" (UID: \"74cb03fe-bacc-4d19-971d-db6542495050\") " pod="opendatahub/opendatahub-operator-controller-manager-b96c58b47-ws4xb" Jul 09 07:09:16.686605 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:16.686562 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/74cb03fe-bacc-4d19-971d-db6542495050-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b96c58b47-ws4xb\" (UID: \"74cb03fe-bacc-4d19-971d-db6542495050\") " pod="opendatahub/opendatahub-operator-controller-manager-b96c58b47-ws4xb" Jul 09 07:09:16.686658 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:16.686641 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/74cb03fe-bacc-4d19-971d-db6542495050-webhook-cert\") pod \"opendatahub-operator-controller-manager-b96c58b47-ws4xb\" (UID: \"74cb03fe-bacc-4d19-971d-db6542495050\") " pod="opendatahub/opendatahub-operator-controller-manager-b96c58b47-ws4xb" Jul 09 07:09:16.694152 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:16.694132 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6lwm\" (UniqueName: \"kubernetes.io/projected/74cb03fe-bacc-4d19-971d-db6542495050-kube-api-access-j6lwm\") pod \"opendatahub-operator-controller-manager-b96c58b47-ws4xb\" (UID: \"74cb03fe-bacc-4d19-971d-db6542495050\") " pod="opendatahub/opendatahub-operator-controller-manager-b96c58b47-ws4xb" Jul 09 07:09:16.717792 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:16.717773 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-b96c58b47-ws4xb" Jul 09 07:09:16.844122 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:16.844099 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b96c58b47-ws4xb"] Jul 09 07:09:16.846631 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:09:16.846603 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74cb03fe_bacc_4d19_971d_db6542495050.slice/crio-563b1ef45ed7ccf8ef482cc5c4ac0263967d08d8b7f212709698bb346ad85c8a WatchSource:0}: Error finding container 563b1ef45ed7ccf8ef482cc5c4ac0263967d08d8b7f212709698bb346ad85c8a: Status 404 returned error can't find the container with id 563b1ef45ed7ccf8ef482cc5c4ac0263967d08d8b7f212709698bb346ad85c8a Jul 09 07:09:16.933275 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:16.933231 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b96c58b47-ws4xb" event={"ID":"74cb03fe-bacc-4d19-971d-db6542495050","Type":"ContainerStarted","Data":"563b1ef45ed7ccf8ef482cc5c4ac0263967d08d8b7f212709698bb346ad85c8a"} Jul 09 07:09:20.951300 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:20.951251 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b96c58b47-ws4xb" event={"ID":"74cb03fe-bacc-4d19-971d-db6542495050","Type":"ContainerStarted","Data":"544676997f86993a0b977df179c9eb41cc146471fd83bf2a801629dc48e0d5c8"} Jul 09 07:09:21.961555 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:21.961522 2564 generic.go:356] "Generic (PLEG): container finished" podID="74cb03fe-bacc-4d19-971d-db6542495050" containerID="544676997f86993a0b977df179c9eb41cc146471fd83bf2a801629dc48e0d5c8" exitCode=0 Jul 09 07:09:21.961947 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:21.961584 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b96c58b47-ws4xb" event={"ID":"74cb03fe-bacc-4d19-971d-db6542495050","Type":"ContainerDied","Data":"544676997f86993a0b977df179c9eb41cc146471fd83bf2a801629dc48e0d5c8"} Jul 09 07:09:22.967583 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:22.967548 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b96c58b47-ws4xb" event={"ID":"74cb03fe-bacc-4d19-971d-db6542495050","Type":"ContainerStarted","Data":"3cea3adacb7493b9c95fcd471bf8ebd163bf44ea71671704988a8fdb8b62042d"} Jul 09 07:09:22.967990 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:22.967780 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-b96c58b47-ws4xb" Jul 09 07:09:22.988539 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:22.988488 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-b96c58b47-ws4xb" podStartSLOduration=2.444403264 podStartE2EDuration="6.988469116s" totalImagesPullingTime="3.490565622s" totalInitContainerRuntime="1.05350023s" isStatefulPod=false podCreationTimestamp="2026-07-09 07:09:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 07:09:22.988113317 +0000 UTC m=+422.039190546" watchObservedRunningTime="2026-07-09 07:09:22.988469116 +0000 UTC m=+422.039546348" Jul 09 07:09:33.973967 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:33.973938 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-b96c58b47-ws4xb" Jul 09 07:09:36.463251 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:36.463220 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-dmzzk"] Jul 09 07:09:36.469779 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:36.469759 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-dmzzk" Jul 09 07:09:36.473248 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:36.473213 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-l52zw\"" Jul 09 07:09:36.473248 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:36.473232 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 09 07:09:36.480550 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:36.480527 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-dmzzk"] Jul 09 07:09:36.528373 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:36.528338 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr4cx\" (UniqueName: \"kubernetes.io/projected/57e6b153-89fc-4cd1-a2ad-19c575990d30-kube-api-access-dr4cx\") pod \"odh-model-controller-7998c67dc4-dmzzk\" (UID: \"57e6b153-89fc-4cd1-a2ad-19c575990d30\") " pod="opendatahub/odh-model-controller-7998c67dc4-dmzzk" Jul 09 07:09:36.528509 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:36.528452 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/57e6b153-89fc-4cd1-a2ad-19c575990d30-cert\") pod \"odh-model-controller-7998c67dc4-dmzzk\" (UID: \"57e6b153-89fc-4cd1-a2ad-19c575990d30\") " pod="opendatahub/odh-model-controller-7998c67dc4-dmzzk" Jul 09 07:09:36.629332 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:36.629292 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/57e6b153-89fc-4cd1-a2ad-19c575990d30-cert\") pod \"odh-model-controller-7998c67dc4-dmzzk\" (UID: \"57e6b153-89fc-4cd1-a2ad-19c575990d30\") " pod="opendatahub/odh-model-controller-7998c67dc4-dmzzk" Jul 09 07:09:36.629509 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:36.629341 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr4cx\" (UniqueName: \"kubernetes.io/projected/57e6b153-89fc-4cd1-a2ad-19c575990d30-kube-api-access-dr4cx\") pod \"odh-model-controller-7998c67dc4-dmzzk\" (UID: \"57e6b153-89fc-4cd1-a2ad-19c575990d30\") " pod="opendatahub/odh-model-controller-7998c67dc4-dmzzk" Jul 09 07:09:36.629509 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:09:36.629443 2564 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 09 07:09:36.629627 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:09:36.629512 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57e6b153-89fc-4cd1-a2ad-19c575990d30-cert podName:57e6b153-89fc-4cd1-a2ad-19c575990d30 nodeName:}" failed. No retries permitted until 2026-07-09 07:09:37.129496543 +0000 UTC m=+436.180573749 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/57e6b153-89fc-4cd1-a2ad-19c575990d30-cert") pod "odh-model-controller-7998c67dc4-dmzzk" (UID: "57e6b153-89fc-4cd1-a2ad-19c575990d30") : secret "odh-model-controller-webhook-cert" not found Jul 09 07:09:36.655975 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:36.655946 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr4cx\" (UniqueName: \"kubernetes.io/projected/57e6b153-89fc-4cd1-a2ad-19c575990d30-kube-api-access-dr4cx\") pod \"odh-model-controller-7998c67dc4-dmzzk\" (UID: \"57e6b153-89fc-4cd1-a2ad-19c575990d30\") " pod="opendatahub/odh-model-controller-7998c67dc4-dmzzk" Jul 09 07:09:37.102899 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:37.102868 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-n5xxf"] Jul 09 07:09:37.105123 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:37.105107 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-n5xxf" Jul 09 07:09:37.110168 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:37.110142 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 09 07:09:37.121642 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:37.121618 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-n5xxf"] Jul 09 07:09:37.133501 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:37.133477 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/57e6b153-89fc-4cd1-a2ad-19c575990d30-cert\") pod \"odh-model-controller-7998c67dc4-dmzzk\" (UID: \"57e6b153-89fc-4cd1-a2ad-19c575990d30\") " pod="opendatahub/odh-model-controller-7998c67dc4-dmzzk" Jul 09 07:09:37.133582 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:37.133556 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b68a13c8-faf4-4dfb-bdfb-dc4cfab00314-tls-certs\") pod \"model-serving-api-54d89bd5fc-n5xxf\" (UID: \"b68a13c8-faf4-4dfb-bdfb-dc4cfab00314\") " pod="opendatahub/model-serving-api-54d89bd5fc-n5xxf" Jul 09 07:09:37.133633 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:37.133592 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7grj6\" (UniqueName: \"kubernetes.io/projected/b68a13c8-faf4-4dfb-bdfb-dc4cfab00314-kube-api-access-7grj6\") pod \"model-serving-api-54d89bd5fc-n5xxf\" (UID: \"b68a13c8-faf4-4dfb-bdfb-dc4cfab00314\") " pod="opendatahub/model-serving-api-54d89bd5fc-n5xxf" Jul 09 07:09:37.135707 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:37.135680 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/57e6b153-89fc-4cd1-a2ad-19c575990d30-cert\") pod \"odh-model-controller-7998c67dc4-dmzzk\" (UID: \"57e6b153-89fc-4cd1-a2ad-19c575990d30\") " pod="opendatahub/odh-model-controller-7998c67dc4-dmzzk" Jul 09 07:09:37.234649 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:37.234614 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7grj6\" (UniqueName: \"kubernetes.io/projected/b68a13c8-faf4-4dfb-bdfb-dc4cfab00314-kube-api-access-7grj6\") pod \"model-serving-api-54d89bd5fc-n5xxf\" (UID: \"b68a13c8-faf4-4dfb-bdfb-dc4cfab00314\") " pod="opendatahub/model-serving-api-54d89bd5fc-n5xxf" Jul 09 07:09:37.234820 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:37.234756 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b68a13c8-faf4-4dfb-bdfb-dc4cfab00314-tls-certs\") pod \"model-serving-api-54d89bd5fc-n5xxf\" (UID: \"b68a13c8-faf4-4dfb-bdfb-dc4cfab00314\") " pod="opendatahub/model-serving-api-54d89bd5fc-n5xxf" Jul 09 07:09:37.237173 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:37.237145 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b68a13c8-faf4-4dfb-bdfb-dc4cfab00314-tls-certs\") pod \"model-serving-api-54d89bd5fc-n5xxf\" (UID: \"b68a13c8-faf4-4dfb-bdfb-dc4cfab00314\") " pod="opendatahub/model-serving-api-54d89bd5fc-n5xxf" Jul 09 07:09:37.242863 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:37.242844 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7grj6\" (UniqueName: \"kubernetes.io/projected/b68a13c8-faf4-4dfb-bdfb-dc4cfab00314-kube-api-access-7grj6\") pod \"model-serving-api-54d89bd5fc-n5xxf\" (UID: \"b68a13c8-faf4-4dfb-bdfb-dc4cfab00314\") " pod="opendatahub/model-serving-api-54d89bd5fc-n5xxf" Jul 09 07:09:37.381669 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:37.381636 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-dmzzk" Jul 09 07:09:37.415292 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:37.415252 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-n5xxf" Jul 09 07:09:37.521515 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:37.517368 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-dmzzk"] Jul 09 07:09:37.551540 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:37.551507 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-n5xxf"] Jul 09 07:09:37.554098 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:09:37.554073 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb68a13c8_faf4_4dfb_bdfb_dc4cfab00314.slice/crio-195915c42154c027623bdace74920be2eba3170cee184a476ac0eda1819488f1 WatchSource:0}: Error finding container 195915c42154c027623bdace74920be2eba3170cee184a476ac0eda1819488f1: Status 404 returned error can't find the container with id 195915c42154c027623bdace74920be2eba3170cee184a476ac0eda1819488f1 Jul 09 07:09:38.030738 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:38.030697 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-dmzzk" event={"ID":"57e6b153-89fc-4cd1-a2ad-19c575990d30","Type":"ContainerStarted","Data":"5eef4e31005b087043047919466abc143f7a2bfdb9ee261e4550cd7954f87f66"} Jul 09 07:09:38.031731 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:38.031707 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-n5xxf" event={"ID":"b68a13c8-faf4-4dfb-bdfb-dc4cfab00314","Type":"ContainerStarted","Data":"195915c42154c027623bdace74920be2eba3170cee184a476ac0eda1819488f1"} Jul 09 07:09:41.047078 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:41.046949 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-dmzzk" event={"ID":"57e6b153-89fc-4cd1-a2ad-19c575990d30","Type":"ContainerStarted","Data":"40cbb7be71b567ef536d08a281ac808ae93a7feeeb21931e00659e4456c501a6"} Jul 09 07:09:41.047609 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:41.047150 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-dmzzk" Jul 09 07:09:41.048687 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:41.048653 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-n5xxf" event={"ID":"b68a13c8-faf4-4dfb-bdfb-dc4cfab00314","Type":"ContainerStarted","Data":"27edf093aedaa412e77d8016ac3ba170d8faa3a70804a3eff32c30bc2069b3e5"} Jul 09 07:09:41.048814 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:41.048803 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-n5xxf" Jul 09 07:09:41.064214 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:41.064169 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-dmzzk" podStartSLOduration=1.922866536 podStartE2EDuration="5.064155271s" totalImagesPullingTime="3.141288735s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 07:09:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 07:09:41.062858802 +0000 UTC m=+440.113936031" watchObservedRunningTime="2026-07-09 07:09:41.064155271 +0000 UTC m=+440.115232498" Jul 09 07:09:41.082431 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:41.082392 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-n5xxf" podStartSLOduration=0.977874466 podStartE2EDuration="4.082377977s" totalImagesPullingTime="3.104503511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 07:09:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 07:09:41.081089622 +0000 UTC m=+440.132167072" watchObservedRunningTime="2026-07-09 07:09:41.082377977 +0000 UTC m=+440.133455206" Jul 09 07:09:45.853540 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:45.853416 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-w7hkl"] Jul 09 07:09:45.859211 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:45.859186 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-w7hkl" Jul 09 07:09:45.863301 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:45.863253 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-t4lsx"] Jul 09 07:09:45.863977 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:45.863952 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 09 07:09:45.864208 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:45.864189 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-lqs4x\"" Jul 09 07:09:45.867012 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:45.866992 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t4lsx" Jul 09 07:09:45.869962 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:45.869940 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 09 07:09:45.870133 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:45.870117 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-9wmck\"" Jul 09 07:09:45.875456 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:45.875435 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-w7hkl"] Jul 09 07:09:45.885211 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:45.885189 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-t4lsx"] Jul 09 07:09:45.908284 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:45.908232 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/56186d32-c8ce-464d-952a-e99ede3877fb-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-t4lsx\" (UID: \"56186d32-c8ce-464d-952a-e99ede3877fb\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t4lsx" Jul 09 07:09:45.908436 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:45.908310 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg7qm\" (UniqueName: \"kubernetes.io/projected/56186d32-c8ce-464d-952a-e99ede3877fb-kube-api-access-pg7qm\") pod \"llmisvc-controller-manager-777c6bd7d5-t4lsx\" (UID: \"56186d32-c8ce-464d-952a-e99ede3877fb\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t4lsx" Jul 09 07:09:45.908436 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:45.908354 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d180add6-9fcc-42bd-a214-5b594e1a35ee-cert\") pod \"kserve-controller-manager-d6f4b65b8-w7hkl\" (UID: \"d180add6-9fcc-42bd-a214-5b594e1a35ee\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-w7hkl" Jul 09 07:09:45.908436 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:45.908423 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp74g\" (UniqueName: \"kubernetes.io/projected/d180add6-9fcc-42bd-a214-5b594e1a35ee-kube-api-access-fp74g\") pod \"kserve-controller-manager-d6f4b65b8-w7hkl\" (UID: \"d180add6-9fcc-42bd-a214-5b594e1a35ee\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-w7hkl" Jul 09 07:09:46.009807 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:46.009765 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg7qm\" (UniqueName: \"kubernetes.io/projected/56186d32-c8ce-464d-952a-e99ede3877fb-kube-api-access-pg7qm\") pod \"llmisvc-controller-manager-777c6bd7d5-t4lsx\" (UID: \"56186d32-c8ce-464d-952a-e99ede3877fb\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t4lsx" Jul 09 07:09:46.009977 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:46.009821 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d180add6-9fcc-42bd-a214-5b594e1a35ee-cert\") pod \"kserve-controller-manager-d6f4b65b8-w7hkl\" (UID: \"d180add6-9fcc-42bd-a214-5b594e1a35ee\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-w7hkl" Jul 09 07:09:46.009977 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:46.009887 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp74g\" (UniqueName: \"kubernetes.io/projected/d180add6-9fcc-42bd-a214-5b594e1a35ee-kube-api-access-fp74g\") pod \"kserve-controller-manager-d6f4b65b8-w7hkl\" (UID: \"d180add6-9fcc-42bd-a214-5b594e1a35ee\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-w7hkl" Jul 09 07:09:46.009977 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:46.009926 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/56186d32-c8ce-464d-952a-e99ede3877fb-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-t4lsx\" (UID: \"56186d32-c8ce-464d-952a-e99ede3877fb\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t4lsx" Jul 09 07:09:46.010162 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:09:46.009994 2564 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 09 07:09:46.010162 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:09:46.010032 2564 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 09 07:09:46.010162 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:09:46.010063 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d180add6-9fcc-42bd-a214-5b594e1a35ee-cert podName:d180add6-9fcc-42bd-a214-5b594e1a35ee nodeName:}" failed. No retries permitted until 2026-07-09 07:09:46.510042733 +0000 UTC m=+445.561119956 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/d180add6-9fcc-42bd-a214-5b594e1a35ee-cert") pod "kserve-controller-manager-d6f4b65b8-w7hkl" (UID: "d180add6-9fcc-42bd-a214-5b594e1a35ee") : secret "kserve-webhook-server-cert" not found Jul 09 07:09:46.010162 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:09:46.010081 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56186d32-c8ce-464d-952a-e99ede3877fb-cert podName:56186d32-c8ce-464d-952a-e99ede3877fb nodeName:}" failed. No retries permitted until 2026-07-09 07:09:46.510072644 +0000 UTC m=+445.561149855 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/56186d32-c8ce-464d-952a-e99ede3877fb-cert") pod "llmisvc-controller-manager-777c6bd7d5-t4lsx" (UID: "56186d32-c8ce-464d-952a-e99ede3877fb") : secret "llmisvc-webhook-server-cert" not found Jul 09 07:09:46.023176 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:46.023111 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg7qm\" (UniqueName: \"kubernetes.io/projected/56186d32-c8ce-464d-952a-e99ede3877fb-kube-api-access-pg7qm\") pod \"llmisvc-controller-manager-777c6bd7d5-t4lsx\" (UID: \"56186d32-c8ce-464d-952a-e99ede3877fb\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t4lsx" Jul 09 07:09:46.023176 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:46.023143 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp74g\" (UniqueName: \"kubernetes.io/projected/d180add6-9fcc-42bd-a214-5b594e1a35ee-kube-api-access-fp74g\") pod \"kserve-controller-manager-d6f4b65b8-w7hkl\" (UID: \"d180add6-9fcc-42bd-a214-5b594e1a35ee\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-w7hkl" Jul 09 07:09:46.514046 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:46.514012 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d180add6-9fcc-42bd-a214-5b594e1a35ee-cert\") pod \"kserve-controller-manager-d6f4b65b8-w7hkl\" (UID: \"d180add6-9fcc-42bd-a214-5b594e1a35ee\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-w7hkl" Jul 09 07:09:46.514245 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:46.514220 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/56186d32-c8ce-464d-952a-e99ede3877fb-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-t4lsx\" (UID: \"56186d32-c8ce-464d-952a-e99ede3877fb\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t4lsx" Jul 09 07:09:46.516496 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:46.516473 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/56186d32-c8ce-464d-952a-e99ede3877fb-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-t4lsx\" (UID: \"56186d32-c8ce-464d-952a-e99ede3877fb\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t4lsx" Jul 09 07:09:46.516496 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:46.516492 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d180add6-9fcc-42bd-a214-5b594e1a35ee-cert\") pod \"kserve-controller-manager-d6f4b65b8-w7hkl\" (UID: \"d180add6-9fcc-42bd-a214-5b594e1a35ee\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-w7hkl" Jul 09 07:09:46.568965 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:46.568939 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7f58fd5d48-5p9cd"] Jul 09 07:09:46.571785 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:46.571766 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7f58fd5d48-5p9cd" Jul 09 07:09:46.574081 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:46.574059 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 09 07:09:46.575515 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:46.575479 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 09 07:09:46.575612 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:46.575532 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 09 07:09:46.576742 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:46.576719 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 09 07:09:46.576955 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:46.576937 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-89tpg\"" Jul 09 07:09:46.588848 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:46.588828 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7f58fd5d48-5p9cd"] Jul 09 07:09:46.615623 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:46.615596 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c9dh\" (UniqueName: \"kubernetes.io/projected/f99957f6-8710-4530-9d05-c261c96547d8-kube-api-access-2c9dh\") pod \"kube-auth-proxy-7f58fd5d48-5p9cd\" (UID: \"f99957f6-8710-4530-9d05-c261c96547d8\") " pod="openshift-ingress/kube-auth-proxy-7f58fd5d48-5p9cd" Jul 09 07:09:46.615808 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:46.615631 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f99957f6-8710-4530-9d05-c261c96547d8-tmp\") pod \"kube-auth-proxy-7f58fd5d48-5p9cd\" (UID: \"f99957f6-8710-4530-9d05-c261c96547d8\") " pod="openshift-ingress/kube-auth-proxy-7f58fd5d48-5p9cd" Jul 09 07:09:46.615808 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:46.615673 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f99957f6-8710-4530-9d05-c261c96547d8-tls-certs\") pod \"kube-auth-proxy-7f58fd5d48-5p9cd\" (UID: \"f99957f6-8710-4530-9d05-c261c96547d8\") " pod="openshift-ingress/kube-auth-proxy-7f58fd5d48-5p9cd" Jul 09 07:09:46.716745 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:46.716716 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c9dh\" (UniqueName: \"kubernetes.io/projected/f99957f6-8710-4530-9d05-c261c96547d8-kube-api-access-2c9dh\") pod \"kube-auth-proxy-7f58fd5d48-5p9cd\" (UID: \"f99957f6-8710-4530-9d05-c261c96547d8\") " pod="openshift-ingress/kube-auth-proxy-7f58fd5d48-5p9cd" Jul 09 07:09:46.716930 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:46.716753 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f99957f6-8710-4530-9d05-c261c96547d8-tmp\") pod \"kube-auth-proxy-7f58fd5d48-5p9cd\" (UID: \"f99957f6-8710-4530-9d05-c261c96547d8\") " pod="openshift-ingress/kube-auth-proxy-7f58fd5d48-5p9cd" Jul 09 07:09:46.716930 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:46.716875 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f99957f6-8710-4530-9d05-c261c96547d8-tls-certs\") pod \"kube-auth-proxy-7f58fd5d48-5p9cd\" (UID: \"f99957f6-8710-4530-9d05-c261c96547d8\") " pod="openshift-ingress/kube-auth-proxy-7f58fd5d48-5p9cd" Jul 09 07:09:46.717052 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:09:46.717027 2564 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 09 07:09:46.717116 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:09:46.717098 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f99957f6-8710-4530-9d05-c261c96547d8-tls-certs podName:f99957f6-8710-4530-9d05-c261c96547d8 nodeName:}" failed. No retries permitted until 2026-07-09 07:09:47.217076963 +0000 UTC m=+446.268154184 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/f99957f6-8710-4530-9d05-c261c96547d8-tls-certs") pod "kube-auth-proxy-7f58fd5d48-5p9cd" (UID: "f99957f6-8710-4530-9d05-c261c96547d8") : secret "kube-auth-proxy-tls" not found Jul 09 07:09:46.718982 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:46.718960 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f99957f6-8710-4530-9d05-c261c96547d8-tmp\") pod \"kube-auth-proxy-7f58fd5d48-5p9cd\" (UID: \"f99957f6-8710-4530-9d05-c261c96547d8\") " pod="openshift-ingress/kube-auth-proxy-7f58fd5d48-5p9cd" Jul 09 07:09:46.733890 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:46.733868 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c9dh\" (UniqueName: \"kubernetes.io/projected/f99957f6-8710-4530-9d05-c261c96547d8-kube-api-access-2c9dh\") pod \"kube-auth-proxy-7f58fd5d48-5p9cd\" (UID: \"f99957f6-8710-4530-9d05-c261c96547d8\") " pod="openshift-ingress/kube-auth-proxy-7f58fd5d48-5p9cd" Jul 09 07:09:46.776461 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:46.776388 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-w7hkl" Jul 09 07:09:46.789539 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:46.789514 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t4lsx" Jul 09 07:09:46.916972 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:46.916946 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-w7hkl"] Jul 09 07:09:46.919205 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:09:46.919173 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd180add6_9fcc_42bd_a214_5b594e1a35ee.slice/crio-d578f96057669c92c22bc226a0e07f92526737f837267c997b9dfc99c3a7ff09 WatchSource:0}: Error finding container d578f96057669c92c22bc226a0e07f92526737f837267c997b9dfc99c3a7ff09: Status 404 returned error can't find the container with id d578f96057669c92c22bc226a0e07f92526737f837267c997b9dfc99c3a7ff09 Jul 09 07:09:46.936351 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:46.936326 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-t4lsx"] Jul 09 07:09:46.939610 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:09:46.939585 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod56186d32_c8ce_464d_952a_e99ede3877fb.slice/crio-218e9a0c804855e03a4de8724f2729bd7585e22be28be7e6141e9cd3e68f823c WatchSource:0}: Error finding container 218e9a0c804855e03a4de8724f2729bd7585e22be28be7e6141e9cd3e68f823c: Status 404 returned error can't find the container with id 218e9a0c804855e03a4de8724f2729bd7585e22be28be7e6141e9cd3e68f823c Jul 09 07:09:47.074949 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:47.074851 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t4lsx" event={"ID":"56186d32-c8ce-464d-952a-e99ede3877fb","Type":"ContainerStarted","Data":"218e9a0c804855e03a4de8724f2729bd7585e22be28be7e6141e9cd3e68f823c"} Jul 09 07:09:47.076028 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:47.076002 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-w7hkl" event={"ID":"d180add6-9fcc-42bd-a214-5b594e1a35ee","Type":"ContainerStarted","Data":"d578f96057669c92c22bc226a0e07f92526737f837267c997b9dfc99c3a7ff09"} Jul 09 07:09:47.221965 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:47.221926 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f99957f6-8710-4530-9d05-c261c96547d8-tls-certs\") pod \"kube-auth-proxy-7f58fd5d48-5p9cd\" (UID: \"f99957f6-8710-4530-9d05-c261c96547d8\") " pod="openshift-ingress/kube-auth-proxy-7f58fd5d48-5p9cd" Jul 09 07:09:47.224468 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:47.224440 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f99957f6-8710-4530-9d05-c261c96547d8-tls-certs\") pod \"kube-auth-proxy-7f58fd5d48-5p9cd\" (UID: \"f99957f6-8710-4530-9d05-c261c96547d8\") " pod="openshift-ingress/kube-auth-proxy-7f58fd5d48-5p9cd" Jul 09 07:09:47.481555 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:47.481529 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7f58fd5d48-5p9cd" Jul 09 07:09:47.616581 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:47.616545 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7f58fd5d48-5p9cd"] Jul 09 07:09:47.661381 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:09:47.661344 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf99957f6_8710_4530_9d05_c261c96547d8.slice/crio-f11ee0c10a19824e7c416b2dc84e0b162fa5721961833542b9a18df2c5d3f685 WatchSource:0}: Error finding container f11ee0c10a19824e7c416b2dc84e0b162fa5721961833542b9a18df2c5d3f685: Status 404 returned error can't find the container with id f11ee0c10a19824e7c416b2dc84e0b162fa5721961833542b9a18df2c5d3f685 Jul 09 07:09:48.082028 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:48.081994 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7f58fd5d48-5p9cd" event={"ID":"f99957f6-8710-4530-9d05-c261c96547d8","Type":"ContainerStarted","Data":"f11ee0c10a19824e7c416b2dc84e0b162fa5721961833542b9a18df2c5d3f685"} Jul 09 07:09:48.620493 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:48.620454 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cwstw"] Jul 09 07:09:48.623966 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:48.623943 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cwstw" Jul 09 07:09:48.626729 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:48.626706 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 09 07:09:48.626729 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:48.626719 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 09 07:09:48.626891 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:48.626875 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 09 07:09:48.628596 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:48.628572 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-hhnwx\"" Jul 09 07:09:48.633161 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:48.629063 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 09 07:09:48.637278 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:48.637237 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cwstw"] Jul 09 07:09:48.737346 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:48.737300 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/742bbe04-c57f-4c45-b499-c0a0d4fa7b34-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-cwstw\" (UID: \"742bbe04-c57f-4c45-b499-c0a0d4fa7b34\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cwstw" Jul 09 07:09:48.737531 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:48.737389 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/742bbe04-c57f-4c45-b499-c0a0d4fa7b34-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-cwstw\" (UID: \"742bbe04-c57f-4c45-b499-c0a0d4fa7b34\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cwstw" Jul 09 07:09:48.737531 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:48.737504 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/742bbe04-c57f-4c45-b499-c0a0d4fa7b34-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-cwstw\" (UID: \"742bbe04-c57f-4c45-b499-c0a0d4fa7b34\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cwstw" Jul 09 07:09:48.737662 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:48.737542 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/742bbe04-c57f-4c45-b499-c0a0d4fa7b34-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-cwstw\" (UID: \"742bbe04-c57f-4c45-b499-c0a0d4fa7b34\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cwstw" Jul 09 07:09:48.737662 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:48.737573 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsgh9\" (UniqueName: \"kubernetes.io/projected/742bbe04-c57f-4c45-b499-c0a0d4fa7b34-kube-api-access-zsgh9\") pod \"istiod-openshift-gateway-94bb8fbfd-cwstw\" (UID: \"742bbe04-c57f-4c45-b499-c0a0d4fa7b34\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cwstw" Jul 09 07:09:48.737662 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:48.737625 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/742bbe04-c57f-4c45-b499-c0a0d4fa7b34-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-cwstw\" (UID: \"742bbe04-c57f-4c45-b499-c0a0d4fa7b34\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cwstw" Jul 09 07:09:48.737662 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:48.737655 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/742bbe04-c57f-4c45-b499-c0a0d4fa7b34-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-cwstw\" (UID: \"742bbe04-c57f-4c45-b499-c0a0d4fa7b34\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cwstw" Jul 09 07:09:48.840062 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:48.838564 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/742bbe04-c57f-4c45-b499-c0a0d4fa7b34-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-cwstw\" (UID: \"742bbe04-c57f-4c45-b499-c0a0d4fa7b34\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cwstw" Jul 09 07:09:48.840062 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:48.838641 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/742bbe04-c57f-4c45-b499-c0a0d4fa7b34-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-cwstw\" (UID: \"742bbe04-c57f-4c45-b499-c0a0d4fa7b34\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cwstw" Jul 09 07:09:48.840062 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:48.838753 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/742bbe04-c57f-4c45-b499-c0a0d4fa7b34-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-cwstw\" (UID: \"742bbe04-c57f-4c45-b499-c0a0d4fa7b34\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cwstw" Jul 09 07:09:48.840062 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:48.838795 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsgh9\" (UniqueName: \"kubernetes.io/projected/742bbe04-c57f-4c45-b499-c0a0d4fa7b34-kube-api-access-zsgh9\") pod \"istiod-openshift-gateway-94bb8fbfd-cwstw\" (UID: \"742bbe04-c57f-4c45-b499-c0a0d4fa7b34\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cwstw" Jul 09 07:09:48.840062 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:48.838889 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/742bbe04-c57f-4c45-b499-c0a0d4fa7b34-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-cwstw\" (UID: \"742bbe04-c57f-4c45-b499-c0a0d4fa7b34\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cwstw" Jul 09 07:09:48.840062 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:48.838936 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/742bbe04-c57f-4c45-b499-c0a0d4fa7b34-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-cwstw\" (UID: \"742bbe04-c57f-4c45-b499-c0a0d4fa7b34\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cwstw" Jul 09 07:09:48.840062 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:48.838995 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/742bbe04-c57f-4c45-b499-c0a0d4fa7b34-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-cwstw\" (UID: \"742bbe04-c57f-4c45-b499-c0a0d4fa7b34\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cwstw" Jul 09 07:09:48.841190 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:48.841154 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/742bbe04-c57f-4c45-b499-c0a0d4fa7b34-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-cwstw\" (UID: \"742bbe04-c57f-4c45-b499-c0a0d4fa7b34\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cwstw" Jul 09 07:09:48.842587 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:48.842543 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/742bbe04-c57f-4c45-b499-c0a0d4fa7b34-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-cwstw\" (UID: \"742bbe04-c57f-4c45-b499-c0a0d4fa7b34\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cwstw" Jul 09 07:09:48.844794 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:48.844748 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/742bbe04-c57f-4c45-b499-c0a0d4fa7b34-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-cwstw\" (UID: \"742bbe04-c57f-4c45-b499-c0a0d4fa7b34\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cwstw" Jul 09 07:09:48.844887 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:48.844826 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/742bbe04-c57f-4c45-b499-c0a0d4fa7b34-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-cwstw\" (UID: \"742bbe04-c57f-4c45-b499-c0a0d4fa7b34\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cwstw" Jul 09 07:09:48.848343 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:48.848295 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/742bbe04-c57f-4c45-b499-c0a0d4fa7b34-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-cwstw\" (UID: \"742bbe04-c57f-4c45-b499-c0a0d4fa7b34\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cwstw" Jul 09 07:09:48.850919 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:48.850878 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/742bbe04-c57f-4c45-b499-c0a0d4fa7b34-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-cwstw\" (UID: \"742bbe04-c57f-4c45-b499-c0a0d4fa7b34\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cwstw" Jul 09 07:09:48.852867 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:48.852825 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsgh9\" (UniqueName: \"kubernetes.io/projected/742bbe04-c57f-4c45-b499-c0a0d4fa7b34-kube-api-access-zsgh9\") pod \"istiod-openshift-gateway-94bb8fbfd-cwstw\" (UID: \"742bbe04-c57f-4c45-b499-c0a0d4fa7b34\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cwstw" Jul 09 07:09:48.942229 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:48.942127 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cwstw" Jul 09 07:09:49.159327 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:49.158490 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cwstw"] Jul 09 07:09:49.743649 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:09:49.743610 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod742bbe04_c57f_4c45_b499_c0a0d4fa7b34.slice/crio-207a9fa1b8af500e2419ddb45d8df4375fb5c34bcd5e8b962873b0d5ba81b5ce WatchSource:0}: Error finding container 207a9fa1b8af500e2419ddb45d8df4375fb5c34bcd5e8b962873b0d5ba81b5ce: Status 404 returned error can't find the container with id 207a9fa1b8af500e2419ddb45d8df4375fb5c34bcd5e8b962873b0d5ba81b5ce Jul 09 07:09:50.096165 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:50.096072 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cwstw" event={"ID":"742bbe04-c57f-4c45-b499-c0a0d4fa7b34","Type":"ContainerStarted","Data":"207a9fa1b8af500e2419ddb45d8df4375fb5c34bcd5e8b962873b0d5ba81b5ce"} Jul 09 07:09:52.054749 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:52.054718 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-dmzzk" Jul 09 07:09:52.056939 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:52.056916 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-n5xxf" Jul 09 07:09:52.108303 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:52.108172 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7f58fd5d48-5p9cd" event={"ID":"f99957f6-8710-4530-9d05-c261c96547d8","Type":"ContainerStarted","Data":"bbb62d1417e700824905e331f6035d708d2f40b21aa42992550ff99d7ed829ef"} Jul 09 07:09:52.110598 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:52.110551 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t4lsx" event={"ID":"56186d32-c8ce-464d-952a-e99ede3877fb","Type":"ContainerStarted","Data":"da0c79c2abe99b316d0d0a33f3bf6a6565658a7ef111d402ce21a77b245513a7"} Jul 09 07:09:52.110749 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:52.110623 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t4lsx" Jul 09 07:09:52.112427 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:52.112401 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-w7hkl" event={"ID":"d180add6-9fcc-42bd-a214-5b594e1a35ee","Type":"ContainerStarted","Data":"7ab483fe2d76bcd673275c59b8f5e6f5b59bdb17f98017dfe3f9d5b8d14d6082"} Jul 09 07:09:52.112535 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:52.112453 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-w7hkl" Jul 09 07:09:52.126558 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:52.126254 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7f58fd5d48-5p9cd" podStartSLOduration=2.6460542780000003 podStartE2EDuration="6.126234981s" totalImagesPullingTime="3.480180703s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 07:09:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 07:09:52.125517862 +0000 UTC m=+451.176595093" watchObservedRunningTime="2026-07-09 07:09:52.126234981 +0000 UTC m=+451.177312210" Jul 09 07:09:52.145271 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:52.145216 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t4lsx" podStartSLOduration=2.949730348 podStartE2EDuration="7.145196305s" totalImagesPullingTime="4.195465957s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 07:09:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 07:09:52.143451947 +0000 UTC m=+451.194529200" watchObservedRunningTime="2026-07-09 07:09:52.145196305 +0000 UTC m=+451.196273535" Jul 09 07:09:52.161668 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:52.161617 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-w7hkl" podStartSLOduration=2.9458890110000002 podStartE2EDuration="7.161597873s" totalImagesPullingTime="4.215708862s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 07:09:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 07:09:52.160447072 +0000 UTC m=+451.211524303" watchObservedRunningTime="2026-07-09 07:09:52.161597873 +0000 UTC m=+451.212675101" Jul 09 07:09:52.601689 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:52.601629 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 07:09:52.601828 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:52.601710 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 07:09:53.118208 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:53.118168 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cwstw" event={"ID":"742bbe04-c57f-4c45-b499-c0a0d4fa7b34","Type":"ContainerStarted","Data":"b7dc9915109ca5a5ad3542c3bd4bf548cc0dabd0bf84bcc1390ad9755055fb18"} Jul 09 07:09:53.118686 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:53.118521 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cwstw" Jul 09 07:09:53.119913 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:53.119890 2564 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-cwstw container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 09 07:09:53.120021 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:53.119963 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cwstw" podUID="742bbe04-c57f-4c45-b499-c0a0d4fa7b34" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 07:09:53.141164 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:53.141116 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cwstw" podStartSLOduration=2.285574808 podStartE2EDuration="5.141102272s" totalImagesPullingTime="2.855527464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 07:09:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 07:09:53.140155022 +0000 UTC m=+452.191232251" watchObservedRunningTime="2026-07-09 07:09:53.141102272 +0000 UTC m=+452.192179501" Jul 09 07:09:54.122418 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:54.122382 2564 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-cwstw container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 09 07:09:54.122799 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:54.122437 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cwstw" podUID="742bbe04-c57f-4c45-b499-c0a0d4fa7b34" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 07:09:57.122536 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:09:57.122509 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cwstw" Jul 09 07:10:23.119867 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:10:23.119836 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t4lsx" Jul 09 07:10:23.123475 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:10:23.123455 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-w7hkl" Jul 09 07:11:15.821287 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:11:15.821199 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2dcxc"] Jul 09 07:11:15.823414 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:11:15.823399 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2dcxc" Jul 09 07:11:15.830826 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:11:15.830800 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 09 07:11:15.831882 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:11:15.831861 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 09 07:11:15.831990 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:11:15.831867 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-cd8z6\"" Jul 09 07:11:15.844112 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:11:15.844089 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2dcxc"] Jul 09 07:11:15.888187 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:11:15.888151 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfrnb\" (UniqueName: \"kubernetes.io/projected/df99a728-d8a0-4cc6-9154-fc9eddb44850-kube-api-access-hfrnb\") pod \"limitador-operator-controller-manager-859d7b78bd-2dcxc\" (UID: \"df99a728-d8a0-4cc6-9154-fc9eddb44850\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2dcxc" Jul 09 07:11:15.989153 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:11:15.989121 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfrnb\" (UniqueName: \"kubernetes.io/projected/df99a728-d8a0-4cc6-9154-fc9eddb44850-kube-api-access-hfrnb\") pod \"limitador-operator-controller-manager-859d7b78bd-2dcxc\" (UID: \"df99a728-d8a0-4cc6-9154-fc9eddb44850\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2dcxc" Jul 09 07:11:16.009377 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:11:16.009346 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfrnb\" (UniqueName: \"kubernetes.io/projected/df99a728-d8a0-4cc6-9154-fc9eddb44850-kube-api-access-hfrnb\") pod \"limitador-operator-controller-manager-859d7b78bd-2dcxc\" (UID: \"df99a728-d8a0-4cc6-9154-fc9eddb44850\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2dcxc" Jul 09 07:11:16.135470 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:11:16.135438 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2dcxc" Jul 09 07:11:16.275537 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:11:16.275505 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2dcxc"] Jul 09 07:11:16.279521 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:11:16.279495 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf99a728_d8a0_4cc6_9154_fc9eddb44850.slice/crio-c55badd7ce8ac028e00aec1be29604bd3386bf7bafdf422d9f2d2eb979875740 WatchSource:0}: Error finding container c55badd7ce8ac028e00aec1be29604bd3386bf7bafdf422d9f2d2eb979875740: Status 404 returned error can't find the container with id c55badd7ce8ac028e00aec1be29604bd3386bf7bafdf422d9f2d2eb979875740 Jul 09 07:11:16.461969 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:11:16.461884 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2dcxc" event={"ID":"df99a728-d8a0-4cc6-9154-fc9eddb44850","Type":"ContainerStarted","Data":"c55badd7ce8ac028e00aec1be29604bd3386bf7bafdf422d9f2d2eb979875740"} Jul 09 07:11:19.476012 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:11:19.475983 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2dcxc" event={"ID":"df99a728-d8a0-4cc6-9154-fc9eddb44850","Type":"ContainerStarted","Data":"6d1628ac8ccf68cfcdb0e5c2c0054af5b123c1346911dfba255198ff979ae800"} Jul 09 07:11:19.476396 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:11:19.476090 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2dcxc" Jul 09 07:11:19.539173 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:11:19.539124 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2dcxc" podStartSLOduration=2.388982547 podStartE2EDuration="4.539110784s" totalImagesPullingTime="2.150128237s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 07:11:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 07:11:19.537913449 +0000 UTC m=+538.588990675" watchObservedRunningTime="2026-07-09 07:11:19.539110784 +0000 UTC m=+538.590188011" Jul 09 07:11:30.482394 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:11:30.482364 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2dcxc" Jul 09 07:11:41.314635 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:11:41.314599 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2dcxc"] Jul 09 07:11:41.315040 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:11:41.314837 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2dcxc" podUID="df99a728-d8a0-4cc6-9154-fc9eddb44850" containerName="manager" containerID="cri-o://6d1628ac8ccf68cfcdb0e5c2c0054af5b123c1346911dfba255198ff979ae800" gracePeriod=2 Jul 09 07:11:41.323380 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:11:41.323357 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2dcxc"] Jul 09 07:11:41.359666 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:11:41.359630 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xs7w2"] Jul 09 07:11:41.360006 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:11:41.359991 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df99a728-d8a0-4cc6-9154-fc9eddb44850" containerName="manager" Jul 09 07:11:41.360006 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:11:41.360007 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="df99a728-d8a0-4cc6-9154-fc9eddb44850" containerName="manager" Jul 09 07:11:41.360133 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:11:41.360070 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="df99a728-d8a0-4cc6-9154-fc9eddb44850" containerName="manager" Jul 09 07:11:41.362979 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:11:41.362957 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xs7w2" Jul 09 07:11:41.365231 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:11:41.365200 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-2dcxc\" is forbidden: User \"system:node:ip-10-0-138-0.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-0.ec2.internal' and this object" podUID="df99a728-d8a0-4cc6-9154-fc9eddb44850" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2dcxc" Jul 09 07:11:41.467086 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:11:41.467046 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-2dcxc\" is forbidden: User \"system:node:ip-10-0-138-0.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-0.ec2.internal' and this object" podUID="df99a728-d8a0-4cc6-9154-fc9eddb44850" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2dcxc" Jul 09 07:11:41.468793 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:11:41.468770 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xs7w2"] Jul 09 07:11:41.504875 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:11:41.504851 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkn2f\" (UniqueName: \"kubernetes.io/projected/84c308a6-03ac-4cd9-9e36-cb37b74957a3-kube-api-access-tkn2f\") pod \"limitador-operator-controller-manager-859d7b78bd-xs7w2\" (UID: \"84c308a6-03ac-4cd9-9e36-cb37b74957a3\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xs7w2" Jul 09 07:11:41.556535 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:11:41.556513 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2dcxc" Jul 09 07:11:41.571436 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:11:41.571380 2564 generic.go:356] "Generic (PLEG): container finished" podID="df99a728-d8a0-4cc6-9154-fc9eddb44850" containerID="6d1628ac8ccf68cfcdb0e5c2c0054af5b123c1346911dfba255198ff979ae800" exitCode=0 Jul 09 07:11:41.571436 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:11:41.571412 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2dcxc" Jul 09 07:11:41.571569 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:11:41.571485 2564 scope.go:117] "RemoveContainer" containerID="6d1628ac8ccf68cfcdb0e5c2c0054af5b123c1346911dfba255198ff979ae800" Jul 09 07:11:41.579168 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:11:41.579151 2564 scope.go:117] "RemoveContainer" containerID="6d1628ac8ccf68cfcdb0e5c2c0054af5b123c1346911dfba255198ff979ae800" Jul 09 07:11:41.579475 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:11:41.579457 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d1628ac8ccf68cfcdb0e5c2c0054af5b123c1346911dfba255198ff979ae800\": container with ID starting with 6d1628ac8ccf68cfcdb0e5c2c0054af5b123c1346911dfba255198ff979ae800 not found: ID does not exist" containerID="6d1628ac8ccf68cfcdb0e5c2c0054af5b123c1346911dfba255198ff979ae800" Jul 09 07:11:41.579548 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:11:41.579487 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d1628ac8ccf68cfcdb0e5c2c0054af5b123c1346911dfba255198ff979ae800"} err="failed to get container status \"6d1628ac8ccf68cfcdb0e5c2c0054af5b123c1346911dfba255198ff979ae800\": rpc error: code = NotFound desc = could not find container \"6d1628ac8ccf68cfcdb0e5c2c0054af5b123c1346911dfba255198ff979ae800\": container with ID starting with 6d1628ac8ccf68cfcdb0e5c2c0054af5b123c1346911dfba255198ff979ae800 not found: ID does not exist" Jul 09 07:11:41.606310 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:11:41.606282 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkn2f\" (UniqueName: \"kubernetes.io/projected/84c308a6-03ac-4cd9-9e36-cb37b74957a3-kube-api-access-tkn2f\") pod \"limitador-operator-controller-manager-859d7b78bd-xs7w2\" (UID: \"84c308a6-03ac-4cd9-9e36-cb37b74957a3\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xs7w2" Jul 09 07:11:41.622216 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:11:41.622187 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkn2f\" (UniqueName: \"kubernetes.io/projected/84c308a6-03ac-4cd9-9e36-cb37b74957a3-kube-api-access-tkn2f\") pod \"limitador-operator-controller-manager-859d7b78bd-xs7w2\" (UID: \"84c308a6-03ac-4cd9-9e36-cb37b74957a3\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xs7w2" Jul 09 07:11:41.679638 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:11:41.679611 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xs7w2" Jul 09 07:11:41.707010 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:11:41.706983 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hfrnb\" (UniqueName: \"kubernetes.io/projected/df99a728-d8a0-4cc6-9154-fc9eddb44850-kube-api-access-hfrnb\") pod \"df99a728-d8a0-4cc6-9154-fc9eddb44850\" (UID: \"df99a728-d8a0-4cc6-9154-fc9eddb44850\") " Jul 09 07:11:41.708998 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:11:41.708972 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df99a728-d8a0-4cc6-9154-fc9eddb44850-kube-api-access-hfrnb" (OuterVolumeSpecName: "kube-api-access-hfrnb") pod "df99a728-d8a0-4cc6-9154-fc9eddb44850" (UID: "df99a728-d8a0-4cc6-9154-fc9eddb44850"). InnerVolumeSpecName "kube-api-access-hfrnb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 07:11:41.808169 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:11:41.808140 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfrnb\" (UniqueName: \"kubernetes.io/projected/df99a728-d8a0-4cc6-9154-fc9eddb44850-kube-api-access-hfrnb\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:11:41.826044 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:11:41.825893 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xs7w2"] Jul 09 07:11:41.828718 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:11:41.828695 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84c308a6_03ac_4cd9_9e36_cb37b74957a3.slice/crio-ae3a6d1194dfd340123cfebb8f08c3e85efdf69dd2a25f577bd0c38ce1824702 WatchSource:0}: Error finding container ae3a6d1194dfd340123cfebb8f08c3e85efdf69dd2a25f577bd0c38ce1824702: Status 404 returned error can't find the container with id ae3a6d1194dfd340123cfebb8f08c3e85efdf69dd2a25f577bd0c38ce1824702 Jul 09 07:11:42.453868 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:11:42.453834 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j5mlg"] Jul 09 07:11:42.457010 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:11:42.456989 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j5mlg" Jul 09 07:11:42.459403 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:11:42.459375 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-j5mlg\" is forbidden: User \"system:node:ip-10-0-138-0.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-0.ec2.internal' and this object" podUID="e7c76921-ab10-4b26-8fd2-48d88a3254c5" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j5mlg" Jul 09 07:11:42.459638 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:11:42.459610 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"kuadrant-operator-controller-manager-dockercfg-gw8h4\" is forbidden: User \"system:node:ip-10-0-138-0.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-0.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-gw8h4\"" type="*v1.Secret" Jul 09 07:11:42.478971 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:11:42.478941 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j5mlg"] Jul 09 07:11:42.577012 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:11:42.576984 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xs7w2" event={"ID":"84c308a6-03ac-4cd9-9e36-cb37b74957a3","Type":"ContainerStarted","Data":"ea76fad121107ef95279b823e31212b429f4449aa5dcc9fb46c0d22947952922"} Jul 09 07:11:42.577174 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:11:42.577028 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xs7w2" event={"ID":"84c308a6-03ac-4cd9-9e36-cb37b74957a3","Type":"ContainerStarted","Data":"ae3a6d1194dfd340123cfebb8f08c3e85efdf69dd2a25f577bd0c38ce1824702"} Jul 09 07:11:42.577174 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:11:42.577076 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xs7w2" Jul 09 07:11:42.613594 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:11:42.613503 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xs7w2" podStartSLOduration=1.613486094 podStartE2EDuration="1.613486094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 07:11:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 07:11:42.609825932 +0000 UTC m=+561.660903161" watchObservedRunningTime="2026-07-09 07:11:42.613486094 +0000 UTC m=+561.664563384" Jul 09 07:11:42.614470 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:11:42.614445 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7c76921-ab10-4b26-8fd2-48d88a3254c5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-j5mlg\" (UID: \"e7c76921-ab10-4b26-8fd2-48d88a3254c5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j5mlg" Jul 09 07:11:42.614571 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:11:42.614492 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phfgm\" (UniqueName: \"kubernetes.io/projected/e7c76921-ab10-4b26-8fd2-48d88a3254c5-kube-api-access-phfgm\") pod \"kuadrant-operator-controller-manager-86878c769f-j5mlg\" (UID: \"e7c76921-ab10-4b26-8fd2-48d88a3254c5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j5mlg" Jul 09 07:11:42.715092 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:11:42.715058 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7c76921-ab10-4b26-8fd2-48d88a3254c5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-j5mlg\" (UID: \"e7c76921-ab10-4b26-8fd2-48d88a3254c5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j5mlg" Jul 09 07:11:42.715235 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:11:42.715127 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phfgm\" (UniqueName: \"kubernetes.io/projected/e7c76921-ab10-4b26-8fd2-48d88a3254c5-kube-api-access-phfgm\") pod \"kuadrant-operator-controller-manager-86878c769f-j5mlg\" (UID: \"e7c76921-ab10-4b26-8fd2-48d88a3254c5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j5mlg" Jul 09 07:11:42.715489 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:11:42.715471 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7c76921-ab10-4b26-8fd2-48d88a3254c5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-j5mlg\" (UID: \"e7c76921-ab10-4b26-8fd2-48d88a3254c5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j5mlg" Jul 09 07:11:42.729504 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:11:42.729475 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phfgm\" (UniqueName: \"kubernetes.io/projected/e7c76921-ab10-4b26-8fd2-48d88a3254c5-kube-api-access-phfgm\") pod \"kuadrant-operator-controller-manager-86878c769f-j5mlg\" (UID: \"e7c76921-ab10-4b26-8fd2-48d88a3254c5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j5mlg" Jul 09 07:11:43.463239 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:11:43.463207 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df99a728-d8a0-4cc6-9154-fc9eddb44850" path="/var/lib/kubelet/pods/df99a728-d8a0-4cc6-9154-fc9eddb44850/volumes" Jul 09 07:11:43.611752 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:11:43.611720 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-gw8h4\"" Jul 09 07:11:43.617384 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:11:43.617370 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j5mlg" Jul 09 07:11:43.764835 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:11:43.764758 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j5mlg"] Jul 09 07:11:43.767812 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:11:43.767781 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7c76921_ab10_4b26_8fd2_48d88a3254c5.slice/crio-7820e3ef4964e54ae30cf466dfc830033fe89ca211bd918f2eb320ed99f43922 WatchSource:0}: Error finding container 7820e3ef4964e54ae30cf466dfc830033fe89ca211bd918f2eb320ed99f43922: Status 404 returned error can't find the container with id 7820e3ef4964e54ae30cf466dfc830033fe89ca211bd918f2eb320ed99f43922 Jul 09 07:11:44.588117 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:11:44.588083 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j5mlg" event={"ID":"e7c76921-ab10-4b26-8fd2-48d88a3254c5","Type":"ContainerStarted","Data":"7820e3ef4964e54ae30cf466dfc830033fe89ca211bd918f2eb320ed99f43922"} Jul 09 07:11:49.615233 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:11:49.615199 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j5mlg" event={"ID":"e7c76921-ab10-4b26-8fd2-48d88a3254c5","Type":"ContainerStarted","Data":"8163b0a59d1f3c80dfd2089e403583c38a4928a5cabd4f238ff684fd8dfe7522"} Jul 09 07:11:49.615661 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:11:49.615316 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j5mlg" Jul 09 07:11:49.648231 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:11:49.648170 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j5mlg" podStartSLOduration=2.588721811 podStartE2EDuration="7.648149035s" totalImagesPullingTime="5.059427224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 07:11:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 07:11:49.647156281 +0000 UTC m=+568.698233511" watchObservedRunningTime="2026-07-09 07:11:49.648149035 +0000 UTC m=+568.699226265" Jul 09 07:11:53.585017 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:11:53.584987 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xs7w2" Jul 09 07:12:00.625997 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:12:00.625960 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j5mlg" Jul 09 07:13:00.230544 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:13:00.230509 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5ff476bffb-gmm5r"] Jul 09 07:13:00.233131 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:13:00.233107 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5ff476bffb-gmm5r" Jul 09 07:13:00.235477 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:13:00.235449 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 09 07:13:00.236007 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:13:00.235989 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-mxwxw\"" Jul 09 07:13:00.236847 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:13:00.236824 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 09 07:13:00.243528 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:13:00.243503 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5ff476bffb-gmm5r"] Jul 09 07:13:00.353840 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:13:00.353803 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/aeb6d9e8-ed36-4826-b51c-e368f0d01fdb-webhook-cert\") pod \"maas-controller-5ff476bffb-gmm5r\" (UID: \"aeb6d9e8-ed36-4826-b51c-e368f0d01fdb\") " pod="opendatahub/maas-controller-5ff476bffb-gmm5r" Jul 09 07:13:00.353999 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:13:00.353854 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4dzf\" (UniqueName: \"kubernetes.io/projected/aeb6d9e8-ed36-4826-b51c-e368f0d01fdb-kube-api-access-t4dzf\") pod \"maas-controller-5ff476bffb-gmm5r\" (UID: \"aeb6d9e8-ed36-4826-b51c-e368f0d01fdb\") " pod="opendatahub/maas-controller-5ff476bffb-gmm5r" Jul 09 07:13:00.454616 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:13:00.454548 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/aeb6d9e8-ed36-4826-b51c-e368f0d01fdb-webhook-cert\") pod \"maas-controller-5ff476bffb-gmm5r\" (UID: \"aeb6d9e8-ed36-4826-b51c-e368f0d01fdb\") " pod="opendatahub/maas-controller-5ff476bffb-gmm5r" Jul 09 07:13:00.454616 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:13:00.454603 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4dzf\" (UniqueName: \"kubernetes.io/projected/aeb6d9e8-ed36-4826-b51c-e368f0d01fdb-kube-api-access-t4dzf\") pod \"maas-controller-5ff476bffb-gmm5r\" (UID: \"aeb6d9e8-ed36-4826-b51c-e368f0d01fdb\") " pod="opendatahub/maas-controller-5ff476bffb-gmm5r" Jul 09 07:13:00.457237 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:13:00.457210 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/aeb6d9e8-ed36-4826-b51c-e368f0d01fdb-webhook-cert\") pod \"maas-controller-5ff476bffb-gmm5r\" (UID: \"aeb6d9e8-ed36-4826-b51c-e368f0d01fdb\") " pod="opendatahub/maas-controller-5ff476bffb-gmm5r" Jul 09 07:13:00.462058 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:13:00.462037 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4dzf\" (UniqueName: \"kubernetes.io/projected/aeb6d9e8-ed36-4826-b51c-e368f0d01fdb-kube-api-access-t4dzf\") pod \"maas-controller-5ff476bffb-gmm5r\" (UID: \"aeb6d9e8-ed36-4826-b51c-e368f0d01fdb\") " pod="opendatahub/maas-controller-5ff476bffb-gmm5r" Jul 09 07:13:00.544050 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:13:00.543984 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5ff476bffb-gmm5r" Jul 09 07:13:00.671230 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:13:00.671208 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5ff476bffb-gmm5r"] Jul 09 07:13:00.673620 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:13:00.673593 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeb6d9e8_ed36_4826_b51c_e368f0d01fdb.slice/crio-dbdfe98ff0d7fe0245d2af7df86e156a6ead0ed8a0252b470c1aa0db0d625e7a WatchSource:0}: Error finding container dbdfe98ff0d7fe0245d2af7df86e156a6ead0ed8a0252b470c1aa0db0d625e7a: Status 404 returned error can't find the container with id dbdfe98ff0d7fe0245d2af7df86e156a6ead0ed8a0252b470c1aa0db0d625e7a Jul 09 07:13:00.929452 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:13:00.929417 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5ff476bffb-gmm5r" event={"ID":"aeb6d9e8-ed36-4826-b51c-e368f0d01fdb","Type":"ContainerStarted","Data":"dbdfe98ff0d7fe0245d2af7df86e156a6ead0ed8a0252b470c1aa0db0d625e7a"} Jul 09 07:13:02.941912 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:13:02.941874 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5ff476bffb-gmm5r" event={"ID":"aeb6d9e8-ed36-4826-b51c-e368f0d01fdb","Type":"ContainerStarted","Data":"03c57da8d40063b6ca0b39193af539635bfbb2b9f4d5e819c114f8d17087685f"} Jul 09 07:13:02.942387 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:13:02.942017 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5ff476bffb-gmm5r" Jul 09 07:13:02.966118 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:13:02.966070 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5ff476bffb-gmm5r" podStartSLOduration=0.85802105 podStartE2EDuration="2.966056161s" totalImagesPullingTime="2.108035111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 07:13:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 07:13:02.964879025 +0000 UTC m=+642.015956262" watchObservedRunningTime="2026-07-09 07:13:02.966056161 +0000 UTC m=+642.017133389" Jul 09 07:13:13.952910 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:13:13.952879 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5ff476bffb-gmm5r" Jul 09 07:13:14.741228 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:13:14.741194 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5ff476bffb-gmm5r"] Jul 09 07:13:14.741418 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:13:14.741371 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-5ff476bffb-gmm5r" podUID="aeb6d9e8-ed36-4826-b51c-e368f0d01fdb" containerName="manager" containerID="cri-o://03c57da8d40063b6ca0b39193af539635bfbb2b9f4d5e819c114f8d17087685f" gracePeriod=30 Jul 09 07:13:14.983101 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:13:14.983077 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5ff476bffb-gmm5r" Jul 09 07:13:14.997517 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:13:14.997467 2564 generic.go:356] "Generic (PLEG): container finished" podID="aeb6d9e8-ed36-4826-b51c-e368f0d01fdb" containerID="03c57da8d40063b6ca0b39193af539635bfbb2b9f4d5e819c114f8d17087685f" exitCode=0 Jul 09 07:13:14.997517 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:13:14.997503 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5ff476bffb-gmm5r" Jul 09 07:13:14.997664 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:13:14.997540 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5ff476bffb-gmm5r" event={"ID":"aeb6d9e8-ed36-4826-b51c-e368f0d01fdb","Type":"ContainerDied","Data":"03c57da8d40063b6ca0b39193af539635bfbb2b9f4d5e819c114f8d17087685f"} Jul 09 07:13:14.997664 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:13:14.997582 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5ff476bffb-gmm5r" event={"ID":"aeb6d9e8-ed36-4826-b51c-e368f0d01fdb","Type":"ContainerDied","Data":"dbdfe98ff0d7fe0245d2af7df86e156a6ead0ed8a0252b470c1aa0db0d625e7a"} Jul 09 07:13:14.997664 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:13:14.997606 2564 scope.go:117] "RemoveContainer" containerID="03c57da8d40063b6ca0b39193af539635bfbb2b9f4d5e819c114f8d17087685f" Jul 09 07:13:15.007555 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:13:15.007537 2564 scope.go:117] "RemoveContainer" containerID="03c57da8d40063b6ca0b39193af539635bfbb2b9f4d5e819c114f8d17087685f" Jul 09 07:13:15.007776 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:13:15.007758 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03c57da8d40063b6ca0b39193af539635bfbb2b9f4d5e819c114f8d17087685f\": container with ID starting with 03c57da8d40063b6ca0b39193af539635bfbb2b9f4d5e819c114f8d17087685f not found: ID does not exist" containerID="03c57da8d40063b6ca0b39193af539635bfbb2b9f4d5e819c114f8d17087685f" Jul 09 07:13:15.007869 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:13:15.007783 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03c57da8d40063b6ca0b39193af539635bfbb2b9f4d5e819c114f8d17087685f"} err="failed to get container status \"03c57da8d40063b6ca0b39193af539635bfbb2b9f4d5e819c114f8d17087685f\": rpc error: code = NotFound desc = could not find container \"03c57da8d40063b6ca0b39193af539635bfbb2b9f4d5e819c114f8d17087685f\": container with ID starting with 03c57da8d40063b6ca0b39193af539635bfbb2b9f4d5e819c114f8d17087685f not found: ID does not exist" Jul 09 07:13:15.071804 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:13:15.071779 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t4dzf\" (UniqueName: \"kubernetes.io/projected/aeb6d9e8-ed36-4826-b51c-e368f0d01fdb-kube-api-access-t4dzf\") pod \"aeb6d9e8-ed36-4826-b51c-e368f0d01fdb\" (UID: \"aeb6d9e8-ed36-4826-b51c-e368f0d01fdb\") " Jul 09 07:13:15.071904 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:13:15.071864 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/aeb6d9e8-ed36-4826-b51c-e368f0d01fdb-webhook-cert\") pod \"aeb6d9e8-ed36-4826-b51c-e368f0d01fdb\" (UID: \"aeb6d9e8-ed36-4826-b51c-e368f0d01fdb\") " Jul 09 07:13:15.073877 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:13:15.073855 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeb6d9e8-ed36-4826-b51c-e368f0d01fdb-kube-api-access-t4dzf" (OuterVolumeSpecName: "kube-api-access-t4dzf") pod "aeb6d9e8-ed36-4826-b51c-e368f0d01fdb" (UID: "aeb6d9e8-ed36-4826-b51c-e368f0d01fdb"). InnerVolumeSpecName "kube-api-access-t4dzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 07:13:15.073930 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:13:15.073868 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aeb6d9e8-ed36-4826-b51c-e368f0d01fdb-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "aeb6d9e8-ed36-4826-b51c-e368f0d01fdb" (UID: "aeb6d9e8-ed36-4826-b51c-e368f0d01fdb"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 07:13:15.173139 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:13:15.173111 2564 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/aeb6d9e8-ed36-4826-b51c-e368f0d01fdb-webhook-cert\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:13:15.173139 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:13:15.173135 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4dzf\" (UniqueName: \"kubernetes.io/projected/aeb6d9e8-ed36-4826-b51c-e368f0d01fdb-kube-api-access-t4dzf\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:13:15.318806 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:13:15.318772 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5ff476bffb-gmm5r"] Jul 09 07:13:15.322530 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:13:15.322509 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5ff476bffb-gmm5r"] Jul 09 07:13:15.462207 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:13:15.462172 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aeb6d9e8-ed36-4826-b51c-e368f0d01fdb" path="/var/lib/kubelet/pods/aeb6d9e8-ed36-4826-b51c-e368f0d01fdb/volumes" Jul 09 07:13:36.783495 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:13:36.783462 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-mh754"] Jul 09 07:13:36.783968 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:13:36.783826 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aeb6d9e8-ed36-4826-b51c-e368f0d01fdb" containerName="manager" Jul 09 07:13:36.783968 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:13:36.783840 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="aeb6d9e8-ed36-4826-b51c-e368f0d01fdb" containerName="manager" Jul 09 07:13:36.783968 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:13:36.783891 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="aeb6d9e8-ed36-4826-b51c-e368f0d01fdb" containerName="manager" Jul 09 07:13:36.787564 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:13:36.787542 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-mh754" Jul 09 07:13:36.792285 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:13:36.792249 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 09 07:13:36.792406 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:13:36.792325 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-m45gv\"" Jul 09 07:13:36.798679 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:13:36.798655 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-mh754"] Jul 09 07:13:36.950922 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:13:36.950885 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b09281f9-6b47-4340-999f-5d9e3d19f782-plugins-config-volume\") pod \"payload-pre-processing-7964457696-mh754\" (UID: \"b09281f9-6b47-4340-999f-5d9e3d19f782\") " pod="openshift-ingress/payload-pre-processing-7964457696-mh754" Jul 09 07:13:36.950922 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:13:36.950924 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b5tp\" (UniqueName: \"kubernetes.io/projected/b09281f9-6b47-4340-999f-5d9e3d19f782-kube-api-access-8b5tp\") pod \"payload-pre-processing-7964457696-mh754\" (UID: \"b09281f9-6b47-4340-999f-5d9e3d19f782\") " pod="openshift-ingress/payload-pre-processing-7964457696-mh754" Jul 09 07:13:37.051752 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:13:37.051675 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b09281f9-6b47-4340-999f-5d9e3d19f782-plugins-config-volume\") pod \"payload-pre-processing-7964457696-mh754\" (UID: \"b09281f9-6b47-4340-999f-5d9e3d19f782\") " pod="openshift-ingress/payload-pre-processing-7964457696-mh754" Jul 09 07:13:37.051752 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:13:37.051713 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b5tp\" (UniqueName: \"kubernetes.io/projected/b09281f9-6b47-4340-999f-5d9e3d19f782-kube-api-access-8b5tp\") pod \"payload-pre-processing-7964457696-mh754\" (UID: \"b09281f9-6b47-4340-999f-5d9e3d19f782\") " pod="openshift-ingress/payload-pre-processing-7964457696-mh754" Jul 09 07:13:37.052397 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:13:37.052378 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b09281f9-6b47-4340-999f-5d9e3d19f782-plugins-config-volume\") pod \"payload-pre-processing-7964457696-mh754\" (UID: \"b09281f9-6b47-4340-999f-5d9e3d19f782\") " pod="openshift-ingress/payload-pre-processing-7964457696-mh754" Jul 09 07:13:37.080503 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:13:37.080477 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b5tp\" (UniqueName: \"kubernetes.io/projected/b09281f9-6b47-4340-999f-5d9e3d19f782-kube-api-access-8b5tp\") pod \"payload-pre-processing-7964457696-mh754\" (UID: \"b09281f9-6b47-4340-999f-5d9e3d19f782\") " pod="openshift-ingress/payload-pre-processing-7964457696-mh754" Jul 09 07:13:37.097891 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:13:37.097862 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-mh754" Jul 09 07:13:37.226704 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:13:37.226672 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-mh754"] Jul 09 07:13:37.228403 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:13:37.228376 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb09281f9_6b47_4340_999f_5d9e3d19f782.slice/crio-ca0b239b251d287af2451e0ddfbfa3c02c06f66b7282ffcc14aca4e5efe2e4ab WatchSource:0}: Error finding container ca0b239b251d287af2451e0ddfbfa3c02c06f66b7282ffcc14aca4e5efe2e4ab: Status 404 returned error can't find the container with id ca0b239b251d287af2451e0ddfbfa3c02c06f66b7282ffcc14aca4e5efe2e4ab Jul 09 07:13:37.230211 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:13:37.230195 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 07:13:38.095906 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:13:38.095876 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-mh754" event={"ID":"b09281f9-6b47-4340-999f-5d9e3d19f782","Type":"ContainerStarted","Data":"ca0b239b251d287af2451e0ddfbfa3c02c06f66b7282ffcc14aca4e5efe2e4ab"} Jul 09 07:13:39.100869 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:13:39.100793 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-mh754" event={"ID":"b09281f9-6b47-4340-999f-5d9e3d19f782","Type":"ContainerStarted","Data":"e720f1118a15b7a63a0964c180e8c43d750cec6254f21908ce1b6f92f0cf6cf9"} Jul 09 07:13:39.101220 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:13:39.100890 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7964457696-mh754" Jul 09 07:13:39.116593 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:13:39.116549 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7964457696-mh754" podStartSLOduration=1.6774864809999999 podStartE2EDuration="3.116533539s" totalImagesPullingTime="1.439047058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 07:13:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 07:13:39.115588037 +0000 UTC m=+678.166665266" watchObservedRunningTime="2026-07-09 07:13:39.116533539 +0000 UTC m=+678.167610764" Jul 09 07:13:50.107104 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:13:50.107074 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7964457696-mh754" Jul 09 07:14:13.684774 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:14:13.684697 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bkszb"] Jul 09 07:14:13.688676 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:14:13.688656 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bkszb" Jul 09 07:14:13.691644 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:14:13.691619 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-rch4x\"" Jul 09 07:14:13.691780 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:14:13.691660 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 09 07:14:13.691845 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:14:13.691786 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 09 07:14:13.692505 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:14:13.692485 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 09 07:14:13.700517 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:14:13.700494 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bkszb"] Jul 09 07:14:13.751984 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:14:13.751952 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f003ced9-bea8-4cf3-a8db-42512467bfb5-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bkszb\" (UID: \"f003ced9-bea8-4cf3-a8db-42512467bfb5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bkszb" Jul 09 07:14:13.751984 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:14:13.751985 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f003ced9-bea8-4cf3-a8db-42512467bfb5-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bkszb\" (UID: \"f003ced9-bea8-4cf3-a8db-42512467bfb5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bkszb" Jul 09 07:14:13.752191 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:14:13.752003 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f003ced9-bea8-4cf3-a8db-42512467bfb5-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bkszb\" (UID: \"f003ced9-bea8-4cf3-a8db-42512467bfb5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bkszb" Jul 09 07:14:13.752191 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:14:13.752020 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v5c7\" (UniqueName: \"kubernetes.io/projected/f003ced9-bea8-4cf3-a8db-42512467bfb5-kube-api-access-6v5c7\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bkszb\" (UID: \"f003ced9-bea8-4cf3-a8db-42512467bfb5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bkszb" Jul 09 07:14:13.752191 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:14:13.752135 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f003ced9-bea8-4cf3-a8db-42512467bfb5-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bkszb\" (UID: \"f003ced9-bea8-4cf3-a8db-42512467bfb5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bkszb" Jul 09 07:14:13.752191 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:14:13.752173 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f003ced9-bea8-4cf3-a8db-42512467bfb5-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bkszb\" (UID: \"f003ced9-bea8-4cf3-a8db-42512467bfb5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bkszb" Jul 09 07:14:13.853351 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:14:13.853317 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f003ced9-bea8-4cf3-a8db-42512467bfb5-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bkszb\" (UID: \"f003ced9-bea8-4cf3-a8db-42512467bfb5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bkszb" Jul 09 07:14:13.853512 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:14:13.853358 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f003ced9-bea8-4cf3-a8db-42512467bfb5-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bkszb\" (UID: \"f003ced9-bea8-4cf3-a8db-42512467bfb5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bkszb" Jul 09 07:14:13.853512 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:14:13.853382 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f003ced9-bea8-4cf3-a8db-42512467bfb5-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bkszb\" (UID: \"f003ced9-bea8-4cf3-a8db-42512467bfb5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bkszb" Jul 09 07:14:13.853512 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:14:13.853493 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v5c7\" (UniqueName: \"kubernetes.io/projected/f003ced9-bea8-4cf3-a8db-42512467bfb5-kube-api-access-6v5c7\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bkszb\" (UID: \"f003ced9-bea8-4cf3-a8db-42512467bfb5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bkszb" Jul 09 07:14:13.853661 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:14:13.853558 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f003ced9-bea8-4cf3-a8db-42512467bfb5-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bkszb\" (UID: \"f003ced9-bea8-4cf3-a8db-42512467bfb5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bkszb" Jul 09 07:14:13.853661 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:14:13.853576 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f003ced9-bea8-4cf3-a8db-42512467bfb5-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bkszb\" (UID: \"f003ced9-bea8-4cf3-a8db-42512467bfb5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bkszb" Jul 09 07:14:13.853882 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:14:13.853827 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f003ced9-bea8-4cf3-a8db-42512467bfb5-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bkszb\" (UID: \"f003ced9-bea8-4cf3-a8db-42512467bfb5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bkszb" Jul 09 07:14:13.854012 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:14:13.853857 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f003ced9-bea8-4cf3-a8db-42512467bfb5-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bkszb\" (UID: \"f003ced9-bea8-4cf3-a8db-42512467bfb5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bkszb" Jul 09 07:14:13.854012 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:14:13.853895 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f003ced9-bea8-4cf3-a8db-42512467bfb5-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bkszb\" (UID: \"f003ced9-bea8-4cf3-a8db-42512467bfb5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bkszb" Jul 09 07:14:13.855591 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:14:13.855572 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f003ced9-bea8-4cf3-a8db-42512467bfb5-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bkszb\" (UID: \"f003ced9-bea8-4cf3-a8db-42512467bfb5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bkszb" Jul 09 07:14:13.856019 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:14:13.855998 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f003ced9-bea8-4cf3-a8db-42512467bfb5-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bkszb\" (UID: \"f003ced9-bea8-4cf3-a8db-42512467bfb5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bkszb" Jul 09 07:14:13.861414 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:14:13.861391 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v5c7\" (UniqueName: \"kubernetes.io/projected/f003ced9-bea8-4cf3-a8db-42512467bfb5-kube-api-access-6v5c7\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bkszb\" (UID: \"f003ced9-bea8-4cf3-a8db-42512467bfb5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bkszb" Jul 09 07:14:14.000035 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:14:13.999961 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bkszb" Jul 09 07:14:14.134034 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:14:14.133900 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bkszb"] Jul 09 07:14:14.137050 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:14:14.137018 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf003ced9_bea8_4cf3_a8db_42512467bfb5.slice/crio-8c5fe9803aad66e95678050c3a7735e7a6f6cd66a422613cd5450df33dbf0053 WatchSource:0}: Error finding container 8c5fe9803aad66e95678050c3a7735e7a6f6cd66a422613cd5450df33dbf0053: Status 404 returned error can't find the container with id 8c5fe9803aad66e95678050c3a7735e7a6f6cd66a422613cd5450df33dbf0053 Jul 09 07:14:14.255274 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:14:14.255185 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bkszb" event={"ID":"f003ced9-bea8-4cf3-a8db-42512467bfb5","Type":"ContainerStarted","Data":"8c5fe9803aad66e95678050c3a7735e7a6f6cd66a422613cd5450df33dbf0053"} Jul 09 07:14:16.266413 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:14:16.266370 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bkszb" event={"ID":"f003ced9-bea8-4cf3-a8db-42512467bfb5","Type":"ContainerStarted","Data":"58d26869e2fdf247483530e2a0f16827c99debe523c68c9b6d3a2ab5142fa020"} Jul 09 07:14:16.284376 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:14:16.284329 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bkszb" podStartSLOduration=2.115295176 podStartE2EDuration="3.284313717s" totalImagesPullingTime="1.169018541s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 07:14:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 07:14:16.28297137 +0000 UTC m=+715.334048598" watchObservedRunningTime="2026-07-09 07:14:16.284313717 +0000 UTC m=+715.335390944" Jul 09 07:14:24.001090 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:14:24.001051 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bkszb" Jul 09 07:14:24.001505 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:14:24.001134 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bkszb" Jul 09 07:14:24.013779 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:14:24.013756 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bkszb" Jul 09 07:14:24.312504 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:14:24.312424 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bkszb" Jul 09 07:14:34.283515 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:14:34.283477 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-w8wfk"] Jul 09 07:14:34.287338 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:14:34.287317 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w8wfk" Jul 09 07:14:34.289595 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:14:34.289578 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 09 07:14:34.297128 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:14:34.297109 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-w8wfk"] Jul 09 07:14:34.330472 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:14:34.330441 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f647a6c-f56c-4f9c-b79c-840ed7f57878-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w8wfk\" (UID: \"2f647a6c-f56c-4f9c-b79c-840ed7f57878\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w8wfk" Jul 09 07:14:34.330617 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:14:34.330495 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f647a6c-f56c-4f9c-b79c-840ed7f57878-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w8wfk\" (UID: \"2f647a6c-f56c-4f9c-b79c-840ed7f57878\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w8wfk" Jul 09 07:14:34.330617 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:14:34.330542 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f647a6c-f56c-4f9c-b79c-840ed7f57878-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w8wfk\" (UID: \"2f647a6c-f56c-4f9c-b79c-840ed7f57878\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w8wfk" Jul 09 07:14:34.330690 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:14:34.330607 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f647a6c-f56c-4f9c-b79c-840ed7f57878-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w8wfk\" (UID: \"2f647a6c-f56c-4f9c-b79c-840ed7f57878\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w8wfk" Jul 09 07:14:34.330690 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:14:34.330634 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p282j\" (UniqueName: \"kubernetes.io/projected/2f647a6c-f56c-4f9c-b79c-840ed7f57878-kube-api-access-p282j\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w8wfk\" (UID: \"2f647a6c-f56c-4f9c-b79c-840ed7f57878\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w8wfk" Jul 09 07:14:34.330690 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:14:34.330660 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f647a6c-f56c-4f9c-b79c-840ed7f57878-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w8wfk\" (UID: \"2f647a6c-f56c-4f9c-b79c-840ed7f57878\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w8wfk" Jul 09 07:14:34.431196 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:14:34.431161 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f647a6c-f56c-4f9c-b79c-840ed7f57878-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w8wfk\" (UID: \"2f647a6c-f56c-4f9c-b79c-840ed7f57878\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w8wfk" Jul 09 07:14:34.431196 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:14:34.431206 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f647a6c-f56c-4f9c-b79c-840ed7f57878-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w8wfk\" (UID: \"2f647a6c-f56c-4f9c-b79c-840ed7f57878\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w8wfk" Jul 09 07:14:34.431458 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:14:34.431247 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f647a6c-f56c-4f9c-b79c-840ed7f57878-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w8wfk\" (UID: \"2f647a6c-f56c-4f9c-b79c-840ed7f57878\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w8wfk" Jul 09 07:14:34.431458 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:14:34.431288 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p282j\" (UniqueName: \"kubernetes.io/projected/2f647a6c-f56c-4f9c-b79c-840ed7f57878-kube-api-access-p282j\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w8wfk\" (UID: \"2f647a6c-f56c-4f9c-b79c-840ed7f57878\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w8wfk" Jul 09 07:14:34.431458 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:14:34.431321 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f647a6c-f56c-4f9c-b79c-840ed7f57878-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w8wfk\" (UID: \"2f647a6c-f56c-4f9c-b79c-840ed7f57878\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w8wfk" Jul 09 07:14:34.431458 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:14:34.431355 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f647a6c-f56c-4f9c-b79c-840ed7f57878-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w8wfk\" (UID: \"2f647a6c-f56c-4f9c-b79c-840ed7f57878\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w8wfk" Jul 09 07:14:34.431736 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:14:34.431710 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f647a6c-f56c-4f9c-b79c-840ed7f57878-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w8wfk\" (UID: \"2f647a6c-f56c-4f9c-b79c-840ed7f57878\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w8wfk" Jul 09 07:14:34.431833 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:14:34.431768 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f647a6c-f56c-4f9c-b79c-840ed7f57878-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w8wfk\" (UID: \"2f647a6c-f56c-4f9c-b79c-840ed7f57878\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w8wfk" Jul 09 07:14:34.431999 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:14:34.431965 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f647a6c-f56c-4f9c-b79c-840ed7f57878-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w8wfk\" (UID: \"2f647a6c-f56c-4f9c-b79c-840ed7f57878\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w8wfk" Jul 09 07:14:34.433998 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:14:34.433977 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f647a6c-f56c-4f9c-b79c-840ed7f57878-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w8wfk\" (UID: \"2f647a6c-f56c-4f9c-b79c-840ed7f57878\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w8wfk" Jul 09 07:14:34.434348 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:14:34.434326 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f647a6c-f56c-4f9c-b79c-840ed7f57878-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w8wfk\" (UID: \"2f647a6c-f56c-4f9c-b79c-840ed7f57878\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w8wfk" Jul 09 07:14:34.439858 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:14:34.439836 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p282j\" (UniqueName: \"kubernetes.io/projected/2f647a6c-f56c-4f9c-b79c-840ed7f57878-kube-api-access-p282j\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w8wfk\" (UID: \"2f647a6c-f56c-4f9c-b79c-840ed7f57878\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w8wfk" Jul 09 07:14:34.597720 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:14:34.597636 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w8wfk" Jul 09 07:14:34.727126 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:14:34.727098 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-w8wfk"] Jul 09 07:14:34.729050 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:14:34.729011 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f647a6c_f56c_4f9c_b79c_840ed7f57878.slice/crio-2b650cbfc98eba130860d7d7092aa7b4c1ac038ecd1ff1b401be9bb500b06afd WatchSource:0}: Error finding container 2b650cbfc98eba130860d7d7092aa7b4c1ac038ecd1ff1b401be9bb500b06afd: Status 404 returned error can't find the container with id 2b650cbfc98eba130860d7d7092aa7b4c1ac038ecd1ff1b401be9bb500b06afd Jul 09 07:14:35.352229 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:14:35.352140 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w8wfk" event={"ID":"2f647a6c-f56c-4f9c-b79c-840ed7f57878","Type":"ContainerStarted","Data":"560820c01b44490134e8bcc3af1faa58236dba7d3f1b6a9767cfd146206c2084"} Jul 09 07:14:35.352229 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:14:35.352194 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w8wfk" event={"ID":"2f647a6c-f56c-4f9c-b79c-840ed7f57878","Type":"ContainerStarted","Data":"2b650cbfc98eba130860d7d7092aa7b4c1ac038ecd1ff1b401be9bb500b06afd"} Jul 09 07:14:35.372056 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:14:35.372005 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w8wfk" podStartSLOduration=1.184439287 podStartE2EDuration="1.371990363s" totalImagesPullingTime="187.551076ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 07:14:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 07:14:35.370338826 +0000 UTC m=+734.421416048" watchObservedRunningTime="2026-07-09 07:14:35.371990363 +0000 UTC m=+734.423067590" Jul 09 07:14:44.598415 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:14:44.598382 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w8wfk" Jul 09 07:14:44.598887 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:14:44.598431 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w8wfk" Jul 09 07:14:44.610677 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:14:44.610654 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w8wfk" Jul 09 07:14:45.407226 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:14:45.407197 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w8wfk" Jul 09 07:22:01.954095 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:22:01.954015 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j5mlg"] Jul 09 07:22:01.954659 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:22:01.954222 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j5mlg" podUID="e7c76921-ab10-4b26-8fd2-48d88a3254c5" containerName="manager" containerID="cri-o://8163b0a59d1f3c80dfd2089e403583c38a4928a5cabd4f238ff684fd8dfe7522" gracePeriod=10 Jul 09 07:22:05.606929 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:22:05.606903 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j5mlg" Jul 09 07:22:05.762220 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:22:05.762143 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-phfgm\" (UniqueName: \"kubernetes.io/projected/e7c76921-ab10-4b26-8fd2-48d88a3254c5-kube-api-access-phfgm\") pod \"e7c76921-ab10-4b26-8fd2-48d88a3254c5\" (UID: \"e7c76921-ab10-4b26-8fd2-48d88a3254c5\") " Jul 09 07:22:05.762220 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:22:05.762181 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7c76921-ab10-4b26-8fd2-48d88a3254c5-extensions-socket-volume\") pod \"e7c76921-ab10-4b26-8fd2-48d88a3254c5\" (UID: \"e7c76921-ab10-4b26-8fd2-48d88a3254c5\") " Jul 09 07:22:05.762627 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:22:05.762609 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7c76921-ab10-4b26-8fd2-48d88a3254c5-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e7c76921-ab10-4b26-8fd2-48d88a3254c5" (UID: "e7c76921-ab10-4b26-8fd2-48d88a3254c5"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 07:22:05.764187 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:22:05.764160 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7c76921-ab10-4b26-8fd2-48d88a3254c5-kube-api-access-phfgm" (OuterVolumeSpecName: "kube-api-access-phfgm") pod "e7c76921-ab10-4b26-8fd2-48d88a3254c5" (UID: "e7c76921-ab10-4b26-8fd2-48d88a3254c5"). InnerVolumeSpecName "kube-api-access-phfgm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 07:22:05.863488 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:22:05.863435 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phfgm\" (UniqueName: \"kubernetes.io/projected/e7c76921-ab10-4b26-8fd2-48d88a3254c5-kube-api-access-phfgm\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:22:05.863488 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:22:05.863482 2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7c76921-ab10-4b26-8fd2-48d88a3254c5-extensions-socket-volume\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:22:06.342150 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:22:06.342121 2564 generic.go:356] "Generic (PLEG): container finished" podID="e7c76921-ab10-4b26-8fd2-48d88a3254c5" containerID="8163b0a59d1f3c80dfd2089e403583c38a4928a5cabd4f238ff684fd8dfe7522" exitCode=0 Jul 09 07:22:06.342367 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:22:06.342167 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j5mlg" Jul 09 07:22:06.342367 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:22:06.342211 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j5mlg" event={"ID":"e7c76921-ab10-4b26-8fd2-48d88a3254c5","Type":"ContainerDied","Data":"8163b0a59d1f3c80dfd2089e403583c38a4928a5cabd4f238ff684fd8dfe7522"} Jul 09 07:22:06.342367 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:22:06.342293 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j5mlg" event={"ID":"e7c76921-ab10-4b26-8fd2-48d88a3254c5","Type":"ContainerDied","Data":"7820e3ef4964e54ae30cf466dfc830033fe89ca211bd918f2eb320ed99f43922"} Jul 09 07:22:06.342367 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:22:06.342327 2564 scope.go:117] "RemoveContainer" containerID="8163b0a59d1f3c80dfd2089e403583c38a4928a5cabd4f238ff684fd8dfe7522" Jul 09 07:22:06.351657 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:22:06.351634 2564 scope.go:117] "RemoveContainer" containerID="8163b0a59d1f3c80dfd2089e403583c38a4928a5cabd4f238ff684fd8dfe7522" Jul 09 07:22:06.351912 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:22:06.351895 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8163b0a59d1f3c80dfd2089e403583c38a4928a5cabd4f238ff684fd8dfe7522\": container with ID starting with 8163b0a59d1f3c80dfd2089e403583c38a4928a5cabd4f238ff684fd8dfe7522 not found: ID does not exist" containerID="8163b0a59d1f3c80dfd2089e403583c38a4928a5cabd4f238ff684fd8dfe7522" Jul 09 07:22:06.351969 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:22:06.351920 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8163b0a59d1f3c80dfd2089e403583c38a4928a5cabd4f238ff684fd8dfe7522"} err="failed to get container status \"8163b0a59d1f3c80dfd2089e403583c38a4928a5cabd4f238ff684fd8dfe7522\": rpc error: code = NotFound desc = could not find container \"8163b0a59d1f3c80dfd2089e403583c38a4928a5cabd4f238ff684fd8dfe7522\": container with ID starting with 8163b0a59d1f3c80dfd2089e403583c38a4928a5cabd4f238ff684fd8dfe7522 not found: ID does not exist" Jul 09 07:22:06.365355 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:22:06.365332 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j5mlg"] Jul 09 07:22:06.378474 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:22:06.378451 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j5mlg"] Jul 09 07:22:07.465808 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:22:07.465777 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7c76921-ab10-4b26-8fd2-48d88a3254c5" path="/var/lib/kubelet/pods/e7c76921-ab10-4b26-8fd2-48d88a3254c5/volumes" Jul 09 07:29:12.705631 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:12.705591 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql"] Jul 09 07:29:12.708016 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:12.706104 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7c76921-ab10-4b26-8fd2-48d88a3254c5" containerName="manager" Jul 09 07:29:12.708016 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:12.706123 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7c76921-ab10-4b26-8fd2-48d88a3254c5" containerName="manager" Jul 09 07:29:12.708016 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:12.706232 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7c76921-ab10-4b26-8fd2-48d88a3254c5" containerName="manager" Jul 09 07:29:12.709545 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:12.709521 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql" Jul 09 07:29:12.713480 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:12.713457 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-37ae4b70-openshift-default-dockercfg-xr7hw\"" Jul 09 07:29:12.738028 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:12.738000 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql"] Jul 09 07:29:12.787826 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:12.787792 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d3e7834-caf7-4113-b0a1-d008b32b90f1-istio-data\") pod \"e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql\" (UID: \"1d3e7834-caf7-4113-b0a1-d008b32b90f1\") " pod="openshift-ingress/e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql" Jul 09 07:29:12.788008 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:12.787830 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d3e7834-caf7-4113-b0a1-d008b32b90f1-istio-envoy\") pod \"e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql\" (UID: \"1d3e7834-caf7-4113-b0a1-d008b32b90f1\") " pod="openshift-ingress/e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql" Jul 09 07:29:12.788008 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:12.787863 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d3e7834-caf7-4113-b0a1-d008b32b90f1-istio-token\") pod \"e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql\" (UID: \"1d3e7834-caf7-4113-b0a1-d008b32b90f1\") " pod="openshift-ingress/e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql" Jul 09 07:29:12.788008 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:12.787886 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d3e7834-caf7-4113-b0a1-d008b32b90f1-istiod-ca-cert\") pod \"e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql\" (UID: \"1d3e7834-caf7-4113-b0a1-d008b32b90f1\") " pod="openshift-ingress/e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql" Jul 09 07:29:12.788008 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:12.787933 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d3e7834-caf7-4113-b0a1-d008b32b90f1-workload-socket\") pod \"e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql\" (UID: \"1d3e7834-caf7-4113-b0a1-d008b32b90f1\") " pod="openshift-ingress/e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql" Jul 09 07:29:12.788008 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:12.787962 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d3e7834-caf7-4113-b0a1-d008b32b90f1-istio-podinfo\") pod \"e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql\" (UID: \"1d3e7834-caf7-4113-b0a1-d008b32b90f1\") " pod="openshift-ingress/e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql" Jul 09 07:29:12.788008 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:12.787979 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d3e7834-caf7-4113-b0a1-d008b32b90f1-credential-socket\") pod \"e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql\" (UID: \"1d3e7834-caf7-4113-b0a1-d008b32b90f1\") " pod="openshift-ingress/e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql" Jul 09 07:29:12.788008 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:12.787994 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhvgg\" (UniqueName: \"kubernetes.io/projected/1d3e7834-caf7-4113-b0a1-d008b32b90f1-kube-api-access-rhvgg\") pod \"e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql\" (UID: \"1d3e7834-caf7-4113-b0a1-d008b32b90f1\") " pod="openshift-ingress/e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql" Jul 09 07:29:12.788257 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:12.788020 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d3e7834-caf7-4113-b0a1-d008b32b90f1-workload-certs\") pod \"e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql\" (UID: \"1d3e7834-caf7-4113-b0a1-d008b32b90f1\") " pod="openshift-ingress/e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql" Jul 09 07:29:12.889225 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:12.889191 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d3e7834-caf7-4113-b0a1-d008b32b90f1-istio-podinfo\") pod \"e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql\" (UID: \"1d3e7834-caf7-4113-b0a1-d008b32b90f1\") " pod="openshift-ingress/e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql" Jul 09 07:29:12.889225 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:12.889226 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d3e7834-caf7-4113-b0a1-d008b32b90f1-credential-socket\") pod \"e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql\" (UID: \"1d3e7834-caf7-4113-b0a1-d008b32b90f1\") " pod="openshift-ingress/e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql" Jul 09 07:29:12.889493 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:12.889247 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhvgg\" (UniqueName: \"kubernetes.io/projected/1d3e7834-caf7-4113-b0a1-d008b32b90f1-kube-api-access-rhvgg\") pod \"e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql\" (UID: \"1d3e7834-caf7-4113-b0a1-d008b32b90f1\") " pod="openshift-ingress/e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql" Jul 09 07:29:12.889493 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:12.889298 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d3e7834-caf7-4113-b0a1-d008b32b90f1-workload-certs\") pod \"e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql\" (UID: \"1d3e7834-caf7-4113-b0a1-d008b32b90f1\") " pod="openshift-ingress/e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql" Jul 09 07:29:12.889493 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:12.889343 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d3e7834-caf7-4113-b0a1-d008b32b90f1-istio-data\") pod \"e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql\" (UID: \"1d3e7834-caf7-4113-b0a1-d008b32b90f1\") " pod="openshift-ingress/e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql" Jul 09 07:29:12.889493 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:12.889371 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d3e7834-caf7-4113-b0a1-d008b32b90f1-istio-envoy\") pod \"e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql\" (UID: \"1d3e7834-caf7-4113-b0a1-d008b32b90f1\") " pod="openshift-ingress/e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql" Jul 09 07:29:12.889493 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:12.889417 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d3e7834-caf7-4113-b0a1-d008b32b90f1-istio-token\") pod \"e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql\" (UID: \"1d3e7834-caf7-4113-b0a1-d008b32b90f1\") " pod="openshift-ingress/e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql" Jul 09 07:29:12.889493 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:12.889442 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d3e7834-caf7-4113-b0a1-d008b32b90f1-istiod-ca-cert\") pod \"e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql\" (UID: \"1d3e7834-caf7-4113-b0a1-d008b32b90f1\") " pod="openshift-ingress/e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql" Jul 09 07:29:12.889493 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:12.889483 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d3e7834-caf7-4113-b0a1-d008b32b90f1-workload-socket\") pod \"e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql\" (UID: \"1d3e7834-caf7-4113-b0a1-d008b32b90f1\") " pod="openshift-ingress/e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql" Jul 09 07:29:12.889823 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:12.889688 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d3e7834-caf7-4113-b0a1-d008b32b90f1-credential-socket\") pod \"e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql\" (UID: \"1d3e7834-caf7-4113-b0a1-d008b32b90f1\") " pod="openshift-ingress/e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql" Jul 09 07:29:12.889823 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:12.889703 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d3e7834-caf7-4113-b0a1-d008b32b90f1-istio-data\") pod \"e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql\" (UID: \"1d3e7834-caf7-4113-b0a1-d008b32b90f1\") " pod="openshift-ingress/e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql" Jul 09 07:29:12.889823 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:12.889753 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d3e7834-caf7-4113-b0a1-d008b32b90f1-workload-certs\") pod \"e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql\" (UID: \"1d3e7834-caf7-4113-b0a1-d008b32b90f1\") " pod="openshift-ingress/e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql" Jul 09 07:29:12.890017 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:12.889999 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d3e7834-caf7-4113-b0a1-d008b32b90f1-workload-socket\") pod \"e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql\" (UID: \"1d3e7834-caf7-4113-b0a1-d008b32b90f1\") " pod="openshift-ingress/e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql" Jul 09 07:29:12.890256 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:12.890232 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d3e7834-caf7-4113-b0a1-d008b32b90f1-istiod-ca-cert\") pod \"e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql\" (UID: \"1d3e7834-caf7-4113-b0a1-d008b32b90f1\") " pod="openshift-ingress/e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql" Jul 09 07:29:12.891611 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:12.891593 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d3e7834-caf7-4113-b0a1-d008b32b90f1-istio-envoy\") pod \"e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql\" (UID: \"1d3e7834-caf7-4113-b0a1-d008b32b90f1\") " pod="openshift-ingress/e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql" Jul 09 07:29:12.891975 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:12.891955 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d3e7834-caf7-4113-b0a1-d008b32b90f1-istio-podinfo\") pod \"e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql\" (UID: \"1d3e7834-caf7-4113-b0a1-d008b32b90f1\") " pod="openshift-ingress/e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql" Jul 09 07:29:12.898399 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:12.898343 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d3e7834-caf7-4113-b0a1-d008b32b90f1-istio-token\") pod \"e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql\" (UID: \"1d3e7834-caf7-4113-b0a1-d008b32b90f1\") " pod="openshift-ingress/e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql" Jul 09 07:29:12.898603 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:12.898580 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhvgg\" (UniqueName: \"kubernetes.io/projected/1d3e7834-caf7-4113-b0a1-d008b32b90f1-kube-api-access-rhvgg\") pod \"e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql\" (UID: \"1d3e7834-caf7-4113-b0a1-d008b32b90f1\") " pod="openshift-ingress/e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql" Jul 09 07:29:13.020476 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:13.020392 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql" Jul 09 07:29:13.158562 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:13.158539 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql"] Jul 09 07:29:13.160845 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:29:13.160817 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d3e7834_caf7_4113_b0a1_d008b32b90f1.slice/crio-fe6ba3d8ef5dd5e7be8f838d9d77af4960adae612b9599e2e075be846d98ec69 WatchSource:0}: Error finding container fe6ba3d8ef5dd5e7be8f838d9d77af4960adae612b9599e2e075be846d98ec69: Status 404 returned error can't find the container with id fe6ba3d8ef5dd5e7be8f838d9d77af4960adae612b9599e2e075be846d98ec69 Jul 09 07:29:13.163356 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:13.163342 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 07:29:13.187355 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:13.187325 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql" event={"ID":"1d3e7834-caf7-4113-b0a1-d008b32b90f1","Type":"ContainerStarted","Data":"fe6ba3d8ef5dd5e7be8f838d9d77af4960adae612b9599e2e075be846d98ec69"} Jul 09 07:29:15.929446 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:15.929403 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 07:29:15.929684 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:15.929488 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 07:29:15.929684 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:15.929520 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 07:29:16.205005 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:16.204918 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql" event={"ID":"1d3e7834-caf7-4113-b0a1-d008b32b90f1","Type":"ContainerStarted","Data":"1fea282d2e178148d5554136c0a9c52bfbbeafc8d80f19a75486eef94bf9d193"} Jul 09 07:29:16.225638 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:16.225593 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql" podStartSLOduration=1.45998741 podStartE2EDuration="4.225576033s" totalImagesPullingTime="2.765588623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 07:29:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 07:29:16.224161715 +0000 UTC m=+1615.275238943" watchObservedRunningTime="2026-07-09 07:29:16.225576033 +0000 UTC m=+1615.276653330" Jul 09 07:29:17.020738 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:17.020645 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql" Jul 09 07:29:17.025492 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:17.025468 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql" Jul 09 07:29:17.208863 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:17.208831 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql" Jul 09 07:29:17.209734 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:17.209716 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql" Jul 09 07:29:19.102400 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:19.102364 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql"] Jul 09 07:29:19.218259 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:19.218225 2564 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql" secret="" err="secret \"e2e-ait-37ae4b70-openshift-default-dockercfg-xr7hw\" not found" Jul 09 07:29:20.221915 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:20.221875 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql" podUID="1d3e7834-caf7-4113-b0a1-d008b32b90f1" containerName="istio-proxy" containerID="cri-o://1fea282d2e178148d5554136c0a9c52bfbbeafc8d80f19a75486eef94bf9d193" gracePeriod=30 Jul 09 07:29:25.305913 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:25.305878 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g"] Jul 09 07:29:25.308704 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:25.308669 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g" Jul 09 07:29:25.311449 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:25.311431 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-27d28868-openshift-default-dockercfg-jb2hm\"" Jul 09 07:29:25.343743 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:25.343709 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g"] Jul 09 07:29:25.407371 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:25.407339 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0e68d52-aeb0-4ed6-811e-467a9b1cc24d-workload-certs\") pod \"e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g\" (UID: \"c0e68d52-aeb0-4ed6-811e-467a9b1cc24d\") " pod="openshift-ingress/e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g" Jul 09 07:29:25.407526 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:25.407389 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0e68d52-aeb0-4ed6-811e-467a9b1cc24d-istio-token\") pod \"e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g\" (UID: \"c0e68d52-aeb0-4ed6-811e-467a9b1cc24d\") " pod="openshift-ingress/e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g" Jul 09 07:29:25.407526 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:25.407427 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e68d52-aeb0-4ed6-811e-467a9b1cc24d-credential-socket\") pod \"e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g\" (UID: \"c0e68d52-aeb0-4ed6-811e-467a9b1cc24d\") " pod="openshift-ingress/e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g" Jul 09 07:29:25.407526 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:25.407474 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0e68d52-aeb0-4ed6-811e-467a9b1cc24d-istio-data\") pod \"e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g\" (UID: \"c0e68d52-aeb0-4ed6-811e-467a9b1cc24d\") " pod="openshift-ingress/e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g" Jul 09 07:29:25.407526 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:25.407509 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0e68d52-aeb0-4ed6-811e-467a9b1cc24d-istiod-ca-cert\") pod \"e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g\" (UID: \"c0e68d52-aeb0-4ed6-811e-467a9b1cc24d\") " pod="openshift-ingress/e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g" Jul 09 07:29:25.407741 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:25.407530 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e68d52-aeb0-4ed6-811e-467a9b1cc24d-workload-socket\") pod \"e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g\" (UID: \"c0e68d52-aeb0-4ed6-811e-467a9b1cc24d\") " pod="openshift-ingress/e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g" Jul 09 07:29:25.407741 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:25.407549 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0e68d52-aeb0-4ed6-811e-467a9b1cc24d-istio-envoy\") pod \"e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g\" (UID: \"c0e68d52-aeb0-4ed6-811e-467a9b1cc24d\") " pod="openshift-ingress/e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g" Jul 09 07:29:25.407741 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:25.407588 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0e68d52-aeb0-4ed6-811e-467a9b1cc24d-istio-podinfo\") pod \"e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g\" (UID: \"c0e68d52-aeb0-4ed6-811e-467a9b1cc24d\") " pod="openshift-ingress/e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g" Jul 09 07:29:25.407741 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:25.407661 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hktxm\" (UniqueName: \"kubernetes.io/projected/c0e68d52-aeb0-4ed6-811e-467a9b1cc24d-kube-api-access-hktxm\") pod \"e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g\" (UID: \"c0e68d52-aeb0-4ed6-811e-467a9b1cc24d\") " pod="openshift-ingress/e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g" Jul 09 07:29:25.508785 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:25.508754 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hktxm\" (UniqueName: \"kubernetes.io/projected/c0e68d52-aeb0-4ed6-811e-467a9b1cc24d-kube-api-access-hktxm\") pod \"e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g\" (UID: \"c0e68d52-aeb0-4ed6-811e-467a9b1cc24d\") " pod="openshift-ingress/e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g" Jul 09 07:29:25.508940 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:25.508794 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0e68d52-aeb0-4ed6-811e-467a9b1cc24d-workload-certs\") pod \"e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g\" (UID: \"c0e68d52-aeb0-4ed6-811e-467a9b1cc24d\") " pod="openshift-ingress/e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g" Jul 09 07:29:25.509036 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:25.509012 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0e68d52-aeb0-4ed6-811e-467a9b1cc24d-istio-token\") pod \"e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g\" (UID: \"c0e68d52-aeb0-4ed6-811e-467a9b1cc24d\") " pod="openshift-ingress/e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g" Jul 09 07:29:25.509096 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:25.509074 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e68d52-aeb0-4ed6-811e-467a9b1cc24d-credential-socket\") pod \"e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g\" (UID: \"c0e68d52-aeb0-4ed6-811e-467a9b1cc24d\") " pod="openshift-ingress/e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g" Jul 09 07:29:25.509216 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:25.509199 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0e68d52-aeb0-4ed6-811e-467a9b1cc24d-istio-data\") pod \"e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g\" (UID: \"c0e68d52-aeb0-4ed6-811e-467a9b1cc24d\") " pod="openshift-ingress/e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g" Jul 09 07:29:25.509299 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:25.509229 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0e68d52-aeb0-4ed6-811e-467a9b1cc24d-istiod-ca-cert\") pod \"e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g\" (UID: \"c0e68d52-aeb0-4ed6-811e-467a9b1cc24d\") " pod="openshift-ingress/e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g" Jul 09 07:29:25.509299 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:25.509290 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e68d52-aeb0-4ed6-811e-467a9b1cc24d-workload-socket\") pod \"e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g\" (UID: \"c0e68d52-aeb0-4ed6-811e-467a9b1cc24d\") " pod="openshift-ingress/e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g" Jul 09 07:29:25.509413 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:25.509326 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0e68d52-aeb0-4ed6-811e-467a9b1cc24d-istio-envoy\") pod \"e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g\" (UID: \"c0e68d52-aeb0-4ed6-811e-467a9b1cc24d\") " pod="openshift-ingress/e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g" Jul 09 07:29:25.509413 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:25.509356 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0e68d52-aeb0-4ed6-811e-467a9b1cc24d-istio-podinfo\") pod \"e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g\" (UID: \"c0e68d52-aeb0-4ed6-811e-467a9b1cc24d\") " pod="openshift-ingress/e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g" Jul 09 07:29:25.509536 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:25.509437 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e68d52-aeb0-4ed6-811e-467a9b1cc24d-credential-socket\") pod \"e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g\" (UID: \"c0e68d52-aeb0-4ed6-811e-467a9b1cc24d\") " pod="openshift-ingress/e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g" Jul 09 07:29:25.509972 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:25.509697 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e68d52-aeb0-4ed6-811e-467a9b1cc24d-workload-socket\") pod \"e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g\" (UID: \"c0e68d52-aeb0-4ed6-811e-467a9b1cc24d\") " pod="openshift-ingress/e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g" Jul 09 07:29:25.509972 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:25.509855 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0e68d52-aeb0-4ed6-811e-467a9b1cc24d-istio-data\") pod \"e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g\" (UID: \"c0e68d52-aeb0-4ed6-811e-467a9b1cc24d\") " pod="openshift-ingress/e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g" Jul 09 07:29:25.509972 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:25.509142 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0e68d52-aeb0-4ed6-811e-467a9b1cc24d-workload-certs\") pod \"e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g\" (UID: \"c0e68d52-aeb0-4ed6-811e-467a9b1cc24d\") " pod="openshift-ingress/e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g" Jul 09 07:29:25.509972 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:25.509894 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0e68d52-aeb0-4ed6-811e-467a9b1cc24d-istiod-ca-cert\") pod \"e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g\" (UID: \"c0e68d52-aeb0-4ed6-811e-467a9b1cc24d\") " pod="openshift-ingress/e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g" Jul 09 07:29:25.511830 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:25.511775 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0e68d52-aeb0-4ed6-811e-467a9b1cc24d-istio-envoy\") pod \"e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g\" (UID: \"c0e68d52-aeb0-4ed6-811e-467a9b1cc24d\") " pod="openshift-ingress/e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g" Jul 09 07:29:25.512016 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:25.511999 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0e68d52-aeb0-4ed6-811e-467a9b1cc24d-istio-podinfo\") pod \"e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g\" (UID: \"c0e68d52-aeb0-4ed6-811e-467a9b1cc24d\") " pod="openshift-ingress/e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g" Jul 09 07:29:25.517867 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:25.517844 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0e68d52-aeb0-4ed6-811e-467a9b1cc24d-istio-token\") pod \"e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g\" (UID: \"c0e68d52-aeb0-4ed6-811e-467a9b1cc24d\") " pod="openshift-ingress/e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g" Jul 09 07:29:25.518394 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:25.518374 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hktxm\" (UniqueName: \"kubernetes.io/projected/c0e68d52-aeb0-4ed6-811e-467a9b1cc24d-kube-api-access-hktxm\") pod \"e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g\" (UID: \"c0e68d52-aeb0-4ed6-811e-467a9b1cc24d\") " pod="openshift-ingress/e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g" Jul 09 07:29:25.588637 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:25.588601 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql" Jul 09 07:29:25.626114 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:25.626080 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g" Jul 09 07:29:25.711257 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:25.711224 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d3e7834-caf7-4113-b0a1-d008b32b90f1-istiod-ca-cert\") pod \"1d3e7834-caf7-4113-b0a1-d008b32b90f1\" (UID: \"1d3e7834-caf7-4113-b0a1-d008b32b90f1\") " Jul 09 07:29:25.711257 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:25.711257 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d3e7834-caf7-4113-b0a1-d008b32b90f1-workload-socket\") pod \"1d3e7834-caf7-4113-b0a1-d008b32b90f1\" (UID: \"1d3e7834-caf7-4113-b0a1-d008b32b90f1\") " Jul 09 07:29:25.711509 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:25.711317 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d3e7834-caf7-4113-b0a1-d008b32b90f1-istio-podinfo\") pod \"1d3e7834-caf7-4113-b0a1-d008b32b90f1\" (UID: \"1d3e7834-caf7-4113-b0a1-d008b32b90f1\") " Jul 09 07:29:25.711509 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:25.711350 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d3e7834-caf7-4113-b0a1-d008b32b90f1-istio-envoy\") pod \"1d3e7834-caf7-4113-b0a1-d008b32b90f1\" (UID: \"1d3e7834-caf7-4113-b0a1-d008b32b90f1\") " Jul 09 07:29:25.711509 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:25.711367 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rhvgg\" (UniqueName: \"kubernetes.io/projected/1d3e7834-caf7-4113-b0a1-d008b32b90f1-kube-api-access-rhvgg\") pod \"1d3e7834-caf7-4113-b0a1-d008b32b90f1\" (UID: \"1d3e7834-caf7-4113-b0a1-d008b32b90f1\") " Jul 09 07:29:25.711509 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:25.711424 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d3e7834-caf7-4113-b0a1-d008b32b90f1-istio-token\") pod \"1d3e7834-caf7-4113-b0a1-d008b32b90f1\" (UID: \"1d3e7834-caf7-4113-b0a1-d008b32b90f1\") " Jul 09 07:29:25.711509 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:25.711465 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d3e7834-caf7-4113-b0a1-d008b32b90f1-credential-socket\") pod \"1d3e7834-caf7-4113-b0a1-d008b32b90f1\" (UID: \"1d3e7834-caf7-4113-b0a1-d008b32b90f1\") " Jul 09 07:29:25.711509 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:25.711480 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d3e7834-caf7-4113-b0a1-d008b32b90f1-workload-certs\") pod \"1d3e7834-caf7-4113-b0a1-d008b32b90f1\" (UID: \"1d3e7834-caf7-4113-b0a1-d008b32b90f1\") " Jul 09 07:29:25.711509 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:25.711511 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d3e7834-caf7-4113-b0a1-d008b32b90f1-istio-data\") pod \"1d3e7834-caf7-4113-b0a1-d008b32b90f1\" (UID: \"1d3e7834-caf7-4113-b0a1-d008b32b90f1\") " Jul 09 07:29:25.711870 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:25.711599 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d3e7834-caf7-4113-b0a1-d008b32b90f1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1d3e7834-caf7-4113-b0a1-d008b32b90f1" (UID: "1d3e7834-caf7-4113-b0a1-d008b32b90f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 07:29:25.711870 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:25.711754 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d3e7834-caf7-4113-b0a1-d008b32b90f1-istiod-ca-cert\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:29:25.712378 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:25.712334 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d3e7834-caf7-4113-b0a1-d008b32b90f1-istio-data" (OuterVolumeSpecName: "istio-data") pod "1d3e7834-caf7-4113-b0a1-d008b32b90f1" (UID: "1d3e7834-caf7-4113-b0a1-d008b32b90f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 07:29:25.712527 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:25.712506 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d3e7834-caf7-4113-b0a1-d008b32b90f1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1d3e7834-caf7-4113-b0a1-d008b32b90f1" (UID: "1d3e7834-caf7-4113-b0a1-d008b32b90f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 07:29:25.712668 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:25.712640 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d3e7834-caf7-4113-b0a1-d008b32b90f1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1d3e7834-caf7-4113-b0a1-d008b32b90f1" (UID: "1d3e7834-caf7-4113-b0a1-d008b32b90f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 07:29:25.712787 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:25.712765 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d3e7834-caf7-4113-b0a1-d008b32b90f1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1d3e7834-caf7-4113-b0a1-d008b32b90f1" (UID: "1d3e7834-caf7-4113-b0a1-d008b32b90f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 07:29:25.715332 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:25.715299 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d3e7834-caf7-4113-b0a1-d008b32b90f1-istio-token" (OuterVolumeSpecName: "istio-token") pod "1d3e7834-caf7-4113-b0a1-d008b32b90f1" (UID: "1d3e7834-caf7-4113-b0a1-d008b32b90f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 07:29:25.715332 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:25.715311 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d3e7834-caf7-4113-b0a1-d008b32b90f1-kube-api-access-rhvgg" (OuterVolumeSpecName: "kube-api-access-rhvgg") pod "1d3e7834-caf7-4113-b0a1-d008b32b90f1" (UID: "1d3e7834-caf7-4113-b0a1-d008b32b90f1"). InnerVolumeSpecName "kube-api-access-rhvgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 07:29:25.715332 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:25.715325 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d3e7834-caf7-4113-b0a1-d008b32b90f1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1d3e7834-caf7-4113-b0a1-d008b32b90f1" (UID: "1d3e7834-caf7-4113-b0a1-d008b32b90f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 07:29:25.716940 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:25.716912 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d3e7834-caf7-4113-b0a1-d008b32b90f1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1d3e7834-caf7-4113-b0a1-d008b32b90f1" (UID: "1d3e7834-caf7-4113-b0a1-d008b32b90f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 07:29:25.758459 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:25.758434 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g"] Jul 09 07:29:25.760793 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:29:25.760763 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0e68d52_aeb0_4ed6_811e_467a9b1cc24d.slice/crio-b17f01dfdc3e9f7776291f93ddb1ee10c764987e15d363340115b5481541270f WatchSource:0}: Error finding container b17f01dfdc3e9f7776291f93ddb1ee10c764987e15d363340115b5481541270f: Status 404 returned error can't find the container with id b17f01dfdc3e9f7776291f93ddb1ee10c764987e15d363340115b5481541270f Jul 09 07:29:25.762815 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:25.762782 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 07:29:25.762904 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:25.762848 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 07:29:25.762904 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:25.762877 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 07:29:25.812569 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:25.812542 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d3e7834-caf7-4113-b0a1-d008b32b90f1-istio-podinfo\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:29:25.812569 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:25.812566 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d3e7834-caf7-4113-b0a1-d008b32b90f1-istio-envoy\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:29:25.812796 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:25.812576 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhvgg\" (UniqueName: \"kubernetes.io/projected/1d3e7834-caf7-4113-b0a1-d008b32b90f1-kube-api-access-rhvgg\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:29:25.812796 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:25.812585 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d3e7834-caf7-4113-b0a1-d008b32b90f1-istio-token\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:29:25.812796 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:25.812595 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d3e7834-caf7-4113-b0a1-d008b32b90f1-credential-socket\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:29:25.812796 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:25.812603 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d3e7834-caf7-4113-b0a1-d008b32b90f1-workload-certs\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:29:25.812796 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:25.812612 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d3e7834-caf7-4113-b0a1-d008b32b90f1-istio-data\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:29:25.812796 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:25.812620 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d3e7834-caf7-4113-b0a1-d008b32b90f1-workload-socket\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:29:26.252690 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:26.252659 2564 generic.go:356] "Generic (PLEG): container finished" podID="1d3e7834-caf7-4113-b0a1-d008b32b90f1" containerID="1fea282d2e178148d5554136c0a9c52bfbbeafc8d80f19a75486eef94bf9d193" exitCode=0 Jul 09 07:29:26.252890 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:26.252722 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql" Jul 09 07:29:26.252890 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:26.252742 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql" event={"ID":"1d3e7834-caf7-4113-b0a1-d008b32b90f1","Type":"ContainerDied","Data":"1fea282d2e178148d5554136c0a9c52bfbbeafc8d80f19a75486eef94bf9d193"} Jul 09 07:29:26.252890 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:26.252794 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql" event={"ID":"1d3e7834-caf7-4113-b0a1-d008b32b90f1","Type":"ContainerDied","Data":"fe6ba3d8ef5dd5e7be8f838d9d77af4960adae612b9599e2e075be846d98ec69"} Jul 09 07:29:26.252890 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:26.252817 2564 scope.go:117] "RemoveContainer" containerID="1fea282d2e178148d5554136c0a9c52bfbbeafc8d80f19a75486eef94bf9d193" Jul 09 07:29:26.254301 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:26.254280 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g" event={"ID":"c0e68d52-aeb0-4ed6-811e-467a9b1cc24d","Type":"ContainerStarted","Data":"f6d229f1e389f3d550030072a9cbcfd5c33c02fadc8c527249519e36e2952a3b"} Jul 09 07:29:26.254301 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:26.254317 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g" event={"ID":"c0e68d52-aeb0-4ed6-811e-467a9b1cc24d","Type":"ContainerStarted","Data":"b17f01dfdc3e9f7776291f93ddb1ee10c764987e15d363340115b5481541270f"} Jul 09 07:29:26.261778 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:26.261762 2564 scope.go:117] "RemoveContainer" containerID="1fea282d2e178148d5554136c0a9c52bfbbeafc8d80f19a75486eef94bf9d193" Jul 09 07:29:26.262030 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:29:26.262010 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fea282d2e178148d5554136c0a9c52bfbbeafc8d80f19a75486eef94bf9d193\": container with ID starting with 1fea282d2e178148d5554136c0a9c52bfbbeafc8d80f19a75486eef94bf9d193 not found: ID does not exist" containerID="1fea282d2e178148d5554136c0a9c52bfbbeafc8d80f19a75486eef94bf9d193" Jul 09 07:29:26.262112 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:26.262036 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fea282d2e178148d5554136c0a9c52bfbbeafc8d80f19a75486eef94bf9d193"} err="failed to get container status \"1fea282d2e178148d5554136c0a9c52bfbbeafc8d80f19a75486eef94bf9d193\": rpc error: code = NotFound desc = could not find container \"1fea282d2e178148d5554136c0a9c52bfbbeafc8d80f19a75486eef94bf9d193\": container with ID starting with 1fea282d2e178148d5554136c0a9c52bfbbeafc8d80f19a75486eef94bf9d193 not found: ID does not exist" Jul 09 07:29:26.277081 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:26.277039 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g" podStartSLOduration=1.277025728 podStartE2EDuration="1.277025728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 07:29:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 07:29:26.274475005 +0000 UTC m=+1625.325552233" watchObservedRunningTime="2026-07-09 07:29:26.277025728 +0000 UTC m=+1625.328102957" Jul 09 07:29:26.289899 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:26.289867 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql"] Jul 09 07:29:26.294828 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:26.294802 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-37ae4b70-openshift-default-7c475764b9-jvkql"] Jul 09 07:29:26.626432 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:26.626401 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g" Jul 09 07:29:26.631434 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:26.631410 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g" Jul 09 07:29:27.260328 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:27.260288 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g" Jul 09 07:29:27.261216 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:27.261197 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g" Jul 09 07:29:27.462831 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:27.462804 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d3e7834-caf7-4113-b0a1-d008b32b90f1" path="/var/lib/kubelet/pods/1d3e7834-caf7-4113-b0a1-d008b32b90f1/volumes" Jul 09 07:29:37.627970 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:37.625394 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g"] Jul 09 07:29:37.627970 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:37.625711 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g" podUID="c0e68d52-aeb0-4ed6-811e-467a9b1cc24d" containerName="istio-proxy" containerID="cri-o://f6d229f1e389f3d550030072a9cbcfd5c33c02fadc8c527249519e36e2952a3b" gracePeriod=30 Jul 09 07:29:42.262636 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:42.262596 2564 patch_prober.go:28] interesting pod/e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 09 07:29:42.263026 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:42.262666 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g" podUID="c0e68d52-aeb0-4ed6-811e-467a9b1cc24d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 07:29:42.875476 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:42.875453 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g" Jul 09 07:29:42.961856 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:42.961788 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0e68d52-aeb0-4ed6-811e-467a9b1cc24d-workload-certs\") pod \"c0e68d52-aeb0-4ed6-811e-467a9b1cc24d\" (UID: \"c0e68d52-aeb0-4ed6-811e-467a9b1cc24d\") " Jul 09 07:29:42.961856 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:42.961837 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0e68d52-aeb0-4ed6-811e-467a9b1cc24d-istio-data\") pod \"c0e68d52-aeb0-4ed6-811e-467a9b1cc24d\" (UID: \"c0e68d52-aeb0-4ed6-811e-467a9b1cc24d\") " Jul 09 07:29:42.962037 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:42.961868 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0e68d52-aeb0-4ed6-811e-467a9b1cc24d-istio-envoy\") pod \"c0e68d52-aeb0-4ed6-811e-467a9b1cc24d\" (UID: \"c0e68d52-aeb0-4ed6-811e-467a9b1cc24d\") " Jul 09 07:29:42.962037 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:42.961889 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e68d52-aeb0-4ed6-811e-467a9b1cc24d-credential-socket\") pod \"c0e68d52-aeb0-4ed6-811e-467a9b1cc24d\" (UID: \"c0e68d52-aeb0-4ed6-811e-467a9b1cc24d\") " Jul 09 07:29:42.962037 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:42.961906 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0e68d52-aeb0-4ed6-811e-467a9b1cc24d-istiod-ca-cert\") pod \"c0e68d52-aeb0-4ed6-811e-467a9b1cc24d\" (UID: \"c0e68d52-aeb0-4ed6-811e-467a9b1cc24d\") " Jul 09 07:29:42.962037 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:42.961958 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hktxm\" (UniqueName: \"kubernetes.io/projected/c0e68d52-aeb0-4ed6-811e-467a9b1cc24d-kube-api-access-hktxm\") pod \"c0e68d52-aeb0-4ed6-811e-467a9b1cc24d\" (UID: \"c0e68d52-aeb0-4ed6-811e-467a9b1cc24d\") " Jul 09 07:29:42.962037 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:42.961986 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e68d52-aeb0-4ed6-811e-467a9b1cc24d-workload-socket\") pod \"c0e68d52-aeb0-4ed6-811e-467a9b1cc24d\" (UID: \"c0e68d52-aeb0-4ed6-811e-467a9b1cc24d\") " Jul 09 07:29:42.962037 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:42.962036 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0e68d52-aeb0-4ed6-811e-467a9b1cc24d-istio-podinfo\") pod \"c0e68d52-aeb0-4ed6-811e-467a9b1cc24d\" (UID: \"c0e68d52-aeb0-4ed6-811e-467a9b1cc24d\") " Jul 09 07:29:42.962351 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:42.962048 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0e68d52-aeb0-4ed6-811e-467a9b1cc24d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c0e68d52-aeb0-4ed6-811e-467a9b1cc24d" (UID: "c0e68d52-aeb0-4ed6-811e-467a9b1cc24d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 07:29:42.962351 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:42.962059 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0e68d52-aeb0-4ed6-811e-467a9b1cc24d-istio-token\") pod \"c0e68d52-aeb0-4ed6-811e-467a9b1cc24d\" (UID: \"c0e68d52-aeb0-4ed6-811e-467a9b1cc24d\") " Jul 09 07:29:42.962351 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:42.962163 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0e68d52-aeb0-4ed6-811e-467a9b1cc24d-istio-data" (OuterVolumeSpecName: "istio-data") pod "c0e68d52-aeb0-4ed6-811e-467a9b1cc24d" (UID: "c0e68d52-aeb0-4ed6-811e-467a9b1cc24d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 07:29:42.962351 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:42.962184 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0e68d52-aeb0-4ed6-811e-467a9b1cc24d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c0e68d52-aeb0-4ed6-811e-467a9b1cc24d" (UID: "c0e68d52-aeb0-4ed6-811e-467a9b1cc24d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 07:29:42.962531 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:42.962415 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e68d52-aeb0-4ed6-811e-467a9b1cc24d-credential-socket\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:29:42.962531 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:42.962435 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0e68d52-aeb0-4ed6-811e-467a9b1cc24d-workload-certs\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:29:42.962531 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:42.962451 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0e68d52-aeb0-4ed6-811e-467a9b1cc24d-istio-data\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:29:42.962672 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:42.962643 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0e68d52-aeb0-4ed6-811e-467a9b1cc24d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c0e68d52-aeb0-4ed6-811e-467a9b1cc24d" (UID: "c0e68d52-aeb0-4ed6-811e-467a9b1cc24d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 07:29:42.962720 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:42.962686 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0e68d52-aeb0-4ed6-811e-467a9b1cc24d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c0e68d52-aeb0-4ed6-811e-467a9b1cc24d" (UID: "c0e68d52-aeb0-4ed6-811e-467a9b1cc24d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 07:29:42.964097 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:42.964067 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0e68d52-aeb0-4ed6-811e-467a9b1cc24d-istio-token" (OuterVolumeSpecName: "istio-token") pod "c0e68d52-aeb0-4ed6-811e-467a9b1cc24d" (UID: "c0e68d52-aeb0-4ed6-811e-467a9b1cc24d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 07:29:42.964215 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:42.964164 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0e68d52-aeb0-4ed6-811e-467a9b1cc24d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c0e68d52-aeb0-4ed6-811e-467a9b1cc24d" (UID: "c0e68d52-aeb0-4ed6-811e-467a9b1cc24d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 07:29:42.964379 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:42.964359 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0e68d52-aeb0-4ed6-811e-467a9b1cc24d-kube-api-access-hktxm" (OuterVolumeSpecName: "kube-api-access-hktxm") pod "c0e68d52-aeb0-4ed6-811e-467a9b1cc24d" (UID: "c0e68d52-aeb0-4ed6-811e-467a9b1cc24d"). InnerVolumeSpecName "kube-api-access-hktxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 07:29:42.964446 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:42.964372 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0e68d52-aeb0-4ed6-811e-467a9b1cc24d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c0e68d52-aeb0-4ed6-811e-467a9b1cc24d" (UID: "c0e68d52-aeb0-4ed6-811e-467a9b1cc24d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 07:29:43.063157 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:43.063134 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0e68d52-aeb0-4ed6-811e-467a9b1cc24d-istio-podinfo\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:29:43.063157 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:43.063155 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0e68d52-aeb0-4ed6-811e-467a9b1cc24d-istio-token\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:29:43.063325 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:43.063164 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0e68d52-aeb0-4ed6-811e-467a9b1cc24d-istio-envoy\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:29:43.063325 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:43.063171 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0e68d52-aeb0-4ed6-811e-467a9b1cc24d-istiod-ca-cert\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:29:43.063325 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:43.063180 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hktxm\" (UniqueName: \"kubernetes.io/projected/c0e68d52-aeb0-4ed6-811e-467a9b1cc24d-kube-api-access-hktxm\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:29:43.063325 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:43.063189 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e68d52-aeb0-4ed6-811e-467a9b1cc24d-workload-socket\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:29:43.331476 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:43.331387 2564 generic.go:356] "Generic (PLEG): container finished" podID="c0e68d52-aeb0-4ed6-811e-467a9b1cc24d" containerID="f6d229f1e389f3d550030072a9cbcfd5c33c02fadc8c527249519e36e2952a3b" exitCode=0 Jul 09 07:29:43.331476 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:43.331442 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g" Jul 09 07:29:43.331900 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:43.331481 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g" event={"ID":"c0e68d52-aeb0-4ed6-811e-467a9b1cc24d","Type":"ContainerDied","Data":"f6d229f1e389f3d550030072a9cbcfd5c33c02fadc8c527249519e36e2952a3b"} Jul 09 07:29:43.331900 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:43.331537 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g" event={"ID":"c0e68d52-aeb0-4ed6-811e-467a9b1cc24d","Type":"ContainerDied","Data":"b17f01dfdc3e9f7776291f93ddb1ee10c764987e15d363340115b5481541270f"} Jul 09 07:29:43.331900 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:43.331558 2564 scope.go:117] "RemoveContainer" containerID="f6d229f1e389f3d550030072a9cbcfd5c33c02fadc8c527249519e36e2952a3b" Jul 09 07:29:43.340705 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:43.340689 2564 scope.go:117] "RemoveContainer" containerID="f6d229f1e389f3d550030072a9cbcfd5c33c02fadc8c527249519e36e2952a3b" Jul 09 07:29:43.340943 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:29:43.340924 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6d229f1e389f3d550030072a9cbcfd5c33c02fadc8c527249519e36e2952a3b\": container with ID starting with f6d229f1e389f3d550030072a9cbcfd5c33c02fadc8c527249519e36e2952a3b not found: ID does not exist" containerID="f6d229f1e389f3d550030072a9cbcfd5c33c02fadc8c527249519e36e2952a3b" Jul 09 07:29:43.340997 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:43.340957 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6d229f1e389f3d550030072a9cbcfd5c33c02fadc8c527249519e36e2952a3b"} err="failed to get container status \"f6d229f1e389f3d550030072a9cbcfd5c33c02fadc8c527249519e36e2952a3b\": rpc error: code = NotFound desc = could not find container \"f6d229f1e389f3d550030072a9cbcfd5c33c02fadc8c527249519e36e2952a3b\": container with ID starting with f6d229f1e389f3d550030072a9cbcfd5c33c02fadc8c527249519e36e2952a3b not found: ID does not exist" Jul 09 07:29:43.358759 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:43.358740 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g"] Jul 09 07:29:43.365292 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:43.365258 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-27d28868-openshift-default-745f88cbf-9tv9g"] Jul 09 07:29:43.469102 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:43.469071 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0e68d52-aeb0-4ed6-811e-467a9b1cc24d" path="/var/lib/kubelet/pods/c0e68d52-aeb0-4ed6-811e-467a9b1cc24d/volumes" Jul 09 07:29:44.057532 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:44.057495 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl"] Jul 09 07:29:44.058236 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:44.058212 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0e68d52-aeb0-4ed6-811e-467a9b1cc24d" containerName="istio-proxy" Jul 09 07:29:44.058236 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:44.058235 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0e68d52-aeb0-4ed6-811e-467a9b1cc24d" containerName="istio-proxy" Jul 09 07:29:44.058400 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:44.058253 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d3e7834-caf7-4113-b0a1-d008b32b90f1" containerName="istio-proxy" Jul 09 07:29:44.058400 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:44.058276 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d3e7834-caf7-4113-b0a1-d008b32b90f1" containerName="istio-proxy" Jul 09 07:29:44.058400 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:44.058359 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d3e7834-caf7-4113-b0a1-d008b32b90f1" containerName="istio-proxy" Jul 09 07:29:44.058400 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:44.058373 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0e68d52-aeb0-4ed6-811e-467a9b1cc24d" containerName="istio-proxy" Jul 09 07:29:44.061339 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:44.061319 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl" Jul 09 07:29:44.063914 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:44.063867 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-b1f9470e-openshift-default-dockercfg-9svnf\"" Jul 09 07:29:44.070427 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:44.070405 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl"] Jul 09 07:29:44.173127 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:44.173076 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9228ca3-2cfc-4adc-970b-9fd4b986fc71-istio-podinfo\") pod \"e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl\" (UID: \"a9228ca3-2cfc-4adc-970b-9fd4b986fc71\") " pod="openshift-ingress/e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl" Jul 09 07:29:44.173361 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:44.173151 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9228ca3-2cfc-4adc-970b-9fd4b986fc71-istio-data\") pod \"e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl\" (UID: \"a9228ca3-2cfc-4adc-970b-9fd4b986fc71\") " pod="openshift-ingress/e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl" Jul 09 07:29:44.173361 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:44.173184 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9228ca3-2cfc-4adc-970b-9fd4b986fc71-istiod-ca-cert\") pod \"e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl\" (UID: \"a9228ca3-2cfc-4adc-970b-9fd4b986fc71\") " pod="openshift-ingress/e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl" Jul 09 07:29:44.173361 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:44.173209 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9228ca3-2cfc-4adc-970b-9fd4b986fc71-workload-certs\") pod \"e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl\" (UID: \"a9228ca3-2cfc-4adc-970b-9fd4b986fc71\") " pod="openshift-ingress/e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl" Jul 09 07:29:44.173361 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:44.173277 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9228ca3-2cfc-4adc-970b-9fd4b986fc71-credential-socket\") pod \"e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl\" (UID: \"a9228ca3-2cfc-4adc-970b-9fd4b986fc71\") " pod="openshift-ingress/e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl" Jul 09 07:29:44.173573 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:44.173406 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9228ca3-2cfc-4adc-970b-9fd4b986fc71-workload-socket\") pod \"e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl\" (UID: \"a9228ca3-2cfc-4adc-970b-9fd4b986fc71\") " pod="openshift-ingress/e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl" Jul 09 07:29:44.173573 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:44.173508 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9228ca3-2cfc-4adc-970b-9fd4b986fc71-istio-token\") pod \"e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl\" (UID: \"a9228ca3-2cfc-4adc-970b-9fd4b986fc71\") " pod="openshift-ingress/e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl" Jul 09 07:29:44.173681 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:44.173576 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksgf2\" (UniqueName: \"kubernetes.io/projected/a9228ca3-2cfc-4adc-970b-9fd4b986fc71-kube-api-access-ksgf2\") pod \"e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl\" (UID: \"a9228ca3-2cfc-4adc-970b-9fd4b986fc71\") " pod="openshift-ingress/e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl" Jul 09 07:29:44.173681 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:44.173606 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9228ca3-2cfc-4adc-970b-9fd4b986fc71-istio-envoy\") pod \"e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl\" (UID: \"a9228ca3-2cfc-4adc-970b-9fd4b986fc71\") " pod="openshift-ingress/e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl" Jul 09 07:29:44.274598 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:44.274563 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9228ca3-2cfc-4adc-970b-9fd4b986fc71-credential-socket\") pod \"e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl\" (UID: \"a9228ca3-2cfc-4adc-970b-9fd4b986fc71\") " pod="openshift-ingress/e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl" Jul 09 07:29:44.274598 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:44.274604 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9228ca3-2cfc-4adc-970b-9fd4b986fc71-workload-socket\") pod \"e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl\" (UID: \"a9228ca3-2cfc-4adc-970b-9fd4b986fc71\") " pod="openshift-ingress/e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl" Jul 09 07:29:44.274816 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:44.274627 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9228ca3-2cfc-4adc-970b-9fd4b986fc71-istio-token\") pod \"e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl\" (UID: \"a9228ca3-2cfc-4adc-970b-9fd4b986fc71\") " pod="openshift-ingress/e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl" Jul 09 07:29:44.274816 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:44.274644 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksgf2\" (UniqueName: \"kubernetes.io/projected/a9228ca3-2cfc-4adc-970b-9fd4b986fc71-kube-api-access-ksgf2\") pod \"e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl\" (UID: \"a9228ca3-2cfc-4adc-970b-9fd4b986fc71\") " pod="openshift-ingress/e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl" Jul 09 07:29:44.274816 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:44.274668 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9228ca3-2cfc-4adc-970b-9fd4b986fc71-istio-envoy\") pod \"e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl\" (UID: \"a9228ca3-2cfc-4adc-970b-9fd4b986fc71\") " pod="openshift-ingress/e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl" Jul 09 07:29:44.274816 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:44.274716 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9228ca3-2cfc-4adc-970b-9fd4b986fc71-istio-podinfo\") pod \"e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl\" (UID: \"a9228ca3-2cfc-4adc-970b-9fd4b986fc71\") " pod="openshift-ingress/e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl" Jul 09 07:29:44.274816 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:44.274761 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9228ca3-2cfc-4adc-970b-9fd4b986fc71-istio-data\") pod \"e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl\" (UID: \"a9228ca3-2cfc-4adc-970b-9fd4b986fc71\") " pod="openshift-ingress/e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl" Jul 09 07:29:44.274816 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:44.274804 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9228ca3-2cfc-4adc-970b-9fd4b986fc71-istiod-ca-cert\") pod \"e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl\" (UID: \"a9228ca3-2cfc-4adc-970b-9fd4b986fc71\") " pod="openshift-ingress/e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl" Jul 09 07:29:44.275116 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:44.274883 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9228ca3-2cfc-4adc-970b-9fd4b986fc71-workload-certs\") pod \"e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl\" (UID: \"a9228ca3-2cfc-4adc-970b-9fd4b986fc71\") " pod="openshift-ingress/e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl" Jul 09 07:29:44.275116 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:44.274998 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9228ca3-2cfc-4adc-970b-9fd4b986fc71-workload-socket\") pod \"e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl\" (UID: \"a9228ca3-2cfc-4adc-970b-9fd4b986fc71\") " pod="openshift-ingress/e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl" Jul 09 07:29:44.275116 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:44.275029 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9228ca3-2cfc-4adc-970b-9fd4b986fc71-credential-socket\") pod \"e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl\" (UID: \"a9228ca3-2cfc-4adc-970b-9fd4b986fc71\") " pod="openshift-ingress/e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl" Jul 09 07:29:44.275302 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:44.275199 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9228ca3-2cfc-4adc-970b-9fd4b986fc71-istio-data\") pod \"e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl\" (UID: \"a9228ca3-2cfc-4adc-970b-9fd4b986fc71\") " pod="openshift-ingress/e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl" Jul 09 07:29:44.275302 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:44.275204 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9228ca3-2cfc-4adc-970b-9fd4b986fc71-workload-certs\") pod \"e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl\" (UID: \"a9228ca3-2cfc-4adc-970b-9fd4b986fc71\") " pod="openshift-ingress/e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl" Jul 09 07:29:44.275523 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:44.275503 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9228ca3-2cfc-4adc-970b-9fd4b986fc71-istiod-ca-cert\") pod \"e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl\" (UID: \"a9228ca3-2cfc-4adc-970b-9fd4b986fc71\") " pod="openshift-ingress/e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl" Jul 09 07:29:44.277095 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:44.277071 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9228ca3-2cfc-4adc-970b-9fd4b986fc71-istio-podinfo\") pod \"e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl\" (UID: \"a9228ca3-2cfc-4adc-970b-9fd4b986fc71\") " pod="openshift-ingress/e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl" Jul 09 07:29:44.277198 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:44.277168 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9228ca3-2cfc-4adc-970b-9fd4b986fc71-istio-envoy\") pod \"e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl\" (UID: \"a9228ca3-2cfc-4adc-970b-9fd4b986fc71\") " pod="openshift-ingress/e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl" Jul 09 07:29:44.284898 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:44.284879 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9228ca3-2cfc-4adc-970b-9fd4b986fc71-istio-token\") pod \"e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl\" (UID: \"a9228ca3-2cfc-4adc-970b-9fd4b986fc71\") " pod="openshift-ingress/e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl" Jul 09 07:29:44.286696 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:44.286679 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksgf2\" (UniqueName: \"kubernetes.io/projected/a9228ca3-2cfc-4adc-970b-9fd4b986fc71-kube-api-access-ksgf2\") pod \"e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl\" (UID: \"a9228ca3-2cfc-4adc-970b-9fd4b986fc71\") " pod="openshift-ingress/e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl" Jul 09 07:29:44.373127 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:44.373053 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl" Jul 09 07:29:44.505358 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:44.505334 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl"] Jul 09 07:29:44.506863 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:29:44.506840 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9228ca3_2cfc_4adc_970b_9fd4b986fc71.slice/crio-bb6f414a5b2893994338aa6040c0d769676af2a69241b218205527b5300dd510 WatchSource:0}: Error finding container bb6f414a5b2893994338aa6040c0d769676af2a69241b218205527b5300dd510: Status 404 returned error can't find the container with id bb6f414a5b2893994338aa6040c0d769676af2a69241b218205527b5300dd510 Jul 09 07:29:44.508889 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:44.508853 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 07:29:44.508970 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:44.508914 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 07:29:44.508970 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:44.508943 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 07:29:45.343871 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:45.343838 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl" event={"ID":"a9228ca3-2cfc-4adc-970b-9fd4b986fc71","Type":"ContainerStarted","Data":"fcec64107ab8da5b1be30d81093a9908269da45e4b2e9cf993f44e43a943c9e4"} Jul 09 07:29:45.344063 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:45.343890 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl" event={"ID":"a9228ca3-2cfc-4adc-970b-9fd4b986fc71","Type":"ContainerStarted","Data":"bb6f414a5b2893994338aa6040c0d769676af2a69241b218205527b5300dd510"} Jul 09 07:29:45.371390 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:45.371337 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl" podStartSLOduration=1.371318661 podStartE2EDuration="1.371318661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 07:29:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 07:29:45.368883843 +0000 UTC m=+1644.419961063" watchObservedRunningTime="2026-07-09 07:29:45.371318661 +0000 UTC m=+1644.422395889" Jul 09 07:29:45.373500 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:45.373471 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl" Jul 09 07:29:45.398103 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:45.398077 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl" Jul 09 07:29:46.348213 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:46.348182 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl" Jul 09 07:29:46.349192 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:46.349175 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl" Jul 09 07:29:49.922896 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:49.922863 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl"] Jul 09 07:29:49.923324 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:49.923036 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl" podUID="a9228ca3-2cfc-4adc-970b-9fd4b986fc71" containerName="istio-proxy" containerID="cri-o://fcec64107ab8da5b1be30d81093a9908269da45e4b2e9cf993f44e43a943c9e4" gracePeriod=30 Jul 09 07:29:55.973598 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:55.973546 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl" Jul 09 07:29:55.988464 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:55.988442 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9228ca3-2cfc-4adc-970b-9fd4b986fc71-istiod-ca-cert\") pod \"a9228ca3-2cfc-4adc-970b-9fd4b986fc71\" (UID: \"a9228ca3-2cfc-4adc-970b-9fd4b986fc71\") " Jul 09 07:29:55.988566 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:55.988478 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9228ca3-2cfc-4adc-970b-9fd4b986fc71-workload-socket\") pod \"a9228ca3-2cfc-4adc-970b-9fd4b986fc71\" (UID: \"a9228ca3-2cfc-4adc-970b-9fd4b986fc71\") " Jul 09 07:29:55.988566 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:55.988507 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9228ca3-2cfc-4adc-970b-9fd4b986fc71-istio-data\") pod \"a9228ca3-2cfc-4adc-970b-9fd4b986fc71\" (UID: \"a9228ca3-2cfc-4adc-970b-9fd4b986fc71\") " Jul 09 07:29:55.988566 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:55.988529 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ksgf2\" (UniqueName: \"kubernetes.io/projected/a9228ca3-2cfc-4adc-970b-9fd4b986fc71-kube-api-access-ksgf2\") pod \"a9228ca3-2cfc-4adc-970b-9fd4b986fc71\" (UID: \"a9228ca3-2cfc-4adc-970b-9fd4b986fc71\") " Jul 09 07:29:55.988566 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:55.988548 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9228ca3-2cfc-4adc-970b-9fd4b986fc71-istio-token\") pod \"a9228ca3-2cfc-4adc-970b-9fd4b986fc71\" (UID: \"a9228ca3-2cfc-4adc-970b-9fd4b986fc71\") " Jul 09 07:29:55.988566 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:55.988563 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9228ca3-2cfc-4adc-970b-9fd4b986fc71-istio-envoy\") pod \"a9228ca3-2cfc-4adc-970b-9fd4b986fc71\" (UID: \"a9228ca3-2cfc-4adc-970b-9fd4b986fc71\") " Jul 09 07:29:55.988808 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:55.988609 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9228ca3-2cfc-4adc-970b-9fd4b986fc71-istio-podinfo\") pod \"a9228ca3-2cfc-4adc-970b-9fd4b986fc71\" (UID: \"a9228ca3-2cfc-4adc-970b-9fd4b986fc71\") " Jul 09 07:29:55.988808 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:55.988679 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9228ca3-2cfc-4adc-970b-9fd4b986fc71-workload-certs\") pod \"a9228ca3-2cfc-4adc-970b-9fd4b986fc71\" (UID: \"a9228ca3-2cfc-4adc-970b-9fd4b986fc71\") " Jul 09 07:29:55.988808 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:55.988745 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9228ca3-2cfc-4adc-970b-9fd4b986fc71-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a9228ca3-2cfc-4adc-970b-9fd4b986fc71" (UID: "a9228ca3-2cfc-4adc-970b-9fd4b986fc71"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 07:29:55.988973 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:55.988853 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9228ca3-2cfc-4adc-970b-9fd4b986fc71-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a9228ca3-2cfc-4adc-970b-9fd4b986fc71" (UID: "a9228ca3-2cfc-4adc-970b-9fd4b986fc71"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 07:29:55.989054 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:55.989032 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9228ca3-2cfc-4adc-970b-9fd4b986fc71-credential-socket\") pod \"a9228ca3-2cfc-4adc-970b-9fd4b986fc71\" (UID: \"a9228ca3-2cfc-4adc-970b-9fd4b986fc71\") " Jul 09 07:29:55.989120 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:55.989065 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9228ca3-2cfc-4adc-970b-9fd4b986fc71-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a9228ca3-2cfc-4adc-970b-9fd4b986fc71" (UID: "a9228ca3-2cfc-4adc-970b-9fd4b986fc71"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 07:29:55.989370 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:55.989325 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9228ca3-2cfc-4adc-970b-9fd4b986fc71-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a9228ca3-2cfc-4adc-970b-9fd4b986fc71" (UID: "a9228ca3-2cfc-4adc-970b-9fd4b986fc71"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 07:29:55.989370 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:55.989336 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9228ca3-2cfc-4adc-970b-9fd4b986fc71-workload-certs\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:29:55.989370 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:55.989368 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9228ca3-2cfc-4adc-970b-9fd4b986fc71-istiod-ca-cert\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:29:55.989608 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:55.989372 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9228ca3-2cfc-4adc-970b-9fd4b986fc71-istio-data" (OuterVolumeSpecName: "istio-data") pod "a9228ca3-2cfc-4adc-970b-9fd4b986fc71" (UID: "a9228ca3-2cfc-4adc-970b-9fd4b986fc71"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 07:29:55.989608 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:55.989385 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9228ca3-2cfc-4adc-970b-9fd4b986fc71-workload-socket\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:29:55.990690 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:55.990665 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9228ca3-2cfc-4adc-970b-9fd4b986fc71-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a9228ca3-2cfc-4adc-970b-9fd4b986fc71" (UID: "a9228ca3-2cfc-4adc-970b-9fd4b986fc71"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 07:29:55.990797 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:55.990765 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9228ca3-2cfc-4adc-970b-9fd4b986fc71-istio-token" (OuterVolumeSpecName: "istio-token") pod "a9228ca3-2cfc-4adc-970b-9fd4b986fc71" (UID: "a9228ca3-2cfc-4adc-970b-9fd4b986fc71"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 07:29:55.991050 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:55.991031 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a9228ca3-2cfc-4adc-970b-9fd4b986fc71-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a9228ca3-2cfc-4adc-970b-9fd4b986fc71" (UID: "a9228ca3-2cfc-4adc-970b-9fd4b986fc71"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 07:29:55.991107 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:55.991058 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9228ca3-2cfc-4adc-970b-9fd4b986fc71-kube-api-access-ksgf2" (OuterVolumeSpecName: "kube-api-access-ksgf2") pod "a9228ca3-2cfc-4adc-970b-9fd4b986fc71" (UID: "a9228ca3-2cfc-4adc-970b-9fd4b986fc71"). InnerVolumeSpecName "kube-api-access-ksgf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 07:29:56.090662 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:56.090632 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9228ca3-2cfc-4adc-970b-9fd4b986fc71-istio-podinfo\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:29:56.090662 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:56.090660 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9228ca3-2cfc-4adc-970b-9fd4b986fc71-credential-socket\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:29:56.090662 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:56.090670 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9228ca3-2cfc-4adc-970b-9fd4b986fc71-istio-data\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:29:56.090874 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:56.090678 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksgf2\" (UniqueName: \"kubernetes.io/projected/a9228ca3-2cfc-4adc-970b-9fd4b986fc71-kube-api-access-ksgf2\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:29:56.090874 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:56.090687 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9228ca3-2cfc-4adc-970b-9fd4b986fc71-istio-token\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:29:56.090874 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:56.090696 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9228ca3-2cfc-4adc-970b-9fd4b986fc71-istio-envoy\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:29:56.392852 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:56.392818 2564 generic.go:356] "Generic (PLEG): container finished" podID="a9228ca3-2cfc-4adc-970b-9fd4b986fc71" containerID="fcec64107ab8da5b1be30d81093a9908269da45e4b2e9cf993f44e43a943c9e4" exitCode=0 Jul 09 07:29:56.393033 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:56.392874 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl" Jul 09 07:29:56.393033 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:56.392897 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl" event={"ID":"a9228ca3-2cfc-4adc-970b-9fd4b986fc71","Type":"ContainerDied","Data":"fcec64107ab8da5b1be30d81093a9908269da45e4b2e9cf993f44e43a943c9e4"} Jul 09 07:29:56.393033 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:56.392951 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl" event={"ID":"a9228ca3-2cfc-4adc-970b-9fd4b986fc71","Type":"ContainerDied","Data":"bb6f414a5b2893994338aa6040c0d769676af2a69241b218205527b5300dd510"} Jul 09 07:29:56.393033 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:56.392982 2564 scope.go:117] "RemoveContainer" containerID="fcec64107ab8da5b1be30d81093a9908269da45e4b2e9cf993f44e43a943c9e4" Jul 09 07:29:56.402190 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:56.402166 2564 scope.go:117] "RemoveContainer" containerID="fcec64107ab8da5b1be30d81093a9908269da45e4b2e9cf993f44e43a943c9e4" Jul 09 07:29:56.402480 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:29:56.402454 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcec64107ab8da5b1be30d81093a9908269da45e4b2e9cf993f44e43a943c9e4\": container with ID starting with fcec64107ab8da5b1be30d81093a9908269da45e4b2e9cf993f44e43a943c9e4 not found: ID does not exist" containerID="fcec64107ab8da5b1be30d81093a9908269da45e4b2e9cf993f44e43a943c9e4" Jul 09 07:29:56.402567 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:56.402491 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcec64107ab8da5b1be30d81093a9908269da45e4b2e9cf993f44e43a943c9e4"} err="failed to get container status \"fcec64107ab8da5b1be30d81093a9908269da45e4b2e9cf993f44e43a943c9e4\": rpc error: code = NotFound desc = could not find container \"fcec64107ab8da5b1be30d81093a9908269da45e4b2e9cf993f44e43a943c9e4\": container with ID starting with fcec64107ab8da5b1be30d81093a9908269da45e4b2e9cf993f44e43a943c9e4 not found: ID does not exist" Jul 09 07:29:56.421088 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:56.421061 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl"] Jul 09 07:29:56.427346 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:56.427321 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-b1f9470e-openshift-default-655d8bd6f9-gx5tl"] Jul 09 07:29:57.462225 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:29:57.462185 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9228ca3-2cfc-4adc-970b-9fd4b986fc71" path="/var/lib/kubelet/pods/a9228ca3-2cfc-4adc-970b-9fd4b986fc71/volumes" Jul 09 07:30:00.128525 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:30:00.128496 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726370-pf2bf"] Jul 09 07:30:00.128995 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:30:00.128977 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9228ca3-2cfc-4adc-970b-9fd4b986fc71" containerName="istio-proxy" Jul 09 07:30:00.129072 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:30:00.128999 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9228ca3-2cfc-4adc-970b-9fd4b986fc71" containerName="istio-proxy" Jul 09 07:30:00.129131 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:30:00.129114 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9228ca3-2cfc-4adc-970b-9fd4b986fc71" containerName="istio-proxy" Jul 09 07:30:00.150774 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:30:00.150749 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726370-pf2bf"] Jul 09 07:30:00.150922 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:30:00.150854 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726370-pf2bf" Jul 09 07:30:00.153727 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:30:00.153707 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 09 07:30:00.154669 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:30:00.154649 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 09 07:30:00.154768 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:30:00.154656 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-tsnn4\"" Jul 09 07:30:00.225310 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:30:00.225259 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glw6g\" (UniqueName: \"kubernetes.io/projected/09c68e3c-c223-470d-a1b9-35142159cee7-kube-api-access-glw6g\") pod \"maas-api-key-cleanup-29726370-pf2bf\" (UID: \"09c68e3c-c223-470d-a1b9-35142159cee7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726370-pf2bf" Jul 09 07:30:00.326018 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:30:00.325989 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glw6g\" (UniqueName: \"kubernetes.io/projected/09c68e3c-c223-470d-a1b9-35142159cee7-kube-api-access-glw6g\") pod \"maas-api-key-cleanup-29726370-pf2bf\" (UID: \"09c68e3c-c223-470d-a1b9-35142159cee7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726370-pf2bf" Jul 09 07:30:00.334836 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:30:00.334805 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glw6g\" (UniqueName: \"kubernetes.io/projected/09c68e3c-c223-470d-a1b9-35142159cee7-kube-api-access-glw6g\") pod \"maas-api-key-cleanup-29726370-pf2bf\" (UID: \"09c68e3c-c223-470d-a1b9-35142159cee7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726370-pf2bf" Jul 09 07:30:00.460896 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:30:00.460825 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726370-pf2bf" Jul 09 07:30:00.588136 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:30:00.588109 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726370-pf2bf"] Jul 09 07:30:00.590244 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:30:00.590219 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09c68e3c_c223_470d_a1b9_35142159cee7.slice/crio-680bda533f90fd50a87d369d07d2f665f621a30c097cd9f8fdd8d576970a4d42 WatchSource:0}: Error finding container 680bda533f90fd50a87d369d07d2f665f621a30c097cd9f8fdd8d576970a4d42: Status 404 returned error can't find the container with id 680bda533f90fd50a87d369d07d2f665f621a30c097cd9f8fdd8d576970a4d42 Jul 09 07:30:01.416347 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:30:01.416314 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726370-pf2bf" event={"ID":"09c68e3c-c223-470d-a1b9-35142159cee7","Type":"ContainerStarted","Data":"680bda533f90fd50a87d369d07d2f665f621a30c097cd9f8fdd8d576970a4d42"} Jul 09 07:30:03.427542 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:30:03.427456 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726370-pf2bf" event={"ID":"09c68e3c-c223-470d-a1b9-35142159cee7","Type":"ContainerStarted","Data":"bd3649c98edf2b7e078a21d68a81e449cabca227b7993e497371d21ed9472939"} Jul 09 07:30:03.443577 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:30:03.443522 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726370-pf2bf" podStartSLOduration=1.221883605 podStartE2EDuration="3.443503968s" totalImagesPullingTime="2.221620363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 07:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 07:30:03.44269163 +0000 UTC m=+1662.493768859" watchObservedRunningTime="2026-07-09 07:30:03.443503968 +0000 UTC m=+1662.494581196" Jul 09 07:31:00.010418 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:00.010342 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726370-pf2bf"] Jul 09 07:31:00.010905 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:00.010540 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726370-pf2bf" podUID="09c68e3c-c223-470d-a1b9-35142159cee7" containerName="cleanup" containerID="cri-o://bd3649c98edf2b7e078a21d68a81e449cabca227b7993e497371d21ed9472939" gracePeriod=30 Jul 09 07:31:30.174063 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:30.174041 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726370-pf2bf" Jul 09 07:31:30.266277 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:30.266191 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-glw6g\" (UniqueName: \"kubernetes.io/projected/09c68e3c-c223-470d-a1b9-35142159cee7-kube-api-access-glw6g\") pod \"09c68e3c-c223-470d-a1b9-35142159cee7\" (UID: \"09c68e3c-c223-470d-a1b9-35142159cee7\") " Jul 09 07:31:30.268297 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:30.268243 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09c68e3c-c223-470d-a1b9-35142159cee7-kube-api-access-glw6g" (OuterVolumeSpecName: "kube-api-access-glw6g") pod "09c68e3c-c223-470d-a1b9-35142159cee7" (UID: "09c68e3c-c223-470d-a1b9-35142159cee7"). InnerVolumeSpecName "kube-api-access-glw6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 07:31:30.367012 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:30.366983 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glw6g\" (UniqueName: \"kubernetes.io/projected/09c68e3c-c223-470d-a1b9-35142159cee7-kube-api-access-glw6g\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:31:30.821235 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:30.821195 2564 generic.go:356] "Generic (PLEG): container finished" podID="09c68e3c-c223-470d-a1b9-35142159cee7" containerID="bd3649c98edf2b7e078a21d68a81e449cabca227b7993e497371d21ed9472939" exitCode=137 Jul 09 07:31:30.821235 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:30.821240 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726370-pf2bf" Jul 09 07:31:30.821510 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:30.821297 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726370-pf2bf" event={"ID":"09c68e3c-c223-470d-a1b9-35142159cee7","Type":"ContainerDied","Data":"bd3649c98edf2b7e078a21d68a81e449cabca227b7993e497371d21ed9472939"} Jul 09 07:31:30.821510 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:30.821363 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726370-pf2bf" event={"ID":"09c68e3c-c223-470d-a1b9-35142159cee7","Type":"ContainerDied","Data":"680bda533f90fd50a87d369d07d2f665f621a30c097cd9f8fdd8d576970a4d42"} Jul 09 07:31:30.821510 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:30.821394 2564 scope.go:117] "RemoveContainer" containerID="bd3649c98edf2b7e078a21d68a81e449cabca227b7993e497371d21ed9472939" Jul 09 07:31:30.830065 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:30.830046 2564 scope.go:117] "RemoveContainer" containerID="bd3649c98edf2b7e078a21d68a81e449cabca227b7993e497371d21ed9472939" Jul 09 07:31:30.830339 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:31:30.830317 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd3649c98edf2b7e078a21d68a81e449cabca227b7993e497371d21ed9472939\": container with ID starting with bd3649c98edf2b7e078a21d68a81e449cabca227b7993e497371d21ed9472939 not found: ID does not exist" containerID="bd3649c98edf2b7e078a21d68a81e449cabca227b7993e497371d21ed9472939" Jul 09 07:31:30.830391 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:30.830349 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd3649c98edf2b7e078a21d68a81e449cabca227b7993e497371d21ed9472939"} err="failed to get container status \"bd3649c98edf2b7e078a21d68a81e449cabca227b7993e497371d21ed9472939\": rpc error: code = NotFound desc = could not find container \"bd3649c98edf2b7e078a21d68a81e449cabca227b7993e497371d21ed9472939\": container with ID starting with bd3649c98edf2b7e078a21d68a81e449cabca227b7993e497371d21ed9472939 not found: ID does not exist" Jul 09 07:31:30.843244 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:30.843222 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726370-pf2bf"] Jul 09 07:31:30.849474 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:30.849455 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726370-pf2bf"] Jul 09 07:31:31.462509 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:31.462482 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09c68e3c-c223-470d-a1b9-35142159cee7" path="/var/lib/kubelet/pods/09c68e3c-c223-470d-a1b9-35142159cee7/volumes" Jul 09 07:31:31.969046 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:31.969016 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx"] Jul 09 07:31:31.969454 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:31.969440 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09c68e3c-c223-470d-a1b9-35142159cee7" containerName="cleanup" Jul 09 07:31:31.969512 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:31.969456 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="09c68e3c-c223-470d-a1b9-35142159cee7" containerName="cleanup" Jul 09 07:31:31.969547 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:31.969518 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="09c68e3c-c223-470d-a1b9-35142159cee7" containerName="cleanup" Jul 09 07:31:31.973706 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:31.973685 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx" Jul 09 07:31:31.976623 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:31.976603 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-0c3ae94c-openshift-default-dockercfg-gj7hf\"" Jul 09 07:31:31.984673 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:31.984652 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx"] Jul 09 07:31:32.080291 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:32.080247 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc6d719-63c1-4ec7-8bf9-efc41adbafc2-workload-socket\") pod \"e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx\" (UID: \"cbc6d719-63c1-4ec7-8bf9-efc41adbafc2\") " pod="openshift-ingress/e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx" Jul 09 07:31:32.080291 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:32.080296 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pddqt\" (UniqueName: \"kubernetes.io/projected/cbc6d719-63c1-4ec7-8bf9-efc41adbafc2-kube-api-access-pddqt\") pod \"e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx\" (UID: \"cbc6d719-63c1-4ec7-8bf9-efc41adbafc2\") " pod="openshift-ingress/e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx" Jul 09 07:31:32.080523 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:32.080344 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbc6d719-63c1-4ec7-8bf9-efc41adbafc2-istio-envoy\") pod \"e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx\" (UID: \"cbc6d719-63c1-4ec7-8bf9-efc41adbafc2\") " pod="openshift-ingress/e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx" Jul 09 07:31:32.080523 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:32.080365 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbc6d719-63c1-4ec7-8bf9-efc41adbafc2-workload-certs\") pod \"e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx\" (UID: \"cbc6d719-63c1-4ec7-8bf9-efc41adbafc2\") " pod="openshift-ingress/e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx" Jul 09 07:31:32.080523 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:32.080397 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbc6d719-63c1-4ec7-8bf9-efc41adbafc2-istio-data\") pod \"e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx\" (UID: \"cbc6d719-63c1-4ec7-8bf9-efc41adbafc2\") " pod="openshift-ingress/e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx" Jul 09 07:31:32.080523 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:32.080430 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbc6d719-63c1-4ec7-8bf9-efc41adbafc2-istiod-ca-cert\") pod \"e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx\" (UID: \"cbc6d719-63c1-4ec7-8bf9-efc41adbafc2\") " pod="openshift-ingress/e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx" Jul 09 07:31:32.080523 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:32.080454 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbc6d719-63c1-4ec7-8bf9-efc41adbafc2-istio-podinfo\") pod \"e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx\" (UID: \"cbc6d719-63c1-4ec7-8bf9-efc41adbafc2\") " pod="openshift-ingress/e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx" Jul 09 07:31:32.080686 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:32.080557 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc6d719-63c1-4ec7-8bf9-efc41adbafc2-credential-socket\") pod \"e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx\" (UID: \"cbc6d719-63c1-4ec7-8bf9-efc41adbafc2\") " pod="openshift-ingress/e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx" Jul 09 07:31:32.080686 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:32.080628 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbc6d719-63c1-4ec7-8bf9-efc41adbafc2-istio-token\") pod \"e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx\" (UID: \"cbc6d719-63c1-4ec7-8bf9-efc41adbafc2\") " pod="openshift-ingress/e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx" Jul 09 07:31:32.181319 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:32.181283 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbc6d719-63c1-4ec7-8bf9-efc41adbafc2-istio-data\") pod \"e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx\" (UID: \"cbc6d719-63c1-4ec7-8bf9-efc41adbafc2\") " pod="openshift-ingress/e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx" Jul 09 07:31:32.181486 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:32.181347 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbc6d719-63c1-4ec7-8bf9-efc41adbafc2-istiod-ca-cert\") pod \"e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx\" (UID: \"cbc6d719-63c1-4ec7-8bf9-efc41adbafc2\") " pod="openshift-ingress/e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx" Jul 09 07:31:32.181486 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:32.181375 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbc6d719-63c1-4ec7-8bf9-efc41adbafc2-istio-podinfo\") pod \"e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx\" (UID: \"cbc6d719-63c1-4ec7-8bf9-efc41adbafc2\") " pod="openshift-ingress/e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx" Jul 09 07:31:32.181486 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:32.181415 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc6d719-63c1-4ec7-8bf9-efc41adbafc2-credential-socket\") pod \"e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx\" (UID: \"cbc6d719-63c1-4ec7-8bf9-efc41adbafc2\") " pod="openshift-ingress/e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx" Jul 09 07:31:32.181486 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:32.181461 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbc6d719-63c1-4ec7-8bf9-efc41adbafc2-istio-token\") pod \"e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx\" (UID: \"cbc6d719-63c1-4ec7-8bf9-efc41adbafc2\") " pod="openshift-ingress/e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx" Jul 09 07:31:32.181699 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:32.181512 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc6d719-63c1-4ec7-8bf9-efc41adbafc2-workload-socket\") pod \"e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx\" (UID: \"cbc6d719-63c1-4ec7-8bf9-efc41adbafc2\") " pod="openshift-ingress/e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx" Jul 09 07:31:32.181756 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:32.181715 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbc6d719-63c1-4ec7-8bf9-efc41adbafc2-istio-data\") pod \"e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx\" (UID: \"cbc6d719-63c1-4ec7-8bf9-efc41adbafc2\") " pod="openshift-ingress/e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx" Jul 09 07:31:32.181830 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:32.181789 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pddqt\" (UniqueName: \"kubernetes.io/projected/cbc6d719-63c1-4ec7-8bf9-efc41adbafc2-kube-api-access-pddqt\") pod \"e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx\" (UID: \"cbc6d719-63c1-4ec7-8bf9-efc41adbafc2\") " pod="openshift-ingress/e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx" Jul 09 07:31:32.181830 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:32.181813 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc6d719-63c1-4ec7-8bf9-efc41adbafc2-credential-socket\") pod \"e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx\" (UID: \"cbc6d719-63c1-4ec7-8bf9-efc41adbafc2\") " pod="openshift-ingress/e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx" Jul 09 07:31:32.181948 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:32.181851 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc6d719-63c1-4ec7-8bf9-efc41adbafc2-workload-socket\") pod \"e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx\" (UID: \"cbc6d719-63c1-4ec7-8bf9-efc41adbafc2\") " pod="openshift-ingress/e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx" Jul 09 07:31:32.181948 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:32.181853 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbc6d719-63c1-4ec7-8bf9-efc41adbafc2-istio-envoy\") pod \"e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx\" (UID: \"cbc6d719-63c1-4ec7-8bf9-efc41adbafc2\") " pod="openshift-ingress/e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx" Jul 09 07:31:32.181948 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:32.181910 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbc6d719-63c1-4ec7-8bf9-efc41adbafc2-workload-certs\") pod \"e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx\" (UID: \"cbc6d719-63c1-4ec7-8bf9-efc41adbafc2\") " pod="openshift-ingress/e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx" Jul 09 07:31:32.182114 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:32.182054 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbc6d719-63c1-4ec7-8bf9-efc41adbafc2-istiod-ca-cert\") pod \"e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx\" (UID: \"cbc6d719-63c1-4ec7-8bf9-efc41adbafc2\") " pod="openshift-ingress/e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx" Jul 09 07:31:32.182204 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:32.182187 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbc6d719-63c1-4ec7-8bf9-efc41adbafc2-workload-certs\") pod \"e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx\" (UID: \"cbc6d719-63c1-4ec7-8bf9-efc41adbafc2\") " pod="openshift-ingress/e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx" Jul 09 07:31:32.183883 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:32.183860 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbc6d719-63c1-4ec7-8bf9-efc41adbafc2-istio-envoy\") pod \"e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx\" (UID: \"cbc6d719-63c1-4ec7-8bf9-efc41adbafc2\") " pod="openshift-ingress/e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx" Jul 09 07:31:32.184178 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:32.184159 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbc6d719-63c1-4ec7-8bf9-efc41adbafc2-istio-podinfo\") pod \"e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx\" (UID: \"cbc6d719-63c1-4ec7-8bf9-efc41adbafc2\") " pod="openshift-ingress/e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx" Jul 09 07:31:32.190435 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:32.190408 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbc6d719-63c1-4ec7-8bf9-efc41adbafc2-istio-token\") pod \"e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx\" (UID: \"cbc6d719-63c1-4ec7-8bf9-efc41adbafc2\") " pod="openshift-ingress/e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx" Jul 09 07:31:32.190729 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:32.190708 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pddqt\" (UniqueName: \"kubernetes.io/projected/cbc6d719-63c1-4ec7-8bf9-efc41adbafc2-kube-api-access-pddqt\") pod \"e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx\" (UID: \"cbc6d719-63c1-4ec7-8bf9-efc41adbafc2\") " pod="openshift-ingress/e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx" Jul 09 07:31:32.284320 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:32.284237 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx" Jul 09 07:31:32.421799 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:32.421763 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx"] Jul 09 07:31:32.423220 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:31:32.423176 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbc6d719_63c1_4ec7_8bf9_efc41adbafc2.slice/crio-28b8a089902b04e6e56855cb6bfb6818d41d167a285a3788758f621c4b171054 WatchSource:0}: Error finding container 28b8a089902b04e6e56855cb6bfb6818d41d167a285a3788758f621c4b171054: Status 404 returned error can't find the container with id 28b8a089902b04e6e56855cb6bfb6818d41d167a285a3788758f621c4b171054 Jul 09 07:31:32.425661 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:32.425622 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 07:31:32.425743 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:32.425732 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 07:31:32.425789 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:32.425774 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 07:31:32.839359 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:32.839318 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx" event={"ID":"cbc6d719-63c1-4ec7-8bf9-efc41adbafc2","Type":"ContainerStarted","Data":"d91df7b801c0d21d35a9189fa45fc8805886f825f35bbc309b714221f619263b"} Jul 09 07:31:32.839810 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:32.839412 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx" event={"ID":"cbc6d719-63c1-4ec7-8bf9-efc41adbafc2","Type":"ContainerStarted","Data":"28b8a089902b04e6e56855cb6bfb6818d41d167a285a3788758f621c4b171054"} Jul 09 07:31:32.859604 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:32.859552 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx" podStartSLOduration=1.859536472 podStartE2EDuration="1.859536472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 07:31:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 07:31:32.858200308 +0000 UTC m=+1751.909277538" watchObservedRunningTime="2026-07-09 07:31:32.859536472 +0000 UTC m=+1751.910613699" Jul 09 07:31:33.284950 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:33.284904 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx" Jul 09 07:31:33.290017 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:33.289984 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx" Jul 09 07:31:33.844222 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:33.844189 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx" Jul 09 07:31:33.845036 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:33.845018 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx" Jul 09 07:31:45.900948 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:45.900915 2564 generic.go:356] "Generic (PLEG): container finished" podID="cbc6d719-63c1-4ec7-8bf9-efc41adbafc2" containerID="d91df7b801c0d21d35a9189fa45fc8805886f825f35bbc309b714221f619263b" exitCode=0 Jul 09 07:31:45.901364 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:45.900973 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx" event={"ID":"cbc6d719-63c1-4ec7-8bf9-efc41adbafc2","Type":"ContainerDied","Data":"d91df7b801c0d21d35a9189fa45fc8805886f825f35bbc309b714221f619263b"} Jul 09 07:31:45.901364 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:45.901352 2564 scope.go:117] "RemoveContainer" containerID="d91df7b801c0d21d35a9189fa45fc8805886f825f35bbc309b714221f619263b" Jul 09 07:31:45.902227 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:45.902197 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 07:31:45.902304 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:45.902286 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 07:31:45.902349 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:45.902326 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 07:31:46.909052 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:46.909021 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx" event={"ID":"cbc6d719-63c1-4ec7-8bf9-efc41adbafc2","Type":"ContainerStarted","Data":"fbbb0e47a2b98ef76dcb7d000f7c8bba9769170fce7ca17a816af9098011fe92"} Jul 09 07:31:47.284661 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:47.284565 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx" Jul 09 07:31:47.285766 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:47.285739 2564 patch_prober.go:28] interesting pod/e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Jul 09 07:31:47.285893 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:47.285787 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx" podUID="cbc6d719-63c1-4ec7-8bf9-efc41adbafc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Jul 09 07:31:48.284814 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:48.284776 2564 patch_prober.go:28] interesting pod/e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Jul 09 07:31:48.285226 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:48.284834 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx" podUID="cbc6d719-63c1-4ec7-8bf9-efc41adbafc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Jul 09 07:31:48.844386 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:48.844350 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx" Jul 09 07:31:49.285329 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:49.285294 2564 patch_prober.go:28] interesting pod/e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Jul 09 07:31:49.285707 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:49.285365 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx" podUID="cbc6d719-63c1-4ec7-8bf9-efc41adbafc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Jul 09 07:31:50.285247 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:50.285209 2564 patch_prober.go:28] interesting pod/e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Jul 09 07:31:50.285447 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:50.285289 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx" podUID="cbc6d719-63c1-4ec7-8bf9-efc41adbafc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Jul 09 07:31:51.288961 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:51.288932 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx" Jul 09 07:31:51.289833 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:51.289816 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx" Jul 09 07:31:52.888920 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:52.888888 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx"] Jul 09 07:31:52.941062 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:52.941021 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx" podUID="cbc6d719-63c1-4ec7-8bf9-efc41adbafc2" containerName="istio-proxy" containerID="cri-o://fbbb0e47a2b98ef76dcb7d000f7c8bba9769170fce7ca17a816af9098011fe92" gracePeriod=30 Jul 09 07:31:58.196502 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:58.196476 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx" Jul 09 07:31:58.207480 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:58.207422 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbc6d719-63c1-4ec7-8bf9-efc41adbafc2-istiod-ca-cert\") pod \"cbc6d719-63c1-4ec7-8bf9-efc41adbafc2\" (UID: \"cbc6d719-63c1-4ec7-8bf9-efc41adbafc2\") " Jul 09 07:31:58.207480 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:58.207469 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pddqt\" (UniqueName: \"kubernetes.io/projected/cbc6d719-63c1-4ec7-8bf9-efc41adbafc2-kube-api-access-pddqt\") pod \"cbc6d719-63c1-4ec7-8bf9-efc41adbafc2\" (UID: \"cbc6d719-63c1-4ec7-8bf9-efc41adbafc2\") " Jul 09 07:31:58.207657 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:58.207505 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbc6d719-63c1-4ec7-8bf9-efc41adbafc2-istio-data\") pod \"cbc6d719-63c1-4ec7-8bf9-efc41adbafc2\" (UID: \"cbc6d719-63c1-4ec7-8bf9-efc41adbafc2\") " Jul 09 07:31:58.207657 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:58.207616 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbc6d719-63c1-4ec7-8bf9-efc41adbafc2-istio-podinfo\") pod \"cbc6d719-63c1-4ec7-8bf9-efc41adbafc2\" (UID: \"cbc6d719-63c1-4ec7-8bf9-efc41adbafc2\") " Jul 09 07:31:58.207743 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:58.207673 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbc6d719-63c1-4ec7-8bf9-efc41adbafc2-istio-token\") pod \"cbc6d719-63c1-4ec7-8bf9-efc41adbafc2\" (UID: \"cbc6d719-63c1-4ec7-8bf9-efc41adbafc2\") " Jul 09 07:31:58.207805 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:58.207774 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbc6d719-63c1-4ec7-8bf9-efc41adbafc2-workload-certs\") pod \"cbc6d719-63c1-4ec7-8bf9-efc41adbafc2\" (UID: \"cbc6d719-63c1-4ec7-8bf9-efc41adbafc2\") " Jul 09 07:31:58.207858 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:58.207825 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc6d719-63c1-4ec7-8bf9-efc41adbafc2-credential-socket\") pod \"cbc6d719-63c1-4ec7-8bf9-efc41adbafc2\" (UID: \"cbc6d719-63c1-4ec7-8bf9-efc41adbafc2\") " Jul 09 07:31:58.207910 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:58.207862 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc6d719-63c1-4ec7-8bf9-efc41adbafc2-workload-socket\") pod \"cbc6d719-63c1-4ec7-8bf9-efc41adbafc2\" (UID: \"cbc6d719-63c1-4ec7-8bf9-efc41adbafc2\") " Jul 09 07:31:58.207910 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:58.207858 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbc6d719-63c1-4ec7-8bf9-efc41adbafc2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cbc6d719-63c1-4ec7-8bf9-efc41adbafc2" (UID: "cbc6d719-63c1-4ec7-8bf9-efc41adbafc2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 07:31:58.207910 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:58.207901 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbc6d719-63c1-4ec7-8bf9-efc41adbafc2-istio-envoy\") pod \"cbc6d719-63c1-4ec7-8bf9-efc41adbafc2\" (UID: \"cbc6d719-63c1-4ec7-8bf9-efc41adbafc2\") " Jul 09 07:31:58.208099 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:58.208063 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbc6d719-63c1-4ec7-8bf9-efc41adbafc2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cbc6d719-63c1-4ec7-8bf9-efc41adbafc2" (UID: "cbc6d719-63c1-4ec7-8bf9-efc41adbafc2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 07:31:58.208154 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:58.208125 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbc6d719-63c1-4ec7-8bf9-efc41adbafc2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cbc6d719-63c1-4ec7-8bf9-efc41adbafc2" (UID: "cbc6d719-63c1-4ec7-8bf9-efc41adbafc2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 07:31:58.208248 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:58.208234 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc6d719-63c1-4ec7-8bf9-efc41adbafc2-credential-socket\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:31:58.208345 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:58.208252 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbc6d719-63c1-4ec7-8bf9-efc41adbafc2-istiod-ca-cert\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:31:58.208345 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:58.208251 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbc6d719-63c1-4ec7-8bf9-efc41adbafc2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cbc6d719-63c1-4ec7-8bf9-efc41adbafc2" (UID: "cbc6d719-63c1-4ec7-8bf9-efc41adbafc2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 07:31:58.208345 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:58.208288 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbc6d719-63c1-4ec7-8bf9-efc41adbafc2-workload-certs\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:31:58.208490 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:58.208338 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbc6d719-63c1-4ec7-8bf9-efc41adbafc2-istio-data" (OuterVolumeSpecName: "istio-data") pod "cbc6d719-63c1-4ec7-8bf9-efc41adbafc2" (UID: "cbc6d719-63c1-4ec7-8bf9-efc41adbafc2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 07:31:58.209726 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:58.209701 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cbc6d719-63c1-4ec7-8bf9-efc41adbafc2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cbc6d719-63c1-4ec7-8bf9-efc41adbafc2" (UID: "cbc6d719-63c1-4ec7-8bf9-efc41adbafc2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 07:31:58.209816 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:58.209767 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbc6d719-63c1-4ec7-8bf9-efc41adbafc2-kube-api-access-pddqt" (OuterVolumeSpecName: "kube-api-access-pddqt") pod "cbc6d719-63c1-4ec7-8bf9-efc41adbafc2" (UID: "cbc6d719-63c1-4ec7-8bf9-efc41adbafc2"). InnerVolumeSpecName "kube-api-access-pddqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 07:31:58.210057 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:58.210041 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbc6d719-63c1-4ec7-8bf9-efc41adbafc2-istio-token" (OuterVolumeSpecName: "istio-token") pod "cbc6d719-63c1-4ec7-8bf9-efc41adbafc2" (UID: "cbc6d719-63c1-4ec7-8bf9-efc41adbafc2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 07:31:58.210212 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:58.210182 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbc6d719-63c1-4ec7-8bf9-efc41adbafc2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cbc6d719-63c1-4ec7-8bf9-efc41adbafc2" (UID: "cbc6d719-63c1-4ec7-8bf9-efc41adbafc2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 07:31:58.308985 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:58.308954 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbc6d719-63c1-4ec7-8bf9-efc41adbafc2-istio-token\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:31:58.308985 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:58.308983 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc6d719-63c1-4ec7-8bf9-efc41adbafc2-workload-socket\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:31:58.308985 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:58.308991 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbc6d719-63c1-4ec7-8bf9-efc41adbafc2-istio-envoy\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:31:58.309222 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:58.309000 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pddqt\" (UniqueName: \"kubernetes.io/projected/cbc6d719-63c1-4ec7-8bf9-efc41adbafc2-kube-api-access-pddqt\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:31:58.309222 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:58.309008 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbc6d719-63c1-4ec7-8bf9-efc41adbafc2-istio-data\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:31:58.309222 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:58.309016 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbc6d719-63c1-4ec7-8bf9-efc41adbafc2-istio-podinfo\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:31:58.969802 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:58.969770 2564 generic.go:356] "Generic (PLEG): container finished" podID="cbc6d719-63c1-4ec7-8bf9-efc41adbafc2" containerID="fbbb0e47a2b98ef76dcb7d000f7c8bba9769170fce7ca17a816af9098011fe92" exitCode=0 Jul 09 07:31:58.969987 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:58.969819 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx" Jul 09 07:31:58.969987 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:58.969860 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx" event={"ID":"cbc6d719-63c1-4ec7-8bf9-efc41adbafc2","Type":"ContainerDied","Data":"fbbb0e47a2b98ef76dcb7d000f7c8bba9769170fce7ca17a816af9098011fe92"} Jul 09 07:31:58.969987 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:58.969918 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx" event={"ID":"cbc6d719-63c1-4ec7-8bf9-efc41adbafc2","Type":"ContainerDied","Data":"28b8a089902b04e6e56855cb6bfb6818d41d167a285a3788758f621c4b171054"} Jul 09 07:31:58.969987 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:58.969948 2564 scope.go:117] "RemoveContainer" containerID="fbbb0e47a2b98ef76dcb7d000f7c8bba9769170fce7ca17a816af9098011fe92" Jul 09 07:31:58.979451 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:58.979433 2564 scope.go:117] "RemoveContainer" containerID="d91df7b801c0d21d35a9189fa45fc8805886f825f35bbc309b714221f619263b" Jul 09 07:31:58.986867 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:58.986849 2564 scope.go:117] "RemoveContainer" containerID="fbbb0e47a2b98ef76dcb7d000f7c8bba9769170fce7ca17a816af9098011fe92" Jul 09 07:31:58.987112 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:31:58.987093 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbbb0e47a2b98ef76dcb7d000f7c8bba9769170fce7ca17a816af9098011fe92\": container with ID starting with fbbb0e47a2b98ef76dcb7d000f7c8bba9769170fce7ca17a816af9098011fe92 not found: ID does not exist" containerID="fbbb0e47a2b98ef76dcb7d000f7c8bba9769170fce7ca17a816af9098011fe92" Jul 09 07:31:58.987152 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:58.987121 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbbb0e47a2b98ef76dcb7d000f7c8bba9769170fce7ca17a816af9098011fe92"} err="failed to get container status \"fbbb0e47a2b98ef76dcb7d000f7c8bba9769170fce7ca17a816af9098011fe92\": rpc error: code = NotFound desc = could not find container \"fbbb0e47a2b98ef76dcb7d000f7c8bba9769170fce7ca17a816af9098011fe92\": container with ID starting with fbbb0e47a2b98ef76dcb7d000f7c8bba9769170fce7ca17a816af9098011fe92 not found: ID does not exist" Jul 09 07:31:58.987152 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:58.987138 2564 scope.go:117] "RemoveContainer" containerID="d91df7b801c0d21d35a9189fa45fc8805886f825f35bbc309b714221f619263b" Jul 09 07:31:58.987378 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:31:58.987362 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d91df7b801c0d21d35a9189fa45fc8805886f825f35bbc309b714221f619263b\": container with ID starting with d91df7b801c0d21d35a9189fa45fc8805886f825f35bbc309b714221f619263b not found: ID does not exist" containerID="d91df7b801c0d21d35a9189fa45fc8805886f825f35bbc309b714221f619263b" Jul 09 07:31:58.987423 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:58.987384 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d91df7b801c0d21d35a9189fa45fc8805886f825f35bbc309b714221f619263b"} err="failed to get container status \"d91df7b801c0d21d35a9189fa45fc8805886f825f35bbc309b714221f619263b\": rpc error: code = NotFound desc = could not find container \"d91df7b801c0d21d35a9189fa45fc8805886f825f35bbc309b714221f619263b\": container with ID starting with d91df7b801c0d21d35a9189fa45fc8805886f825f35bbc309b714221f619263b not found: ID does not exist" Jul 09 07:31:58.991124 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:58.991105 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx"] Jul 09 07:31:58.995737 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:58.995711 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-0c3ae94c-openshift-default-5f7887d9b7-wj6xx"] Jul 09 07:31:59.463331 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:31:59.463289 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbc6d719-63c1-4ec7-8bf9-efc41adbafc2" path="/var/lib/kubelet/pods/cbc6d719-63c1-4ec7-8bf9-efc41adbafc2/volumes" Jul 09 07:32:12.810547 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:12.810514 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8"] Jul 09 07:32:12.811073 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:12.810872 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbc6d719-63c1-4ec7-8bf9-efc41adbafc2" containerName="istio-proxy" Jul 09 07:32:12.811073 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:12.810886 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbc6d719-63c1-4ec7-8bf9-efc41adbafc2" containerName="istio-proxy" Jul 09 07:32:12.811073 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:12.810895 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbc6d719-63c1-4ec7-8bf9-efc41adbafc2" containerName="istio-proxy" Jul 09 07:32:12.811073 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:12.810900 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbc6d719-63c1-4ec7-8bf9-efc41adbafc2" containerName="istio-proxy" Jul 09 07:32:12.811073 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:12.810965 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbc6d719-63c1-4ec7-8bf9-efc41adbafc2" containerName="istio-proxy" Jul 09 07:32:12.813966 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:12.813948 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8" Jul 09 07:32:12.816589 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:12.816567 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-b0405232-openshift-default-dockercfg-5p27v\"" Jul 09 07:32:12.825168 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:12.825149 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8"] Jul 09 07:32:12.921436 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:12.921385 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6796c3b4-7535-469c-9fe5-b62477582184-istio-data\") pod \"e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8\" (UID: \"6796c3b4-7535-469c-9fe5-b62477582184\") " pod="openshift-ingress/e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8" Jul 09 07:32:12.921436 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:12.921422 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6796c3b4-7535-469c-9fe5-b62477582184-istio-token\") pod \"e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8\" (UID: \"6796c3b4-7535-469c-9fe5-b62477582184\") " pod="openshift-ingress/e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8" Jul 09 07:32:12.921642 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:12.921450 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6796c3b4-7535-469c-9fe5-b62477582184-credential-socket\") pod \"e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8\" (UID: \"6796c3b4-7535-469c-9fe5-b62477582184\") " pod="openshift-ingress/e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8" Jul 09 07:32:12.921642 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:12.921498 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2sn4\" (UniqueName: \"kubernetes.io/projected/6796c3b4-7535-469c-9fe5-b62477582184-kube-api-access-n2sn4\") pod \"e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8\" (UID: \"6796c3b4-7535-469c-9fe5-b62477582184\") " pod="openshift-ingress/e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8" Jul 09 07:32:12.921642 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:12.921539 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6796c3b4-7535-469c-9fe5-b62477582184-workload-socket\") pod \"e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8\" (UID: \"6796c3b4-7535-469c-9fe5-b62477582184\") " pod="openshift-ingress/e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8" Jul 09 07:32:12.921642 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:12.921559 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6796c3b4-7535-469c-9fe5-b62477582184-istiod-ca-cert\") pod \"e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8\" (UID: \"6796c3b4-7535-469c-9fe5-b62477582184\") " pod="openshift-ingress/e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8" Jul 09 07:32:12.921642 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:12.921574 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6796c3b4-7535-469c-9fe5-b62477582184-istio-envoy\") pod \"e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8\" (UID: \"6796c3b4-7535-469c-9fe5-b62477582184\") " pod="openshift-ingress/e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8" Jul 09 07:32:12.921797 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:12.921657 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6796c3b4-7535-469c-9fe5-b62477582184-istio-podinfo\") pod \"e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8\" (UID: \"6796c3b4-7535-469c-9fe5-b62477582184\") " pod="openshift-ingress/e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8" Jul 09 07:32:12.921797 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:12.921691 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6796c3b4-7535-469c-9fe5-b62477582184-workload-certs\") pod \"e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8\" (UID: \"6796c3b4-7535-469c-9fe5-b62477582184\") " pod="openshift-ingress/e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8" Jul 09 07:32:13.022950 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:13.022909 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6796c3b4-7535-469c-9fe5-b62477582184-credential-socket\") pod \"e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8\" (UID: \"6796c3b4-7535-469c-9fe5-b62477582184\") " pod="openshift-ingress/e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8" Jul 09 07:32:13.023137 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:13.022970 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2sn4\" (UniqueName: \"kubernetes.io/projected/6796c3b4-7535-469c-9fe5-b62477582184-kube-api-access-n2sn4\") pod \"e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8\" (UID: \"6796c3b4-7535-469c-9fe5-b62477582184\") " pod="openshift-ingress/e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8" Jul 09 07:32:13.023137 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:13.023024 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6796c3b4-7535-469c-9fe5-b62477582184-workload-socket\") pod \"e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8\" (UID: \"6796c3b4-7535-469c-9fe5-b62477582184\") " pod="openshift-ingress/e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8" Jul 09 07:32:13.023137 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:13.023051 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6796c3b4-7535-469c-9fe5-b62477582184-istiod-ca-cert\") pod \"e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8\" (UID: \"6796c3b4-7535-469c-9fe5-b62477582184\") " pod="openshift-ingress/e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8" Jul 09 07:32:13.023328 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:13.023188 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6796c3b4-7535-469c-9fe5-b62477582184-istio-envoy\") pod \"e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8\" (UID: \"6796c3b4-7535-469c-9fe5-b62477582184\") " pod="openshift-ingress/e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8" Jul 09 07:32:13.023328 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:13.023258 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6796c3b4-7535-469c-9fe5-b62477582184-istio-podinfo\") pod \"e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8\" (UID: \"6796c3b4-7535-469c-9fe5-b62477582184\") " pod="openshift-ingress/e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8" Jul 09 07:32:13.023328 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:13.023311 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6796c3b4-7535-469c-9fe5-b62477582184-workload-certs\") pod \"e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8\" (UID: \"6796c3b4-7535-469c-9fe5-b62477582184\") " pod="openshift-ingress/e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8" Jul 09 07:32:13.023476 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:13.023346 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6796c3b4-7535-469c-9fe5-b62477582184-istio-data\") pod \"e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8\" (UID: \"6796c3b4-7535-469c-9fe5-b62477582184\") " pod="openshift-ingress/e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8" Jul 09 07:32:13.023476 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:13.023387 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6796c3b4-7535-469c-9fe5-b62477582184-istio-token\") pod \"e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8\" (UID: \"6796c3b4-7535-469c-9fe5-b62477582184\") " pod="openshift-ingress/e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8" Jul 09 07:32:13.023476 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:13.023416 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6796c3b4-7535-469c-9fe5-b62477582184-credential-socket\") pod \"e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8\" (UID: \"6796c3b4-7535-469c-9fe5-b62477582184\") " pod="openshift-ingress/e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8" Jul 09 07:32:13.023476 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:13.023445 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6796c3b4-7535-469c-9fe5-b62477582184-workload-socket\") pod \"e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8\" (UID: \"6796c3b4-7535-469c-9fe5-b62477582184\") " pod="openshift-ingress/e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8" Jul 09 07:32:13.023711 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:13.023683 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6796c3b4-7535-469c-9fe5-b62477582184-workload-certs\") pod \"e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8\" (UID: \"6796c3b4-7535-469c-9fe5-b62477582184\") " pod="openshift-ingress/e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8" Jul 09 07:32:13.023845 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:13.023718 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6796c3b4-7535-469c-9fe5-b62477582184-istio-data\") pod \"e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8\" (UID: \"6796c3b4-7535-469c-9fe5-b62477582184\") " pod="openshift-ingress/e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8" Jul 09 07:32:13.023845 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:13.023823 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6796c3b4-7535-469c-9fe5-b62477582184-istiod-ca-cert\") pod \"e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8\" (UID: \"6796c3b4-7535-469c-9fe5-b62477582184\") " pod="openshift-ingress/e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8" Jul 09 07:32:13.025473 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:13.025450 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6796c3b4-7535-469c-9fe5-b62477582184-istio-envoy\") pod \"e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8\" (UID: \"6796c3b4-7535-469c-9fe5-b62477582184\") " pod="openshift-ingress/e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8" Jul 09 07:32:13.026142 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:13.026122 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6796c3b4-7535-469c-9fe5-b62477582184-istio-podinfo\") pod \"e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8\" (UID: \"6796c3b4-7535-469c-9fe5-b62477582184\") " pod="openshift-ingress/e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8" Jul 09 07:32:13.031358 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:13.031335 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6796c3b4-7535-469c-9fe5-b62477582184-istio-token\") pod \"e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8\" (UID: \"6796c3b4-7535-469c-9fe5-b62477582184\") " pod="openshift-ingress/e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8" Jul 09 07:32:13.031358 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:13.031353 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2sn4\" (UniqueName: \"kubernetes.io/projected/6796c3b4-7535-469c-9fe5-b62477582184-kube-api-access-n2sn4\") pod \"e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8\" (UID: \"6796c3b4-7535-469c-9fe5-b62477582184\") " pod="openshift-ingress/e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8" Jul 09 07:32:13.124912 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:13.124797 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8" Jul 09 07:32:13.260082 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:13.260055 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8"] Jul 09 07:32:13.261746 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:32:13.261718 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6796c3b4_7535_469c_9fe5_b62477582184.slice/crio-51a935fad17ad9cb12065865eeac525a2bd0f1fb0cb9108a47c9527295ab4fcf WatchSource:0}: Error finding container 51a935fad17ad9cb12065865eeac525a2bd0f1fb0cb9108a47c9527295ab4fcf: Status 404 returned error can't find the container with id 51a935fad17ad9cb12065865eeac525a2bd0f1fb0cb9108a47c9527295ab4fcf Jul 09 07:32:13.263842 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:13.263809 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 07:32:13.263949 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:13.263877 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 07:32:13.263949 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:13.263905 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 07:32:14.037344 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:14.037313 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8" event={"ID":"6796c3b4-7535-469c-9fe5-b62477582184","Type":"ContainerStarted","Data":"9ed1fd5b0a0ee877da7dfc0a008c49c85fc361d3253242f9f7c756ce49d23d3e"} Jul 09 07:32:14.037776 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:14.037363 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8" event={"ID":"6796c3b4-7535-469c-9fe5-b62477582184","Type":"ContainerStarted","Data":"51a935fad17ad9cb12065865eeac525a2bd0f1fb0cb9108a47c9527295ab4fcf"} Jul 09 07:32:14.059400 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:14.059349 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8" podStartSLOduration=2.059335814 podStartE2EDuration="2.059335814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 07:32:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 07:32:14.057851656 +0000 UTC m=+1793.108928884" watchObservedRunningTime="2026-07-09 07:32:14.059335814 +0000 UTC m=+1793.110413042" Jul 09 07:32:14.125320 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:14.125287 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8" Jul 09 07:32:14.130553 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:14.130530 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8" Jul 09 07:32:15.045588 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:15.045543 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8" Jul 09 07:32:15.048348 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:15.048324 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8" Jul 09 07:32:29.108223 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:29.108130 2564 generic.go:356] "Generic (PLEG): container finished" podID="6796c3b4-7535-469c-9fe5-b62477582184" containerID="9ed1fd5b0a0ee877da7dfc0a008c49c85fc361d3253242f9f7c756ce49d23d3e" exitCode=0 Jul 09 07:32:29.108735 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:29.108214 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8" event={"ID":"6796c3b4-7535-469c-9fe5-b62477582184","Type":"ContainerDied","Data":"9ed1fd5b0a0ee877da7dfc0a008c49c85fc361d3253242f9f7c756ce49d23d3e"} Jul 09 07:32:29.108802 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:29.108742 2564 scope.go:117] "RemoveContainer" containerID="9ed1fd5b0a0ee877da7dfc0a008c49c85fc361d3253242f9f7c756ce49d23d3e" Jul 09 07:32:29.109742 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:29.109715 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 07:32:29.109830 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:29.109779 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 07:32:29.109830 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:29.109809 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 07:32:30.045493 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:30.045446 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8" Jul 09 07:32:30.114524 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:30.114488 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8" event={"ID":"6796c3b4-7535-469c-9fe5-b62477582184","Type":"ContainerStarted","Data":"101900fa6881bfee814b83cd6b4e2a4885dae6b1f05284f1203414aeca7e2b40"} Jul 09 07:32:30.124977 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:30.124951 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8" Jul 09 07:32:30.125967 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:30.125947 2564 patch_prober.go:28] interesting pod/e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Jul 09 07:32:30.126042 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:30.125999 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8" podUID="6796c3b4-7535-469c-9fe5-b62477582184" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Jul 09 07:32:31.125721 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:31.125689 2564 patch_prober.go:28] interesting pod/e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Jul 09 07:32:31.126090 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:31.125742 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8" podUID="6796c3b4-7535-469c-9fe5-b62477582184" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Jul 09 07:32:32.125107 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:32.125078 2564 patch_prober.go:28] interesting pod/e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Jul 09 07:32:32.125357 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:32.125128 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8" podUID="6796c3b4-7535-469c-9fe5-b62477582184" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Jul 09 07:32:33.125778 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:33.125742 2564 patch_prober.go:28] interesting pod/e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Jul 09 07:32:33.126157 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:33.125800 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8" podUID="6796c3b4-7535-469c-9fe5-b62477582184" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Jul 09 07:32:34.129249 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:34.129218 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8" Jul 09 07:32:34.129673 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:34.129517 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8" Jul 09 07:32:34.130204 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:34.130186 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8" Jul 09 07:32:37.790526 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:37.790490 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-b0405232/e2e-lifecycle-model-b0405232-kserve-d77bcb7f-p2wtj"] Jul 09 07:32:37.791094 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:37.791074 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbc6d719-63c1-4ec7-8bf9-efc41adbafc2" containerName="istio-proxy" Jul 09 07:32:37.793506 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:37.793487 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b0405232/e2e-lifecycle-model-b0405232-kserve-d77bcb7f-p2wtj" Jul 09 07:32:37.797036 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:37.797013 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-b0405232\"/\"openshift-service-ca.crt\"" Jul 09 07:32:37.798133 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:37.798116 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-b0405232\"/\"e2e-lifecycle-model-b0405232-kserve-self-signed-certs\"" Jul 09 07:32:37.798246 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:37.798129 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-b0405232\"/\"default-dockercfg-ckdmw\"" Jul 09 07:32:37.798246 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:37.798150 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-b0405232\"/\"kube-root-ca.crt\"" Jul 09 07:32:37.805391 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:37.805371 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-b0405232/e2e-lifecycle-model-b0405232-kserve-d77bcb7f-p2wtj"] Jul 09 07:32:37.941330 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:37.941296 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/468e77a9-8396-4302-90de-8306089fcee9-tls-certs\") pod \"e2e-lifecycle-model-b0405232-kserve-d77bcb7f-p2wtj\" (UID: \"468e77a9-8396-4302-90de-8306089fcee9\") " pod="ai-tenant-e2e-mt-b0405232/e2e-lifecycle-model-b0405232-kserve-d77bcb7f-p2wtj" Jul 09 07:32:37.941492 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:37.941342 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/468e77a9-8396-4302-90de-8306089fcee9-model-cache\") pod \"e2e-lifecycle-model-b0405232-kserve-d77bcb7f-p2wtj\" (UID: \"468e77a9-8396-4302-90de-8306089fcee9\") " pod="ai-tenant-e2e-mt-b0405232/e2e-lifecycle-model-b0405232-kserve-d77bcb7f-p2wtj" Jul 09 07:32:37.941492 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:37.941382 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/468e77a9-8396-4302-90de-8306089fcee9-dshm\") pod \"e2e-lifecycle-model-b0405232-kserve-d77bcb7f-p2wtj\" (UID: \"468e77a9-8396-4302-90de-8306089fcee9\") " pod="ai-tenant-e2e-mt-b0405232/e2e-lifecycle-model-b0405232-kserve-d77bcb7f-p2wtj" Jul 09 07:32:37.941492 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:37.941421 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hbrv\" (UniqueName: \"kubernetes.io/projected/468e77a9-8396-4302-90de-8306089fcee9-kube-api-access-9hbrv\") pod \"e2e-lifecycle-model-b0405232-kserve-d77bcb7f-p2wtj\" (UID: \"468e77a9-8396-4302-90de-8306089fcee9\") " pod="ai-tenant-e2e-mt-b0405232/e2e-lifecycle-model-b0405232-kserve-d77bcb7f-p2wtj" Jul 09 07:32:37.941492 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:37.941482 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/468e77a9-8396-4302-90de-8306089fcee9-home\") pod \"e2e-lifecycle-model-b0405232-kserve-d77bcb7f-p2wtj\" (UID: \"468e77a9-8396-4302-90de-8306089fcee9\") " pod="ai-tenant-e2e-mt-b0405232/e2e-lifecycle-model-b0405232-kserve-d77bcb7f-p2wtj" Jul 09 07:32:37.941627 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:37.941512 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/468e77a9-8396-4302-90de-8306089fcee9-tmp-dir\") pod \"e2e-lifecycle-model-b0405232-kserve-d77bcb7f-p2wtj\" (UID: \"468e77a9-8396-4302-90de-8306089fcee9\") " pod="ai-tenant-e2e-mt-b0405232/e2e-lifecycle-model-b0405232-kserve-d77bcb7f-p2wtj" Jul 09 07:32:38.042895 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:38.042820 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/468e77a9-8396-4302-90de-8306089fcee9-home\") pod \"e2e-lifecycle-model-b0405232-kserve-d77bcb7f-p2wtj\" (UID: \"468e77a9-8396-4302-90de-8306089fcee9\") " pod="ai-tenant-e2e-mt-b0405232/e2e-lifecycle-model-b0405232-kserve-d77bcb7f-p2wtj" Jul 09 07:32:38.042895 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:38.042857 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/468e77a9-8396-4302-90de-8306089fcee9-tmp-dir\") pod \"e2e-lifecycle-model-b0405232-kserve-d77bcb7f-p2wtj\" (UID: \"468e77a9-8396-4302-90de-8306089fcee9\") " pod="ai-tenant-e2e-mt-b0405232/e2e-lifecycle-model-b0405232-kserve-d77bcb7f-p2wtj" Jul 09 07:32:38.043063 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:38.043045 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/468e77a9-8396-4302-90de-8306089fcee9-tls-certs\") pod \"e2e-lifecycle-model-b0405232-kserve-d77bcb7f-p2wtj\" (UID: \"468e77a9-8396-4302-90de-8306089fcee9\") " pod="ai-tenant-e2e-mt-b0405232/e2e-lifecycle-model-b0405232-kserve-d77bcb7f-p2wtj" Jul 09 07:32:38.043111 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:38.043093 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/468e77a9-8396-4302-90de-8306089fcee9-model-cache\") pod \"e2e-lifecycle-model-b0405232-kserve-d77bcb7f-p2wtj\" (UID: \"468e77a9-8396-4302-90de-8306089fcee9\") " pod="ai-tenant-e2e-mt-b0405232/e2e-lifecycle-model-b0405232-kserve-d77bcb7f-p2wtj" Jul 09 07:32:38.043164 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:38.043127 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/468e77a9-8396-4302-90de-8306089fcee9-dshm\") pod \"e2e-lifecycle-model-b0405232-kserve-d77bcb7f-p2wtj\" (UID: \"468e77a9-8396-4302-90de-8306089fcee9\") " pod="ai-tenant-e2e-mt-b0405232/e2e-lifecycle-model-b0405232-kserve-d77bcb7f-p2wtj" Jul 09 07:32:38.043218 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:38.043171 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hbrv\" (UniqueName: \"kubernetes.io/projected/468e77a9-8396-4302-90de-8306089fcee9-kube-api-access-9hbrv\") pod \"e2e-lifecycle-model-b0405232-kserve-d77bcb7f-p2wtj\" (UID: \"468e77a9-8396-4302-90de-8306089fcee9\") " pod="ai-tenant-e2e-mt-b0405232/e2e-lifecycle-model-b0405232-kserve-d77bcb7f-p2wtj" Jul 09 07:32:38.043218 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:38.043199 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/468e77a9-8396-4302-90de-8306089fcee9-home\") pod \"e2e-lifecycle-model-b0405232-kserve-d77bcb7f-p2wtj\" (UID: \"468e77a9-8396-4302-90de-8306089fcee9\") " pod="ai-tenant-e2e-mt-b0405232/e2e-lifecycle-model-b0405232-kserve-d77bcb7f-p2wtj" Jul 09 07:32:38.043351 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:38.043250 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/468e77a9-8396-4302-90de-8306089fcee9-tmp-dir\") pod \"e2e-lifecycle-model-b0405232-kserve-d77bcb7f-p2wtj\" (UID: \"468e77a9-8396-4302-90de-8306089fcee9\") " pod="ai-tenant-e2e-mt-b0405232/e2e-lifecycle-model-b0405232-kserve-d77bcb7f-p2wtj" Jul 09 07:32:38.043403 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:38.043385 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/468e77a9-8396-4302-90de-8306089fcee9-model-cache\") pod \"e2e-lifecycle-model-b0405232-kserve-d77bcb7f-p2wtj\" (UID: \"468e77a9-8396-4302-90de-8306089fcee9\") " pod="ai-tenant-e2e-mt-b0405232/e2e-lifecycle-model-b0405232-kserve-d77bcb7f-p2wtj" Jul 09 07:32:38.045380 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:38.045359 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/468e77a9-8396-4302-90de-8306089fcee9-dshm\") pod \"e2e-lifecycle-model-b0405232-kserve-d77bcb7f-p2wtj\" (UID: \"468e77a9-8396-4302-90de-8306089fcee9\") " pod="ai-tenant-e2e-mt-b0405232/e2e-lifecycle-model-b0405232-kserve-d77bcb7f-p2wtj" Jul 09 07:32:38.045567 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:38.045550 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/468e77a9-8396-4302-90de-8306089fcee9-tls-certs\") pod \"e2e-lifecycle-model-b0405232-kserve-d77bcb7f-p2wtj\" (UID: \"468e77a9-8396-4302-90de-8306089fcee9\") " pod="ai-tenant-e2e-mt-b0405232/e2e-lifecycle-model-b0405232-kserve-d77bcb7f-p2wtj" Jul 09 07:32:38.051213 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:38.051194 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hbrv\" (UniqueName: \"kubernetes.io/projected/468e77a9-8396-4302-90de-8306089fcee9-kube-api-access-9hbrv\") pod \"e2e-lifecycle-model-b0405232-kserve-d77bcb7f-p2wtj\" (UID: \"468e77a9-8396-4302-90de-8306089fcee9\") " pod="ai-tenant-e2e-mt-b0405232/e2e-lifecycle-model-b0405232-kserve-d77bcb7f-p2wtj" Jul 09 07:32:38.103007 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:38.102980 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b0405232/e2e-lifecycle-model-b0405232-kserve-d77bcb7f-p2wtj" Jul 09 07:32:38.238613 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:38.238570 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-b0405232/e2e-lifecycle-model-b0405232-kserve-d77bcb7f-p2wtj"] Jul 09 07:32:38.240962 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:32:38.240933 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod468e77a9_8396_4302_90de_8306089fcee9.slice/crio-b1a76de1d309178d9c795f9d654bf58adfebdd69b4c1d756a2b1f58d0d6b7b99 WatchSource:0}: Error finding container b1a76de1d309178d9c795f9d654bf58adfebdd69b4c1d756a2b1f58d0d6b7b99: Status 404 returned error can't find the container with id b1a76de1d309178d9c795f9d654bf58adfebdd69b4c1d756a2b1f58d0d6b7b99 Jul 09 07:32:39.159999 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:39.159966 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b0405232/e2e-lifecycle-model-b0405232-kserve-d77bcb7f-p2wtj" event={"ID":"468e77a9-8396-4302-90de-8306089fcee9","Type":"ContainerStarted","Data":"bedebc8086b6a9d56f409700dac32c3180bb007f0ab0390e6588622f97a409e4"} Jul 09 07:32:39.159999 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:39.160004 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b0405232/e2e-lifecycle-model-b0405232-kserve-d77bcb7f-p2wtj" event={"ID":"468e77a9-8396-4302-90de-8306089fcee9","Type":"ContainerStarted","Data":"b1a76de1d309178d9c795f9d654bf58adfebdd69b4c1d756a2b1f58d0d6b7b99"} Jul 09 07:32:39.178753 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:39.178700 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-b0405232/e2e-lifecycle-model-b0405232-kserve-d77bcb7f-p2wtj" podStartSLOduration=2.178684778 podStartE2EDuration="2.178684778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 07:32:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 07:32:39.177490839 +0000 UTC m=+1818.228568068" watchObservedRunningTime="2026-07-09 07:32:39.178684778 +0000 UTC m=+1818.229762006" Jul 09 07:32:48.103094 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:48.103058 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-b0405232/e2e-lifecycle-model-b0405232-kserve-d77bcb7f-p2wtj" Jul 09 07:32:48.103589 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:48.103108 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-b0405232/e2e-lifecycle-model-b0405232-kserve-d77bcb7f-p2wtj" Jul 09 07:32:48.115790 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:48.115761 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-b0405232/e2e-lifecycle-model-b0405232-kserve-d77bcb7f-p2wtj" Jul 09 07:32:48.210331 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:32:48.210303 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-b0405232/e2e-lifecycle-model-b0405232-kserve-d77bcb7f-p2wtj" Jul 09 07:33:09.017431 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:09.017395 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-b0405232/e2e-lifecycle-model-b0405232-kserve-d77bcb7f-p2wtj"] Jul 09 07:33:24.037971 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:24.037904 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-mt-b0405232/e2e-lifecycle-model-b0405232-kserve-d77bcb7f-p2wtj" podUID="468e77a9-8396-4302-90de-8306089fcee9" containerName="main" containerID="cri-o://bedebc8086b6a9d56f409700dac32c3180bb007f0ab0390e6588622f97a409e4" gracePeriod=45 Jul 09 07:33:24.281585 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:24.281563 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b0405232/e2e-lifecycle-model-b0405232-kserve-d77bcb7f-p2wtj" Jul 09 07:33:24.359407 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:24.359323 2564 generic.go:356] "Generic (PLEG): container finished" podID="468e77a9-8396-4302-90de-8306089fcee9" containerID="bedebc8086b6a9d56f409700dac32c3180bb007f0ab0390e6588622f97a409e4" exitCode=0 Jul 09 07:33:24.359407 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:24.359366 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b0405232/e2e-lifecycle-model-b0405232-kserve-d77bcb7f-p2wtj" Jul 09 07:33:24.359407 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:24.359389 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b0405232/e2e-lifecycle-model-b0405232-kserve-d77bcb7f-p2wtj" event={"ID":"468e77a9-8396-4302-90de-8306089fcee9","Type":"ContainerDied","Data":"bedebc8086b6a9d56f409700dac32c3180bb007f0ab0390e6588622f97a409e4"} Jul 09 07:33:24.359628 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:24.359424 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b0405232/e2e-lifecycle-model-b0405232-kserve-d77bcb7f-p2wtj" event={"ID":"468e77a9-8396-4302-90de-8306089fcee9","Type":"ContainerDied","Data":"b1a76de1d309178d9c795f9d654bf58adfebdd69b4c1d756a2b1f58d0d6b7b99"} Jul 09 07:33:24.359628 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:24.359441 2564 scope.go:117] "RemoveContainer" containerID="bedebc8086b6a9d56f409700dac32c3180bb007f0ab0390e6588622f97a409e4" Jul 09 07:33:24.367414 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:24.367398 2564 scope.go:117] "RemoveContainer" containerID="bedebc8086b6a9d56f409700dac32c3180bb007f0ab0390e6588622f97a409e4" Jul 09 07:33:24.367693 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:33:24.367671 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bedebc8086b6a9d56f409700dac32c3180bb007f0ab0390e6588622f97a409e4\": container with ID starting with bedebc8086b6a9d56f409700dac32c3180bb007f0ab0390e6588622f97a409e4 not found: ID does not exist" containerID="bedebc8086b6a9d56f409700dac32c3180bb007f0ab0390e6588622f97a409e4" Jul 09 07:33:24.367739 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:24.367702 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bedebc8086b6a9d56f409700dac32c3180bb007f0ab0390e6588622f97a409e4"} err="failed to get container status \"bedebc8086b6a9d56f409700dac32c3180bb007f0ab0390e6588622f97a409e4\": rpc error: code = NotFound desc = could not find container \"bedebc8086b6a9d56f409700dac32c3180bb007f0ab0390e6588622f97a409e4\": container with ID starting with bedebc8086b6a9d56f409700dac32c3180bb007f0ab0390e6588622f97a409e4 not found: ID does not exist" Jul 09 07:33:24.427181 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:24.427156 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/468e77a9-8396-4302-90de-8306089fcee9-dshm\") pod \"468e77a9-8396-4302-90de-8306089fcee9\" (UID: \"468e77a9-8396-4302-90de-8306089fcee9\") " Jul 09 07:33:24.427352 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:24.427189 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/468e77a9-8396-4302-90de-8306089fcee9-model-cache\") pod \"468e77a9-8396-4302-90de-8306089fcee9\" (UID: \"468e77a9-8396-4302-90de-8306089fcee9\") " Jul 09 07:33:24.427352 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:24.427205 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/468e77a9-8396-4302-90de-8306089fcee9-home\") pod \"468e77a9-8396-4302-90de-8306089fcee9\" (UID: \"468e77a9-8396-4302-90de-8306089fcee9\") " Jul 09 07:33:24.427352 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:24.427243 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9hbrv\" (UniqueName: \"kubernetes.io/projected/468e77a9-8396-4302-90de-8306089fcee9-kube-api-access-9hbrv\") pod \"468e77a9-8396-4302-90de-8306089fcee9\" (UID: \"468e77a9-8396-4302-90de-8306089fcee9\") " Jul 09 07:33:24.427352 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:24.427294 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/468e77a9-8396-4302-90de-8306089fcee9-tmp-dir\") pod \"468e77a9-8396-4302-90de-8306089fcee9\" (UID: \"468e77a9-8396-4302-90de-8306089fcee9\") " Jul 09 07:33:24.427352 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:24.427329 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/468e77a9-8396-4302-90de-8306089fcee9-tls-certs\") pod \"468e77a9-8396-4302-90de-8306089fcee9\" (UID: \"468e77a9-8396-4302-90de-8306089fcee9\") " Jul 09 07:33:24.427609 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:24.427509 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/468e77a9-8396-4302-90de-8306089fcee9-home" (OuterVolumeSpecName: "home") pod "468e77a9-8396-4302-90de-8306089fcee9" (UID: "468e77a9-8396-4302-90de-8306089fcee9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 07:33:24.427609 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:24.427527 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/468e77a9-8396-4302-90de-8306089fcee9-model-cache" (OuterVolumeSpecName: "model-cache") pod "468e77a9-8396-4302-90de-8306089fcee9" (UID: "468e77a9-8396-4302-90de-8306089fcee9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 07:33:24.427733 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:24.427606 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/468e77a9-8396-4302-90de-8306089fcee9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "468e77a9-8396-4302-90de-8306089fcee9" (UID: "468e77a9-8396-4302-90de-8306089fcee9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 07:33:24.427733 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:24.427625 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/468e77a9-8396-4302-90de-8306089fcee9-model-cache\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:33:24.427733 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:24.427642 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/468e77a9-8396-4302-90de-8306089fcee9-home\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:33:24.429513 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:24.429485 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/468e77a9-8396-4302-90de-8306089fcee9-kube-api-access-9hbrv" (OuterVolumeSpecName: "kube-api-access-9hbrv") pod "468e77a9-8396-4302-90de-8306089fcee9" (UID: "468e77a9-8396-4302-90de-8306089fcee9"). InnerVolumeSpecName "kube-api-access-9hbrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 07:33:24.429513 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:24.429490 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/468e77a9-8396-4302-90de-8306089fcee9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "468e77a9-8396-4302-90de-8306089fcee9" (UID: "468e77a9-8396-4302-90de-8306089fcee9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 07:33:24.429630 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:24.429573 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/468e77a9-8396-4302-90de-8306089fcee9-dshm" (OuterVolumeSpecName: "dshm") pod "468e77a9-8396-4302-90de-8306089fcee9" (UID: "468e77a9-8396-4302-90de-8306089fcee9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 07:33:24.528666 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:24.528640 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hbrv\" (UniqueName: \"kubernetes.io/projected/468e77a9-8396-4302-90de-8306089fcee9-kube-api-access-9hbrv\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:33:24.528775 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:24.528682 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/468e77a9-8396-4302-90de-8306089fcee9-tmp-dir\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:33:24.528775 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:24.528692 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/468e77a9-8396-4302-90de-8306089fcee9-tls-certs\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:33:24.528775 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:24.528701 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/468e77a9-8396-4302-90de-8306089fcee9-dshm\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:33:24.681099 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:24.681071 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-b0405232/e2e-lifecycle-model-b0405232-kserve-d77bcb7f-p2wtj"] Jul 09 07:33:24.686362 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:24.686338 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-b0405232/e2e-lifecycle-model-b0405232-kserve-d77bcb7f-p2wtj"] Jul 09 07:33:25.462881 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:25.462847 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="468e77a9-8396-4302-90de-8306089fcee9" path="/var/lib/kubelet/pods/468e77a9-8396-4302-90de-8306089fcee9/volumes" Jul 09 07:33:47.545457 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:47.545376 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8"] Jul 09 07:33:47.545962 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:47.545581 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8" podUID="6796c3b4-7535-469c-9fe5-b62477582184" containerName="istio-proxy" containerID="cri-o://101900fa6881bfee814b83cd6b4e2a4885dae6b1f05284f1203414aeca7e2b40" gracePeriod=30 Jul 09 07:33:49.131989 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:49.131955 2564 patch_prober.go:28] interesting pod/e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 09 07:33:49.132366 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:49.132012 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8" podUID="6796c3b4-7535-469c-9fe5-b62477582184" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 07:33:52.794021 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:52.793991 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8" Jul 09 07:33:52.867761 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:52.867679 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6796c3b4-7535-469c-9fe5-b62477582184-workload-certs\") pod \"6796c3b4-7535-469c-9fe5-b62477582184\" (UID: \"6796c3b4-7535-469c-9fe5-b62477582184\") " Jul 09 07:33:52.867761 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:52.867746 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6796c3b4-7535-469c-9fe5-b62477582184-istiod-ca-cert\") pod \"6796c3b4-7535-469c-9fe5-b62477582184\" (UID: \"6796c3b4-7535-469c-9fe5-b62477582184\") " Jul 09 07:33:52.867963 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:52.867784 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n2sn4\" (UniqueName: \"kubernetes.io/projected/6796c3b4-7535-469c-9fe5-b62477582184-kube-api-access-n2sn4\") pod \"6796c3b4-7535-469c-9fe5-b62477582184\" (UID: \"6796c3b4-7535-469c-9fe5-b62477582184\") " Jul 09 07:33:52.867963 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:52.867808 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6796c3b4-7535-469c-9fe5-b62477582184-istio-token\") pod \"6796c3b4-7535-469c-9fe5-b62477582184\" (UID: \"6796c3b4-7535-469c-9fe5-b62477582184\") " Jul 09 07:33:52.867963 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:52.867831 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6796c3b4-7535-469c-9fe5-b62477582184-credential-socket\") pod \"6796c3b4-7535-469c-9fe5-b62477582184\" (UID: \"6796c3b4-7535-469c-9fe5-b62477582184\") " Jul 09 07:33:52.867963 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:52.867863 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6796c3b4-7535-469c-9fe5-b62477582184-istio-envoy\") pod \"6796c3b4-7535-469c-9fe5-b62477582184\" (UID: \"6796c3b4-7535-469c-9fe5-b62477582184\") " Jul 09 07:33:52.868143 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:52.867978 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6796c3b4-7535-469c-9fe5-b62477582184-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6796c3b4-7535-469c-9fe5-b62477582184" (UID: "6796c3b4-7535-469c-9fe5-b62477582184"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 07:33:52.868143 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:52.868007 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6796c3b4-7535-469c-9fe5-b62477582184-workload-socket\") pod \"6796c3b4-7535-469c-9fe5-b62477582184\" (UID: \"6796c3b4-7535-469c-9fe5-b62477582184\") " Jul 09 07:33:52.868143 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:52.868056 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6796c3b4-7535-469c-9fe5-b62477582184-istio-data\") pod \"6796c3b4-7535-469c-9fe5-b62477582184\" (UID: \"6796c3b4-7535-469c-9fe5-b62477582184\") " Jul 09 07:33:52.868143 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:52.868083 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6796c3b4-7535-469c-9fe5-b62477582184-istio-podinfo\") pod \"6796c3b4-7535-469c-9fe5-b62477582184\" (UID: \"6796c3b4-7535-469c-9fe5-b62477582184\") " Jul 09 07:33:52.868441 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:52.868288 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6796c3b4-7535-469c-9fe5-b62477582184-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6796c3b4-7535-469c-9fe5-b62477582184" (UID: "6796c3b4-7535-469c-9fe5-b62477582184"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 07:33:52.868441 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:52.868322 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6796c3b4-7535-469c-9fe5-b62477582184-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6796c3b4-7535-469c-9fe5-b62477582184" (UID: "6796c3b4-7535-469c-9fe5-b62477582184"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 07:33:52.868540 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:52.868442 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6796c3b4-7535-469c-9fe5-b62477582184-istiod-ca-cert\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:33:52.868540 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:52.868463 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6796c3b4-7535-469c-9fe5-b62477582184-workload-socket\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:33:52.868540 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:52.868476 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6796c3b4-7535-469c-9fe5-b62477582184-workload-certs\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:33:52.868689 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:52.868669 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6796c3b4-7535-469c-9fe5-b62477582184-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6796c3b4-7535-469c-9fe5-b62477582184" (UID: "6796c3b4-7535-469c-9fe5-b62477582184"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 07:33:52.868886 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:52.868865 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6796c3b4-7535-469c-9fe5-b62477582184-istio-data" (OuterVolumeSpecName: "istio-data") pod "6796c3b4-7535-469c-9fe5-b62477582184" (UID: "6796c3b4-7535-469c-9fe5-b62477582184"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 07:33:52.870452 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:52.870410 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6796c3b4-7535-469c-9fe5-b62477582184-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6796c3b4-7535-469c-9fe5-b62477582184" (UID: "6796c3b4-7535-469c-9fe5-b62477582184"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 07:33:52.870452 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:52.870423 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6796c3b4-7535-469c-9fe5-b62477582184-kube-api-access-n2sn4" (OuterVolumeSpecName: "kube-api-access-n2sn4") pod "6796c3b4-7535-469c-9fe5-b62477582184" (UID: "6796c3b4-7535-469c-9fe5-b62477582184"). InnerVolumeSpecName "kube-api-access-n2sn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 07:33:52.870594 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:52.870494 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6796c3b4-7535-469c-9fe5-b62477582184-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6796c3b4-7535-469c-9fe5-b62477582184" (UID: "6796c3b4-7535-469c-9fe5-b62477582184"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 07:33:52.870594 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:52.870494 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6796c3b4-7535-469c-9fe5-b62477582184-istio-token" (OuterVolumeSpecName: "istio-token") pod "6796c3b4-7535-469c-9fe5-b62477582184" (UID: "6796c3b4-7535-469c-9fe5-b62477582184"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 07:33:52.969183 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:52.969152 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2sn4\" (UniqueName: \"kubernetes.io/projected/6796c3b4-7535-469c-9fe5-b62477582184-kube-api-access-n2sn4\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:33:52.969183 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:52.969179 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6796c3b4-7535-469c-9fe5-b62477582184-istio-token\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:33:52.969373 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:52.969197 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6796c3b4-7535-469c-9fe5-b62477582184-credential-socket\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:33:52.969373 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:52.969205 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6796c3b4-7535-469c-9fe5-b62477582184-istio-envoy\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:33:52.969373 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:52.969214 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6796c3b4-7535-469c-9fe5-b62477582184-istio-data\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:33:52.969373 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:52.969222 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6796c3b4-7535-469c-9fe5-b62477582184-istio-podinfo\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:33:53.493538 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:53.493510 2564 generic.go:356] "Generic (PLEG): container finished" podID="6796c3b4-7535-469c-9fe5-b62477582184" containerID="101900fa6881bfee814b83cd6b4e2a4885dae6b1f05284f1203414aeca7e2b40" exitCode=0 Jul 09 07:33:53.493678 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:53.493562 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8" Jul 09 07:33:53.493678 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:53.493593 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8" event={"ID":"6796c3b4-7535-469c-9fe5-b62477582184","Type":"ContainerDied","Data":"101900fa6881bfee814b83cd6b4e2a4885dae6b1f05284f1203414aeca7e2b40"} Jul 09 07:33:53.493678 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:53.493647 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8" event={"ID":"6796c3b4-7535-469c-9fe5-b62477582184","Type":"ContainerDied","Data":"51a935fad17ad9cb12065865eeac525a2bd0f1fb0cb9108a47c9527295ab4fcf"} Jul 09 07:33:53.493678 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:53.493670 2564 scope.go:117] "RemoveContainer" containerID="101900fa6881bfee814b83cd6b4e2a4885dae6b1f05284f1203414aeca7e2b40" Jul 09 07:33:53.501853 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:53.501837 2564 scope.go:117] "RemoveContainer" containerID="9ed1fd5b0a0ee877da7dfc0a008c49c85fc361d3253242f9f7c756ce49d23d3e" Jul 09 07:33:53.509277 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:53.509248 2564 scope.go:117] "RemoveContainer" containerID="101900fa6881bfee814b83cd6b4e2a4885dae6b1f05284f1203414aeca7e2b40" Jul 09 07:33:53.509559 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:33:53.509542 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"101900fa6881bfee814b83cd6b4e2a4885dae6b1f05284f1203414aeca7e2b40\": container with ID starting with 101900fa6881bfee814b83cd6b4e2a4885dae6b1f05284f1203414aeca7e2b40 not found: ID does not exist" containerID="101900fa6881bfee814b83cd6b4e2a4885dae6b1f05284f1203414aeca7e2b40" Jul 09 07:33:53.509617 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:53.509567 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"101900fa6881bfee814b83cd6b4e2a4885dae6b1f05284f1203414aeca7e2b40"} err="failed to get container status \"101900fa6881bfee814b83cd6b4e2a4885dae6b1f05284f1203414aeca7e2b40\": rpc error: code = NotFound desc = could not find container \"101900fa6881bfee814b83cd6b4e2a4885dae6b1f05284f1203414aeca7e2b40\": container with ID starting with 101900fa6881bfee814b83cd6b4e2a4885dae6b1f05284f1203414aeca7e2b40 not found: ID does not exist" Jul 09 07:33:53.509617 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:53.509583 2564 scope.go:117] "RemoveContainer" containerID="9ed1fd5b0a0ee877da7dfc0a008c49c85fc361d3253242f9f7c756ce49d23d3e" Jul 09 07:33:53.509823 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:33:53.509806 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ed1fd5b0a0ee877da7dfc0a008c49c85fc361d3253242f9f7c756ce49d23d3e\": container with ID starting with 9ed1fd5b0a0ee877da7dfc0a008c49c85fc361d3253242f9f7c756ce49d23d3e not found: ID does not exist" containerID="9ed1fd5b0a0ee877da7dfc0a008c49c85fc361d3253242f9f7c756ce49d23d3e" Jul 09 07:33:53.509900 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:53.509835 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ed1fd5b0a0ee877da7dfc0a008c49c85fc361d3253242f9f7c756ce49d23d3e"} err="failed to get container status \"9ed1fd5b0a0ee877da7dfc0a008c49c85fc361d3253242f9f7c756ce49d23d3e\": rpc error: code = NotFound desc = could not find container \"9ed1fd5b0a0ee877da7dfc0a008c49c85fc361d3253242f9f7c756ce49d23d3e\": container with ID starting with 9ed1fd5b0a0ee877da7dfc0a008c49c85fc361d3253242f9f7c756ce49d23d3e not found: ID does not exist" Jul 09 07:33:53.517826 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:53.517804 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8"] Jul 09 07:33:53.525256 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:53.525229 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-b0405232-openshift-default-d4c66f55f-4csm8"] Jul 09 07:33:55.462810 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:33:55.462778 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6796c3b4-7535-469c-9fe5-b62477582184" path="/var/lib/kubelet/pods/6796c3b4-7535-469c-9fe5-b62477582184/volumes" Jul 09 07:34:45.087241 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:45.087204 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk"] Jul 09 07:34:45.087690 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:45.087559 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6796c3b4-7535-469c-9fe5-b62477582184" containerName="istio-proxy" Jul 09 07:34:45.087690 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:45.087572 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6796c3b4-7535-469c-9fe5-b62477582184" containerName="istio-proxy" Jul 09 07:34:45.087690 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:45.087581 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="468e77a9-8396-4302-90de-8306089fcee9" containerName="main" Jul 09 07:34:45.087690 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:45.087586 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="468e77a9-8396-4302-90de-8306089fcee9" containerName="main" Jul 09 07:34:45.087690 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:45.087657 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="468e77a9-8396-4302-90de-8306089fcee9" containerName="main" Jul 09 07:34:45.087690 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:45.087668 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6796c3b4-7535-469c-9fe5-b62477582184" containerName="istio-proxy" Jul 09 07:34:45.090626 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:45.090599 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk" Jul 09 07:34:45.097444 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:45.097415 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-9c5220-openshift-default-dockercfg-lrtwx\"" Jul 09 07:34:45.125615 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:45.125589 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk"] Jul 09 07:34:45.187826 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:45.187793 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2caf8729-d7e2-4eba-b289-b3c0364d25bf-credential-socket\") pod \"e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk\" (UID: \"2caf8729-d7e2-4eba-b289-b3c0364d25bf\") " pod="openshift-ingress/e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk" Jul 09 07:34:45.187826 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:45.187831 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2caf8729-d7e2-4eba-b289-b3c0364d25bf-istio-data\") pod \"e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk\" (UID: \"2caf8729-d7e2-4eba-b289-b3c0364d25bf\") " pod="openshift-ingress/e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk" Jul 09 07:34:45.187826 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:45.187859 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2caf8729-d7e2-4eba-b289-b3c0364d25bf-istiod-ca-cert\") pod \"e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk\" (UID: \"2caf8729-d7e2-4eba-b289-b3c0364d25bf\") " pod="openshift-ingress/e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk" Jul 09 07:34:45.188160 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:45.187911 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2caf8729-d7e2-4eba-b289-b3c0364d25bf-workload-certs\") pod \"e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk\" (UID: \"2caf8729-d7e2-4eba-b289-b3c0364d25bf\") " pod="openshift-ingress/e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk" Jul 09 07:34:45.188160 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:45.187976 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2caf8729-d7e2-4eba-b289-b3c0364d25bf-istio-podinfo\") pod \"e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk\" (UID: \"2caf8729-d7e2-4eba-b289-b3c0364d25bf\") " pod="openshift-ingress/e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk" Jul 09 07:34:45.188160 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:45.188003 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2caf8729-d7e2-4eba-b289-b3c0364d25bf-istio-token\") pod \"e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk\" (UID: \"2caf8729-d7e2-4eba-b289-b3c0364d25bf\") " pod="openshift-ingress/e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk" Jul 09 07:34:45.188160 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:45.188022 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2caf8729-d7e2-4eba-b289-b3c0364d25bf-istio-envoy\") pod \"e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk\" (UID: \"2caf8729-d7e2-4eba-b289-b3c0364d25bf\") " pod="openshift-ingress/e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk" Jul 09 07:34:45.188160 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:45.188053 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2g48\" (UniqueName: \"kubernetes.io/projected/2caf8729-d7e2-4eba-b289-b3c0364d25bf-kube-api-access-r2g48\") pod \"e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk\" (UID: \"2caf8729-d7e2-4eba-b289-b3c0364d25bf\") " pod="openshift-ingress/e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk" Jul 09 07:34:45.188160 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:45.188107 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2caf8729-d7e2-4eba-b289-b3c0364d25bf-workload-socket\") pod \"e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk\" (UID: \"2caf8729-d7e2-4eba-b289-b3c0364d25bf\") " pod="openshift-ingress/e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk" Jul 09 07:34:45.289127 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:45.289092 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2caf8729-d7e2-4eba-b289-b3c0364d25bf-workload-socket\") pod \"e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk\" (UID: \"2caf8729-d7e2-4eba-b289-b3c0364d25bf\") " pod="openshift-ingress/e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk" Jul 09 07:34:45.289333 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:45.289145 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2caf8729-d7e2-4eba-b289-b3c0364d25bf-credential-socket\") pod \"e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk\" (UID: \"2caf8729-d7e2-4eba-b289-b3c0364d25bf\") " pod="openshift-ingress/e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk" Jul 09 07:34:45.289333 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:45.289172 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2caf8729-d7e2-4eba-b289-b3c0364d25bf-istio-data\") pod \"e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk\" (UID: \"2caf8729-d7e2-4eba-b289-b3c0364d25bf\") " pod="openshift-ingress/e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk" Jul 09 07:34:45.289333 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:45.289196 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2caf8729-d7e2-4eba-b289-b3c0364d25bf-istiod-ca-cert\") pod \"e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk\" (UID: \"2caf8729-d7e2-4eba-b289-b3c0364d25bf\") " pod="openshift-ingress/e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk" Jul 09 07:34:45.289333 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:45.289216 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2caf8729-d7e2-4eba-b289-b3c0364d25bf-workload-certs\") pod \"e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk\" (UID: \"2caf8729-d7e2-4eba-b289-b3c0364d25bf\") " pod="openshift-ingress/e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk" Jul 09 07:34:45.289333 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:45.289253 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2caf8729-d7e2-4eba-b289-b3c0364d25bf-istio-podinfo\") pod \"e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk\" (UID: \"2caf8729-d7e2-4eba-b289-b3c0364d25bf\") " pod="openshift-ingress/e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk" Jul 09 07:34:45.289333 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:45.289290 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2caf8729-d7e2-4eba-b289-b3c0364d25bf-istio-token\") pod \"e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk\" (UID: \"2caf8729-d7e2-4eba-b289-b3c0364d25bf\") " pod="openshift-ingress/e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk" Jul 09 07:34:45.289651 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:45.289382 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2caf8729-d7e2-4eba-b289-b3c0364d25bf-istio-envoy\") pod \"e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk\" (UID: \"2caf8729-d7e2-4eba-b289-b3c0364d25bf\") " pod="openshift-ingress/e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk" Jul 09 07:34:45.289651 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:45.289617 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2caf8729-d7e2-4eba-b289-b3c0364d25bf-workload-socket\") pod \"e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk\" (UID: \"2caf8729-d7e2-4eba-b289-b3c0364d25bf\") " pod="openshift-ingress/e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk" Jul 09 07:34:45.289759 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:45.289673 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2g48\" (UniqueName: \"kubernetes.io/projected/2caf8729-d7e2-4eba-b289-b3c0364d25bf-kube-api-access-r2g48\") pod \"e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk\" (UID: \"2caf8729-d7e2-4eba-b289-b3c0364d25bf\") " pod="openshift-ingress/e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk" Jul 09 07:34:45.289759 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:45.289690 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2caf8729-d7e2-4eba-b289-b3c0364d25bf-workload-certs\") pod \"e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk\" (UID: \"2caf8729-d7e2-4eba-b289-b3c0364d25bf\") " pod="openshift-ingress/e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk" Jul 09 07:34:45.289759 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:45.289712 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2caf8729-d7e2-4eba-b289-b3c0364d25bf-istio-data\") pod \"e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk\" (UID: \"2caf8729-d7e2-4eba-b289-b3c0364d25bf\") " pod="openshift-ingress/e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk" Jul 09 07:34:45.289908 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:45.289882 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2caf8729-d7e2-4eba-b289-b3c0364d25bf-credential-socket\") pod \"e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk\" (UID: \"2caf8729-d7e2-4eba-b289-b3c0364d25bf\") " pod="openshift-ingress/e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk" Jul 09 07:34:45.289981 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:45.289961 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2caf8729-d7e2-4eba-b289-b3c0364d25bf-istiod-ca-cert\") pod \"e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk\" (UID: \"2caf8729-d7e2-4eba-b289-b3c0364d25bf\") " pod="openshift-ingress/e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk" Jul 09 07:34:45.291668 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:45.291649 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2caf8729-d7e2-4eba-b289-b3c0364d25bf-istio-envoy\") pod \"e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk\" (UID: \"2caf8729-d7e2-4eba-b289-b3c0364d25bf\") " pod="openshift-ingress/e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk" Jul 09 07:34:45.291885 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:45.291867 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2caf8729-d7e2-4eba-b289-b3c0364d25bf-istio-podinfo\") pod \"e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk\" (UID: \"2caf8729-d7e2-4eba-b289-b3c0364d25bf\") " pod="openshift-ingress/e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk" Jul 09 07:34:45.302693 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:45.302673 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2caf8729-d7e2-4eba-b289-b3c0364d25bf-istio-token\") pod \"e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk\" (UID: \"2caf8729-d7e2-4eba-b289-b3c0364d25bf\") " pod="openshift-ingress/e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk" Jul 09 07:34:45.304236 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:45.304215 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2g48\" (UniqueName: \"kubernetes.io/projected/2caf8729-d7e2-4eba-b289-b3c0364d25bf-kube-api-access-r2g48\") pod \"e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk\" (UID: \"2caf8729-d7e2-4eba-b289-b3c0364d25bf\") " pod="openshift-ingress/e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk" Jul 09 07:34:45.402185 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:45.402153 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk" Jul 09 07:34:45.572927 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:45.572900 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk"] Jul 09 07:34:45.575703 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:34:45.575673 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2caf8729_d7e2_4eba_b289_b3c0364d25bf.slice/crio-87974d06b2967412b34984c8c3908024210508f86bf2be76ad7513264b80f6c8 WatchSource:0}: Error finding container 87974d06b2967412b34984c8c3908024210508f86bf2be76ad7513264b80f6c8: Status 404 returned error can't find the container with id 87974d06b2967412b34984c8c3908024210508f86bf2be76ad7513264b80f6c8 Jul 09 07:34:45.577598 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:45.577562 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 07:34:45.577686 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:45.577645 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 07:34:45.577686 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:45.577676 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 07:34:45.724937 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:45.724853 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk" event={"ID":"2caf8729-d7e2-4eba-b289-b3c0364d25bf","Type":"ContainerStarted","Data":"9dbd512cc8a4d74400884ec3cc806f1f6b2cf4e3c2319ef31566024e11243a6d"} Jul 09 07:34:45.725077 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:45.724943 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk" event={"ID":"2caf8729-d7e2-4eba-b289-b3c0364d25bf","Type":"ContainerStarted","Data":"87974d06b2967412b34984c8c3908024210508f86bf2be76ad7513264b80f6c8"} Jul 09 07:34:45.754119 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:45.753367 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk" podStartSLOduration=0.753346409 podStartE2EDuration="753.346409ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 07:34:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 07:34:45.752217275 +0000 UTC m=+1944.803294502" watchObservedRunningTime="2026-07-09 07:34:45.753346409 +0000 UTC m=+1944.804423641" Jul 09 07:34:46.402946 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:46.402911 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk" Jul 09 07:34:46.407507 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:46.407487 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk" Jul 09 07:34:46.729770 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:46.729685 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk" Jul 09 07:34:46.730587 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:46.730567 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk" Jul 09 07:34:52.848785 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:52.848753 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv"] Jul 09 07:34:52.849325 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:52.849143 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6796c3b4-7535-469c-9fe5-b62477582184" containerName="istio-proxy" Jul 09 07:34:52.849325 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:52.849160 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6796c3b4-7535-469c-9fe5-b62477582184" containerName="istio-proxy" Jul 09 07:34:52.849325 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:52.849228 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6796c3b4-7535-469c-9fe5-b62477582184" containerName="istio-proxy" Jul 09 07:34:52.852370 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:52.852354 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv" Jul 09 07:34:52.854986 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:52.854962 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-055963-openshift-default-dockercfg-nk5wz\"" Jul 09 07:34:52.869063 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:52.869042 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv"] Jul 09 07:34:52.959387 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:52.959353 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57973303-54ce-41d6-b0c1-57998b6c4feb-workload-certs\") pod \"e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv\" (UID: \"57973303-54ce-41d6-b0c1-57998b6c4feb\") " pod="openshift-ingress/e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv" Jul 09 07:34:52.959387 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:52.959388 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57973303-54ce-41d6-b0c1-57998b6c4feb-istio-podinfo\") pod \"e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv\" (UID: \"57973303-54ce-41d6-b0c1-57998b6c4feb\") " pod="openshift-ingress/e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv" Jul 09 07:34:52.959620 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:52.959408 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57973303-54ce-41d6-b0c1-57998b6c4feb-istio-data\") pod \"e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv\" (UID: \"57973303-54ce-41d6-b0c1-57998b6c4feb\") " pod="openshift-ingress/e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv" Jul 09 07:34:52.959620 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:52.959429 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57973303-54ce-41d6-b0c1-57998b6c4feb-istio-token\") pod \"e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv\" (UID: \"57973303-54ce-41d6-b0c1-57998b6c4feb\") " pod="openshift-ingress/e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv" Jul 09 07:34:52.959620 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:52.959522 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57973303-54ce-41d6-b0c1-57998b6c4feb-istio-envoy\") pod \"e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv\" (UID: \"57973303-54ce-41d6-b0c1-57998b6c4feb\") " pod="openshift-ingress/e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv" Jul 09 07:34:52.959620 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:52.959556 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7spsf\" (UniqueName: \"kubernetes.io/projected/57973303-54ce-41d6-b0c1-57998b6c4feb-kube-api-access-7spsf\") pod \"e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv\" (UID: \"57973303-54ce-41d6-b0c1-57998b6c4feb\") " pod="openshift-ingress/e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv" Jul 09 07:34:52.959620 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:52.959586 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57973303-54ce-41d6-b0c1-57998b6c4feb-credential-socket\") pod \"e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv\" (UID: \"57973303-54ce-41d6-b0c1-57998b6c4feb\") " pod="openshift-ingress/e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv" Jul 09 07:34:52.959620 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:52.959611 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57973303-54ce-41d6-b0c1-57998b6c4feb-istiod-ca-cert\") pod \"e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv\" (UID: \"57973303-54ce-41d6-b0c1-57998b6c4feb\") " pod="openshift-ingress/e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv" Jul 09 07:34:52.959883 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:52.959661 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57973303-54ce-41d6-b0c1-57998b6c4feb-workload-socket\") pod \"e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv\" (UID: \"57973303-54ce-41d6-b0c1-57998b6c4feb\") " pod="openshift-ingress/e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv" Jul 09 07:34:53.060452 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:53.060418 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57973303-54ce-41d6-b0c1-57998b6c4feb-workload-certs\") pod \"e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv\" (UID: \"57973303-54ce-41d6-b0c1-57998b6c4feb\") " pod="openshift-ingress/e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv" Jul 09 07:34:53.060452 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:53.060458 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57973303-54ce-41d6-b0c1-57998b6c4feb-istio-podinfo\") pod \"e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv\" (UID: \"57973303-54ce-41d6-b0c1-57998b6c4feb\") " pod="openshift-ingress/e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv" Jul 09 07:34:53.060683 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:53.060478 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57973303-54ce-41d6-b0c1-57998b6c4feb-istio-data\") pod \"e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv\" (UID: \"57973303-54ce-41d6-b0c1-57998b6c4feb\") " pod="openshift-ingress/e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv" Jul 09 07:34:53.060683 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:53.060501 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57973303-54ce-41d6-b0c1-57998b6c4feb-istio-token\") pod \"e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv\" (UID: \"57973303-54ce-41d6-b0c1-57998b6c4feb\") " pod="openshift-ingress/e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv" Jul 09 07:34:53.060683 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:53.060529 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57973303-54ce-41d6-b0c1-57998b6c4feb-istio-envoy\") pod \"e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv\" (UID: \"57973303-54ce-41d6-b0c1-57998b6c4feb\") " pod="openshift-ingress/e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv" Jul 09 07:34:53.060683 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:53.060558 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7spsf\" (UniqueName: \"kubernetes.io/projected/57973303-54ce-41d6-b0c1-57998b6c4feb-kube-api-access-7spsf\") pod \"e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv\" (UID: \"57973303-54ce-41d6-b0c1-57998b6c4feb\") " pod="openshift-ingress/e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv" Jul 09 07:34:53.060683 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:53.060585 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57973303-54ce-41d6-b0c1-57998b6c4feb-credential-socket\") pod \"e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv\" (UID: \"57973303-54ce-41d6-b0c1-57998b6c4feb\") " pod="openshift-ingress/e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv" Jul 09 07:34:53.060683 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:53.060607 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57973303-54ce-41d6-b0c1-57998b6c4feb-istiod-ca-cert\") pod \"e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv\" (UID: \"57973303-54ce-41d6-b0c1-57998b6c4feb\") " pod="openshift-ingress/e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv" Jul 09 07:34:53.060683 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:53.060646 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57973303-54ce-41d6-b0c1-57998b6c4feb-workload-socket\") pod \"e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv\" (UID: \"57973303-54ce-41d6-b0c1-57998b6c4feb\") " pod="openshift-ingress/e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv" Jul 09 07:34:53.061027 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:53.060849 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57973303-54ce-41d6-b0c1-57998b6c4feb-workload-certs\") pod \"e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv\" (UID: \"57973303-54ce-41d6-b0c1-57998b6c4feb\") " pod="openshift-ingress/e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv" Jul 09 07:34:53.061027 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:53.060905 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57973303-54ce-41d6-b0c1-57998b6c4feb-istio-data\") pod \"e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv\" (UID: \"57973303-54ce-41d6-b0c1-57998b6c4feb\") " pod="openshift-ingress/e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv" Jul 09 07:34:53.061127 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:53.061064 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57973303-54ce-41d6-b0c1-57998b6c4feb-credential-socket\") pod \"e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv\" (UID: \"57973303-54ce-41d6-b0c1-57998b6c4feb\") " pod="openshift-ingress/e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv" Jul 09 07:34:53.061178 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:53.061135 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57973303-54ce-41d6-b0c1-57998b6c4feb-workload-socket\") pod \"e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv\" (UID: \"57973303-54ce-41d6-b0c1-57998b6c4feb\") " pod="openshift-ingress/e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv" Jul 09 07:34:53.061415 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:53.061395 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57973303-54ce-41d6-b0c1-57998b6c4feb-istiod-ca-cert\") pod \"e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv\" (UID: \"57973303-54ce-41d6-b0c1-57998b6c4feb\") " pod="openshift-ingress/e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv" Jul 09 07:34:53.062834 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:53.062812 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57973303-54ce-41d6-b0c1-57998b6c4feb-istio-envoy\") pod \"e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv\" (UID: \"57973303-54ce-41d6-b0c1-57998b6c4feb\") " pod="openshift-ingress/e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv" Jul 09 07:34:53.062948 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:53.062932 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57973303-54ce-41d6-b0c1-57998b6c4feb-istio-podinfo\") pod \"e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv\" (UID: \"57973303-54ce-41d6-b0c1-57998b6c4feb\") " pod="openshift-ingress/e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv" Jul 09 07:34:53.070428 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:53.070407 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57973303-54ce-41d6-b0c1-57998b6c4feb-istio-token\") pod \"e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv\" (UID: \"57973303-54ce-41d6-b0c1-57998b6c4feb\") " pod="openshift-ingress/e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv" Jul 09 07:34:53.072850 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:53.072831 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7spsf\" (UniqueName: \"kubernetes.io/projected/57973303-54ce-41d6-b0c1-57998b6c4feb-kube-api-access-7spsf\") pod \"e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv\" (UID: \"57973303-54ce-41d6-b0c1-57998b6c4feb\") " pod="openshift-ingress/e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv" Jul 09 07:34:53.162727 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:53.162697 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv" Jul 09 07:34:53.305126 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:53.305099 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv"] Jul 09 07:34:53.307396 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:34:53.307366 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57973303_54ce_41d6_b0c1_57998b6c4feb.slice/crio-2b87f11b4e8238ad5d729d659833388a2ef79fa04053e8827b5cf49da988725c WatchSource:0}: Error finding container 2b87f11b4e8238ad5d729d659833388a2ef79fa04053e8827b5cf49da988725c: Status 404 returned error can't find the container with id 2b87f11b4e8238ad5d729d659833388a2ef79fa04053e8827b5cf49da988725c Jul 09 07:34:53.309486 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:53.309450 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 07:34:53.309592 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:53.309526 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 07:34:53.309592 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:53.309574 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 07:34:53.761425 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:53.761391 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv" event={"ID":"57973303-54ce-41d6-b0c1-57998b6c4feb","Type":"ContainerStarted","Data":"cd63b5848aab2ef91c07caea95eec0c09acf1b8209b43b68c5136b165c07f627"} Jul 09 07:34:53.761595 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:53.761453 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv" event={"ID":"57973303-54ce-41d6-b0c1-57998b6c4feb","Type":"ContainerStarted","Data":"2b87f11b4e8238ad5d729d659833388a2ef79fa04053e8827b5cf49da988725c"} Jul 09 07:34:53.797573 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:53.797502 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv" podStartSLOduration=1.7974856529999999 podStartE2EDuration="1.797485653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 07:34:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 07:34:53.796529933 +0000 UTC m=+1952.847607163" watchObservedRunningTime="2026-07-09 07:34:53.797485653 +0000 UTC m=+1952.848562880" Jul 09 07:34:54.163518 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:54.163481 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv" Jul 09 07:34:54.168668 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:54.168646 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv" Jul 09 07:34:54.765733 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:54.765706 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv" Jul 09 07:34:54.766686 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:34:54.766666 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv" Jul 09 07:35:04.810520 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:04.810428 2564 generic.go:356] "Generic (PLEG): container finished" podID="2caf8729-d7e2-4eba-b289-b3c0364d25bf" containerID="9dbd512cc8a4d74400884ec3cc806f1f6b2cf4e3c2319ef31566024e11243a6d" exitCode=0 Jul 09 07:35:04.810520 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:04.810507 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk" event={"ID":"2caf8729-d7e2-4eba-b289-b3c0364d25bf","Type":"ContainerDied","Data":"9dbd512cc8a4d74400884ec3cc806f1f6b2cf4e3c2319ef31566024e11243a6d"} Jul 09 07:35:04.810951 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:04.810910 2564 scope.go:117] "RemoveContainer" containerID="9dbd512cc8a4d74400884ec3cc806f1f6b2cf4e3c2319ef31566024e11243a6d" Jul 09 07:35:04.811851 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:04.811825 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 07:35:04.811905 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:04.811887 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 07:35:04.811940 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:04.811919 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 07:35:05.817208 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:05.817172 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk" event={"ID":"2caf8729-d7e2-4eba-b289-b3c0364d25bf","Type":"ContainerStarted","Data":"0bfcb7cc04fb4e713c7694bb42e7038554ba5b1b2dbee8fcfb9533d6fe2bfa24"} Jul 09 07:35:06.402635 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:06.402598 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk" Jul 09 07:35:06.402843 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:06.402818 2564 patch_prober.go:28] interesting pod/e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Jul 09 07:35:06.402905 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:06.402880 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk" podUID="2caf8729-d7e2-4eba-b289-b3c0364d25bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Jul 09 07:35:07.403367 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:07.403333 2564 patch_prober.go:28] interesting pod/e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Jul 09 07:35:07.403762 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:07.403391 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk" podUID="2caf8729-d7e2-4eba-b289-b3c0364d25bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Jul 09 07:35:08.403323 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:08.403280 2564 patch_prober.go:28] interesting pod/e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Jul 09 07:35:08.403503 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:08.403360 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk" podUID="2caf8729-d7e2-4eba-b289-b3c0364d25bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Jul 09 07:35:09.403255 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:09.403214 2564 patch_prober.go:28] interesting pod/e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Jul 09 07:35:09.403448 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:09.403302 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk" podUID="2caf8729-d7e2-4eba-b289-b3c0364d25bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Jul 09 07:35:10.406394 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:10.406359 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk" Jul 09 07:35:10.406840 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:10.406758 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk" Jul 09 07:35:10.407467 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:10.407442 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk" Jul 09 07:35:10.808491 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:10.808400 2564 patch_prober.go:28] interesting pod/e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 09 07:35:10.808640 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:10.808486 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv" podUID="57973303-54ce-41d6-b0c1-57998b6c4feb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 09 07:35:11.850630 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:11.850594 2564 generic.go:356] "Generic (PLEG): container finished" podID="57973303-54ce-41d6-b0c1-57998b6c4feb" containerID="cd63b5848aab2ef91c07caea95eec0c09acf1b8209b43b68c5136b165c07f627" exitCode=0 Jul 09 07:35:11.851134 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:11.850664 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv" event={"ID":"57973303-54ce-41d6-b0c1-57998b6c4feb","Type":"ContainerDied","Data":"cd63b5848aab2ef91c07caea95eec0c09acf1b8209b43b68c5136b165c07f627"} Jul 09 07:35:11.851203 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:11.851176 2564 scope.go:117] "RemoveContainer" containerID="cd63b5848aab2ef91c07caea95eec0c09acf1b8209b43b68c5136b165c07f627" Jul 09 07:35:11.852114 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:11.852086 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 07:35:11.852205 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:11.852153 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 07:35:11.852205 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:11.852188 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 07:35:12.858422 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:12.858388 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv" event={"ID":"57973303-54ce-41d6-b0c1-57998b6c4feb","Type":"ContainerStarted","Data":"1d38a9fd6dbeea18b0cf2b02eb0c4a5974af29f7b938be72794e537ff8c32854"} Jul 09 07:35:13.163097 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:13.163059 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv" Jul 09 07:35:13.163359 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:13.163340 2564 patch_prober.go:28] interesting pod/e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Jul 09 07:35:13.163446 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:13.163426 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv" podUID="57973303-54ce-41d6-b0c1-57998b6c4feb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Jul 09 07:35:14.163830 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:14.163793 2564 patch_prober.go:28] interesting pod/e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Jul 09 07:35:14.164214 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:14.163878 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv" podUID="57973303-54ce-41d6-b0c1-57998b6c4feb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Jul 09 07:35:15.163174 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:15.163141 2564 patch_prober.go:28] interesting pod/e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Jul 09 07:35:15.163389 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:15.163196 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv" podUID="57973303-54ce-41d6-b0c1-57998b6c4feb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Jul 09 07:35:16.163277 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:16.163235 2564 patch_prober.go:28] interesting pod/e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Jul 09 07:35:16.163669 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:16.163321 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv" podUID="57973303-54ce-41d6-b0c1-57998b6c4feb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Jul 09 07:35:17.166852 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:17.166823 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv" Jul 09 07:35:17.167297 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:17.167082 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv" Jul 09 07:35:17.167727 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:17.167712 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv" Jul 09 07:35:35.988042 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:35.988005 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-055963/test-model-055963-kserve-6dd9cf7769-lbctp"] Jul 09 07:35:35.990737 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:35.990716 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-055963/test-model-055963-kserve-6dd9cf7769-lbctp" Jul 09 07:35:35.994203 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:35.994178 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-055963\"/\"openshift-service-ca.crt\"" Jul 09 07:35:35.994388 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:35.994258 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-055963\"/\"default-dockercfg-gh8m4\"" Jul 09 07:35:35.994388 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:35.994258 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-055963\"/\"test-model-055963-kserve-self-signed-certs\"" Jul 09 07:35:35.994579 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:35.994288 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-055963\"/\"kube-root-ca.crt\"" Jul 09 07:35:36.001033 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:36.001011 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-055963/test-model-055963-kserve-6dd9cf7769-lbctp"] Jul 09 07:35:36.147047 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:36.147014 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e66ae8a5-f4d2-4b2c-aba4-dd85508b0775-home\") pod \"test-model-055963-kserve-6dd9cf7769-lbctp\" (UID: \"e66ae8a5-f4d2-4b2c-aba4-dd85508b0775\") " pod="ai-tenant-e2e-inf-b-055963/test-model-055963-kserve-6dd9cf7769-lbctp" Jul 09 07:35:36.147215 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:36.147058 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4jzh\" (UniqueName: \"kubernetes.io/projected/e66ae8a5-f4d2-4b2c-aba4-dd85508b0775-kube-api-access-g4jzh\") pod \"test-model-055963-kserve-6dd9cf7769-lbctp\" (UID: \"e66ae8a5-f4d2-4b2c-aba4-dd85508b0775\") " pod="ai-tenant-e2e-inf-b-055963/test-model-055963-kserve-6dd9cf7769-lbctp" Jul 09 07:35:36.147215 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:36.147085 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e66ae8a5-f4d2-4b2c-aba4-dd85508b0775-tmp-dir\") pod \"test-model-055963-kserve-6dd9cf7769-lbctp\" (UID: \"e66ae8a5-f4d2-4b2c-aba4-dd85508b0775\") " pod="ai-tenant-e2e-inf-b-055963/test-model-055963-kserve-6dd9cf7769-lbctp" Jul 09 07:35:36.147215 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:36.147144 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e66ae8a5-f4d2-4b2c-aba4-dd85508b0775-model-cache\") pod \"test-model-055963-kserve-6dd9cf7769-lbctp\" (UID: \"e66ae8a5-f4d2-4b2c-aba4-dd85508b0775\") " pod="ai-tenant-e2e-inf-b-055963/test-model-055963-kserve-6dd9cf7769-lbctp" Jul 09 07:35:36.147215 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:36.147160 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e66ae8a5-f4d2-4b2c-aba4-dd85508b0775-tls-certs\") pod \"test-model-055963-kserve-6dd9cf7769-lbctp\" (UID: \"e66ae8a5-f4d2-4b2c-aba4-dd85508b0775\") " pod="ai-tenant-e2e-inf-b-055963/test-model-055963-kserve-6dd9cf7769-lbctp" Jul 09 07:35:36.147215 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:36.147186 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e66ae8a5-f4d2-4b2c-aba4-dd85508b0775-dshm\") pod \"test-model-055963-kserve-6dd9cf7769-lbctp\" (UID: \"e66ae8a5-f4d2-4b2c-aba4-dd85508b0775\") " pod="ai-tenant-e2e-inf-b-055963/test-model-055963-kserve-6dd9cf7769-lbctp" Jul 09 07:35:36.248345 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:36.248247 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4jzh\" (UniqueName: \"kubernetes.io/projected/e66ae8a5-f4d2-4b2c-aba4-dd85508b0775-kube-api-access-g4jzh\") pod \"test-model-055963-kserve-6dd9cf7769-lbctp\" (UID: \"e66ae8a5-f4d2-4b2c-aba4-dd85508b0775\") " pod="ai-tenant-e2e-inf-b-055963/test-model-055963-kserve-6dd9cf7769-lbctp" Jul 09 07:35:36.248345 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:36.248304 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e66ae8a5-f4d2-4b2c-aba4-dd85508b0775-tmp-dir\") pod \"test-model-055963-kserve-6dd9cf7769-lbctp\" (UID: \"e66ae8a5-f4d2-4b2c-aba4-dd85508b0775\") " pod="ai-tenant-e2e-inf-b-055963/test-model-055963-kserve-6dd9cf7769-lbctp" Jul 09 07:35:36.248575 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:36.248366 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e66ae8a5-f4d2-4b2c-aba4-dd85508b0775-model-cache\") pod \"test-model-055963-kserve-6dd9cf7769-lbctp\" (UID: \"e66ae8a5-f4d2-4b2c-aba4-dd85508b0775\") " pod="ai-tenant-e2e-inf-b-055963/test-model-055963-kserve-6dd9cf7769-lbctp" Jul 09 07:35:36.248575 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:36.248388 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e66ae8a5-f4d2-4b2c-aba4-dd85508b0775-tls-certs\") pod \"test-model-055963-kserve-6dd9cf7769-lbctp\" (UID: \"e66ae8a5-f4d2-4b2c-aba4-dd85508b0775\") " pod="ai-tenant-e2e-inf-b-055963/test-model-055963-kserve-6dd9cf7769-lbctp" Jul 09 07:35:36.248575 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:36.248419 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e66ae8a5-f4d2-4b2c-aba4-dd85508b0775-dshm\") pod \"test-model-055963-kserve-6dd9cf7769-lbctp\" (UID: \"e66ae8a5-f4d2-4b2c-aba4-dd85508b0775\") " pod="ai-tenant-e2e-inf-b-055963/test-model-055963-kserve-6dd9cf7769-lbctp" Jul 09 07:35:36.248575 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:36.248473 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e66ae8a5-f4d2-4b2c-aba4-dd85508b0775-home\") pod \"test-model-055963-kserve-6dd9cf7769-lbctp\" (UID: \"e66ae8a5-f4d2-4b2c-aba4-dd85508b0775\") " pod="ai-tenant-e2e-inf-b-055963/test-model-055963-kserve-6dd9cf7769-lbctp" Jul 09 07:35:36.248798 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:36.248752 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e66ae8a5-f4d2-4b2c-aba4-dd85508b0775-tmp-dir\") pod \"test-model-055963-kserve-6dd9cf7769-lbctp\" (UID: \"e66ae8a5-f4d2-4b2c-aba4-dd85508b0775\") " pod="ai-tenant-e2e-inf-b-055963/test-model-055963-kserve-6dd9cf7769-lbctp" Jul 09 07:35:36.248849 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:36.248794 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e66ae8a5-f4d2-4b2c-aba4-dd85508b0775-home\") pod \"test-model-055963-kserve-6dd9cf7769-lbctp\" (UID: \"e66ae8a5-f4d2-4b2c-aba4-dd85508b0775\") " pod="ai-tenant-e2e-inf-b-055963/test-model-055963-kserve-6dd9cf7769-lbctp" Jul 09 07:35:36.248849 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:36.248838 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e66ae8a5-f4d2-4b2c-aba4-dd85508b0775-model-cache\") pod \"test-model-055963-kserve-6dd9cf7769-lbctp\" (UID: \"e66ae8a5-f4d2-4b2c-aba4-dd85508b0775\") " pod="ai-tenant-e2e-inf-b-055963/test-model-055963-kserve-6dd9cf7769-lbctp" Jul 09 07:35:36.250698 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:36.250677 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e66ae8a5-f4d2-4b2c-aba4-dd85508b0775-dshm\") pod \"test-model-055963-kserve-6dd9cf7769-lbctp\" (UID: \"e66ae8a5-f4d2-4b2c-aba4-dd85508b0775\") " pod="ai-tenant-e2e-inf-b-055963/test-model-055963-kserve-6dd9cf7769-lbctp" Jul 09 07:35:36.251000 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:36.250978 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e66ae8a5-f4d2-4b2c-aba4-dd85508b0775-tls-certs\") pod \"test-model-055963-kserve-6dd9cf7769-lbctp\" (UID: \"e66ae8a5-f4d2-4b2c-aba4-dd85508b0775\") " pod="ai-tenant-e2e-inf-b-055963/test-model-055963-kserve-6dd9cf7769-lbctp" Jul 09 07:35:36.256687 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:36.256668 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4jzh\" (UniqueName: \"kubernetes.io/projected/e66ae8a5-f4d2-4b2c-aba4-dd85508b0775-kube-api-access-g4jzh\") pod \"test-model-055963-kserve-6dd9cf7769-lbctp\" (UID: \"e66ae8a5-f4d2-4b2c-aba4-dd85508b0775\") " pod="ai-tenant-e2e-inf-b-055963/test-model-055963-kserve-6dd9cf7769-lbctp" Jul 09 07:35:36.301658 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:36.301630 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-055963/test-model-055963-kserve-6dd9cf7769-lbctp" Jul 09 07:35:36.431433 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:36.431404 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-055963/test-model-055963-kserve-6dd9cf7769-lbctp"] Jul 09 07:35:36.433452 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:35:36.433425 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode66ae8a5_f4d2_4b2c_aba4_dd85508b0775.slice/crio-a3ff01bbc68604d2b7f7215382cc6414626c9624d087728f41dda7c7153bb0a2 WatchSource:0}: Error finding container a3ff01bbc68604d2b7f7215382cc6414626c9624d087728f41dda7c7153bb0a2: Status 404 returned error can't find the container with id a3ff01bbc68604d2b7f7215382cc6414626c9624d087728f41dda7c7153bb0a2 Jul 09 07:35:36.966583 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:36.966548 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-055963/test-model-055963-kserve-6dd9cf7769-lbctp" event={"ID":"e66ae8a5-f4d2-4b2c-aba4-dd85508b0775","Type":"ContainerStarted","Data":"f19586928711749b132ff45996331b14f9acada6dfb2c9afa3b3da6b4bfbff7e"} Jul 09 07:35:36.966749 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:36.966592 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-055963/test-model-055963-kserve-6dd9cf7769-lbctp" event={"ID":"e66ae8a5-f4d2-4b2c-aba4-dd85508b0775","Type":"ContainerStarted","Data":"a3ff01bbc68604d2b7f7215382cc6414626c9624d087728f41dda7c7153bb0a2"} Jul 09 07:35:36.986234 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:36.986181 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-055963/test-model-055963-kserve-6dd9cf7769-lbctp" podStartSLOduration=1.986166768 podStartE2EDuration="1.986166768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 07:35:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 07:35:36.98420494 +0000 UTC m=+1996.035282169" watchObservedRunningTime="2026-07-09 07:35:36.986166768 +0000 UTC m=+1996.037243996" Jul 09 07:35:46.302029 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:46.301997 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-055963/test-model-055963-kserve-6dd9cf7769-lbctp" Jul 09 07:35:46.302639 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:46.302051 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-055963/test-model-055963-kserve-6dd9cf7769-lbctp" Jul 09 07:35:46.314436 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:46.314410 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-055963/test-model-055963-kserve-6dd9cf7769-lbctp" Jul 09 07:35:47.025827 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:47.025800 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-055963/test-model-055963-kserve-6dd9cf7769-lbctp" Jul 09 07:35:54.654202 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:35:54.654172 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-055963/test-model-055963-kserve-6dd9cf7769-lbctp"] Jul 09 07:36:09.674588 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:09.674520 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-055963/test-model-055963-kserve-6dd9cf7769-lbctp" podUID="e66ae8a5-f4d2-4b2c-aba4-dd85508b0775" containerName="main" containerID="cri-o://f19586928711749b132ff45996331b14f9acada6dfb2c9afa3b3da6b4bfbff7e" gracePeriod=45 Jul 09 07:36:09.913499 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:09.913475 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-055963/test-model-055963-kserve-6dd9cf7769-lbctp" Jul 09 07:36:10.034906 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:10.034825 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e66ae8a5-f4d2-4b2c-aba4-dd85508b0775-model-cache\") pod \"e66ae8a5-f4d2-4b2c-aba4-dd85508b0775\" (UID: \"e66ae8a5-f4d2-4b2c-aba4-dd85508b0775\") " Jul 09 07:36:10.034906 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:10.034857 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e66ae8a5-f4d2-4b2c-aba4-dd85508b0775-tls-certs\") pod \"e66ae8a5-f4d2-4b2c-aba4-dd85508b0775\" (UID: \"e66ae8a5-f4d2-4b2c-aba4-dd85508b0775\") " Jul 09 07:36:10.034906 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:10.034877 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e66ae8a5-f4d2-4b2c-aba4-dd85508b0775-dshm\") pod \"e66ae8a5-f4d2-4b2c-aba4-dd85508b0775\" (UID: \"e66ae8a5-f4d2-4b2c-aba4-dd85508b0775\") " Jul 09 07:36:10.035189 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:10.034958 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e66ae8a5-f4d2-4b2c-aba4-dd85508b0775-home\") pod \"e66ae8a5-f4d2-4b2c-aba4-dd85508b0775\" (UID: \"e66ae8a5-f4d2-4b2c-aba4-dd85508b0775\") " Jul 09 07:36:10.035189 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:10.034986 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e66ae8a5-f4d2-4b2c-aba4-dd85508b0775-tmp-dir\") pod \"e66ae8a5-f4d2-4b2c-aba4-dd85508b0775\" (UID: \"e66ae8a5-f4d2-4b2c-aba4-dd85508b0775\") " Jul 09 07:36:10.035189 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:10.035044 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g4jzh\" (UniqueName: \"kubernetes.io/projected/e66ae8a5-f4d2-4b2c-aba4-dd85508b0775-kube-api-access-g4jzh\") pod \"e66ae8a5-f4d2-4b2c-aba4-dd85508b0775\" (UID: \"e66ae8a5-f4d2-4b2c-aba4-dd85508b0775\") " Jul 09 07:36:10.035189 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:10.035126 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e66ae8a5-f4d2-4b2c-aba4-dd85508b0775-model-cache" (OuterVolumeSpecName: "model-cache") pod "e66ae8a5-f4d2-4b2c-aba4-dd85508b0775" (UID: "e66ae8a5-f4d2-4b2c-aba4-dd85508b0775"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 07:36:10.035411 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:10.035213 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e66ae8a5-f4d2-4b2c-aba4-dd85508b0775-home" (OuterVolumeSpecName: "home") pod "e66ae8a5-f4d2-4b2c-aba4-dd85508b0775" (UID: "e66ae8a5-f4d2-4b2c-aba4-dd85508b0775"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 07:36:10.035411 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:10.035295 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e66ae8a5-f4d2-4b2c-aba4-dd85508b0775-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e66ae8a5-f4d2-4b2c-aba4-dd85508b0775" (UID: "e66ae8a5-f4d2-4b2c-aba4-dd85508b0775"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 07:36:10.035411 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:10.035397 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e66ae8a5-f4d2-4b2c-aba4-dd85508b0775-home\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:36:10.035512 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:10.035412 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e66ae8a5-f4d2-4b2c-aba4-dd85508b0775-tmp-dir\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:36:10.035512 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:10.035426 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e66ae8a5-f4d2-4b2c-aba4-dd85508b0775-model-cache\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:36:10.037083 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:10.037053 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e66ae8a5-f4d2-4b2c-aba4-dd85508b0775-dshm" (OuterVolumeSpecName: "dshm") pod "e66ae8a5-f4d2-4b2c-aba4-dd85508b0775" (UID: "e66ae8a5-f4d2-4b2c-aba4-dd85508b0775"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 07:36:10.037201 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:10.037097 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e66ae8a5-f4d2-4b2c-aba4-dd85508b0775-kube-api-access-g4jzh" (OuterVolumeSpecName: "kube-api-access-g4jzh") pod "e66ae8a5-f4d2-4b2c-aba4-dd85508b0775" (UID: "e66ae8a5-f4d2-4b2c-aba4-dd85508b0775"). InnerVolumeSpecName "kube-api-access-g4jzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 07:36:10.037201 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:10.037107 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e66ae8a5-f4d2-4b2c-aba4-dd85508b0775-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e66ae8a5-f4d2-4b2c-aba4-dd85508b0775" (UID: "e66ae8a5-f4d2-4b2c-aba4-dd85508b0775"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 07:36:10.120254 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:10.120219 2564 generic.go:356] "Generic (PLEG): container finished" podID="e66ae8a5-f4d2-4b2c-aba4-dd85508b0775" containerID="f19586928711749b132ff45996331b14f9acada6dfb2c9afa3b3da6b4bfbff7e" exitCode=0 Jul 09 07:36:10.120254 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:10.120277 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-055963/test-model-055963-kserve-6dd9cf7769-lbctp" Jul 09 07:36:10.120521 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:10.120311 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-055963/test-model-055963-kserve-6dd9cf7769-lbctp" event={"ID":"e66ae8a5-f4d2-4b2c-aba4-dd85508b0775","Type":"ContainerDied","Data":"f19586928711749b132ff45996331b14f9acada6dfb2c9afa3b3da6b4bfbff7e"} Jul 09 07:36:10.120521 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:10.120358 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-055963/test-model-055963-kserve-6dd9cf7769-lbctp" event={"ID":"e66ae8a5-f4d2-4b2c-aba4-dd85508b0775","Type":"ContainerDied","Data":"a3ff01bbc68604d2b7f7215382cc6414626c9624d087728f41dda7c7153bb0a2"} Jul 09 07:36:10.120521 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:10.120380 2564 scope.go:117] "RemoveContainer" containerID="f19586928711749b132ff45996331b14f9acada6dfb2c9afa3b3da6b4bfbff7e" Jul 09 07:36:10.129313 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:10.129256 2564 scope.go:117] "RemoveContainer" containerID="f19586928711749b132ff45996331b14f9acada6dfb2c9afa3b3da6b4bfbff7e" Jul 09 07:36:10.129544 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:36:10.129527 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f19586928711749b132ff45996331b14f9acada6dfb2c9afa3b3da6b4bfbff7e\": container with ID starting with f19586928711749b132ff45996331b14f9acada6dfb2c9afa3b3da6b4bfbff7e not found: ID does not exist" containerID="f19586928711749b132ff45996331b14f9acada6dfb2c9afa3b3da6b4bfbff7e" Jul 09 07:36:10.129609 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:10.129550 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f19586928711749b132ff45996331b14f9acada6dfb2c9afa3b3da6b4bfbff7e"} err="failed to get container status \"f19586928711749b132ff45996331b14f9acada6dfb2c9afa3b3da6b4bfbff7e\": rpc error: code = NotFound desc = could not find container \"f19586928711749b132ff45996331b14f9acada6dfb2c9afa3b3da6b4bfbff7e\": container with ID starting with f19586928711749b132ff45996331b14f9acada6dfb2c9afa3b3da6b4bfbff7e not found: ID does not exist" Jul 09 07:36:10.136788 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:10.136769 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e66ae8a5-f4d2-4b2c-aba4-dd85508b0775-tls-certs\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:36:10.136788 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:10.136788 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e66ae8a5-f4d2-4b2c-aba4-dd85508b0775-dshm\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:36:10.136906 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:10.136797 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4jzh\" (UniqueName: \"kubernetes.io/projected/e66ae8a5-f4d2-4b2c-aba4-dd85508b0775-kube-api-access-g4jzh\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:36:10.143380 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:10.143357 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-055963/test-model-055963-kserve-6dd9cf7769-lbctp"] Jul 09 07:36:10.147071 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:10.147052 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-055963/test-model-055963-kserve-6dd9cf7769-lbctp"] Jul 09 07:36:11.464191 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:11.464159 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e66ae8a5-f4d2-4b2c-aba4-dd85508b0775" path="/var/lib/kubelet/pods/e66ae8a5-f4d2-4b2c-aba4-dd85508b0775/volumes" Jul 09 07:36:37.868668 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:37.868633 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk"] Jul 09 07:36:37.869257 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:37.868838 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk" podUID="2caf8729-d7e2-4eba-b289-b3c0364d25bf" containerName="istio-proxy" containerID="cri-o://0bfcb7cc04fb4e713c7694bb42e7038554ba5b1b2dbee8fcfb9533d6fe2bfa24" gracePeriod=30 Jul 09 07:36:40.409469 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:40.409427 2564 patch_prober.go:28] interesting pod/e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 09 07:36:40.409843 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:40.409491 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk" podUID="2caf8729-d7e2-4eba-b289-b3c0364d25bf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 07:36:43.119993 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:43.119968 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk" Jul 09 07:36:43.217623 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:43.217537 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2caf8729-d7e2-4eba-b289-b3c0364d25bf-credential-socket\") pod \"2caf8729-d7e2-4eba-b289-b3c0364d25bf\" (UID: \"2caf8729-d7e2-4eba-b289-b3c0364d25bf\") " Jul 09 07:36:43.217623 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:43.217571 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2caf8729-d7e2-4eba-b289-b3c0364d25bf-istio-envoy\") pod \"2caf8729-d7e2-4eba-b289-b3c0364d25bf\" (UID: \"2caf8729-d7e2-4eba-b289-b3c0364d25bf\") " Jul 09 07:36:43.217623 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:43.217592 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r2g48\" (UniqueName: \"kubernetes.io/projected/2caf8729-d7e2-4eba-b289-b3c0364d25bf-kube-api-access-r2g48\") pod \"2caf8729-d7e2-4eba-b289-b3c0364d25bf\" (UID: \"2caf8729-d7e2-4eba-b289-b3c0364d25bf\") " Jul 09 07:36:43.217623 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:43.217620 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2caf8729-d7e2-4eba-b289-b3c0364d25bf-istiod-ca-cert\") pod \"2caf8729-d7e2-4eba-b289-b3c0364d25bf\" (UID: \"2caf8729-d7e2-4eba-b289-b3c0364d25bf\") " Jul 09 07:36:43.218067 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:43.217663 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2caf8729-d7e2-4eba-b289-b3c0364d25bf-workload-certs\") pod \"2caf8729-d7e2-4eba-b289-b3c0364d25bf\" (UID: \"2caf8729-d7e2-4eba-b289-b3c0364d25bf\") " Jul 09 07:36:43.218067 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:43.217700 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2caf8729-d7e2-4eba-b289-b3c0364d25bf-istio-data\") pod \"2caf8729-d7e2-4eba-b289-b3c0364d25bf\" (UID: \"2caf8729-d7e2-4eba-b289-b3c0364d25bf\") " Jul 09 07:36:43.218067 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:43.217723 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2caf8729-d7e2-4eba-b289-b3c0364d25bf-istio-podinfo\") pod \"2caf8729-d7e2-4eba-b289-b3c0364d25bf\" (UID: \"2caf8729-d7e2-4eba-b289-b3c0364d25bf\") " Jul 09 07:36:43.218067 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:43.217774 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2caf8729-d7e2-4eba-b289-b3c0364d25bf-istio-token\") pod \"2caf8729-d7e2-4eba-b289-b3c0364d25bf\" (UID: \"2caf8729-d7e2-4eba-b289-b3c0364d25bf\") " Jul 09 07:36:43.218067 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:43.217832 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2caf8729-d7e2-4eba-b289-b3c0364d25bf-workload-socket\") pod \"2caf8729-d7e2-4eba-b289-b3c0364d25bf\" (UID: \"2caf8729-d7e2-4eba-b289-b3c0364d25bf\") " Jul 09 07:36:43.218067 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:43.217854 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2caf8729-d7e2-4eba-b289-b3c0364d25bf-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2caf8729-d7e2-4eba-b289-b3c0364d25bf" (UID: "2caf8729-d7e2-4eba-b289-b3c0364d25bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 07:36:43.218403 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:43.218088 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2caf8729-d7e2-4eba-b289-b3c0364d25bf-credential-socket\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:36:43.218403 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:43.218114 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2caf8729-d7e2-4eba-b289-b3c0364d25bf-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2caf8729-d7e2-4eba-b289-b3c0364d25bf" (UID: "2caf8729-d7e2-4eba-b289-b3c0364d25bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 07:36:43.218403 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:43.218327 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2caf8729-d7e2-4eba-b289-b3c0364d25bf-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2caf8729-d7e2-4eba-b289-b3c0364d25bf" (UID: "2caf8729-d7e2-4eba-b289-b3c0364d25bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 07:36:43.218665 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:43.218637 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2caf8729-d7e2-4eba-b289-b3c0364d25bf-istio-data" (OuterVolumeSpecName: "istio-data") pod "2caf8729-d7e2-4eba-b289-b3c0364d25bf" (UID: "2caf8729-d7e2-4eba-b289-b3c0364d25bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 07:36:43.218786 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:43.218759 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2caf8729-d7e2-4eba-b289-b3c0364d25bf-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2caf8729-d7e2-4eba-b289-b3c0364d25bf" (UID: "2caf8729-d7e2-4eba-b289-b3c0364d25bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 07:36:43.219964 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:43.219933 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2caf8729-d7e2-4eba-b289-b3c0364d25bf-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2caf8729-d7e2-4eba-b289-b3c0364d25bf" (UID: "2caf8729-d7e2-4eba-b289-b3c0364d25bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 07:36:43.219964 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:43.219950 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2caf8729-d7e2-4eba-b289-b3c0364d25bf-istio-token" (OuterVolumeSpecName: "istio-token") pod "2caf8729-d7e2-4eba-b289-b3c0364d25bf" (UID: "2caf8729-d7e2-4eba-b289-b3c0364d25bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 07:36:43.220215 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:43.220190 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2caf8729-d7e2-4eba-b289-b3c0364d25bf-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2caf8729-d7e2-4eba-b289-b3c0364d25bf" (UID: "2caf8729-d7e2-4eba-b289-b3c0364d25bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 07:36:43.220295 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:43.220208 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2caf8729-d7e2-4eba-b289-b3c0364d25bf-kube-api-access-r2g48" (OuterVolumeSpecName: "kube-api-access-r2g48") pod "2caf8729-d7e2-4eba-b289-b3c0364d25bf" (UID: "2caf8729-d7e2-4eba-b289-b3c0364d25bf"). InnerVolumeSpecName "kube-api-access-r2g48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 07:36:43.277547 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:43.277507 2564 generic.go:356] "Generic (PLEG): container finished" podID="2caf8729-d7e2-4eba-b289-b3c0364d25bf" containerID="0bfcb7cc04fb4e713c7694bb42e7038554ba5b1b2dbee8fcfb9533d6fe2bfa24" exitCode=0 Jul 09 07:36:43.277668 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:43.277560 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk" Jul 09 07:36:43.277668 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:43.277562 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk" event={"ID":"2caf8729-d7e2-4eba-b289-b3c0364d25bf","Type":"ContainerDied","Data":"0bfcb7cc04fb4e713c7694bb42e7038554ba5b1b2dbee8fcfb9533d6fe2bfa24"} Jul 09 07:36:43.277668 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:43.277631 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk" event={"ID":"2caf8729-d7e2-4eba-b289-b3c0364d25bf","Type":"ContainerDied","Data":"87974d06b2967412b34984c8c3908024210508f86bf2be76ad7513264b80f6c8"} Jul 09 07:36:43.277668 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:43.277655 2564 scope.go:117] "RemoveContainer" containerID="0bfcb7cc04fb4e713c7694bb42e7038554ba5b1b2dbee8fcfb9533d6fe2bfa24" Jul 09 07:36:43.286540 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:43.286522 2564 scope.go:117] "RemoveContainer" containerID="9dbd512cc8a4d74400884ec3cc806f1f6b2cf4e3c2319ef31566024e11243a6d" Jul 09 07:36:43.294095 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:43.294079 2564 scope.go:117] "RemoveContainer" containerID="0bfcb7cc04fb4e713c7694bb42e7038554ba5b1b2dbee8fcfb9533d6fe2bfa24" Jul 09 07:36:43.294358 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:36:43.294335 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bfcb7cc04fb4e713c7694bb42e7038554ba5b1b2dbee8fcfb9533d6fe2bfa24\": container with ID starting with 0bfcb7cc04fb4e713c7694bb42e7038554ba5b1b2dbee8fcfb9533d6fe2bfa24 not found: ID does not exist" containerID="0bfcb7cc04fb4e713c7694bb42e7038554ba5b1b2dbee8fcfb9533d6fe2bfa24" Jul 09 07:36:43.294455 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:43.294365 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bfcb7cc04fb4e713c7694bb42e7038554ba5b1b2dbee8fcfb9533d6fe2bfa24"} err="failed to get container status \"0bfcb7cc04fb4e713c7694bb42e7038554ba5b1b2dbee8fcfb9533d6fe2bfa24\": rpc error: code = NotFound desc = could not find container \"0bfcb7cc04fb4e713c7694bb42e7038554ba5b1b2dbee8fcfb9533d6fe2bfa24\": container with ID starting with 0bfcb7cc04fb4e713c7694bb42e7038554ba5b1b2dbee8fcfb9533d6fe2bfa24 not found: ID does not exist" Jul 09 07:36:43.294455 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:43.294381 2564 scope.go:117] "RemoveContainer" containerID="9dbd512cc8a4d74400884ec3cc806f1f6b2cf4e3c2319ef31566024e11243a6d" Jul 09 07:36:43.294611 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:36:43.294596 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dbd512cc8a4d74400884ec3cc806f1f6b2cf4e3c2319ef31566024e11243a6d\": container with ID starting with 9dbd512cc8a4d74400884ec3cc806f1f6b2cf4e3c2319ef31566024e11243a6d not found: ID does not exist" containerID="9dbd512cc8a4d74400884ec3cc806f1f6b2cf4e3c2319ef31566024e11243a6d" Jul 09 07:36:43.294662 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:43.294617 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dbd512cc8a4d74400884ec3cc806f1f6b2cf4e3c2319ef31566024e11243a6d"} err="failed to get container status \"9dbd512cc8a4d74400884ec3cc806f1f6b2cf4e3c2319ef31566024e11243a6d\": rpc error: code = NotFound desc = could not find container \"9dbd512cc8a4d74400884ec3cc806f1f6b2cf4e3c2319ef31566024e11243a6d\": container with ID starting with 9dbd512cc8a4d74400884ec3cc806f1f6b2cf4e3c2319ef31566024e11243a6d not found: ID does not exist" Jul 09 07:36:43.300144 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:43.300121 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk"] Jul 09 07:36:43.304884 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:43.304863 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-9c5220-openshift-default-7596d7c875-knzhk"] Jul 09 07:36:43.318977 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:43.318956 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2caf8729-d7e2-4eba-b289-b3c0364d25bf-istio-token\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:36:43.318977 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:43.318977 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2caf8729-d7e2-4eba-b289-b3c0364d25bf-workload-socket\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:36:43.319081 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:43.318986 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2caf8729-d7e2-4eba-b289-b3c0364d25bf-istio-envoy\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:36:43.319081 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:43.318994 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2g48\" (UniqueName: \"kubernetes.io/projected/2caf8729-d7e2-4eba-b289-b3c0364d25bf-kube-api-access-r2g48\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:36:43.319081 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:43.319004 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2caf8729-d7e2-4eba-b289-b3c0364d25bf-istiod-ca-cert\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:36:43.319081 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:43.319013 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2caf8729-d7e2-4eba-b289-b3c0364d25bf-workload-certs\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:36:43.319081 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:43.319020 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2caf8729-d7e2-4eba-b289-b3c0364d25bf-istio-data\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:36:43.319081 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:43.319028 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2caf8729-d7e2-4eba-b289-b3c0364d25bf-istio-podinfo\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:36:43.464029 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:43.463998 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2caf8729-d7e2-4eba-b289-b3c0364d25bf" path="/var/lib/kubelet/pods/2caf8729-d7e2-4eba-b289-b3c0364d25bf/volumes" Jul 09 07:36:45.059599 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:45.059568 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv"] Jul 09 07:36:45.059998 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:45.059770 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv" podUID="57973303-54ce-41d6-b0c1-57998b6c4feb" containerName="istio-proxy" containerID="cri-o://1d38a9fd6dbeea18b0cf2b02eb0c4a5974af29f7b938be72794e537ff8c32854" gracePeriod=30 Jul 09 07:36:45.931513 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:45.931470 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc"] Jul 09 07:36:45.932223 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:45.932198 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2caf8729-d7e2-4eba-b289-b3c0364d25bf" containerName="istio-proxy" Jul 09 07:36:45.932223 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:45.932225 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2caf8729-d7e2-4eba-b289-b3c0364d25bf" containerName="istio-proxy" Jul 09 07:36:45.932382 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:45.932245 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2caf8729-d7e2-4eba-b289-b3c0364d25bf" containerName="istio-proxy" Jul 09 07:36:45.932382 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:45.932255 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2caf8729-d7e2-4eba-b289-b3c0364d25bf" containerName="istio-proxy" Jul 09 07:36:45.932382 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:45.932300 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e66ae8a5-f4d2-4b2c-aba4-dd85508b0775" containerName="main" Jul 09 07:36:45.932382 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:45.932309 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e66ae8a5-f4d2-4b2c-aba4-dd85508b0775" containerName="main" Jul 09 07:36:45.932513 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:45.932389 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e66ae8a5-f4d2-4b2c-aba4-dd85508b0775" containerName="main" Jul 09 07:36:45.932513 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:45.932401 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2caf8729-d7e2-4eba-b289-b3c0364d25bf" containerName="istio-proxy" Jul 09 07:36:45.935633 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:45.935616 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc" Jul 09 07:36:45.938069 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:45.938050 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-42f976-openshift-default-dockercfg-4bh48\"" Jul 09 07:36:45.947494 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:45.947472 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc"] Jul 09 07:36:46.042458 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:46.042417 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf6f503f-48f5-4993-b6f6-e53bd6b11aac-istio-data\") pod \"e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc\" (UID: \"cf6f503f-48f5-4993-b6f6-e53bd6b11aac\") " pod="openshift-ingress/e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc" Jul 09 07:36:46.042646 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:46.042481 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6f503f-48f5-4993-b6f6-e53bd6b11aac-credential-socket\") pod \"e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc\" (UID: \"cf6f503f-48f5-4993-b6f6-e53bd6b11aac\") " pod="openshift-ingress/e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc" Jul 09 07:36:46.042646 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:46.042515 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf6f503f-48f5-4993-b6f6-e53bd6b11aac-istiod-ca-cert\") pod \"e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc\" (UID: \"cf6f503f-48f5-4993-b6f6-e53bd6b11aac\") " pod="openshift-ingress/e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc" Jul 09 07:36:46.042646 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:46.042542 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf6f503f-48f5-4993-b6f6-e53bd6b11aac-istio-token\") pod \"e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc\" (UID: \"cf6f503f-48f5-4993-b6f6-e53bd6b11aac\") " pod="openshift-ingress/e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc" Jul 09 07:36:46.042646 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:46.042569 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6f503f-48f5-4993-b6f6-e53bd6b11aac-workload-socket\") pod \"e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc\" (UID: \"cf6f503f-48f5-4993-b6f6-e53bd6b11aac\") " pod="openshift-ingress/e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc" Jul 09 07:36:46.042646 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:46.042607 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf6f503f-48f5-4993-b6f6-e53bd6b11aac-workload-certs\") pod \"e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc\" (UID: \"cf6f503f-48f5-4993-b6f6-e53bd6b11aac\") " pod="openshift-ingress/e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc" Jul 09 07:36:46.042646 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:46.042641 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf6f503f-48f5-4993-b6f6-e53bd6b11aac-istio-envoy\") pod \"e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc\" (UID: \"cf6f503f-48f5-4993-b6f6-e53bd6b11aac\") " pod="openshift-ingress/e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc" Jul 09 07:36:46.042950 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:46.042663 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7gn4\" (UniqueName: \"kubernetes.io/projected/cf6f503f-48f5-4993-b6f6-e53bd6b11aac-kube-api-access-q7gn4\") pod \"e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc\" (UID: \"cf6f503f-48f5-4993-b6f6-e53bd6b11aac\") " pod="openshift-ingress/e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc" Jul 09 07:36:46.042950 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:46.042697 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf6f503f-48f5-4993-b6f6-e53bd6b11aac-istio-podinfo\") pod \"e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc\" (UID: \"cf6f503f-48f5-4993-b6f6-e53bd6b11aac\") " pod="openshift-ingress/e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc" Jul 09 07:36:46.143463 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:46.143424 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf6f503f-48f5-4993-b6f6-e53bd6b11aac-istio-envoy\") pod \"e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc\" (UID: \"cf6f503f-48f5-4993-b6f6-e53bd6b11aac\") " pod="openshift-ingress/e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc" Jul 09 07:36:46.143463 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:46.143466 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7gn4\" (UniqueName: \"kubernetes.io/projected/cf6f503f-48f5-4993-b6f6-e53bd6b11aac-kube-api-access-q7gn4\") pod \"e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc\" (UID: \"cf6f503f-48f5-4993-b6f6-e53bd6b11aac\") " pod="openshift-ingress/e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc" Jul 09 07:36:46.143909 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:46.143594 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf6f503f-48f5-4993-b6f6-e53bd6b11aac-istio-podinfo\") pod \"e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc\" (UID: \"cf6f503f-48f5-4993-b6f6-e53bd6b11aac\") " pod="openshift-ingress/e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc" Jul 09 07:36:46.143909 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:46.143703 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf6f503f-48f5-4993-b6f6-e53bd6b11aac-istio-data\") pod \"e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc\" (UID: \"cf6f503f-48f5-4993-b6f6-e53bd6b11aac\") " pod="openshift-ingress/e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc" Jul 09 07:36:46.143909 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:46.143749 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6f503f-48f5-4993-b6f6-e53bd6b11aac-credential-socket\") pod \"e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc\" (UID: \"cf6f503f-48f5-4993-b6f6-e53bd6b11aac\") " pod="openshift-ingress/e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc" Jul 09 07:36:46.143909 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:46.143781 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf6f503f-48f5-4993-b6f6-e53bd6b11aac-istiod-ca-cert\") pod \"e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc\" (UID: \"cf6f503f-48f5-4993-b6f6-e53bd6b11aac\") " pod="openshift-ingress/e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc" Jul 09 07:36:46.143909 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:46.143807 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf6f503f-48f5-4993-b6f6-e53bd6b11aac-istio-token\") pod \"e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc\" (UID: \"cf6f503f-48f5-4993-b6f6-e53bd6b11aac\") " pod="openshift-ingress/e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc" Jul 09 07:36:46.143909 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:46.143834 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6f503f-48f5-4993-b6f6-e53bd6b11aac-workload-socket\") pod \"e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc\" (UID: \"cf6f503f-48f5-4993-b6f6-e53bd6b11aac\") " pod="openshift-ingress/e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc" Jul 09 07:36:46.143909 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:46.143873 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf6f503f-48f5-4993-b6f6-e53bd6b11aac-workload-certs\") pod \"e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc\" (UID: \"cf6f503f-48f5-4993-b6f6-e53bd6b11aac\") " pod="openshift-ingress/e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc" Jul 09 07:36:46.144321 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:46.144096 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6f503f-48f5-4993-b6f6-e53bd6b11aac-credential-socket\") pod \"e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc\" (UID: \"cf6f503f-48f5-4993-b6f6-e53bd6b11aac\") " pod="openshift-ingress/e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc" Jul 09 07:36:46.144321 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:46.144107 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf6f503f-48f5-4993-b6f6-e53bd6b11aac-istio-data\") pod \"e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc\" (UID: \"cf6f503f-48f5-4993-b6f6-e53bd6b11aac\") " pod="openshift-ingress/e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc" Jul 09 07:36:46.144321 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:46.144160 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf6f503f-48f5-4993-b6f6-e53bd6b11aac-workload-certs\") pod \"e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc\" (UID: \"cf6f503f-48f5-4993-b6f6-e53bd6b11aac\") " pod="openshift-ingress/e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc" Jul 09 07:36:46.144490 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:46.144410 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6f503f-48f5-4993-b6f6-e53bd6b11aac-workload-socket\") pod \"e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc\" (UID: \"cf6f503f-48f5-4993-b6f6-e53bd6b11aac\") " pod="openshift-ingress/e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc" Jul 09 07:36:46.144946 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:46.144920 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf6f503f-48f5-4993-b6f6-e53bd6b11aac-istiod-ca-cert\") pod \"e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc\" (UID: \"cf6f503f-48f5-4993-b6f6-e53bd6b11aac\") " pod="openshift-ingress/e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc" Jul 09 07:36:46.145772 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:46.145746 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf6f503f-48f5-4993-b6f6-e53bd6b11aac-istio-envoy\") pod \"e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc\" (UID: \"cf6f503f-48f5-4993-b6f6-e53bd6b11aac\") " pod="openshift-ingress/e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc" Jul 09 07:36:46.146005 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:46.145987 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf6f503f-48f5-4993-b6f6-e53bd6b11aac-istio-podinfo\") pod \"e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc\" (UID: \"cf6f503f-48f5-4993-b6f6-e53bd6b11aac\") " pod="openshift-ingress/e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc" Jul 09 07:36:46.151640 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:46.151620 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7gn4\" (UniqueName: \"kubernetes.io/projected/cf6f503f-48f5-4993-b6f6-e53bd6b11aac-kube-api-access-q7gn4\") pod \"e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc\" (UID: \"cf6f503f-48f5-4993-b6f6-e53bd6b11aac\") " pod="openshift-ingress/e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc" Jul 09 07:36:46.151788 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:46.151771 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf6f503f-48f5-4993-b6f6-e53bd6b11aac-istio-token\") pod \"e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc\" (UID: \"cf6f503f-48f5-4993-b6f6-e53bd6b11aac\") " pod="openshift-ingress/e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc" Jul 09 07:36:46.247362 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:46.247290 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc" Jul 09 07:36:46.386810 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:46.386783 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc"] Jul 09 07:36:46.389446 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:36:46.389419 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf6f503f_48f5_4993_b6f6_e53bd6b11aac.slice/crio-3a229c8f7494d55dd3c1ea9ea2c02c89fa0ff889f31d5e9cbd0e91da717d6737 WatchSource:0}: Error finding container 3a229c8f7494d55dd3c1ea9ea2c02c89fa0ff889f31d5e9cbd0e91da717d6737: Status 404 returned error can't find the container with id 3a229c8f7494d55dd3c1ea9ea2c02c89fa0ff889f31d5e9cbd0e91da717d6737 Jul 09 07:36:46.391739 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:46.391706 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 07:36:46.391850 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:46.391768 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 07:36:46.391850 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:46.391797 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 07:36:47.169635 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:47.169602 2564 patch_prober.go:28] interesting pod/e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 09 07:36:47.169981 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:47.169657 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv" podUID="57973303-54ce-41d6-b0c1-57998b6c4feb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 07:36:47.299943 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:47.299905 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc" event={"ID":"cf6f503f-48f5-4993-b6f6-e53bd6b11aac","Type":"ContainerStarted","Data":"8653b6b8eb20c21bd77a246170dd892581a2c76a57d9c6a98ffabe3b019a4d51"} Jul 09 07:36:47.300139 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:47.299963 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc" event={"ID":"cf6f503f-48f5-4993-b6f6-e53bd6b11aac","Type":"ContainerStarted","Data":"3a229c8f7494d55dd3c1ea9ea2c02c89fa0ff889f31d5e9cbd0e91da717d6737"} Jul 09 07:36:47.320858 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:47.320786 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc" podStartSLOduration=2.320765684 podStartE2EDuration="2.320765684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 07:36:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 07:36:47.31991822 +0000 UTC m=+2066.370995445" watchObservedRunningTime="2026-07-09 07:36:47.320765684 +0000 UTC m=+2066.371842914" Jul 09 07:36:48.247470 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:48.247427 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc" Jul 09 07:36:48.252206 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:48.252181 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc" Jul 09 07:36:48.305120 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:48.305086 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc" Jul 09 07:36:48.305879 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:48.305862 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc" Jul 09 07:36:50.299520 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:50.299494 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv" Jul 09 07:36:50.315716 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:50.315676 2564 generic.go:356] "Generic (PLEG): container finished" podID="57973303-54ce-41d6-b0c1-57998b6c4feb" containerID="1d38a9fd6dbeea18b0cf2b02eb0c4a5974af29f7b938be72794e537ff8c32854" exitCode=0 Jul 09 07:36:50.315853 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:50.315723 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv" Jul 09 07:36:50.315853 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:50.315726 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv" event={"ID":"57973303-54ce-41d6-b0c1-57998b6c4feb","Type":"ContainerDied","Data":"1d38a9fd6dbeea18b0cf2b02eb0c4a5974af29f7b938be72794e537ff8c32854"} Jul 09 07:36:50.315853 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:50.315805 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv" event={"ID":"57973303-54ce-41d6-b0c1-57998b6c4feb","Type":"ContainerDied","Data":"2b87f11b4e8238ad5d729d659833388a2ef79fa04053e8827b5cf49da988725c"} Jul 09 07:36:50.315853 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:50.315830 2564 scope.go:117] "RemoveContainer" containerID="1d38a9fd6dbeea18b0cf2b02eb0c4a5974af29f7b938be72794e537ff8c32854" Jul 09 07:36:50.325531 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:50.325509 2564 scope.go:117] "RemoveContainer" containerID="cd63b5848aab2ef91c07caea95eec0c09acf1b8209b43b68c5136b165c07f627" Jul 09 07:36:50.336420 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:50.336399 2564 scope.go:117] "RemoveContainer" containerID="1d38a9fd6dbeea18b0cf2b02eb0c4a5974af29f7b938be72794e537ff8c32854" Jul 09 07:36:50.336715 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:36:50.336694 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d38a9fd6dbeea18b0cf2b02eb0c4a5974af29f7b938be72794e537ff8c32854\": container with ID starting with 1d38a9fd6dbeea18b0cf2b02eb0c4a5974af29f7b938be72794e537ff8c32854 not found: ID does not exist" containerID="1d38a9fd6dbeea18b0cf2b02eb0c4a5974af29f7b938be72794e537ff8c32854" Jul 09 07:36:50.336771 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:50.336724 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d38a9fd6dbeea18b0cf2b02eb0c4a5974af29f7b938be72794e537ff8c32854"} err="failed to get container status \"1d38a9fd6dbeea18b0cf2b02eb0c4a5974af29f7b938be72794e537ff8c32854\": rpc error: code = NotFound desc = could not find container \"1d38a9fd6dbeea18b0cf2b02eb0c4a5974af29f7b938be72794e537ff8c32854\": container with ID starting with 1d38a9fd6dbeea18b0cf2b02eb0c4a5974af29f7b938be72794e537ff8c32854 not found: ID does not exist" Jul 09 07:36:50.336771 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:50.336742 2564 scope.go:117] "RemoveContainer" containerID="cd63b5848aab2ef91c07caea95eec0c09acf1b8209b43b68c5136b165c07f627" Jul 09 07:36:50.336994 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:36:50.336981 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd63b5848aab2ef91c07caea95eec0c09acf1b8209b43b68c5136b165c07f627\": container with ID starting with cd63b5848aab2ef91c07caea95eec0c09acf1b8209b43b68c5136b165c07f627 not found: ID does not exist" containerID="cd63b5848aab2ef91c07caea95eec0c09acf1b8209b43b68c5136b165c07f627" Jul 09 07:36:50.337039 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:50.336999 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd63b5848aab2ef91c07caea95eec0c09acf1b8209b43b68c5136b165c07f627"} err="failed to get container status \"cd63b5848aab2ef91c07caea95eec0c09acf1b8209b43b68c5136b165c07f627\": rpc error: code = NotFound desc = could not find container \"cd63b5848aab2ef91c07caea95eec0c09acf1b8209b43b68c5136b165c07f627\": container with ID starting with cd63b5848aab2ef91c07caea95eec0c09acf1b8209b43b68c5136b165c07f627 not found: ID does not exist" Jul 09 07:36:50.380038 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:50.380014 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57973303-54ce-41d6-b0c1-57998b6c4feb-credential-socket\") pod \"57973303-54ce-41d6-b0c1-57998b6c4feb\" (UID: \"57973303-54ce-41d6-b0c1-57998b6c4feb\") " Jul 09 07:36:50.380174 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:50.380043 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57973303-54ce-41d6-b0c1-57998b6c4feb-istiod-ca-cert\") pod \"57973303-54ce-41d6-b0c1-57998b6c4feb\" (UID: \"57973303-54ce-41d6-b0c1-57998b6c4feb\") " Jul 09 07:36:50.380174 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:50.380065 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57973303-54ce-41d6-b0c1-57998b6c4feb-istio-envoy\") pod \"57973303-54ce-41d6-b0c1-57998b6c4feb\" (UID: \"57973303-54ce-41d6-b0c1-57998b6c4feb\") " Jul 09 07:36:50.380174 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:50.380084 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57973303-54ce-41d6-b0c1-57998b6c4feb-workload-socket\") pod \"57973303-54ce-41d6-b0c1-57998b6c4feb\" (UID: \"57973303-54ce-41d6-b0c1-57998b6c4feb\") " Jul 09 07:36:50.380174 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:50.380106 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57973303-54ce-41d6-b0c1-57998b6c4feb-istio-podinfo\") pod \"57973303-54ce-41d6-b0c1-57998b6c4feb\" (UID: \"57973303-54ce-41d6-b0c1-57998b6c4feb\") " Jul 09 07:36:50.380174 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:50.380129 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57973303-54ce-41d6-b0c1-57998b6c4feb-istio-data\") pod \"57973303-54ce-41d6-b0c1-57998b6c4feb\" (UID: \"57973303-54ce-41d6-b0c1-57998b6c4feb\") " Jul 09 07:36:50.380174 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:50.380149 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57973303-54ce-41d6-b0c1-57998b6c4feb-istio-token\") pod \"57973303-54ce-41d6-b0c1-57998b6c4feb\" (UID: \"57973303-54ce-41d6-b0c1-57998b6c4feb\") " Jul 09 07:36:50.380540 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:50.380197 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7spsf\" (UniqueName: \"kubernetes.io/projected/57973303-54ce-41d6-b0c1-57998b6c4feb-kube-api-access-7spsf\") pod \"57973303-54ce-41d6-b0c1-57998b6c4feb\" (UID: \"57973303-54ce-41d6-b0c1-57998b6c4feb\") " Jul 09 07:36:50.380540 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:50.380302 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57973303-54ce-41d6-b0c1-57998b6c4feb-workload-certs\") pod \"57973303-54ce-41d6-b0c1-57998b6c4feb\" (UID: \"57973303-54ce-41d6-b0c1-57998b6c4feb\") " Jul 09 07:36:50.380540 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:50.380352 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57973303-54ce-41d6-b0c1-57998b6c4feb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "57973303-54ce-41d6-b0c1-57998b6c4feb" (UID: "57973303-54ce-41d6-b0c1-57998b6c4feb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 07:36:50.380540 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:50.380467 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57973303-54ce-41d6-b0c1-57998b6c4feb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "57973303-54ce-41d6-b0c1-57998b6c4feb" (UID: "57973303-54ce-41d6-b0c1-57998b6c4feb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 07:36:50.380788 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:50.380584 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57973303-54ce-41d6-b0c1-57998b6c4feb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "57973303-54ce-41d6-b0c1-57998b6c4feb" (UID: "57973303-54ce-41d6-b0c1-57998b6c4feb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 07:36:50.380788 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:50.380670 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57973303-54ce-41d6-b0c1-57998b6c4feb-credential-socket\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:36:50.380788 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:50.380689 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57973303-54ce-41d6-b0c1-57998b6c4feb-istiod-ca-cert\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:36:50.380788 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:50.380706 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57973303-54ce-41d6-b0c1-57998b6c4feb-workload-socket\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:36:50.380988 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:50.380915 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57973303-54ce-41d6-b0c1-57998b6c4feb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "57973303-54ce-41d6-b0c1-57998b6c4feb" (UID: "57973303-54ce-41d6-b0c1-57998b6c4feb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 07:36:50.381342 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:50.381319 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57973303-54ce-41d6-b0c1-57998b6c4feb-istio-data" (OuterVolumeSpecName: "istio-data") pod "57973303-54ce-41d6-b0c1-57998b6c4feb" (UID: "57973303-54ce-41d6-b0c1-57998b6c4feb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 07:36:50.382414 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:50.382390 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57973303-54ce-41d6-b0c1-57998b6c4feb-istio-token" (OuterVolumeSpecName: "istio-token") pod "57973303-54ce-41d6-b0c1-57998b6c4feb" (UID: "57973303-54ce-41d6-b0c1-57998b6c4feb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 07:36:50.382610 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:50.382587 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57973303-54ce-41d6-b0c1-57998b6c4feb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "57973303-54ce-41d6-b0c1-57998b6c4feb" (UID: "57973303-54ce-41d6-b0c1-57998b6c4feb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 07:36:50.382764 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:50.382740 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57973303-54ce-41d6-b0c1-57998b6c4feb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "57973303-54ce-41d6-b0c1-57998b6c4feb" (UID: "57973303-54ce-41d6-b0c1-57998b6c4feb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 07:36:50.382764 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:50.382739 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57973303-54ce-41d6-b0c1-57998b6c4feb-kube-api-access-7spsf" (OuterVolumeSpecName: "kube-api-access-7spsf") pod "57973303-54ce-41d6-b0c1-57998b6c4feb" (UID: "57973303-54ce-41d6-b0c1-57998b6c4feb"). InnerVolumeSpecName "kube-api-access-7spsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 07:36:50.481357 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:50.481329 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57973303-54ce-41d6-b0c1-57998b6c4feb-workload-certs\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:36:50.481357 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:50.481352 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57973303-54ce-41d6-b0c1-57998b6c4feb-istio-envoy\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:36:50.481357 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:50.481361 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57973303-54ce-41d6-b0c1-57998b6c4feb-istio-podinfo\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:36:50.481569 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:50.481369 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57973303-54ce-41d6-b0c1-57998b6c4feb-istio-data\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:36:50.481569 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:50.481378 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57973303-54ce-41d6-b0c1-57998b6c4feb-istio-token\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:36:50.481569 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:50.481386 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7spsf\" (UniqueName: \"kubernetes.io/projected/57973303-54ce-41d6-b0c1-57998b6c4feb-kube-api-access-7spsf\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:36:50.641281 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:50.641239 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv"] Jul 09 07:36:50.657167 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:50.657139 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-055963-openshift-default-769689dcc9-5fxrv"] Jul 09 07:36:51.462518 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:51.462488 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57973303-54ce-41d6-b0c1-57998b6c4feb" path="/var/lib/kubelet/pods/57973303-54ce-41d6-b0c1-57998b6c4feb/volumes" Jul 09 07:36:54.498995 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:54.498966 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-ab11a1-openshift-default-689f477559-6qznw"] Jul 09 07:36:54.499516 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:54.499491 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57973303-54ce-41d6-b0c1-57998b6c4feb" containerName="istio-proxy" Jul 09 07:36:54.499516 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:54.499515 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="57973303-54ce-41d6-b0c1-57998b6c4feb" containerName="istio-proxy" Jul 09 07:36:54.499676 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:54.499535 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57973303-54ce-41d6-b0c1-57998b6c4feb" containerName="istio-proxy" Jul 09 07:36:54.499676 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:54.499541 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="57973303-54ce-41d6-b0c1-57998b6c4feb" containerName="istio-proxy" Jul 09 07:36:54.499676 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:54.499604 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2caf8729-d7e2-4eba-b289-b3c0364d25bf" containerName="istio-proxy" Jul 09 07:36:54.499676 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:54.499613 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="57973303-54ce-41d6-b0c1-57998b6c4feb" containerName="istio-proxy" Jul 09 07:36:54.502455 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:54.502435 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ab11a1-openshift-default-689f477559-6qznw" Jul 09 07:36:54.504904 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:54.504885 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-ab11a1-openshift-default-dockercfg-6r28n\"" Jul 09 07:36:54.514716 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:54.514696 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ab11a1-openshift-default-689f477559-6qznw"] Jul 09 07:36:54.617154 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:54.617116 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af7b1769-b214-4224-8ac6-58d2933a113d-istio-token\") pod \"e2e-api-b-ab11a1-openshift-default-689f477559-6qznw\" (UID: \"af7b1769-b214-4224-8ac6-58d2933a113d\") " pod="openshift-ingress/e2e-api-b-ab11a1-openshift-default-689f477559-6qznw" Jul 09 07:36:54.617756 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:54.617604 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt6g8\" (UniqueName: \"kubernetes.io/projected/af7b1769-b214-4224-8ac6-58d2933a113d-kube-api-access-wt6g8\") pod \"e2e-api-b-ab11a1-openshift-default-689f477559-6qznw\" (UID: \"af7b1769-b214-4224-8ac6-58d2933a113d\") " pod="openshift-ingress/e2e-api-b-ab11a1-openshift-default-689f477559-6qznw" Jul 09 07:36:54.617756 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:54.617658 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af7b1769-b214-4224-8ac6-58d2933a113d-istio-envoy\") pod \"e2e-api-b-ab11a1-openshift-default-689f477559-6qznw\" (UID: \"af7b1769-b214-4224-8ac6-58d2933a113d\") " pod="openshift-ingress/e2e-api-b-ab11a1-openshift-default-689f477559-6qznw" Jul 09 07:36:54.617866 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:54.617758 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af7b1769-b214-4224-8ac6-58d2933a113d-istio-podinfo\") pod \"e2e-api-b-ab11a1-openshift-default-689f477559-6qznw\" (UID: \"af7b1769-b214-4224-8ac6-58d2933a113d\") " pod="openshift-ingress/e2e-api-b-ab11a1-openshift-default-689f477559-6qznw" Jul 09 07:36:54.617866 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:54.617785 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af7b1769-b214-4224-8ac6-58d2933a113d-workload-certs\") pod \"e2e-api-b-ab11a1-openshift-default-689f477559-6qznw\" (UID: \"af7b1769-b214-4224-8ac6-58d2933a113d\") " pod="openshift-ingress/e2e-api-b-ab11a1-openshift-default-689f477559-6qznw" Jul 09 07:36:54.617866 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:54.617832 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af7b1769-b214-4224-8ac6-58d2933a113d-istiod-ca-cert\") pod \"e2e-api-b-ab11a1-openshift-default-689f477559-6qznw\" (UID: \"af7b1769-b214-4224-8ac6-58d2933a113d\") " pod="openshift-ingress/e2e-api-b-ab11a1-openshift-default-689f477559-6qznw" Jul 09 07:36:54.617973 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:54.617869 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af7b1769-b214-4224-8ac6-58d2933a113d-workload-socket\") pod \"e2e-api-b-ab11a1-openshift-default-689f477559-6qznw\" (UID: \"af7b1769-b214-4224-8ac6-58d2933a113d\") " pod="openshift-ingress/e2e-api-b-ab11a1-openshift-default-689f477559-6qznw" Jul 09 07:36:54.617973 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:54.617919 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af7b1769-b214-4224-8ac6-58d2933a113d-istio-data\") pod \"e2e-api-b-ab11a1-openshift-default-689f477559-6qznw\" (UID: \"af7b1769-b214-4224-8ac6-58d2933a113d\") " pod="openshift-ingress/e2e-api-b-ab11a1-openshift-default-689f477559-6qznw" Jul 09 07:36:54.617973 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:54.617939 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af7b1769-b214-4224-8ac6-58d2933a113d-credential-socket\") pod \"e2e-api-b-ab11a1-openshift-default-689f477559-6qznw\" (UID: \"af7b1769-b214-4224-8ac6-58d2933a113d\") " pod="openshift-ingress/e2e-api-b-ab11a1-openshift-default-689f477559-6qznw" Jul 09 07:36:54.718365 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:54.718329 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af7b1769-b214-4224-8ac6-58d2933a113d-istio-data\") pod \"e2e-api-b-ab11a1-openshift-default-689f477559-6qznw\" (UID: \"af7b1769-b214-4224-8ac6-58d2933a113d\") " pod="openshift-ingress/e2e-api-b-ab11a1-openshift-default-689f477559-6qznw" Jul 09 07:36:54.718543 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:54.718377 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af7b1769-b214-4224-8ac6-58d2933a113d-credential-socket\") pod \"e2e-api-b-ab11a1-openshift-default-689f477559-6qznw\" (UID: \"af7b1769-b214-4224-8ac6-58d2933a113d\") " pod="openshift-ingress/e2e-api-b-ab11a1-openshift-default-689f477559-6qznw" Jul 09 07:36:54.718543 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:54.718397 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af7b1769-b214-4224-8ac6-58d2933a113d-istio-token\") pod \"e2e-api-b-ab11a1-openshift-default-689f477559-6qznw\" (UID: \"af7b1769-b214-4224-8ac6-58d2933a113d\") " pod="openshift-ingress/e2e-api-b-ab11a1-openshift-default-689f477559-6qznw" Jul 09 07:36:54.718543 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:54.718423 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt6g8\" (UniqueName: \"kubernetes.io/projected/af7b1769-b214-4224-8ac6-58d2933a113d-kube-api-access-wt6g8\") pod \"e2e-api-b-ab11a1-openshift-default-689f477559-6qznw\" (UID: \"af7b1769-b214-4224-8ac6-58d2933a113d\") " pod="openshift-ingress/e2e-api-b-ab11a1-openshift-default-689f477559-6qznw" Jul 09 07:36:54.718543 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:54.718459 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af7b1769-b214-4224-8ac6-58d2933a113d-istio-envoy\") pod \"e2e-api-b-ab11a1-openshift-default-689f477559-6qznw\" (UID: \"af7b1769-b214-4224-8ac6-58d2933a113d\") " pod="openshift-ingress/e2e-api-b-ab11a1-openshift-default-689f477559-6qznw" Jul 09 07:36:54.718543 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:54.718515 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af7b1769-b214-4224-8ac6-58d2933a113d-istio-podinfo\") pod \"e2e-api-b-ab11a1-openshift-default-689f477559-6qznw\" (UID: \"af7b1769-b214-4224-8ac6-58d2933a113d\") " pod="openshift-ingress/e2e-api-b-ab11a1-openshift-default-689f477559-6qznw" Jul 09 07:36:54.718543 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:54.718540 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af7b1769-b214-4224-8ac6-58d2933a113d-workload-certs\") pod \"e2e-api-b-ab11a1-openshift-default-689f477559-6qznw\" (UID: \"af7b1769-b214-4224-8ac6-58d2933a113d\") " pod="openshift-ingress/e2e-api-b-ab11a1-openshift-default-689f477559-6qznw" Jul 09 07:36:54.718842 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:54.718580 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af7b1769-b214-4224-8ac6-58d2933a113d-istiod-ca-cert\") pod \"e2e-api-b-ab11a1-openshift-default-689f477559-6qznw\" (UID: \"af7b1769-b214-4224-8ac6-58d2933a113d\") " pod="openshift-ingress/e2e-api-b-ab11a1-openshift-default-689f477559-6qznw" Jul 09 07:36:54.718842 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:54.718632 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af7b1769-b214-4224-8ac6-58d2933a113d-workload-socket\") pod \"e2e-api-b-ab11a1-openshift-default-689f477559-6qznw\" (UID: \"af7b1769-b214-4224-8ac6-58d2933a113d\") " pod="openshift-ingress/e2e-api-b-ab11a1-openshift-default-689f477559-6qznw" Jul 09 07:36:54.718842 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:54.718699 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af7b1769-b214-4224-8ac6-58d2933a113d-istio-data\") pod \"e2e-api-b-ab11a1-openshift-default-689f477559-6qznw\" (UID: \"af7b1769-b214-4224-8ac6-58d2933a113d\") " pod="openshift-ingress/e2e-api-b-ab11a1-openshift-default-689f477559-6qznw" Jul 09 07:36:54.719026 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:54.719006 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af7b1769-b214-4224-8ac6-58d2933a113d-workload-certs\") pod \"e2e-api-b-ab11a1-openshift-default-689f477559-6qznw\" (UID: \"af7b1769-b214-4224-8ac6-58d2933a113d\") " pod="openshift-ingress/e2e-api-b-ab11a1-openshift-default-689f477559-6qznw" Jul 09 07:36:54.719083 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:54.719012 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af7b1769-b214-4224-8ac6-58d2933a113d-workload-socket\") pod \"e2e-api-b-ab11a1-openshift-default-689f477559-6qznw\" (UID: \"af7b1769-b214-4224-8ac6-58d2933a113d\") " pod="openshift-ingress/e2e-api-b-ab11a1-openshift-default-689f477559-6qznw" Jul 09 07:36:54.719123 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:54.719083 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af7b1769-b214-4224-8ac6-58d2933a113d-credential-socket\") pod \"e2e-api-b-ab11a1-openshift-default-689f477559-6qznw\" (UID: \"af7b1769-b214-4224-8ac6-58d2933a113d\") " pod="openshift-ingress/e2e-api-b-ab11a1-openshift-default-689f477559-6qznw" Jul 09 07:36:54.719386 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:54.719369 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af7b1769-b214-4224-8ac6-58d2933a113d-istiod-ca-cert\") pod \"e2e-api-b-ab11a1-openshift-default-689f477559-6qznw\" (UID: \"af7b1769-b214-4224-8ac6-58d2933a113d\") " pod="openshift-ingress/e2e-api-b-ab11a1-openshift-default-689f477559-6qznw" Jul 09 07:36:54.720974 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:54.720953 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af7b1769-b214-4224-8ac6-58d2933a113d-istio-podinfo\") pod \"e2e-api-b-ab11a1-openshift-default-689f477559-6qznw\" (UID: \"af7b1769-b214-4224-8ac6-58d2933a113d\") " pod="openshift-ingress/e2e-api-b-ab11a1-openshift-default-689f477559-6qznw" Jul 09 07:36:54.721099 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:54.721082 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af7b1769-b214-4224-8ac6-58d2933a113d-istio-envoy\") pod \"e2e-api-b-ab11a1-openshift-default-689f477559-6qznw\" (UID: \"af7b1769-b214-4224-8ac6-58d2933a113d\") " pod="openshift-ingress/e2e-api-b-ab11a1-openshift-default-689f477559-6qznw" Jul 09 07:36:54.732716 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:54.732693 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af7b1769-b214-4224-8ac6-58d2933a113d-istio-token\") pod \"e2e-api-b-ab11a1-openshift-default-689f477559-6qznw\" (UID: \"af7b1769-b214-4224-8ac6-58d2933a113d\") " pod="openshift-ingress/e2e-api-b-ab11a1-openshift-default-689f477559-6qznw" Jul 09 07:36:54.734707 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:54.734679 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt6g8\" (UniqueName: \"kubernetes.io/projected/af7b1769-b214-4224-8ac6-58d2933a113d-kube-api-access-wt6g8\") pod \"e2e-api-b-ab11a1-openshift-default-689f477559-6qznw\" (UID: \"af7b1769-b214-4224-8ac6-58d2933a113d\") " pod="openshift-ingress/e2e-api-b-ab11a1-openshift-default-689f477559-6qznw" Jul 09 07:36:54.814025 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:54.813945 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ab11a1-openshift-default-689f477559-6qznw" Jul 09 07:36:54.953011 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:36:54.952976 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf7b1769_b214_4224_8ac6_58d2933a113d.slice/crio-5def9c0bf8f433a090e82039ab0c9aa3cca611baf604fe48f111d2254f92d536 WatchSource:0}: Error finding container 5def9c0bf8f433a090e82039ab0c9aa3cca611baf604fe48f111d2254f92d536: Status 404 returned error can't find the container with id 5def9c0bf8f433a090e82039ab0c9aa3cca611baf604fe48f111d2254f92d536 Jul 09 07:36:54.953303 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:54.953281 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ab11a1-openshift-default-689f477559-6qznw"] Jul 09 07:36:54.954914 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:54.954881 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 07:36:54.954988 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:54.954945 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 07:36:54.954988 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:54.954974 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 07:36:55.345243 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:55.345208 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ab11a1-openshift-default-689f477559-6qznw" event={"ID":"af7b1769-b214-4224-8ac6-58d2933a113d","Type":"ContainerStarted","Data":"c5ef774845dab1ff88781ae3b8700db4d986cbfcb92cd96250284880e57ece8d"} Jul 09 07:36:55.345471 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:55.345319 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ab11a1-openshift-default-689f477559-6qznw" event={"ID":"af7b1769-b214-4224-8ac6-58d2933a113d","Type":"ContainerStarted","Data":"5def9c0bf8f433a090e82039ab0c9aa3cca611baf604fe48f111d2254f92d536"} Jul 09 07:36:55.387997 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:55.387951 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-ab11a1-openshift-default-689f477559-6qznw" podStartSLOduration=1.387938312 podStartE2EDuration="1.387938312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 07:36:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 07:36:55.387090536 +0000 UTC m=+2074.438167764" watchObservedRunningTime="2026-07-09 07:36:55.387938312 +0000 UTC m=+2074.439015540" Jul 09 07:36:55.814746 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:55.814712 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-ab11a1-openshift-default-689f477559-6qznw" Jul 09 07:36:56.819514 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:56.819486 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-ab11a1-openshift-default-689f477559-6qznw" Jul 09 07:36:57.354941 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:57.354908 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-ab11a1-openshift-default-689f477559-6qznw" Jul 09 07:36:57.355962 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:36:57.355938 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-ab11a1-openshift-default-689f477559-6qznw" Jul 09 07:37:04.347394 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:04.347357 2564 patch_prober.go:28] interesting pod/e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 09 07:37:04.347777 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:04.347416 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc" podUID="cf6f503f-48f5-4993-b6f6-e53bd6b11aac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 09 07:37:05.394118 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:05.394085 2564 generic.go:356] "Generic (PLEG): container finished" podID="cf6f503f-48f5-4993-b6f6-e53bd6b11aac" containerID="8653b6b8eb20c21bd77a246170dd892581a2c76a57d9c6a98ffabe3b019a4d51" exitCode=0 Jul 09 07:37:05.394563 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:05.394163 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc" event={"ID":"cf6f503f-48f5-4993-b6f6-e53bd6b11aac","Type":"ContainerDied","Data":"8653b6b8eb20c21bd77a246170dd892581a2c76a57d9c6a98ffabe3b019a4d51"} Jul 09 07:37:05.394662 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:05.394643 2564 scope.go:117] "RemoveContainer" containerID="8653b6b8eb20c21bd77a246170dd892581a2c76a57d9c6a98ffabe3b019a4d51" Jul 09 07:37:05.395683 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:05.395656 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 07:37:05.395763 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:05.395731 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 07:37:05.395800 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:05.395762 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 07:37:06.401127 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:06.401093 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc" event={"ID":"cf6f503f-48f5-4993-b6f6-e53bd6b11aac","Type":"ContainerStarted","Data":"ad5a9ab3cb6c83a8a701454dace3bb2ae289f210a9fe8212dbbcb7dbe84584ea"} Jul 09 07:37:07.248180 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:07.248134 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc" Jul 09 07:37:07.248418 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:07.248285 2564 patch_prober.go:28] interesting pod/e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Jul 09 07:37:07.248418 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:07.248343 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc" podUID="cf6f503f-48f5-4993-b6f6-e53bd6b11aac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Jul 09 07:37:08.247911 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:08.247865 2564 patch_prober.go:28] interesting pod/e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Jul 09 07:37:08.248432 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:08.247933 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc" podUID="cf6f503f-48f5-4993-b6f6-e53bd6b11aac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Jul 09 07:37:09.247785 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:09.247751 2564 patch_prober.go:28] interesting pod/e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Jul 09 07:37:09.248101 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:09.248057 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc" podUID="cf6f503f-48f5-4993-b6f6-e53bd6b11aac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Jul 09 07:37:10.248098 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:10.248056 2564 patch_prober.go:28] interesting pod/e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Jul 09 07:37:10.248726 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:10.248686 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc" podUID="cf6f503f-48f5-4993-b6f6-e53bd6b11aac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Jul 09 07:37:11.252164 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:11.252132 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc" Jul 09 07:37:11.252632 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:11.252490 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc" Jul 09 07:37:11.253255 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:11.253234 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc" Jul 09 07:37:13.052117 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:13.052084 2564 patch_prober.go:28] interesting pod/e2e-api-b-ab11a1-openshift-default-689f477559-6qznw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": read tcp 10.132.0.2:37278->10.132.0.43:15021: read: connection reset by peer" start-of-body= Jul 09 07:37:13.052550 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:13.052148 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-ab11a1-openshift-default-689f477559-6qznw" podUID="af7b1769-b214-4224-8ac6-58d2933a113d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": read tcp 10.132.0.2:37278->10.132.0.43:15021: read: connection reset by peer" Jul 09 07:37:13.434888 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:13.434857 2564 generic.go:356] "Generic (PLEG): container finished" podID="af7b1769-b214-4224-8ac6-58d2933a113d" containerID="c5ef774845dab1ff88781ae3b8700db4d986cbfcb92cd96250284880e57ece8d" exitCode=0 Jul 09 07:37:13.435056 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:13.434930 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ab11a1-openshift-default-689f477559-6qznw" event={"ID":"af7b1769-b214-4224-8ac6-58d2933a113d","Type":"ContainerDied","Data":"c5ef774845dab1ff88781ae3b8700db4d986cbfcb92cd96250284880e57ece8d"} Jul 09 07:37:13.435345 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:13.435332 2564 scope.go:117] "RemoveContainer" containerID="c5ef774845dab1ff88781ae3b8700db4d986cbfcb92cd96250284880e57ece8d" Jul 09 07:37:13.436358 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:13.436327 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 07:37:13.436470 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:13.436408 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 07:37:13.436470 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:13.436457 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 07:37:14.445848 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:14.445800 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ab11a1-openshift-default-689f477559-6qznw" event={"ID":"af7b1769-b214-4224-8ac6-58d2933a113d","Type":"ContainerStarted","Data":"2f8a32e96c3c03e8e363362ee241aba94a43ce8219d90ff69ee2f460b5b7eff7"} Jul 09 07:37:14.814318 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:14.814197 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-ab11a1-openshift-default-689f477559-6qznw" Jul 09 07:37:14.814500 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:14.814432 2564 patch_prober.go:28] interesting pod/e2e-api-b-ab11a1-openshift-default-689f477559-6qznw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 09 07:37:14.814570 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:14.814489 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ab11a1-openshift-default-689f477559-6qznw" podUID="af7b1769-b214-4224-8ac6-58d2933a113d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 09 07:37:15.815203 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:15.815169 2564 patch_prober.go:28] interesting pod/e2e-api-b-ab11a1-openshift-default-689f477559-6qznw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 09 07:37:15.815632 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:15.815221 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ab11a1-openshift-default-689f477559-6qznw" podUID="af7b1769-b214-4224-8ac6-58d2933a113d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 09 07:37:16.598964 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:16.598930 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc"] Jul 09 07:37:16.599175 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:16.599150 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc" podUID="cf6f503f-48f5-4993-b6f6-e53bd6b11aac" containerName="istio-proxy" containerID="cri-o://ad5a9ab3cb6c83a8a701454dace3bb2ae289f210a9fe8212dbbcb7dbe84584ea" gracePeriod=30 Jul 09 07:37:16.815025 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:16.814989 2564 patch_prober.go:28] interesting pod/e2e-api-b-ab11a1-openshift-default-689f477559-6qznw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 09 07:37:16.815202 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:16.815045 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ab11a1-openshift-default-689f477559-6qznw" podUID="af7b1769-b214-4224-8ac6-58d2933a113d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 09 07:37:17.814895 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:17.814858 2564 patch_prober.go:28] interesting pod/e2e-api-b-ab11a1-openshift-default-689f477559-6qznw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 09 07:37:17.815290 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:17.814913 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ab11a1-openshift-default-689f477559-6qznw" podUID="af7b1769-b214-4224-8ac6-58d2933a113d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 09 07:37:18.817491 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:18.817459 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-ab11a1-openshift-default-689f477559-6qznw" Jul 09 07:37:18.817960 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:18.817942 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-ab11a1-openshift-default-689f477559-6qznw" Jul 09 07:37:18.818712 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:18.818694 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-ab11a1-openshift-default-689f477559-6qznw" Jul 09 07:37:21.864460 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:21.864436 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc" Jul 09 07:37:21.995193 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:21.995163 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf6f503f-48f5-4993-b6f6-e53bd6b11aac-istio-envoy\") pod \"cf6f503f-48f5-4993-b6f6-e53bd6b11aac\" (UID: \"cf6f503f-48f5-4993-b6f6-e53bd6b11aac\") " Jul 09 07:37:21.995397 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:21.995228 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf6f503f-48f5-4993-b6f6-e53bd6b11aac-istio-data\") pod \"cf6f503f-48f5-4993-b6f6-e53bd6b11aac\" (UID: \"cf6f503f-48f5-4993-b6f6-e53bd6b11aac\") " Jul 09 07:37:21.995397 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:21.995243 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf6f503f-48f5-4993-b6f6-e53bd6b11aac-workload-certs\") pod \"cf6f503f-48f5-4993-b6f6-e53bd6b11aac\" (UID: \"cf6f503f-48f5-4993-b6f6-e53bd6b11aac\") " Jul 09 07:37:21.995397 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:21.995291 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf6f503f-48f5-4993-b6f6-e53bd6b11aac-istio-token\") pod \"cf6f503f-48f5-4993-b6f6-e53bd6b11aac\" (UID: \"cf6f503f-48f5-4993-b6f6-e53bd6b11aac\") " Jul 09 07:37:21.995397 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:21.995326 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf6f503f-48f5-4993-b6f6-e53bd6b11aac-istio-podinfo\") pod \"cf6f503f-48f5-4993-b6f6-e53bd6b11aac\" (UID: \"cf6f503f-48f5-4993-b6f6-e53bd6b11aac\") " Jul 09 07:37:21.995397 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:21.995364 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf6f503f-48f5-4993-b6f6-e53bd6b11aac-istiod-ca-cert\") pod \"cf6f503f-48f5-4993-b6f6-e53bd6b11aac\" (UID: \"cf6f503f-48f5-4993-b6f6-e53bd6b11aac\") " Jul 09 07:37:21.995397 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:21.995393 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7gn4\" (UniqueName: \"kubernetes.io/projected/cf6f503f-48f5-4993-b6f6-e53bd6b11aac-kube-api-access-q7gn4\") pod \"cf6f503f-48f5-4993-b6f6-e53bd6b11aac\" (UID: \"cf6f503f-48f5-4993-b6f6-e53bd6b11aac\") " Jul 09 07:37:21.995740 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:21.995439 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6f503f-48f5-4993-b6f6-e53bd6b11aac-workload-socket\") pod \"cf6f503f-48f5-4993-b6f6-e53bd6b11aac\" (UID: \"cf6f503f-48f5-4993-b6f6-e53bd6b11aac\") " Jul 09 07:37:21.995740 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:21.995590 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf6f503f-48f5-4993-b6f6-e53bd6b11aac-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cf6f503f-48f5-4993-b6f6-e53bd6b11aac" (UID: "cf6f503f-48f5-4993-b6f6-e53bd6b11aac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 07:37:21.995854 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:21.995797 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf6f503f-48f5-4993-b6f6-e53bd6b11aac-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cf6f503f-48f5-4993-b6f6-e53bd6b11aac" (UID: "cf6f503f-48f5-4993-b6f6-e53bd6b11aac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 07:37:21.995854 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:21.995563 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6f503f-48f5-4993-b6f6-e53bd6b11aac-credential-socket\") pod \"cf6f503f-48f5-4993-b6f6-e53bd6b11aac\" (UID: \"cf6f503f-48f5-4993-b6f6-e53bd6b11aac\") " Jul 09 07:37:21.995951 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:21.995855 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf6f503f-48f5-4993-b6f6-e53bd6b11aac-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cf6f503f-48f5-4993-b6f6-e53bd6b11aac" (UID: "cf6f503f-48f5-4993-b6f6-e53bd6b11aac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 07:37:21.996089 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:21.996069 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf6f503f-48f5-4993-b6f6-e53bd6b11aac-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cf6f503f-48f5-4993-b6f6-e53bd6b11aac" (UID: "cf6f503f-48f5-4993-b6f6-e53bd6b11aac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 07:37:21.996143 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:21.996084 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf6f503f-48f5-4993-b6f6-e53bd6b11aac-istiod-ca-cert\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:37:21.996143 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:21.996103 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6f503f-48f5-4993-b6f6-e53bd6b11aac-workload-socket\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:37:21.996143 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:21.996118 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf6f503f-48f5-4993-b6f6-e53bd6b11aac-workload-certs\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:37:21.996310 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:21.996242 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf6f503f-48f5-4993-b6f6-e53bd6b11aac-istio-data" (OuterVolumeSpecName: "istio-data") pod "cf6f503f-48f5-4993-b6f6-e53bd6b11aac" (UID: "cf6f503f-48f5-4993-b6f6-e53bd6b11aac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 07:37:21.997500 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:21.997480 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf6f503f-48f5-4993-b6f6-e53bd6b11aac-istio-token" (OuterVolumeSpecName: "istio-token") pod "cf6f503f-48f5-4993-b6f6-e53bd6b11aac" (UID: "cf6f503f-48f5-4993-b6f6-e53bd6b11aac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 07:37:21.997816 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:21.997788 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf6f503f-48f5-4993-b6f6-e53bd6b11aac-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cf6f503f-48f5-4993-b6f6-e53bd6b11aac" (UID: "cf6f503f-48f5-4993-b6f6-e53bd6b11aac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 07:37:21.997816 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:21.997791 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf6f503f-48f5-4993-b6f6-e53bd6b11aac-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cf6f503f-48f5-4993-b6f6-e53bd6b11aac" (UID: "cf6f503f-48f5-4993-b6f6-e53bd6b11aac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 07:37:21.997951 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:21.997812 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf6f503f-48f5-4993-b6f6-e53bd6b11aac-kube-api-access-q7gn4" (OuterVolumeSpecName: "kube-api-access-q7gn4") pod "cf6f503f-48f5-4993-b6f6-e53bd6b11aac" (UID: "cf6f503f-48f5-4993-b6f6-e53bd6b11aac"). InnerVolumeSpecName "kube-api-access-q7gn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 07:37:22.097191 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:22.097124 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6f503f-48f5-4993-b6f6-e53bd6b11aac-credential-socket\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:37:22.097191 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:22.097146 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf6f503f-48f5-4993-b6f6-e53bd6b11aac-istio-envoy\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:37:22.097191 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:22.097157 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf6f503f-48f5-4993-b6f6-e53bd6b11aac-istio-data\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:37:22.097191 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:22.097165 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf6f503f-48f5-4993-b6f6-e53bd6b11aac-istio-token\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:37:22.097191 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:22.097173 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf6f503f-48f5-4993-b6f6-e53bd6b11aac-istio-podinfo\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:37:22.097191 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:22.097181 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7gn4\" (UniqueName: \"kubernetes.io/projected/cf6f503f-48f5-4993-b6f6-e53bd6b11aac-kube-api-access-q7gn4\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:37:22.486333 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:22.486298 2564 generic.go:356] "Generic (PLEG): container finished" podID="cf6f503f-48f5-4993-b6f6-e53bd6b11aac" containerID="ad5a9ab3cb6c83a8a701454dace3bb2ae289f210a9fe8212dbbcb7dbe84584ea" exitCode=0 Jul 09 07:37:22.486536 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:22.486344 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc" Jul 09 07:37:22.486536 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:22.486349 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc" event={"ID":"cf6f503f-48f5-4993-b6f6-e53bd6b11aac","Type":"ContainerDied","Data":"ad5a9ab3cb6c83a8a701454dace3bb2ae289f210a9fe8212dbbcb7dbe84584ea"} Jul 09 07:37:22.486536 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:22.486398 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc" event={"ID":"cf6f503f-48f5-4993-b6f6-e53bd6b11aac","Type":"ContainerDied","Data":"3a229c8f7494d55dd3c1ea9ea2c02c89fa0ff889f31d5e9cbd0e91da717d6737"} Jul 09 07:37:22.486536 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:22.486427 2564 scope.go:117] "RemoveContainer" containerID="ad5a9ab3cb6c83a8a701454dace3bb2ae289f210a9fe8212dbbcb7dbe84584ea" Jul 09 07:37:22.496095 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:22.496077 2564 scope.go:117] "RemoveContainer" containerID="8653b6b8eb20c21bd77a246170dd892581a2c76a57d9c6a98ffabe3b019a4d51" Jul 09 07:37:22.505210 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:22.505193 2564 scope.go:117] "RemoveContainer" containerID="ad5a9ab3cb6c83a8a701454dace3bb2ae289f210a9fe8212dbbcb7dbe84584ea" Jul 09 07:37:22.505552 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:37:22.505535 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad5a9ab3cb6c83a8a701454dace3bb2ae289f210a9fe8212dbbcb7dbe84584ea\": container with ID starting with ad5a9ab3cb6c83a8a701454dace3bb2ae289f210a9fe8212dbbcb7dbe84584ea not found: ID does not exist" containerID="ad5a9ab3cb6c83a8a701454dace3bb2ae289f210a9fe8212dbbcb7dbe84584ea" Jul 09 07:37:22.505606 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:22.505559 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad5a9ab3cb6c83a8a701454dace3bb2ae289f210a9fe8212dbbcb7dbe84584ea"} err="failed to get container status \"ad5a9ab3cb6c83a8a701454dace3bb2ae289f210a9fe8212dbbcb7dbe84584ea\": rpc error: code = NotFound desc = could not find container \"ad5a9ab3cb6c83a8a701454dace3bb2ae289f210a9fe8212dbbcb7dbe84584ea\": container with ID starting with ad5a9ab3cb6c83a8a701454dace3bb2ae289f210a9fe8212dbbcb7dbe84584ea not found: ID does not exist" Jul 09 07:37:22.505606 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:22.505575 2564 scope.go:117] "RemoveContainer" containerID="8653b6b8eb20c21bd77a246170dd892581a2c76a57d9c6a98ffabe3b019a4d51" Jul 09 07:37:22.505783 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:37:22.505763 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8653b6b8eb20c21bd77a246170dd892581a2c76a57d9c6a98ffabe3b019a4d51\": container with ID starting with 8653b6b8eb20c21bd77a246170dd892581a2c76a57d9c6a98ffabe3b019a4d51 not found: ID does not exist" containerID="8653b6b8eb20c21bd77a246170dd892581a2c76a57d9c6a98ffabe3b019a4d51" Jul 09 07:37:22.505837 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:22.505789 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8653b6b8eb20c21bd77a246170dd892581a2c76a57d9c6a98ffabe3b019a4d51"} err="failed to get container status \"8653b6b8eb20c21bd77a246170dd892581a2c76a57d9c6a98ffabe3b019a4d51\": rpc error: code = NotFound desc = could not find container \"8653b6b8eb20c21bd77a246170dd892581a2c76a57d9c6a98ffabe3b019a4d51\": container with ID starting with 8653b6b8eb20c21bd77a246170dd892581a2c76a57d9c6a98ffabe3b019a4d51 not found: ID does not exist" Jul 09 07:37:22.508917 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:22.508895 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc"] Jul 09 07:37:22.513495 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:22.513476 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-42f976-openshift-default-7f6c47559f-bsjvc"] Jul 09 07:37:23.463899 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:23.463868 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf6f503f-48f5-4993-b6f6-e53bd6b11aac" path="/var/lib/kubelet/pods/cf6f503f-48f5-4993-b6f6-e53bd6b11aac/volumes" Jul 09 07:37:23.554909 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:23.554870 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ab11a1-openshift-default-689f477559-6qznw"] Jul 09 07:37:23.555118 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:23.555092 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-ab11a1-openshift-default-689f477559-6qznw" podUID="af7b1769-b214-4224-8ac6-58d2933a113d" containerName="istio-proxy" containerID="cri-o://2f8a32e96c3c03e8e363362ee241aba94a43ce8219d90ff69ee2f460b5b7eff7" gracePeriod=30 Jul 09 07:37:24.473891 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:24.473859 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx"] Jul 09 07:37:24.474250 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:24.474196 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf6f503f-48f5-4993-b6f6-e53bd6b11aac" containerName="istio-proxy" Jul 09 07:37:24.474250 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:24.474206 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf6f503f-48f5-4993-b6f6-e53bd6b11aac" containerName="istio-proxy" Jul 09 07:37:24.474250 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:24.474213 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf6f503f-48f5-4993-b6f6-e53bd6b11aac" containerName="istio-proxy" Jul 09 07:37:24.474250 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:24.474219 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf6f503f-48f5-4993-b6f6-e53bd6b11aac" containerName="istio-proxy" Jul 09 07:37:24.474410 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:24.474299 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf6f503f-48f5-4993-b6f6-e53bd6b11aac" containerName="istio-proxy" Jul 09 07:37:24.474410 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:24.474309 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="57973303-54ce-41d6-b0c1-57998b6c4feb" containerName="istio-proxy" Jul 09 07:37:24.477197 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:24.477180 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx" Jul 09 07:37:24.479909 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:24.479889 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-62f792-openshift-default-dockercfg-jh84v\"" Jul 09 07:37:24.490039 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:24.490016 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx"] Jul 09 07:37:24.520054 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:24.520029 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4470e3e5-d2c2-4821-b880-eac72b2bf5d1-istio-token\") pod \"e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx\" (UID: \"4470e3e5-d2c2-4821-b880-eac72b2bf5d1\") " pod="openshift-ingress/e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx" Jul 09 07:37:24.520251 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:24.520085 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4470e3e5-d2c2-4821-b880-eac72b2bf5d1-istio-podinfo\") pod \"e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx\" (UID: \"4470e3e5-d2c2-4821-b880-eac72b2bf5d1\") " pod="openshift-ingress/e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx" Jul 09 07:37:24.520251 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:24.520140 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4470e3e5-d2c2-4821-b880-eac72b2bf5d1-workload-socket\") pod \"e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx\" (UID: \"4470e3e5-d2c2-4821-b880-eac72b2bf5d1\") " pod="openshift-ingress/e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx" Jul 09 07:37:24.520251 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:24.520222 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4470e3e5-d2c2-4821-b880-eac72b2bf5d1-workload-certs\") pod \"e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx\" (UID: \"4470e3e5-d2c2-4821-b880-eac72b2bf5d1\") " pod="openshift-ingress/e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx" Jul 09 07:37:24.520488 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:24.520251 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4470e3e5-d2c2-4821-b880-eac72b2bf5d1-credential-socket\") pod \"e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx\" (UID: \"4470e3e5-d2c2-4821-b880-eac72b2bf5d1\") " pod="openshift-ingress/e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx" Jul 09 07:37:24.520488 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:24.520303 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb67f\" (UniqueName: \"kubernetes.io/projected/4470e3e5-d2c2-4821-b880-eac72b2bf5d1-kube-api-access-gb67f\") pod \"e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx\" (UID: \"4470e3e5-d2c2-4821-b880-eac72b2bf5d1\") " pod="openshift-ingress/e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx" Jul 09 07:37:24.520488 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:24.520345 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4470e3e5-d2c2-4821-b880-eac72b2bf5d1-istio-data\") pod \"e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx\" (UID: \"4470e3e5-d2c2-4821-b880-eac72b2bf5d1\") " pod="openshift-ingress/e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx" Jul 09 07:37:24.520488 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:24.520443 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4470e3e5-d2c2-4821-b880-eac72b2bf5d1-istio-envoy\") pod \"e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx\" (UID: \"4470e3e5-d2c2-4821-b880-eac72b2bf5d1\") " pod="openshift-ingress/e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx" Jul 09 07:37:24.520488 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:24.520480 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4470e3e5-d2c2-4821-b880-eac72b2bf5d1-istiod-ca-cert\") pod \"e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx\" (UID: \"4470e3e5-d2c2-4821-b880-eac72b2bf5d1\") " pod="openshift-ingress/e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx" Jul 09 07:37:24.621158 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:24.621132 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4470e3e5-d2c2-4821-b880-eac72b2bf5d1-istio-envoy\") pod \"e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx\" (UID: \"4470e3e5-d2c2-4821-b880-eac72b2bf5d1\") " pod="openshift-ingress/e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx" Jul 09 07:37:24.621338 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:24.621165 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4470e3e5-d2c2-4821-b880-eac72b2bf5d1-istiod-ca-cert\") pod \"e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx\" (UID: \"4470e3e5-d2c2-4821-b880-eac72b2bf5d1\") " pod="openshift-ingress/e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx" Jul 09 07:37:24.621338 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:24.621208 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4470e3e5-d2c2-4821-b880-eac72b2bf5d1-istio-token\") pod \"e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx\" (UID: \"4470e3e5-d2c2-4821-b880-eac72b2bf5d1\") " pod="openshift-ingress/e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx" Jul 09 07:37:24.621338 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:24.621258 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4470e3e5-d2c2-4821-b880-eac72b2bf5d1-istio-podinfo\") pod \"e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx\" (UID: \"4470e3e5-d2c2-4821-b880-eac72b2bf5d1\") " pod="openshift-ingress/e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx" Jul 09 07:37:24.621338 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:24.621299 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4470e3e5-d2c2-4821-b880-eac72b2bf5d1-workload-socket\") pod \"e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx\" (UID: \"4470e3e5-d2c2-4821-b880-eac72b2bf5d1\") " pod="openshift-ingress/e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx" Jul 09 07:37:24.621590 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:24.621355 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4470e3e5-d2c2-4821-b880-eac72b2bf5d1-workload-certs\") pod \"e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx\" (UID: \"4470e3e5-d2c2-4821-b880-eac72b2bf5d1\") " pod="openshift-ingress/e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx" Jul 09 07:37:24.621590 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:24.621393 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4470e3e5-d2c2-4821-b880-eac72b2bf5d1-credential-socket\") pod \"e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx\" (UID: \"4470e3e5-d2c2-4821-b880-eac72b2bf5d1\") " pod="openshift-ingress/e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx" Jul 09 07:37:24.621590 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:24.621421 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb67f\" (UniqueName: \"kubernetes.io/projected/4470e3e5-d2c2-4821-b880-eac72b2bf5d1-kube-api-access-gb67f\") pod \"e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx\" (UID: \"4470e3e5-d2c2-4821-b880-eac72b2bf5d1\") " pod="openshift-ingress/e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx" Jul 09 07:37:24.621590 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:24.621448 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4470e3e5-d2c2-4821-b880-eac72b2bf5d1-istio-data\") pod \"e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx\" (UID: \"4470e3e5-d2c2-4821-b880-eac72b2bf5d1\") " pod="openshift-ingress/e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx" Jul 09 07:37:24.621798 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:24.621739 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4470e3e5-d2c2-4821-b880-eac72b2bf5d1-workload-socket\") pod \"e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx\" (UID: \"4470e3e5-d2c2-4821-b880-eac72b2bf5d1\") " pod="openshift-ingress/e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx" Jul 09 07:37:24.621798 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:24.621753 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4470e3e5-d2c2-4821-b880-eac72b2bf5d1-credential-socket\") pod \"e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx\" (UID: \"4470e3e5-d2c2-4821-b880-eac72b2bf5d1\") " pod="openshift-ingress/e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx" Jul 09 07:37:24.621909 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:24.621792 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4470e3e5-d2c2-4821-b880-eac72b2bf5d1-istio-data\") pod \"e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx\" (UID: \"4470e3e5-d2c2-4821-b880-eac72b2bf5d1\") " pod="openshift-ingress/e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx" Jul 09 07:37:24.621909 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:24.621871 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4470e3e5-d2c2-4821-b880-eac72b2bf5d1-workload-certs\") pod \"e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx\" (UID: \"4470e3e5-d2c2-4821-b880-eac72b2bf5d1\") " pod="openshift-ingress/e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx" Jul 09 07:37:24.622329 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:24.622256 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4470e3e5-d2c2-4821-b880-eac72b2bf5d1-istiod-ca-cert\") pod \"e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx\" (UID: \"4470e3e5-d2c2-4821-b880-eac72b2bf5d1\") " pod="openshift-ingress/e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx" Jul 09 07:37:24.623358 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:24.623330 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4470e3e5-d2c2-4821-b880-eac72b2bf5d1-istio-envoy\") pod \"e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx\" (UID: \"4470e3e5-d2c2-4821-b880-eac72b2bf5d1\") " pod="openshift-ingress/e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx" Jul 09 07:37:24.624035 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:24.624016 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4470e3e5-d2c2-4821-b880-eac72b2bf5d1-istio-podinfo\") pod \"e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx\" (UID: \"4470e3e5-d2c2-4821-b880-eac72b2bf5d1\") " pod="openshift-ingress/e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx" Jul 09 07:37:24.631352 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:24.631307 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4470e3e5-d2c2-4821-b880-eac72b2bf5d1-istio-token\") pod \"e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx\" (UID: \"4470e3e5-d2c2-4821-b880-eac72b2bf5d1\") " pod="openshift-ingress/e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx" Jul 09 07:37:24.631947 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:24.631927 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb67f\" (UniqueName: \"kubernetes.io/projected/4470e3e5-d2c2-4821-b880-eac72b2bf5d1-kube-api-access-gb67f\") pod \"e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx\" (UID: \"4470e3e5-d2c2-4821-b880-eac72b2bf5d1\") " pod="openshift-ingress/e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx" Jul 09 07:37:24.787176 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:24.787098 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx" Jul 09 07:37:24.918603 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:24.918577 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx"] Jul 09 07:37:24.920827 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:37:24.920800 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4470e3e5_d2c2_4821_b880_eac72b2bf5d1.slice/crio-9952126e861e9b01e6947905b22f08762223a5c2437a451ca9ce36a513b37da2 WatchSource:0}: Error finding container 9952126e861e9b01e6947905b22f08762223a5c2437a451ca9ce36a513b37da2: Status 404 returned error can't find the container with id 9952126e861e9b01e6947905b22f08762223a5c2437a451ca9ce36a513b37da2 Jul 09 07:37:24.923154 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:24.923117 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 07:37:24.923216 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:24.923199 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 07:37:24.923284 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:24.923251 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 07:37:25.503715 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:25.503671 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx" event={"ID":"4470e3e5-d2c2-4821-b880-eac72b2bf5d1","Type":"ContainerStarted","Data":"b779abcd7adc5716d8a2056d0c26cbda84f54398c22d0e23f1e3fe414ce29540"} Jul 09 07:37:25.504103 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:25.503739 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx" event={"ID":"4470e3e5-d2c2-4821-b880-eac72b2bf5d1","Type":"ContainerStarted","Data":"9952126e861e9b01e6947905b22f08762223a5c2437a451ca9ce36a513b37da2"} Jul 09 07:37:25.527948 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:25.527890 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx" podStartSLOduration=1.527871588 podStartE2EDuration="1.527871588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 07:37:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 07:37:25.526183677 +0000 UTC m=+2104.577260929" watchObservedRunningTime="2026-07-09 07:37:25.527871588 +0000 UTC m=+2104.578948819" Jul 09 07:37:25.788254 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:25.788160 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx" Jul 09 07:37:25.792929 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:25.792906 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx" Jul 09 07:37:26.509015 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:26.508980 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx" Jul 09 07:37:26.509938 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:26.509919 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx" Jul 09 07:37:28.796450 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:28.796428 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ab11a1-openshift-default-689f477559-6qznw" Jul 09 07:37:28.864897 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:28.864825 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wt6g8\" (UniqueName: \"kubernetes.io/projected/af7b1769-b214-4224-8ac6-58d2933a113d-kube-api-access-wt6g8\") pod \"af7b1769-b214-4224-8ac6-58d2933a113d\" (UID: \"af7b1769-b214-4224-8ac6-58d2933a113d\") " Jul 09 07:37:28.864897 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:28.864869 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af7b1769-b214-4224-8ac6-58d2933a113d-istio-data\") pod \"af7b1769-b214-4224-8ac6-58d2933a113d\" (UID: \"af7b1769-b214-4224-8ac6-58d2933a113d\") " Jul 09 07:37:28.864897 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:28.864897 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af7b1769-b214-4224-8ac6-58d2933a113d-istio-envoy\") pod \"af7b1769-b214-4224-8ac6-58d2933a113d\" (UID: \"af7b1769-b214-4224-8ac6-58d2933a113d\") " Jul 09 07:37:28.865165 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:28.864921 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af7b1769-b214-4224-8ac6-58d2933a113d-istiod-ca-cert\") pod \"af7b1769-b214-4224-8ac6-58d2933a113d\" (UID: \"af7b1769-b214-4224-8ac6-58d2933a113d\") " Jul 09 07:37:28.865165 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:28.864967 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af7b1769-b214-4224-8ac6-58d2933a113d-workload-certs\") pod \"af7b1769-b214-4224-8ac6-58d2933a113d\" (UID: \"af7b1769-b214-4224-8ac6-58d2933a113d\") " Jul 09 07:37:28.865165 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:28.865004 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af7b1769-b214-4224-8ac6-58d2933a113d-workload-socket\") pod \"af7b1769-b214-4224-8ac6-58d2933a113d\" (UID: \"af7b1769-b214-4224-8ac6-58d2933a113d\") " Jul 09 07:37:28.865165 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:28.865047 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af7b1769-b214-4224-8ac6-58d2933a113d-credential-socket\") pod \"af7b1769-b214-4224-8ac6-58d2933a113d\" (UID: \"af7b1769-b214-4224-8ac6-58d2933a113d\") " Jul 09 07:37:28.865165 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:28.865082 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af7b1769-b214-4224-8ac6-58d2933a113d-istio-podinfo\") pod \"af7b1769-b214-4224-8ac6-58d2933a113d\" (UID: \"af7b1769-b214-4224-8ac6-58d2933a113d\") " Jul 09 07:37:28.865165 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:28.865112 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af7b1769-b214-4224-8ac6-58d2933a113d-istio-token\") pod \"af7b1769-b214-4224-8ac6-58d2933a113d\" (UID: \"af7b1769-b214-4224-8ac6-58d2933a113d\") " Jul 09 07:37:28.865513 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:28.865306 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af7b1769-b214-4224-8ac6-58d2933a113d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "af7b1769-b214-4224-8ac6-58d2933a113d" (UID: "af7b1769-b214-4224-8ac6-58d2933a113d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 07:37:28.865513 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:28.865392 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af7b1769-b214-4224-8ac6-58d2933a113d-workload-certs\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:37:28.865513 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:28.865398 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af7b1769-b214-4224-8ac6-58d2933a113d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "af7b1769-b214-4224-8ac6-58d2933a113d" (UID: "af7b1769-b214-4224-8ac6-58d2933a113d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 07:37:28.865513 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:28.865500 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af7b1769-b214-4224-8ac6-58d2933a113d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "af7b1769-b214-4224-8ac6-58d2933a113d" (UID: "af7b1769-b214-4224-8ac6-58d2933a113d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 07:37:28.865722 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:28.865568 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af7b1769-b214-4224-8ac6-58d2933a113d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "af7b1769-b214-4224-8ac6-58d2933a113d" (UID: "af7b1769-b214-4224-8ac6-58d2933a113d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 07:37:28.865777 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:28.865722 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af7b1769-b214-4224-8ac6-58d2933a113d-istio-data" (OuterVolumeSpecName: "istio-data") pod "af7b1769-b214-4224-8ac6-58d2933a113d" (UID: "af7b1769-b214-4224-8ac6-58d2933a113d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 07:37:28.867129 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:28.867104 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af7b1769-b214-4224-8ac6-58d2933a113d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "af7b1769-b214-4224-8ac6-58d2933a113d" (UID: "af7b1769-b214-4224-8ac6-58d2933a113d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 07:37:28.867328 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:28.867303 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af7b1769-b214-4224-8ac6-58d2933a113d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "af7b1769-b214-4224-8ac6-58d2933a113d" (UID: "af7b1769-b214-4224-8ac6-58d2933a113d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 07:37:28.867489 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:28.867468 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af7b1769-b214-4224-8ac6-58d2933a113d-kube-api-access-wt6g8" (OuterVolumeSpecName: "kube-api-access-wt6g8") pod "af7b1769-b214-4224-8ac6-58d2933a113d" (UID: "af7b1769-b214-4224-8ac6-58d2933a113d"). InnerVolumeSpecName "kube-api-access-wt6g8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 07:37:28.867556 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:28.867540 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af7b1769-b214-4224-8ac6-58d2933a113d-istio-token" (OuterVolumeSpecName: "istio-token") pod "af7b1769-b214-4224-8ac6-58d2933a113d" (UID: "af7b1769-b214-4224-8ac6-58d2933a113d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 07:37:28.966095 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:28.966068 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af7b1769-b214-4224-8ac6-58d2933a113d-istio-data\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:37:28.966095 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:28.966092 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af7b1769-b214-4224-8ac6-58d2933a113d-istio-envoy\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:37:28.966295 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:28.966102 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af7b1769-b214-4224-8ac6-58d2933a113d-istiod-ca-cert\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:37:28.966295 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:28.966112 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af7b1769-b214-4224-8ac6-58d2933a113d-workload-socket\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:37:28.966295 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:28.966120 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af7b1769-b214-4224-8ac6-58d2933a113d-credential-socket\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:37:28.966295 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:28.966129 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af7b1769-b214-4224-8ac6-58d2933a113d-istio-podinfo\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:37:28.966295 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:28.966137 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af7b1769-b214-4224-8ac6-58d2933a113d-istio-token\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:37:28.966295 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:28.966145 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wt6g8\" (UniqueName: \"kubernetes.io/projected/af7b1769-b214-4224-8ac6-58d2933a113d-kube-api-access-wt6g8\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:37:29.524215 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:29.524178 2564 generic.go:356] "Generic (PLEG): container finished" podID="af7b1769-b214-4224-8ac6-58d2933a113d" containerID="2f8a32e96c3c03e8e363362ee241aba94a43ce8219d90ff69ee2f460b5b7eff7" exitCode=0 Jul 09 07:37:29.524416 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:29.524233 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ab11a1-openshift-default-689f477559-6qznw" Jul 09 07:37:29.524416 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:29.524241 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ab11a1-openshift-default-689f477559-6qznw" event={"ID":"af7b1769-b214-4224-8ac6-58d2933a113d","Type":"ContainerDied","Data":"2f8a32e96c3c03e8e363362ee241aba94a43ce8219d90ff69ee2f460b5b7eff7"} Jul 09 07:37:29.524416 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:29.524313 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ab11a1-openshift-default-689f477559-6qznw" event={"ID":"af7b1769-b214-4224-8ac6-58d2933a113d","Type":"ContainerDied","Data":"5def9c0bf8f433a090e82039ab0c9aa3cca611baf604fe48f111d2254f92d536"} Jul 09 07:37:29.524416 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:29.524344 2564 scope.go:117] "RemoveContainer" containerID="2f8a32e96c3c03e8e363362ee241aba94a43ce8219d90ff69ee2f460b5b7eff7" Jul 09 07:37:29.532782 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:29.532751 2564 scope.go:117] "RemoveContainer" containerID="c5ef774845dab1ff88781ae3b8700db4d986cbfcb92cd96250284880e57ece8d" Jul 09 07:37:29.540969 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:29.540953 2564 scope.go:117] "RemoveContainer" containerID="2f8a32e96c3c03e8e363362ee241aba94a43ce8219d90ff69ee2f460b5b7eff7" Jul 09 07:37:29.541239 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:37:29.541216 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f8a32e96c3c03e8e363362ee241aba94a43ce8219d90ff69ee2f460b5b7eff7\": container with ID starting with 2f8a32e96c3c03e8e363362ee241aba94a43ce8219d90ff69ee2f460b5b7eff7 not found: ID does not exist" containerID="2f8a32e96c3c03e8e363362ee241aba94a43ce8219d90ff69ee2f460b5b7eff7" Jul 09 07:37:29.541323 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:29.541246 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f8a32e96c3c03e8e363362ee241aba94a43ce8219d90ff69ee2f460b5b7eff7"} err="failed to get container status \"2f8a32e96c3c03e8e363362ee241aba94a43ce8219d90ff69ee2f460b5b7eff7\": rpc error: code = NotFound desc = could not find container \"2f8a32e96c3c03e8e363362ee241aba94a43ce8219d90ff69ee2f460b5b7eff7\": container with ID starting with 2f8a32e96c3c03e8e363362ee241aba94a43ce8219d90ff69ee2f460b5b7eff7 not found: ID does not exist" Jul 09 07:37:29.541323 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:29.541282 2564 scope.go:117] "RemoveContainer" containerID="c5ef774845dab1ff88781ae3b8700db4d986cbfcb92cd96250284880e57ece8d" Jul 09 07:37:29.541513 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:37:29.541497 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5ef774845dab1ff88781ae3b8700db4d986cbfcb92cd96250284880e57ece8d\": container with ID starting with c5ef774845dab1ff88781ae3b8700db4d986cbfcb92cd96250284880e57ece8d not found: ID does not exist" containerID="c5ef774845dab1ff88781ae3b8700db4d986cbfcb92cd96250284880e57ece8d" Jul 09 07:37:29.541556 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:29.541519 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5ef774845dab1ff88781ae3b8700db4d986cbfcb92cd96250284880e57ece8d"} err="failed to get container status \"c5ef774845dab1ff88781ae3b8700db4d986cbfcb92cd96250284880e57ece8d\": rpc error: code = NotFound desc = could not find container \"c5ef774845dab1ff88781ae3b8700db4d986cbfcb92cd96250284880e57ece8d\": container with ID starting with c5ef774845dab1ff88781ae3b8700db4d986cbfcb92cd96250284880e57ece8d not found: ID does not exist" Jul 09 07:37:29.545854 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:29.545833 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ab11a1-openshift-default-689f477559-6qznw"] Jul 09 07:37:29.549631 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:29.549611 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-ab11a1-openshift-default-689f477559-6qznw"] Jul 09 07:37:31.462672 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:31.462639 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af7b1769-b214-4224-8ac6-58d2933a113d" path="/var/lib/kubelet/pods/af7b1769-b214-4224-8ac6-58d2933a113d/volumes" Jul 09 07:37:32.590191 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:32.590154 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7"] Jul 09 07:37:32.590695 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:32.590681 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af7b1769-b214-4224-8ac6-58d2933a113d" containerName="istio-proxy" Jul 09 07:37:32.590745 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:32.590699 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="af7b1769-b214-4224-8ac6-58d2933a113d" containerName="istio-proxy" Jul 09 07:37:32.590810 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:32.590800 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf6f503f-48f5-4993-b6f6-e53bd6b11aac" containerName="istio-proxy" Jul 09 07:37:32.590855 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:32.590813 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="af7b1769-b214-4224-8ac6-58d2933a113d" containerName="istio-proxy" Jul 09 07:37:32.590855 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:32.590825 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="af7b1769-b214-4224-8ac6-58d2933a113d" containerName="istio-proxy" Jul 09 07:37:32.595851 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:32.595830 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7" Jul 09 07:37:32.598287 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:32.598246 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-da4680-openshift-default-dockercfg-5cdqv\"" Jul 09 07:37:32.607868 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:32.605535 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7"] Jul 09 07:37:32.699617 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:32.699570 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4-istiod-ca-cert\") pod \"e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7\" (UID: \"fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4\") " pod="openshift-ingress/e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7" Jul 09 07:37:32.699617 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:32.699617 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr66z\" (UniqueName: \"kubernetes.io/projected/fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4-kube-api-access-jr66z\") pod \"e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7\" (UID: \"fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4\") " pod="openshift-ingress/e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7" Jul 09 07:37:32.699901 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:32.699647 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4-credential-socket\") pod \"e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7\" (UID: \"fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4\") " pod="openshift-ingress/e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7" Jul 09 07:37:32.699901 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:32.699729 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4-workload-socket\") pod \"e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7\" (UID: \"fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4\") " pod="openshift-ingress/e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7" Jul 09 07:37:32.699901 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:32.699783 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4-istio-envoy\") pod \"e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7\" (UID: \"fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4\") " pod="openshift-ingress/e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7" Jul 09 07:37:32.699901 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:32.699836 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4-workload-certs\") pod \"e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7\" (UID: \"fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4\") " pod="openshift-ingress/e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7" Jul 09 07:37:32.699901 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:32.699866 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4-istio-data\") pod \"e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7\" (UID: \"fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4\") " pod="openshift-ingress/e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7" Jul 09 07:37:32.700153 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:32.699916 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4-istio-podinfo\") pod \"e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7\" (UID: \"fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4\") " pod="openshift-ingress/e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7" Jul 09 07:37:32.700153 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:32.699945 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4-istio-token\") pod \"e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7\" (UID: \"fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4\") " pod="openshift-ingress/e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7" Jul 09 07:37:32.801331 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:32.801296 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4-credential-socket\") pod \"e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7\" (UID: \"fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4\") " pod="openshift-ingress/e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7" Jul 09 07:37:32.801331 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:32.801336 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4-workload-socket\") pod \"e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7\" (UID: \"fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4\") " pod="openshift-ingress/e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7" Jul 09 07:37:32.801553 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:32.801352 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4-istio-envoy\") pod \"e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7\" (UID: \"fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4\") " pod="openshift-ingress/e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7" Jul 09 07:37:32.801553 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:32.801377 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4-workload-certs\") pod \"e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7\" (UID: \"fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4\") " pod="openshift-ingress/e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7" Jul 09 07:37:32.801553 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:32.801407 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4-istio-data\") pod \"e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7\" (UID: \"fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4\") " pod="openshift-ingress/e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7" Jul 09 07:37:32.801553 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:32.801536 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4-istio-podinfo\") pod \"e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7\" (UID: \"fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4\") " pod="openshift-ingress/e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7" Jul 09 07:37:32.801755 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:32.801580 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4-istio-token\") pod \"e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7\" (UID: \"fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4\") " pod="openshift-ingress/e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7" Jul 09 07:37:32.801755 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:32.801680 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4-istiod-ca-cert\") pod \"e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7\" (UID: \"fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4\") " pod="openshift-ingress/e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7" Jul 09 07:37:32.801755 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:32.801712 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr66z\" (UniqueName: \"kubernetes.io/projected/fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4-kube-api-access-jr66z\") pod \"e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7\" (UID: \"fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4\") " pod="openshift-ingress/e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7" Jul 09 07:37:32.801755 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:32.801723 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4-credential-socket\") pod \"e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7\" (UID: \"fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4\") " pod="openshift-ingress/e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7" Jul 09 07:37:32.801932 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:32.801789 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4-workload-socket\") pod \"e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7\" (UID: \"fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4\") " pod="openshift-ingress/e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7" Jul 09 07:37:32.802007 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:32.801986 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4-workload-certs\") pod \"e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7\" (UID: \"fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4\") " pod="openshift-ingress/e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7" Jul 09 07:37:32.802069 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:32.801998 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4-istio-data\") pod \"e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7\" (UID: \"fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4\") " pod="openshift-ingress/e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7" Jul 09 07:37:32.802482 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:32.802463 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4-istiod-ca-cert\") pod \"e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7\" (UID: \"fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4\") " pod="openshift-ingress/e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7" Jul 09 07:37:32.803677 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:32.803655 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4-istio-envoy\") pod \"e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7\" (UID: \"fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4\") " pod="openshift-ingress/e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7" Jul 09 07:37:32.804094 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:32.804075 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4-istio-podinfo\") pod \"e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7\" (UID: \"fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4\") " pod="openshift-ingress/e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7" Jul 09 07:37:32.809850 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:32.809829 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4-istio-token\") pod \"e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7\" (UID: \"fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4\") " pod="openshift-ingress/e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7" Jul 09 07:37:32.809942 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:32.809852 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr66z\" (UniqueName: \"kubernetes.io/projected/fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4-kube-api-access-jr66z\") pod \"e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7\" (UID: \"fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4\") " pod="openshift-ingress/e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7" Jul 09 07:37:32.909751 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:32.909721 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7" Jul 09 07:37:33.040911 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:33.040885 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7"] Jul 09 07:37:33.043587 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:37:33.043562 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe5ae6f0_2fe6_456f_b1fd_e05ecb4037a4.slice/crio-a42294fa72bafe2d983e90c95777cf5f14edb496b42adf67749f266dd81dfff1 WatchSource:0}: Error finding container a42294fa72bafe2d983e90c95777cf5f14edb496b42adf67749f266dd81dfff1: Status 404 returned error can't find the container with id a42294fa72bafe2d983e90c95777cf5f14edb496b42adf67749f266dd81dfff1 Jul 09 07:37:33.045586 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:33.045549 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 07:37:33.045691 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:33.045631 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 07:37:33.045691 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:33.045673 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 07:37:33.543633 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:33.543596 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7" event={"ID":"fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4","Type":"ContainerStarted","Data":"e57673c0cac015e1bf66b4be342c5f4bd71b88f2b1a18651503dbcb922af7279"} Jul 09 07:37:33.543801 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:33.543653 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7" event={"ID":"fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4","Type":"ContainerStarted","Data":"a42294fa72bafe2d983e90c95777cf5f14edb496b42adf67749f266dd81dfff1"} Jul 09 07:37:33.568483 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:33.568423 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7" podStartSLOduration=1.568402015 podStartE2EDuration="1.568402015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 07:37:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 07:37:33.566389647 +0000 UTC m=+2112.617466886" watchObservedRunningTime="2026-07-09 07:37:33.568402015 +0000 UTC m=+2112.619479245" Jul 09 07:37:33.910326 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:33.910292 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7" Jul 09 07:37:34.915524 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:34.915490 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7" Jul 09 07:37:35.553907 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:35.553850 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7" Jul 09 07:37:35.554985 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:35.554957 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7" Jul 09 07:37:42.550438 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:42.550395 2564 patch_prober.go:28] interesting pod/e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 09 07:37:42.550882 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:42.550457 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx" podUID="4470e3e5-d2c2-4821-b880-eac72b2bf5d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 09 07:37:43.600492 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:43.600456 2564 generic.go:356] "Generic (PLEG): container finished" podID="4470e3e5-d2c2-4821-b880-eac72b2bf5d1" containerID="b779abcd7adc5716d8a2056d0c26cbda84f54398c22d0e23f1e3fe414ce29540" exitCode=0 Jul 09 07:37:43.600894 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:43.600527 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx" event={"ID":"4470e3e5-d2c2-4821-b880-eac72b2bf5d1","Type":"ContainerDied","Data":"b779abcd7adc5716d8a2056d0c26cbda84f54398c22d0e23f1e3fe414ce29540"} Jul 09 07:37:43.600940 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:43.600903 2564 scope.go:117] "RemoveContainer" containerID="b779abcd7adc5716d8a2056d0c26cbda84f54398c22d0e23f1e3fe414ce29540" Jul 09 07:37:43.601872 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:43.601844 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 07:37:43.601935 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:43.601924 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 07:37:43.601976 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:43.601956 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 07:37:44.607281 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:44.607235 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx" event={"ID":"4470e3e5-d2c2-4821-b880-eac72b2bf5d1","Type":"ContainerStarted","Data":"f5353c2202bb6b6493caf833eff886bd8af52d64f5ac699351063c8e3943c305"} Jul 09 07:37:44.787628 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:44.787591 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx" Jul 09 07:37:44.787886 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:44.787857 2564 patch_prober.go:28] interesting pod/e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Jul 09 07:37:44.787936 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:44.787909 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx" podUID="4470e3e5-d2c2-4821-b880-eac72b2bf5d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Jul 09 07:37:45.787832 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:45.787635 2564 patch_prober.go:28] interesting pod/e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Jul 09 07:37:45.787832 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:45.787714 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx" podUID="4470e3e5-d2c2-4821-b880-eac72b2bf5d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Jul 09 07:37:46.787431 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:46.787398 2564 patch_prober.go:28] interesting pod/e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Jul 09 07:37:46.787613 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:46.787450 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx" podUID="4470e3e5-d2c2-4821-b880-eac72b2bf5d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Jul 09 07:37:47.787580 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:47.787501 2564 patch_prober.go:28] interesting pod/e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Jul 09 07:37:47.787580 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:47.787556 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx" podUID="4470e3e5-d2c2-4821-b880-eac72b2bf5d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Jul 09 07:37:48.797588 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:48.797550 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx" Jul 09 07:37:48.798045 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:48.797960 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx" Jul 09 07:37:48.798707 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:48.798685 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx" Jul 09 07:37:50.555517 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:50.555485 2564 patch_prober.go:28] interesting pod/e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Jul 09 07:37:50.555935 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:50.555543 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7" podUID="fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Jul 09 07:37:50.636803 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:50.636771 2564 generic.go:356] "Generic (PLEG): container finished" podID="fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4" containerID="e57673c0cac015e1bf66b4be342c5f4bd71b88f2b1a18651503dbcb922af7279" exitCode=0 Jul 09 07:37:50.636950 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:50.636849 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7" event={"ID":"fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4","Type":"ContainerDied","Data":"e57673c0cac015e1bf66b4be342c5f4bd71b88f2b1a18651503dbcb922af7279"} Jul 09 07:37:50.637357 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:50.637339 2564 scope.go:117] "RemoveContainer" containerID="e57673c0cac015e1bf66b4be342c5f4bd71b88f2b1a18651503dbcb922af7279" Jul 09 07:37:50.638249 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:50.638224 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 07:37:50.638322 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:50.638305 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 07:37:50.638374 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:50.638338 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 07:37:51.648041 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:51.647994 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7" event={"ID":"fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4","Type":"ContainerStarted","Data":"9f3119c0220b14299a0fb5b23e69993620a7f83fe691908eb98bed07d6279da3"} Jul 09 07:37:51.910596 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:51.910501 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7" Jul 09 07:37:51.910804 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:51.910783 2564 patch_prober.go:28] interesting pod/e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Jul 09 07:37:51.910849 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:51.910829 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7" podUID="fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Jul 09 07:37:52.911105 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:52.911068 2564 patch_prober.go:28] interesting pod/e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Jul 09 07:37:52.911594 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:52.911129 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7" podUID="fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Jul 09 07:37:53.910553 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:53.910517 2564 patch_prober.go:28] interesting pod/e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Jul 09 07:37:53.910720 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:53.910573 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7" podUID="fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Jul 09 07:37:54.911047 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:54.911013 2564 patch_prober.go:28] interesting pod/e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Jul 09 07:37:54.911489 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:54.911067 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7" podUID="fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Jul 09 07:37:55.914447 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:55.914420 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7" Jul 09 07:37:55.914849 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:55.914828 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7" Jul 09 07:37:55.915400 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:37:55.915382 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7" Jul 09 07:40:51.575602 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:40:51.575560 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx"] Jul 09 07:40:51.576135 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:40:51.575820 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx" podUID="4470e3e5-d2c2-4821-b880-eac72b2bf5d1" containerName="istio-proxy" containerID="cri-o://f5353c2202bb6b6493caf833eff886bd8af52d64f5ac699351063c8e3943c305" gracePeriod=30 Jul 09 07:40:56.819042 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:40:56.819019 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx" Jul 09 07:40:56.919213 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:40:56.919185 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4470e3e5-d2c2-4821-b880-eac72b2bf5d1-credential-socket\") pod \"4470e3e5-d2c2-4821-b880-eac72b2bf5d1\" (UID: \"4470e3e5-d2c2-4821-b880-eac72b2bf5d1\") " Jul 09 07:40:56.919387 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:40:56.919220 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4470e3e5-d2c2-4821-b880-eac72b2bf5d1-istio-podinfo\") pod \"4470e3e5-d2c2-4821-b880-eac72b2bf5d1\" (UID: \"4470e3e5-d2c2-4821-b880-eac72b2bf5d1\") " Jul 09 07:40:56.919387 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:40:56.919248 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gb67f\" (UniqueName: \"kubernetes.io/projected/4470e3e5-d2c2-4821-b880-eac72b2bf5d1-kube-api-access-gb67f\") pod \"4470e3e5-d2c2-4821-b880-eac72b2bf5d1\" (UID: \"4470e3e5-d2c2-4821-b880-eac72b2bf5d1\") " Jul 09 07:40:56.919387 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:40:56.919298 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4470e3e5-d2c2-4821-b880-eac72b2bf5d1-istio-token\") pod \"4470e3e5-d2c2-4821-b880-eac72b2bf5d1\" (UID: \"4470e3e5-d2c2-4821-b880-eac72b2bf5d1\") " Jul 09 07:40:56.919387 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:40:56.919321 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4470e3e5-d2c2-4821-b880-eac72b2bf5d1-workload-certs\") pod \"4470e3e5-d2c2-4821-b880-eac72b2bf5d1\" (UID: \"4470e3e5-d2c2-4821-b880-eac72b2bf5d1\") " Jul 09 07:40:56.919387 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:40:56.919337 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4470e3e5-d2c2-4821-b880-eac72b2bf5d1-istiod-ca-cert\") pod \"4470e3e5-d2c2-4821-b880-eac72b2bf5d1\" (UID: \"4470e3e5-d2c2-4821-b880-eac72b2bf5d1\") " Jul 09 07:40:56.919632 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:40:56.919496 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4470e3e5-d2c2-4821-b880-eac72b2bf5d1-workload-socket\") pod \"4470e3e5-d2c2-4821-b880-eac72b2bf5d1\" (UID: \"4470e3e5-d2c2-4821-b880-eac72b2bf5d1\") " Jul 09 07:40:56.919632 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:40:56.919565 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4470e3e5-d2c2-4821-b880-eac72b2bf5d1-istio-data\") pod \"4470e3e5-d2c2-4821-b880-eac72b2bf5d1\" (UID: \"4470e3e5-d2c2-4821-b880-eac72b2bf5d1\") " Jul 09 07:40:56.919632 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:40:56.919591 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4470e3e5-d2c2-4821-b880-eac72b2bf5d1-istio-envoy\") pod \"4470e3e5-d2c2-4821-b880-eac72b2bf5d1\" (UID: \"4470e3e5-d2c2-4821-b880-eac72b2bf5d1\") " Jul 09 07:40:56.919791 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:40:56.919498 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4470e3e5-d2c2-4821-b880-eac72b2bf5d1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4470e3e5-d2c2-4821-b880-eac72b2bf5d1" (UID: "4470e3e5-d2c2-4821-b880-eac72b2bf5d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 07:40:56.919791 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:40:56.919662 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4470e3e5-d2c2-4821-b880-eac72b2bf5d1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4470e3e5-d2c2-4821-b880-eac72b2bf5d1" (UID: "4470e3e5-d2c2-4821-b880-eac72b2bf5d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 07:40:56.919791 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:40:56.919722 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4470e3e5-d2c2-4821-b880-eac72b2bf5d1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4470e3e5-d2c2-4821-b880-eac72b2bf5d1" (UID: "4470e3e5-d2c2-4821-b880-eac72b2bf5d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 07:40:56.919791 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:40:56.919754 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4470e3e5-d2c2-4821-b880-eac72b2bf5d1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4470e3e5-d2c2-4821-b880-eac72b2bf5d1" (UID: "4470e3e5-d2c2-4821-b880-eac72b2bf5d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 07:40:56.920064 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:40:56.919983 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4470e3e5-d2c2-4821-b880-eac72b2bf5d1-workload-socket\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:40:56.920064 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:40:56.920006 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4470e3e5-d2c2-4821-b880-eac72b2bf5d1-credential-socket\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:40:56.920064 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:40:56.920020 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4470e3e5-d2c2-4821-b880-eac72b2bf5d1-workload-certs\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:40:56.920064 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:40:56.920035 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4470e3e5-d2c2-4821-b880-eac72b2bf5d1-istiod-ca-cert\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:40:56.920668 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:40:56.920643 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4470e3e5-d2c2-4821-b880-eac72b2bf5d1-istio-data" (OuterVolumeSpecName: "istio-data") pod "4470e3e5-d2c2-4821-b880-eac72b2bf5d1" (UID: "4470e3e5-d2c2-4821-b880-eac72b2bf5d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 07:40:56.921465 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:40:56.921439 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4470e3e5-d2c2-4821-b880-eac72b2bf5d1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4470e3e5-d2c2-4821-b880-eac72b2bf5d1" (UID: "4470e3e5-d2c2-4821-b880-eac72b2bf5d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 07:40:56.921571 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:40:56.921553 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4470e3e5-d2c2-4821-b880-eac72b2bf5d1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4470e3e5-d2c2-4821-b880-eac72b2bf5d1" (UID: "4470e3e5-d2c2-4821-b880-eac72b2bf5d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 07:40:56.921617 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:40:56.921590 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4470e3e5-d2c2-4821-b880-eac72b2bf5d1-kube-api-access-gb67f" (OuterVolumeSpecName: "kube-api-access-gb67f") pod "4470e3e5-d2c2-4821-b880-eac72b2bf5d1" (UID: "4470e3e5-d2c2-4821-b880-eac72b2bf5d1"). InnerVolumeSpecName "kube-api-access-gb67f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 07:40:56.921657 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:40:56.921631 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4470e3e5-d2c2-4821-b880-eac72b2bf5d1-istio-token" (OuterVolumeSpecName: "istio-token") pod "4470e3e5-d2c2-4821-b880-eac72b2bf5d1" (UID: "4470e3e5-d2c2-4821-b880-eac72b2bf5d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 07:40:57.020777 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:40:57.020748 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4470e3e5-d2c2-4821-b880-eac72b2bf5d1-istio-data\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:40:57.020777 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:40:57.020773 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4470e3e5-d2c2-4821-b880-eac72b2bf5d1-istio-envoy\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:40:57.020777 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:40:57.020781 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4470e3e5-d2c2-4821-b880-eac72b2bf5d1-istio-podinfo\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:40:57.020977 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:40:57.020790 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb67f\" (UniqueName: \"kubernetes.io/projected/4470e3e5-d2c2-4821-b880-eac72b2bf5d1-kube-api-access-gb67f\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:40:57.020977 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:40:57.020799 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4470e3e5-d2c2-4821-b880-eac72b2bf5d1-istio-token\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:40:57.494732 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:40:57.494701 2564 generic.go:356] "Generic (PLEG): container finished" podID="4470e3e5-d2c2-4821-b880-eac72b2bf5d1" containerID="f5353c2202bb6b6493caf833eff886bd8af52d64f5ac699351063c8e3943c305" exitCode=0 Jul 09 07:40:57.494892 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:40:57.494746 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx" Jul 09 07:40:57.494892 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:40:57.494753 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx" event={"ID":"4470e3e5-d2c2-4821-b880-eac72b2bf5d1","Type":"ContainerDied","Data":"f5353c2202bb6b6493caf833eff886bd8af52d64f5ac699351063c8e3943c305"} Jul 09 07:40:57.494892 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:40:57.494794 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx" event={"ID":"4470e3e5-d2c2-4821-b880-eac72b2bf5d1","Type":"ContainerDied","Data":"9952126e861e9b01e6947905b22f08762223a5c2437a451ca9ce36a513b37da2"} Jul 09 07:40:57.494892 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:40:57.494814 2564 scope.go:117] "RemoveContainer" containerID="f5353c2202bb6b6493caf833eff886bd8af52d64f5ac699351063c8e3943c305" Jul 09 07:40:57.503329 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:40:57.503314 2564 scope.go:117] "RemoveContainer" containerID="b779abcd7adc5716d8a2056d0c26cbda84f54398c22d0e23f1e3fe414ce29540" Jul 09 07:40:57.511595 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:40:57.511575 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx"] Jul 09 07:40:57.512730 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:40:57.512708 2564 scope.go:117] "RemoveContainer" containerID="f5353c2202bb6b6493caf833eff886bd8af52d64f5ac699351063c8e3943c305" Jul 09 07:40:57.512989 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:40:57.512966 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5353c2202bb6b6493caf833eff886bd8af52d64f5ac699351063c8e3943c305\": container with ID starting with f5353c2202bb6b6493caf833eff886bd8af52d64f5ac699351063c8e3943c305 not found: ID does not exist" containerID="f5353c2202bb6b6493caf833eff886bd8af52d64f5ac699351063c8e3943c305" Jul 09 07:40:57.513042 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:40:57.513002 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5353c2202bb6b6493caf833eff886bd8af52d64f5ac699351063c8e3943c305"} err="failed to get container status \"f5353c2202bb6b6493caf833eff886bd8af52d64f5ac699351063c8e3943c305\": rpc error: code = NotFound desc = could not find container \"f5353c2202bb6b6493caf833eff886bd8af52d64f5ac699351063c8e3943c305\": container with ID starting with f5353c2202bb6b6493caf833eff886bd8af52d64f5ac699351063c8e3943c305 not found: ID does not exist" Jul 09 07:40:57.513042 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:40:57.513024 2564 scope.go:117] "RemoveContainer" containerID="b779abcd7adc5716d8a2056d0c26cbda84f54398c22d0e23f1e3fe414ce29540" Jul 09 07:40:57.513292 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:40:57.513273 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b779abcd7adc5716d8a2056d0c26cbda84f54398c22d0e23f1e3fe414ce29540\": container with ID starting with b779abcd7adc5716d8a2056d0c26cbda84f54398c22d0e23f1e3fe414ce29540 not found: ID does not exist" containerID="b779abcd7adc5716d8a2056d0c26cbda84f54398c22d0e23f1e3fe414ce29540" Jul 09 07:40:57.513357 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:40:57.513298 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b779abcd7adc5716d8a2056d0c26cbda84f54398c22d0e23f1e3fe414ce29540"} err="failed to get container status \"b779abcd7adc5716d8a2056d0c26cbda84f54398c22d0e23f1e3fe414ce29540\": rpc error: code = NotFound desc = could not find container \"b779abcd7adc5716d8a2056d0c26cbda84f54398c22d0e23f1e3fe414ce29540\": container with ID starting with b779abcd7adc5716d8a2056d0c26cbda84f54398c22d0e23f1e3fe414ce29540 not found: ID does not exist" Jul 09 07:40:57.519178 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:40:57.519160 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-62f792-openshift-default-b6769dd55-hrdzx"] Jul 09 07:40:58.586603 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:40:58.586567 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7"] Jul 09 07:40:58.586983 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:40:58.586805 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7" podUID="fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4" containerName="istio-proxy" containerID="cri-o://9f3119c0220b14299a0fb5b23e69993620a7f83fe691908eb98bed07d6279da3" gracePeriod=30 Jul 09 07:40:59.463396 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:40:59.463365 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4470e3e5-d2c2-4821-b880-eac72b2bf5d1" path="/var/lib/kubelet/pods/4470e3e5-d2c2-4821-b880-eac72b2bf5d1/volumes" Jul 09 07:41:03.845087 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:03.845063 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7" Jul 09 07:41:03.877138 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:03.877112 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4-istio-token\") pod \"fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4\" (UID: \"fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4\") " Jul 09 07:41:03.877254 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:03.877173 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4-istio-envoy\") pod \"fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4\" (UID: \"fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4\") " Jul 09 07:41:03.877254 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:03.877208 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4-credential-socket\") pod \"fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4\" (UID: \"fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4\") " Jul 09 07:41:03.877366 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:03.877289 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4-istio-podinfo\") pod \"fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4\" (UID: \"fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4\") " Jul 09 07:41:03.877366 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:03.877324 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jr66z\" (UniqueName: \"kubernetes.io/projected/fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4-kube-api-access-jr66z\") pod \"fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4\" (UID: \"fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4\") " Jul 09 07:41:03.877366 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:03.877362 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4-istiod-ca-cert\") pod \"fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4\" (UID: \"fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4\") " Jul 09 07:41:03.877502 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:03.877395 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4-workload-certs\") pod \"fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4\" (UID: \"fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4\") " Jul 09 07:41:03.877502 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:03.877424 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4-workload-socket\") pod \"fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4\" (UID: \"fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4\") " Jul 09 07:41:03.877502 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:03.877462 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4-istio-data\") pod \"fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4\" (UID: \"fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4\") " Jul 09 07:41:03.877661 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:03.877548 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4" (UID: "fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 07:41:03.877719 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:03.877688 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4" (UID: "fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 07:41:03.877786 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:03.877742 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4-credential-socket\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:41:03.877786 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:03.877745 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4" (UID: "fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 07:41:03.878021 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:03.877998 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4" (UID: "fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 07:41:03.878890 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:03.878862 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4-istio-data" (OuterVolumeSpecName: "istio-data") pod "fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4" (UID: "fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 07:41:03.879684 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:03.879652 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4-kube-api-access-jr66z" (OuterVolumeSpecName: "kube-api-access-jr66z") pod "fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4" (UID: "fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4"). InnerVolumeSpecName "kube-api-access-jr66z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 07:41:03.879684 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:03.879669 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4-istio-token" (OuterVolumeSpecName: "istio-token") pod "fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4" (UID: "fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 07:41:03.879849 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:03.879773 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4" (UID: "fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 07:41:03.879849 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:03.879789 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4" (UID: "fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 07:41:03.978580 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:03.978552 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4-istio-podinfo\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:41:03.978580 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:03.978577 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jr66z\" (UniqueName: \"kubernetes.io/projected/fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4-kube-api-access-jr66z\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:41:03.978754 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:03.978591 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4-istiod-ca-cert\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:41:03.978754 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:03.978602 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4-workload-certs\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:41:03.978754 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:03.978613 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4-workload-socket\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:41:03.978754 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:03.978623 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4-istio-data\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:41:03.978754 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:03.978634 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4-istio-token\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:41:03.978754 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:03.978646 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4-istio-envoy\") on node \"ip-10-0-138-0.ec2.internal\" DevicePath \"\"" Jul 09 07:41:04.024602 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:04.024572 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-w7hkl_d180add6-9fcc-42bd-a214-5b594e1a35ee/manager/0.log" Jul 09 07:41:04.147894 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:04.147823 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-t4lsx_56186d32-c8ce-464d-952a-e99ede3877fb/manager/0.log" Jul 09 07:41:04.382526 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:04.382501 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-n5xxf_b68a13c8-faf4-4dfb-bdfb-dc4cfab00314/server/0.log" Jul 09 07:41:04.508050 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:04.507959 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-dmzzk_57e6b153-89fc-4cd1-a2ad-19c575990d30/manager/0.log" Jul 09 07:41:04.527284 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:04.527237 2564 generic.go:356] "Generic (PLEG): container finished" podID="fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4" containerID="9f3119c0220b14299a0fb5b23e69993620a7f83fe691908eb98bed07d6279da3" exitCode=0 Jul 09 07:41:04.527458 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:04.527296 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7" event={"ID":"fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4","Type":"ContainerDied","Data":"9f3119c0220b14299a0fb5b23e69993620a7f83fe691908eb98bed07d6279da3"} Jul 09 07:41:04.527458 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:04.527320 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7" Jul 09 07:41:04.527458 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:04.527351 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7" event={"ID":"fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4","Type":"ContainerDied","Data":"a42294fa72bafe2d983e90c95777cf5f14edb496b42adf67749f266dd81dfff1"} Jul 09 07:41:04.527458 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:04.527372 2564 scope.go:117] "RemoveContainer" containerID="9f3119c0220b14299a0fb5b23e69993620a7f83fe691908eb98bed07d6279da3" Jul 09 07:41:04.536036 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:04.536023 2564 scope.go:117] "RemoveContainer" containerID="e57673c0cac015e1bf66b4be342c5f4bd71b88f2b1a18651503dbcb922af7279" Jul 09 07:41:04.543658 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:04.543637 2564 scope.go:117] "RemoveContainer" containerID="9f3119c0220b14299a0fb5b23e69993620a7f83fe691908eb98bed07d6279da3" Jul 09 07:41:04.543901 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:41:04.543882 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f3119c0220b14299a0fb5b23e69993620a7f83fe691908eb98bed07d6279da3\": container with ID starting with 9f3119c0220b14299a0fb5b23e69993620a7f83fe691908eb98bed07d6279da3 not found: ID does not exist" containerID="9f3119c0220b14299a0fb5b23e69993620a7f83fe691908eb98bed07d6279da3" Jul 09 07:41:04.543953 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:04.543909 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f3119c0220b14299a0fb5b23e69993620a7f83fe691908eb98bed07d6279da3"} err="failed to get container status \"9f3119c0220b14299a0fb5b23e69993620a7f83fe691908eb98bed07d6279da3\": rpc error: code = NotFound desc = could not find container \"9f3119c0220b14299a0fb5b23e69993620a7f83fe691908eb98bed07d6279da3\": container with ID starting with 9f3119c0220b14299a0fb5b23e69993620a7f83fe691908eb98bed07d6279da3 not found: ID does not exist" Jul 09 07:41:04.543953 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:04.543924 2564 scope.go:117] "RemoveContainer" containerID="e57673c0cac015e1bf66b4be342c5f4bd71b88f2b1a18651503dbcb922af7279" Jul 09 07:41:04.544135 ip-10-0-138-0 kubenswrapper[2564]: E0709 07:41:04.544120 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e57673c0cac015e1bf66b4be342c5f4bd71b88f2b1a18651503dbcb922af7279\": container with ID starting with e57673c0cac015e1bf66b4be342c5f4bd71b88f2b1a18651503dbcb922af7279 not found: ID does not exist" containerID="e57673c0cac015e1bf66b4be342c5f4bd71b88f2b1a18651503dbcb922af7279" Jul 09 07:41:04.544175 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:04.544139 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e57673c0cac015e1bf66b4be342c5f4bd71b88f2b1a18651503dbcb922af7279"} err="failed to get container status \"e57673c0cac015e1bf66b4be342c5f4bd71b88f2b1a18651503dbcb922af7279\": rpc error: code = NotFound desc = could not find container \"e57673c0cac015e1bf66b4be342c5f4bd71b88f2b1a18651503dbcb922af7279\": container with ID starting with e57673c0cac015e1bf66b4be342c5f4bd71b88f2b1a18651503dbcb922af7279 not found: ID does not exist" Jul 09 07:41:04.550252 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:04.550231 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7"] Jul 09 07:41:04.557029 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:04.557009 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-da4680-openshift-default-84df886fbc-kp2g7"] Jul 09 07:41:04.908586 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:04.908557 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b96c58b47-ws4xb_74cb03fe-bacc-4d19-971d-db6542495050/manager/0.log" Jul 09 07:41:04.915695 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:04.915670 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b96c58b47-ws4xb_74cb03fe-bacc-4d19-971d-db6542495050/copy-manifests/0.log" Jul 09 07:41:05.462457 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:05.462424 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4" path="/var/lib/kubelet/pods/fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4/volumes" Jul 09 07:41:07.066538 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:07.066502 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-xs7w2_84c308a6-03ac-4cd9-9e36-cb37b74957a3/manager/0.log" Jul 09 07:41:07.554022 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:07.553989 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-cwstw_742bbe04-c57f-4c45-b499-c0a0d4fa7b34/discovery/0.log" Jul 09 07:41:07.671407 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:07.671377 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f58fd5d48-5p9cd_f99957f6-8710-4530-9d05-c261c96547d8/kube-auth-proxy/0.log" Jul 09 07:41:08.040189 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:08.040160 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-mh754_b09281f9-6b47-4340-999f-5d9e3d19f782/payload-pre-processing/0.log" Jul 09 07:41:08.627203 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:08.627176 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-bkszb_f003ced9-bea8-4cf3-a8db-42512467bfb5/main/0.log" Jul 09 07:41:08.746192 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:08.746160 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-w8wfk_2f647a6c-f56c-4f9c-b79c-840ed7f57878/main/0.log" Jul 09 07:41:16.667116 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:16.667038 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2kgs9_2c7d2ce1-e4bb-4b4f-b4ad-fffb23b6f986/global-pull-secret-syncer/0.log" Jul 09 07:41:16.856632 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:16.856604 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-fcljc_a318fe5c-c8c8-4fff-b429-a7f4aa33b8c0/konnectivity-agent/0.log" Jul 09 07:41:16.926478 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:16.926404 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-0.ec2.internal_a0854266c70379aa8ce321ab152e516e/haproxy/0.log" Jul 09 07:41:22.548625 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:22.548590 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-xs7w2_84c308a6-03ac-4cd9-9e36-cb37b74957a3/manager/0.log" Jul 09 07:41:24.571991 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:24.571958 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-pnpsf_e5b9b919-a410-4786-acf2-8e5b052e0ff3/kube-state-metrics/0.log" Jul 09 07:41:24.594404 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:24.594375 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-pnpsf_e5b9b919-a410-4786-acf2-8e5b052e0ff3/kube-rbac-proxy-main/0.log" Jul 09 07:41:24.623639 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:24.623615 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-pnpsf_e5b9b919-a410-4786-acf2-8e5b052e0ff3/kube-rbac-proxy-self/0.log" Jul 09 07:41:24.690052 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:24.690025 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-frjxp_d713e588-85eb-4e22-93d5-414e9a28aa30/monitoring-plugin/0.log" Jul 09 07:41:24.916559 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:24.916531 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ql4f7_9b494327-fac4-4296-8418-e71c08fbac25/node-exporter/0.log" Jul 09 07:41:24.937769 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:24.937742 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ql4f7_9b494327-fac4-4296-8418-e71c08fbac25/kube-rbac-proxy/0.log" Jul 09 07:41:24.962354 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:24.962337 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ql4f7_9b494327-fac4-4296-8418-e71c08fbac25/init-textfile/0.log" Jul 09 07:41:24.991559 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:24.991537 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-7q8rc_43d4b98d-4299-469f-8fd3-90cafd9aa515/kube-rbac-proxy-main/0.log" Jul 09 07:41:25.015089 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:25.015071 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-7q8rc_43d4b98d-4299-469f-8fd3-90cafd9aa515/kube-rbac-proxy-self/0.log" Jul 09 07:41:25.041314 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:25.041290 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-7q8rc_43d4b98d-4299-469f-8fd3-90cafd9aa515/openshift-state-metrics/0.log" Jul 09 07:41:26.887536 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:26.887505 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-s54zf/perf-node-gather-daemonset-tl6ps"] Jul 09 07:41:26.887916 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:26.887827 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af7b1769-b214-4224-8ac6-58d2933a113d" containerName="istio-proxy" Jul 09 07:41:26.887916 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:26.887838 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="af7b1769-b214-4224-8ac6-58d2933a113d" containerName="istio-proxy" Jul 09 07:41:26.887916 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:26.887845 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4470e3e5-d2c2-4821-b880-eac72b2bf5d1" containerName="istio-proxy" Jul 09 07:41:26.887916 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:26.887850 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4470e3e5-d2c2-4821-b880-eac72b2bf5d1" containerName="istio-proxy" Jul 09 07:41:26.887916 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:26.887857 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4" containerName="istio-proxy" Jul 09 07:41:26.887916 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:26.887862 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4" containerName="istio-proxy" Jul 09 07:41:26.887916 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:26.887874 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4470e3e5-d2c2-4821-b880-eac72b2bf5d1" containerName="istio-proxy" Jul 09 07:41:26.887916 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:26.887880 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4470e3e5-d2c2-4821-b880-eac72b2bf5d1" containerName="istio-proxy" Jul 09 07:41:26.888165 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:26.887942 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4470e3e5-d2c2-4821-b880-eac72b2bf5d1" containerName="istio-proxy" Jul 09 07:41:26.888165 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:26.887950 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4470e3e5-d2c2-4821-b880-eac72b2bf5d1" containerName="istio-proxy" Jul 09 07:41:26.888165 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:26.887957 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe5ae6f0-2fe6-456f-b1fd-e05ecb4037a4" containerName="istio-proxy" Jul 09 07:41:26.892635 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:26.892613 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s54zf/perf-node-gather-daemonset-tl6ps" Jul 09 07:41:26.894867 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:26.894847 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-s54zf\"/\"default-dockercfg-tqqpq\"" Jul 09 07:41:26.894867 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:26.894861 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s54zf\"/\"kube-root-ca.crt\"" Jul 09 07:41:26.895784 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:26.895763 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s54zf\"/\"openshift-service-ca.crt\"" Jul 09 07:41:26.899650 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:26.899398 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s54zf/perf-node-gather-daemonset-tl6ps"] Jul 09 07:41:26.982449 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:26.982424 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07c4d1b2-02b2-4596-8606-6d29b8eeebd8-sys\") pod \"perf-node-gather-daemonset-tl6ps\" (UID: \"07c4d1b2-02b2-4596-8606-6d29b8eeebd8\") " pod="openshift-must-gather-s54zf/perf-node-gather-daemonset-tl6ps" Jul 09 07:41:26.982614 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:26.982486 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/07c4d1b2-02b2-4596-8606-6d29b8eeebd8-lib-modules\") pod \"perf-node-gather-daemonset-tl6ps\" (UID: \"07c4d1b2-02b2-4596-8606-6d29b8eeebd8\") " pod="openshift-must-gather-s54zf/perf-node-gather-daemonset-tl6ps" Jul 09 07:41:26.982614 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:26.982565 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/07c4d1b2-02b2-4596-8606-6d29b8eeebd8-podres\") pod \"perf-node-gather-daemonset-tl6ps\" (UID: \"07c4d1b2-02b2-4596-8606-6d29b8eeebd8\") " pod="openshift-must-gather-s54zf/perf-node-gather-daemonset-tl6ps" Jul 09 07:41:26.982714 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:26.982652 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffpz7\" (UniqueName: \"kubernetes.io/projected/07c4d1b2-02b2-4596-8606-6d29b8eeebd8-kube-api-access-ffpz7\") pod \"perf-node-gather-daemonset-tl6ps\" (UID: \"07c4d1b2-02b2-4596-8606-6d29b8eeebd8\") " pod="openshift-must-gather-s54zf/perf-node-gather-daemonset-tl6ps" Jul 09 07:41:26.982714 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:26.982680 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/07c4d1b2-02b2-4596-8606-6d29b8eeebd8-proc\") pod \"perf-node-gather-daemonset-tl6ps\" (UID: \"07c4d1b2-02b2-4596-8606-6d29b8eeebd8\") " pod="openshift-must-gather-s54zf/perf-node-gather-daemonset-tl6ps" Jul 09 07:41:27.083873 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:27.083841 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/07c4d1b2-02b2-4596-8606-6d29b8eeebd8-podres\") pod \"perf-node-gather-daemonset-tl6ps\" (UID: \"07c4d1b2-02b2-4596-8606-6d29b8eeebd8\") " pod="openshift-must-gather-s54zf/perf-node-gather-daemonset-tl6ps" Jul 09 07:41:27.084039 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:27.083897 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffpz7\" (UniqueName: \"kubernetes.io/projected/07c4d1b2-02b2-4596-8606-6d29b8eeebd8-kube-api-access-ffpz7\") pod \"perf-node-gather-daemonset-tl6ps\" (UID: \"07c4d1b2-02b2-4596-8606-6d29b8eeebd8\") " pod="openshift-must-gather-s54zf/perf-node-gather-daemonset-tl6ps" Jul 09 07:41:27.084039 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:27.083915 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/07c4d1b2-02b2-4596-8606-6d29b8eeebd8-proc\") pod \"perf-node-gather-daemonset-tl6ps\" (UID: \"07c4d1b2-02b2-4596-8606-6d29b8eeebd8\") " pod="openshift-must-gather-s54zf/perf-node-gather-daemonset-tl6ps" Jul 09 07:41:27.084039 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:27.083944 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07c4d1b2-02b2-4596-8606-6d29b8eeebd8-sys\") pod \"perf-node-gather-daemonset-tl6ps\" (UID: \"07c4d1b2-02b2-4596-8606-6d29b8eeebd8\") " pod="openshift-must-gather-s54zf/perf-node-gather-daemonset-tl6ps" Jul 09 07:41:27.084039 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:27.084004 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/07c4d1b2-02b2-4596-8606-6d29b8eeebd8-proc\") pod \"perf-node-gather-daemonset-tl6ps\" (UID: \"07c4d1b2-02b2-4596-8606-6d29b8eeebd8\") " pod="openshift-must-gather-s54zf/perf-node-gather-daemonset-tl6ps" Jul 09 07:41:27.084039 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:27.084003 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/07c4d1b2-02b2-4596-8606-6d29b8eeebd8-lib-modules\") pod \"perf-node-gather-daemonset-tl6ps\" (UID: \"07c4d1b2-02b2-4596-8606-6d29b8eeebd8\") " pod="openshift-must-gather-s54zf/perf-node-gather-daemonset-tl6ps" Jul 09 07:41:27.084039 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:27.084027 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/07c4d1b2-02b2-4596-8606-6d29b8eeebd8-podres\") pod \"perf-node-gather-daemonset-tl6ps\" (UID: \"07c4d1b2-02b2-4596-8606-6d29b8eeebd8\") " pod="openshift-must-gather-s54zf/perf-node-gather-daemonset-tl6ps" Jul 09 07:41:27.084280 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:27.084071 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07c4d1b2-02b2-4596-8606-6d29b8eeebd8-sys\") pod \"perf-node-gather-daemonset-tl6ps\" (UID: \"07c4d1b2-02b2-4596-8606-6d29b8eeebd8\") " pod="openshift-must-gather-s54zf/perf-node-gather-daemonset-tl6ps" Jul 09 07:41:27.084280 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:27.084096 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/07c4d1b2-02b2-4596-8606-6d29b8eeebd8-lib-modules\") pod \"perf-node-gather-daemonset-tl6ps\" (UID: \"07c4d1b2-02b2-4596-8606-6d29b8eeebd8\") " pod="openshift-must-gather-s54zf/perf-node-gather-daemonset-tl6ps" Jul 09 07:41:27.091691 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:27.091663 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffpz7\" (UniqueName: \"kubernetes.io/projected/07c4d1b2-02b2-4596-8606-6d29b8eeebd8-kube-api-access-ffpz7\") pod \"perf-node-gather-daemonset-tl6ps\" (UID: \"07c4d1b2-02b2-4596-8606-6d29b8eeebd8\") " pod="openshift-must-gather-s54zf/perf-node-gather-daemonset-tl6ps" Jul 09 07:41:27.203343 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:27.203227 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s54zf/perf-node-gather-daemonset-tl6ps" Jul 09 07:41:27.335249 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:27.335222 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s54zf/perf-node-gather-daemonset-tl6ps"] Jul 09 07:41:27.337505 ip-10-0-138-0 kubenswrapper[2564]: W0709 07:41:27.337469 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod07c4d1b2_02b2_4596_8606_6d29b8eeebd8.slice/crio-1c2bd2d5cbd101e367a34cd94c77146a973a34302551ef89b16248b10ebccf9f WatchSource:0}: Error finding container 1c2bd2d5cbd101e367a34cd94c77146a973a34302551ef89b16248b10ebccf9f: Status 404 returned error can't find the container with id 1c2bd2d5cbd101e367a34cd94c77146a973a34302551ef89b16248b10ebccf9f Jul 09 07:41:27.648759 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:27.648725 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s54zf/perf-node-gather-daemonset-tl6ps" event={"ID":"07c4d1b2-02b2-4596-8606-6d29b8eeebd8","Type":"ContainerStarted","Data":"18343f70dcfaa266c3194c919a6f8c7cf077914cad9f1be326b32951f0e4a56e"} Jul 09 07:41:27.648929 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:27.648777 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s54zf/perf-node-gather-daemonset-tl6ps" event={"ID":"07c4d1b2-02b2-4596-8606-6d29b8eeebd8","Type":"ContainerStarted","Data":"1c2bd2d5cbd101e367a34cd94c77146a973a34302551ef89b16248b10ebccf9f"} Jul 09 07:41:27.648929 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:27.648866 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-s54zf/perf-node-gather-daemonset-tl6ps" Jul 09 07:41:27.665306 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:27.665241 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-s54zf/perf-node-gather-daemonset-tl6ps" podStartSLOduration=1.6652276339999998 podStartE2EDuration="1.665227634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 07:41:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 07:41:27.662935619 +0000 UTC m=+2346.714012847" watchObservedRunningTime="2026-07-09 07:41:27.665227634 +0000 UTC m=+2346.716304893" Jul 09 07:41:29.616660 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:29.616622 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sd74f_99def812-f7ca-4ac1-b0b9-227eb153b1bb/dns/0.log" Jul 09 07:41:29.639381 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:29.639356 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sd74f_99def812-f7ca-4ac1-b0b9-227eb153b1bb/kube-rbac-proxy/0.log" Jul 09 07:41:29.687458 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:29.687429 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tk76r_9a204676-b17e-4b4b-aeed-d3c3e611f9e3/dns-node-resolver/0.log" Jul 09 07:41:30.308146 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:30.308114 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gxqds_7bac4227-dd99-4984-aa2e-efeaa7a04755/node-ca/0.log" Jul 09 07:41:31.483352 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:31.483324 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-cwstw_742bbe04-c57f-4c45-b499-c0a0d4fa7b34/discovery/0.log" Jul 09 07:41:31.504589 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:31.504564 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f58fd5d48-5p9cd_f99957f6-8710-4530-9d05-c261c96547d8/kube-auth-proxy/0.log" Jul 09 07:41:31.690460 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:31.690434 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-mh754_b09281f9-6b47-4340-999f-5d9e3d19f782/payload-pre-processing/0.log" Jul 09 07:41:32.470178 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:32.470137 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8wqzv_e5d54ab9-973e-4ed3-b78d-d90373e6264a/serve-healthcheck-canary/0.log" Jul 09 07:41:33.097591 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:33.097563 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-w9slp_7d6b67d6-76c8-4abc-a51b-3d7e326051d7/kube-rbac-proxy/0.log" Jul 09 07:41:33.120937 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:33.120917 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-w9slp_7d6b67d6-76c8-4abc-a51b-3d7e326051d7/exporter/0.log" Jul 09 07:41:33.143329 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:33.143306 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-w9slp_7d6b67d6-76c8-4abc-a51b-3d7e326051d7/extractor/0.log" Jul 09 07:41:33.661942 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:33.661909 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-s54zf/perf-node-gather-daemonset-tl6ps" Jul 09 07:41:35.645377 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:35.645348 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-w7hkl_d180add6-9fcc-42bd-a214-5b594e1a35ee/manager/0.log" Jul 09 07:41:35.826447 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:35.826339 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-t4lsx_56186d32-c8ce-464d-952a-e99ede3877fb/manager/0.log" Jul 09 07:41:35.990933 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:35.990872 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-n5xxf_b68a13c8-faf4-4dfb-bdfb-dc4cfab00314/server/0.log" Jul 09 07:41:36.084133 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:36.084036 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-dmzzk_57e6b153-89fc-4cd1-a2ad-19c575990d30/manager/0.log" Jul 09 07:41:36.242671 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:36.242643 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b96c58b47-ws4xb_74cb03fe-bacc-4d19-971d-db6542495050/manager/0.log" Jul 09 07:41:36.263102 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:36.263077 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b96c58b47-ws4xb_74cb03fe-bacc-4d19-971d-db6542495050/copy-manifests/0.log" Jul 09 07:41:37.663592 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:37.663494 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-hjbpw_04588e70-1de7-4a10-8c25-a47e1484b076/openshift-lws-operator/0.log" Jul 09 07:41:37.708135 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:37.708105 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-cwstw_742bbe04-c57f-4c45-b499-c0a0d4fa7b34/discovery/0.log" Jul 09 07:41:37.720787 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:37.720749 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f58fd5d48-5p9cd_f99957f6-8710-4530-9d05-c261c96547d8/kube-auth-proxy/0.log" Jul 09 07:41:37.864915 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:37.864888 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-mh754_b09281f9-6b47-4340-999f-5d9e3d19f782/payload-pre-processing/0.log" Jul 09 07:41:39.244511 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:39.244477 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-cwstw_742bbe04-c57f-4c45-b499-c0a0d4fa7b34/discovery/0.log" Jul 09 07:41:39.258616 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:39.258588 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f58fd5d48-5p9cd_f99957f6-8710-4530-9d05-c261c96547d8/kube-auth-proxy/0.log" Jul 09 07:41:39.437160 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:39.437120 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-mh754_b09281f9-6b47-4340-999f-5d9e3d19f782/payload-pre-processing/0.log" Jul 09 07:41:42.600062 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:42.600033 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-7knwb_5e203da9-5aba-4c20-a42c-45b5cf360ab3/migrator/0.log" Jul 09 07:41:42.623186 ip-10-0-138-0 kubenswrapper[2564]: I0709 07:41:42.623161 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-7knwb_5e203da9-5aba-4c20-a42c-45b5cf360ab3/graceful-termination/0.log"