Jul 10 14:30:54.124041 ip-10-0-140-35 systemd[1]: Starting Kubernetes Kubelet... Jul 10 14:30:54.618022 ip-10-0-140-35 kubenswrapper[2569]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 14:30:54.618022 ip-10-0-140-35 kubenswrapper[2569]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 10 14:30:54.618022 ip-10-0-140-35 kubenswrapper[2569]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 14:30:54.618022 ip-10-0-140-35 kubenswrapper[2569]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 10 14:30:54.618022 ip-10-0-140-35 kubenswrapper[2569]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 14:30:54.618022 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.535666 2569 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 10 14:30:54.618022 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540570 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 14:30:54.618022 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540584 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 14:30:54.618022 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540588 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 14:30:54.618022 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540591 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 14:30:54.618022 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540595 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 14:30:54.618022 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540598 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 14:30:54.618022 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540601 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 14:30:54.618022 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540604 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 14:30:54.615797 ip-10-0-140-35 systemd[1]: Started Kubernetes Kubelet. Jul 10 14:30:54.620003 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540607 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 14:30:54.620003 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540609 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 14:30:54.620003 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540612 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 14:30:54.620003 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540615 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 14:30:54.620003 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540618 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 14:30:54.620003 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540621 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 14:30:54.620003 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540624 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 14:30:54.620003 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540627 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 14:30:54.620003 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540630 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 14:30:54.620003 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540633 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 14:30:54.620003 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540636 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 14:30:54.620003 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540638 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 14:30:54.620003 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540641 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 14:30:54.620003 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540644 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 14:30:54.620003 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540651 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 14:30:54.620003 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540654 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 14:30:54.620003 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540658 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 14:30:54.620003 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540662 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 14:30:54.620003 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540665 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 14:30:54.635025 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540668 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 14:30:54.635025 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540673 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 14:30:54.635025 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540677 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 14:30:54.635025 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540680 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 14:30:54.635025 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540683 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 14:30:54.635025 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540686 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 14:30:54.635025 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540689 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 14:30:54.635025 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540692 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 14:30:54.635025 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540695 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 10 14:30:54.635025 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540698 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 14:30:54.635025 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540701 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 14:30:54.635025 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540704 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 14:30:54.635025 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540706 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 14:30:54.635025 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540709 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 14:30:54.635025 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540712 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 14:30:54.635025 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540714 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 14:30:54.635025 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540717 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 14:30:54.635025 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540720 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 14:30:54.635025 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540722 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 14:30:54.635025 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540725 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 14:30:54.638132 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540728 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 14:30:54.638132 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540730 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 14:30:54.638132 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540733 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 14:30:54.638132 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540735 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 14:30:54.638132 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540738 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 14:30:54.638132 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540741 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 14:30:54.638132 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540743 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 14:30:54.638132 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540746 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 14:30:54.638132 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540765 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 14:30:54.638132 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540769 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 14:30:54.638132 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540773 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 14:30:54.638132 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540776 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 14:30:54.638132 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540779 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 14:30:54.638132 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540782 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 14:30:54.638132 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540785 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 14:30:54.638132 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540787 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 14:30:54.638132 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540790 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 14:30:54.638132 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540793 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 14:30:54.638132 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540796 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 14:30:54.638132 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540799 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 14:30:54.650836 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540801 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 14:30:54.650836 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540804 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 14:30:54.650836 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540807 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 14:30:54.650836 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540809 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 14:30:54.650836 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540812 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 14:30:54.650836 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540815 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 14:30:54.650836 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540817 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 14:30:54.650836 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540820 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 14:30:54.650836 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540823 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 14:30:54.650836 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540826 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 14:30:54.650836 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540829 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 14:30:54.650836 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540832 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 14:30:54.650836 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540835 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 14:30:54.650836 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540837 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 14:30:54.650836 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540840 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 14:30:54.650836 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540844 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 14:30:54.650836 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540847 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 14:30:54.650836 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540850 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 14:30:54.650836 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540853 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 14:30:54.650836 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540855 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 14:30:54.653814 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540858 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 14:30:54.653814 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540860 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 14:30:54.653814 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540863 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 14:30:54.653814 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540865 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 14:30:54.653814 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540868 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 14:30:54.653814 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540871 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 14:30:54.653814 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540873 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 14:30:54.653814 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540876 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 14:30:54.653814 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540879 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 14:30:54.653814 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540883 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 14:30:54.653814 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540886 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 14:30:54.653814 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.540889 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 14:30:54.653814 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.541546 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 14:30:54.653814 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.541552 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 14:30:54.653814 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.541556 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 14:30:54.653814 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.541559 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 14:30:54.653814 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.541562 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 14:30:54.653814 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.541564 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 14:30:54.653814 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.541567 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 14:30:54.653814 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.541570 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 14:30:54.802780 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.541572 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 14:30:54.802780 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.541575 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 14:30:54.802780 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.541577 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 14:30:54.802780 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.541580 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 14:30:54.802780 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.541583 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 14:30:54.802780 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.541585 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 14:30:54.802780 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.541588 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 14:30:54.802780 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.541591 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 14:30:54.802780 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.541594 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 14:30:54.802780 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.541597 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 14:30:54.802780 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.541600 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 14:30:54.802780 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.541602 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 14:30:54.802780 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.541605 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 14:30:54.802780 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.541607 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 14:30:54.802780 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.541610 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 14:30:54.802780 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.541617 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 14:30:54.802780 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.541621 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 14:30:54.802780 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.541624 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 14:30:54.802780 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.541626 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 14:30:54.802780 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.541629 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 14:30:54.805228 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.541633 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 14:30:54.805228 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.541635 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 14:30:54.805228 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.541638 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 14:30:54.805228 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.541641 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 14:30:54.805228 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.541643 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 14:30:54.805228 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.541646 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 14:30:54.805228 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.541649 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 14:30:54.805228 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.541651 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 14:30:54.805228 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.541654 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 14:30:54.805228 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.541656 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 14:30:54.805228 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.541660 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 14:30:54.805228 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.541663 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 14:30:54.805228 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.541666 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 14:30:54.805228 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.541669 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 14:30:54.805228 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.541671 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 14:30:54.805228 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.541674 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 14:30:54.805228 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.541676 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 14:30:54.805228 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.541679 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 14:30:54.805228 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.541681 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 14:30:54.807068 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.541684 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 14:30:54.807068 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.541687 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 14:30:54.807068 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.541690 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 14:30:54.807068 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.541693 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 14:30:54.807068 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.541695 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 14:30:54.807068 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.541698 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 14:30:54.807068 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.541702 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 14:30:54.807068 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.541706 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 14:30:54.807068 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.541710 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 14:30:54.807068 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.541713 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 14:30:54.807068 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.541715 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 14:30:54.807068 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.541728 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 14:30:54.807068 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.541793 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 14:30:54.807068 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.541808 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 14:30:54.807068 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.542014 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 14:30:54.807068 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.542017 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 14:30:54.807068 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.542020 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 14:30:54.807068 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.542023 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 14:30:54.807068 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.542168 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 14:30:54.807068 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.542225 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 14:30:54.810952 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.542228 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 14:30:54.810952 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.542231 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 14:30:54.810952 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.542234 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 14:30:54.810952 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.542237 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 14:30:54.810952 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.542343 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 14:30:54.810952 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.542440 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 10 14:30:54.810952 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.542443 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 14:30:54.810952 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.542447 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 14:30:54.810952 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.542450 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 14:30:54.810952 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.542453 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 14:30:54.810952 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.542653 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 14:30:54.810952 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.542655 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 14:30:54.810952 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.542658 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 14:30:54.810952 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.542660 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 14:30:54.810952 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.542664 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 14:30:54.810952 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.542876 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 14:30:54.810952 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.542996 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 14:30:54.810952 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.543000 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 14:30:54.810952 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.543003 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 14:30:54.824325 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.543005 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 14:30:54.824325 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.543009 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 14:30:54.824325 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.543012 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 14:30:54.824325 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.543014 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 14:30:54.824325 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.543018 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 14:30:54.824325 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.543020 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 14:30:54.824325 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.543024 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 14:30:54.824325 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.543026 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 14:30:54.824325 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.543029 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 14:30:54.824325 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.543031 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 14:30:54.824325 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.543034 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 14:30:54.824325 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.543036 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 14:30:54.824325 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.543039 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 14:30:54.824325 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544312 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 14:30:54.824325 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544319 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 14:30:54.824325 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544323 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 14:30:54.824325 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544326 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 14:30:54.824325 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544328 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 14:30:54.824325 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544332 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 14:30:54.824325 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544335 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 14:30:54.826617 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544339 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 14:30:54.826617 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544344 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 14:30:54.826617 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544348 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 14:30:54.826617 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544351 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 14:30:54.826617 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544356 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 14:30:54.826617 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544359 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 14:30:54.826617 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544361 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 14:30:54.826617 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544364 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 14:30:54.826617 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544367 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 14:30:54.826617 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544371 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 14:30:54.826617 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544374 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 14:30:54.826617 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544378 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 14:30:54.826617 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544381 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 14:30:54.826617 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544386 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 14:30:54.826617 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544390 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 14:30:54.826617 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544394 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 10 14:30:54.826617 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544396 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 14:30:54.826617 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544399 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 14:30:54.826617 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544403 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 14:30:54.829131 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544406 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 14:30:54.829131 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544409 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 14:30:54.829131 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544412 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 14:30:54.829131 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544415 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 14:30:54.829131 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544419 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 14:30:54.829131 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544421 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 14:30:54.829131 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544424 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 14:30:54.829131 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544429 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 14:30:54.829131 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544431 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 14:30:54.829131 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544434 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 14:30:54.829131 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544438 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 14:30:54.829131 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544442 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 14:30:54.829131 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544445 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 14:30:54.829131 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544449 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 14:30:54.829131 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544452 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 14:30:54.829131 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544454 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 14:30:54.829131 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544457 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 14:30:54.829131 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544460 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 14:30:54.829131 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544464 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 14:30:54.829131 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544466 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 14:30:54.829903 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544469 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 14:30:54.829903 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544471 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 14:30:54.829903 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544474 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 14:30:54.829903 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544477 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 14:30:54.829903 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544479 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 14:30:54.829903 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544484 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 14:30:54.829903 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544487 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 14:30:54.829903 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544490 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 14:30:54.829903 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544493 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 14:30:54.829903 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544495 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 14:30:54.829903 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544498 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 14:30:54.829903 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544500 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 14:30:54.829903 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544503 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 14:30:54.829903 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544505 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 14:30:54.829903 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544510 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 14:30:54.829903 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544512 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 14:30:54.829903 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544515 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 14:30:54.829903 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544517 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 14:30:54.829903 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544520 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 14:30:54.829903 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544523 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 14:30:54.832058 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544525 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 14:30:54.832058 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544528 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 14:30:54.832058 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544530 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 14:30:54.832058 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544532 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 14:30:54.832058 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544535 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 14:30:54.832058 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544537 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 14:30:54.832058 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544541 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 14:30:54.832058 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544543 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 14:30:54.832058 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544548 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 14:30:54.832058 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544551 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 14:30:54.832058 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544553 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 14:30:54.832058 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544556 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 14:30:54.832058 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544559 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 14:30:54.832058 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544561 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 14:30:54.832058 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544564 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 14:30:54.832058 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544566 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 14:30:54.832058 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544569 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 14:30:54.832058 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544573 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 14:30:54.832058 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544577 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 14:30:54.833507 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544579 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 14:30:54.833507 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544582 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 14:30:54.833507 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544584 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 14:30:54.833507 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544586 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 14:30:54.833507 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544589 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 14:30:54.833507 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544592 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 14:30:54.833507 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544594 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 14:30:54.833507 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544596 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 14:30:54.833507 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544599 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 14:30:54.833507 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544601 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 14:30:54.833507 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544604 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 14:30:54.833507 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544606 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 14:30:54.833507 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544609 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 14:30:54.833507 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.544611 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 14:30:54.833507 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544686 2569 flags.go:64] FLAG: --address="0.0.0.0" Jul 10 14:30:54.833507 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544701 2569 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 10 14:30:54.833507 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544710 2569 flags.go:64] FLAG: --anonymous-auth="true" Jul 10 14:30:54.833507 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544716 2569 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 10 14:30:54.833507 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544722 2569 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 10 14:30:54.833507 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544727 2569 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 10 14:30:54.833507 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544734 2569 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 10 14:30:54.834375 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544739 2569 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 10 14:30:54.834375 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544742 2569 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 10 14:30:54.834375 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544745 2569 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 10 14:30:54.834375 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544761 2569 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 10 14:30:54.834375 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544765 2569 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 10 14:30:54.834375 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544768 2569 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 10 14:30:54.834375 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544771 2569 flags.go:64] FLAG: --cgroup-root="" Jul 10 14:30:54.834375 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544774 2569 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 10 14:30:54.834375 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544777 2569 flags.go:64] FLAG: --client-ca-file="" Jul 10 14:30:54.834375 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544782 2569 flags.go:64] FLAG: --cloud-provider="external" Jul 10 14:30:54.834375 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544786 2569 flags.go:64] FLAG: --cluster-dns="[]" Jul 10 14:30:54.834375 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544790 2569 flags.go:64] FLAG: --cluster-domain="" Jul 10 14:30:54.834375 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544793 2569 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 10 14:30:54.834375 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544796 2569 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 10 14:30:54.834375 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544800 2569 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 10 14:30:54.834375 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544803 2569 flags.go:64] FLAG: --container-log-max-files="5" Jul 10 14:30:54.834375 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544807 2569 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 10 14:30:54.834375 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544810 2569 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 10 14:30:54.834375 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544813 2569 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 10 14:30:54.834375 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544816 2569 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 10 14:30:54.834375 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544819 2569 flags.go:64] FLAG: --contention-profiling="false" Jul 10 14:30:54.834375 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544822 2569 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 10 14:30:54.834375 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544825 2569 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 10 14:30:54.834375 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544828 2569 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 10 14:30:54.835197 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544830 2569 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 10 14:30:54.835197 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544834 2569 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 10 14:30:54.835197 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544837 2569 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 10 14:30:54.835197 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544840 2569 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 10 14:30:54.835197 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544842 2569 flags.go:64] FLAG: --enable-load-reader="false" Jul 10 14:30:54.835197 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544846 2569 flags.go:64] FLAG: --enable-server="true" Jul 10 14:30:54.835197 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544849 2569 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 10 14:30:54.835197 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544854 2569 flags.go:64] FLAG: --event-burst="100" Jul 10 14:30:54.835197 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544857 2569 flags.go:64] FLAG: --event-qps="50" Jul 10 14:30:54.835197 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544860 2569 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 10 14:30:54.835197 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544863 2569 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 10 14:30:54.835197 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544866 2569 flags.go:64] FLAG: --eviction-hard="" Jul 10 14:30:54.835197 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544869 2569 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 10 14:30:54.835197 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544872 2569 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 10 14:30:54.835197 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544875 2569 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 10 14:30:54.835197 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544878 2569 flags.go:64] FLAG: --eviction-soft="" Jul 10 14:30:54.835197 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544881 2569 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 10 14:30:54.835197 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544886 2569 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 10 14:30:54.835197 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544891 2569 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 10 14:30:54.835197 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544895 2569 flags.go:64] FLAG: --experimental-mounter-path="" Jul 10 14:30:54.835197 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544897 2569 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 10 14:30:54.835197 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544900 2569 flags.go:64] FLAG: --fail-swap-on="true" Jul 10 14:30:54.835197 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544903 2569 flags.go:64] FLAG: --feature-gates="" Jul 10 14:30:54.835197 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544907 2569 flags.go:64] FLAG: --file-check-frequency="20s" Jul 10 14:30:54.835197 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544910 2569 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 10 14:30:54.836025 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544913 2569 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 10 14:30:54.836025 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544916 2569 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 10 14:30:54.836025 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544918 2569 flags.go:64] FLAG: --healthz-port="10248" Jul 10 14:30:54.836025 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544921 2569 flags.go:64] FLAG: --help="false" Jul 10 14:30:54.836025 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544924 2569 flags.go:64] FLAG: --hostname-override="ip-10-0-140-35.ec2.internal" Jul 10 14:30:54.836025 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544927 2569 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 10 14:30:54.836025 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544931 2569 flags.go:64] FLAG: --http-check-frequency="20s" Jul 10 14:30:54.836025 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544933 2569 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 10 14:30:54.836025 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544937 2569 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 10 14:30:54.836025 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544940 2569 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 10 14:30:54.836025 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544943 2569 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 10 14:30:54.836025 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544945 2569 flags.go:64] FLAG: --image-service-endpoint="" Jul 10 14:30:54.836025 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544948 2569 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 10 14:30:54.836025 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544951 2569 flags.go:64] FLAG: --kube-api-burst="100" Jul 10 14:30:54.836025 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544954 2569 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 10 14:30:54.836025 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544957 2569 flags.go:64] FLAG: --kube-api-qps="50" Jul 10 14:30:54.836025 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544960 2569 flags.go:64] FLAG: --kube-reserved="" Jul 10 14:30:54.836025 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544963 2569 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 10 14:30:54.836025 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544966 2569 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 10 14:30:54.836025 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544969 2569 flags.go:64] FLAG: --kubelet-cgroups="" Jul 10 14:30:54.836025 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544972 2569 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 10 14:30:54.836025 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544974 2569 flags.go:64] FLAG: --lock-file="" Jul 10 14:30:54.836025 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544977 2569 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 10 14:30:54.836025 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544980 2569 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 10 14:30:54.837040 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544984 2569 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 10 14:30:54.837040 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544990 2569 flags.go:64] FLAG: --log-json-split-stream="false" Jul 10 14:30:54.837040 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544994 2569 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 10 14:30:54.837040 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.544997 2569 flags.go:64] FLAG: --log-text-split-stream="false" Jul 10 14:30:54.837040 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545000 2569 flags.go:64] FLAG: --logging-format="text" Jul 10 14:30:54.837040 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545003 2569 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 10 14:30:54.837040 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545006 2569 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 10 14:30:54.837040 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545009 2569 flags.go:64] FLAG: --manifest-url="" Jul 10 14:30:54.837040 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545012 2569 flags.go:64] FLAG: --manifest-url-header="" Jul 10 14:30:54.837040 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545016 2569 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 10 14:30:54.837040 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545019 2569 flags.go:64] FLAG: --max-open-files="1000000" Jul 10 14:30:54.837040 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545023 2569 flags.go:64] FLAG: --max-pods="110" Jul 10 14:30:54.837040 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545026 2569 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 10 14:30:54.837040 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545029 2569 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 10 14:30:54.837040 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545032 2569 flags.go:64] FLAG: --memory-manager-policy="None" Jul 10 14:30:54.837040 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545034 2569 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 10 14:30:54.837040 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545037 2569 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 10 14:30:54.837040 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545040 2569 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 10 14:30:54.837040 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545043 2569 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 10 14:30:54.837040 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545050 2569 flags.go:64] FLAG: --node-status-max-images="50" Jul 10 14:30:54.837040 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545053 2569 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 10 14:30:54.837040 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545056 2569 flags.go:64] FLAG: --oom-score-adj="-999" Jul 10 14:30:54.837040 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545059 2569 flags.go:64] FLAG: --pod-cidr="" Jul 10 14:30:54.838616 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545062 2569 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 10 14:30:54.838616 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545066 2569 flags.go:64] FLAG: --pod-manifest-path="" Jul 10 14:30:54.838616 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545069 2569 flags.go:64] FLAG: --pod-max-pids="-1" Jul 10 14:30:54.838616 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545072 2569 flags.go:64] FLAG: --pods-per-core="0" Jul 10 14:30:54.838616 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545075 2569 flags.go:64] FLAG: --port="10250" Jul 10 14:30:54.838616 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545078 2569 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 10 14:30:54.838616 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545081 2569 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d117652ba27b3ebe" Jul 10 14:30:54.838616 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545084 2569 flags.go:64] FLAG: --qos-reserved="" Jul 10 14:30:54.838616 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545087 2569 flags.go:64] FLAG: --read-only-port="10255" Jul 10 14:30:54.838616 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545090 2569 flags.go:64] FLAG: --register-node="true" Jul 10 14:30:54.838616 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545093 2569 flags.go:64] FLAG: --register-with-taints="" Jul 10 14:30:54.838616 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545100 2569 flags.go:64] FLAG: --registry-burst="10" Jul 10 14:30:54.838616 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545104 2569 flags.go:64] FLAG: --registry-qps="5" Jul 10 14:30:54.838616 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545107 2569 flags.go:64] FLAG: --reserved-cpus="" Jul 10 14:30:54.838616 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545109 2569 flags.go:64] FLAG: --reserved-memory="" Jul 10 14:30:54.838616 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545113 2569 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 10 14:30:54.838616 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545116 2569 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 10 14:30:54.838616 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545119 2569 flags.go:64] FLAG: --rotate-certificates="false" Jul 10 14:30:54.838616 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545122 2569 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 10 14:30:54.838616 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545124 2569 flags.go:64] FLAG: --runonce="false" Jul 10 14:30:54.838616 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545127 2569 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 10 14:30:54.838616 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545130 2569 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 10 14:30:54.838616 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545133 2569 flags.go:64] FLAG: --seccomp-default="false" Jul 10 14:30:54.838616 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545136 2569 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 10 14:30:54.838616 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545139 2569 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 10 14:30:54.839333 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545142 2569 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 10 14:30:54.839333 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545145 2569 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 10 14:30:54.839333 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545148 2569 flags.go:64] FLAG: --storage-driver-password="root" Jul 10 14:30:54.839333 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545150 2569 flags.go:64] FLAG: --storage-driver-secure="false" Jul 10 14:30:54.839333 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545153 2569 flags.go:64] FLAG: --storage-driver-table="stats" Jul 10 14:30:54.839333 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545156 2569 flags.go:64] FLAG: --storage-driver-user="root" Jul 10 14:30:54.839333 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545159 2569 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 10 14:30:54.839333 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545162 2569 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 10 14:30:54.839333 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545165 2569 flags.go:64] FLAG: --system-cgroups="" Jul 10 14:30:54.839333 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545167 2569 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 10 14:30:54.839333 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545173 2569 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 10 14:30:54.839333 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545176 2569 flags.go:64] FLAG: --tls-cert-file="" Jul 10 14:30:54.839333 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545179 2569 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 10 14:30:54.839333 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545182 2569 flags.go:64] FLAG: --tls-min-version="" Jul 10 14:30:54.839333 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545185 2569 flags.go:64] FLAG: --tls-private-key-file="" Jul 10 14:30:54.839333 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545188 2569 flags.go:64] FLAG: --topology-manager-policy="none" Jul 10 14:30:54.839333 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545192 2569 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 10 14:30:54.839333 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545195 2569 flags.go:64] FLAG: --topology-manager-scope="container" Jul 10 14:30:54.839333 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545199 2569 flags.go:64] FLAG: --v="2" Jul 10 14:30:54.839333 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545203 2569 flags.go:64] FLAG: --version="false" Jul 10 14:30:54.839333 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545207 2569 flags.go:64] FLAG: --vmodule="" Jul 10 14:30:54.839333 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545211 2569 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 10 14:30:54.839333 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545214 2569 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 10 14:30:54.839333 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545328 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 14:30:54.839333 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545332 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 14:30:55.075253 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545335 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 14:30:55.075253 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545338 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 14:30:55.075253 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545341 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 14:30:55.075253 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545344 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 14:30:55.075253 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545347 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 14:30:55.075253 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545350 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 14:30:55.075253 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545353 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 14:30:55.075253 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545355 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 14:30:55.075253 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545358 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 14:30:55.075253 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545360 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 14:30:55.075253 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545363 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 14:30:55.075253 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545366 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 14:30:55.075253 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545369 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 14:30:55.075253 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545371 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 14:30:55.075253 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545374 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 14:30:55.075253 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545377 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 14:30:55.075253 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545381 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 14:30:55.075253 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545384 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 14:30:55.075253 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545387 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 14:30:55.076032 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545389 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 14:30:55.076032 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545392 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 14:30:55.076032 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545394 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 14:30:55.076032 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545397 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 14:30:55.076032 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545401 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 14:30:55.076032 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545403 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 14:30:55.076032 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545407 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 14:30:55.076032 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545410 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 14:30:55.076032 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545412 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 14:30:55.076032 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545415 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 14:30:55.076032 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545418 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 14:30:55.076032 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545421 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 14:30:55.076032 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545423 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 14:30:55.076032 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545426 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 14:30:55.076032 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545428 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 14:30:55.076032 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545431 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 14:30:55.076032 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545433 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 14:30:55.076032 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545436 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 14:30:55.076032 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545439 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 14:30:55.076032 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545441 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 14:30:55.076635 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545444 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 14:30:55.076635 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545446 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 14:30:55.076635 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545448 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 14:30:55.076635 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545451 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 14:30:55.076635 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545453 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 14:30:55.076635 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545456 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 14:30:55.076635 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545458 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 14:30:55.076635 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545461 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 14:30:55.076635 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545463 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 14:30:55.076635 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545466 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 14:30:55.076635 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545468 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 14:30:55.076635 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545471 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 14:30:55.076635 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545474 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 14:30:55.076635 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545476 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 14:30:55.076635 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545479 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 14:30:55.076635 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545481 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 14:30:55.076635 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545485 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 14:30:55.076635 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545487 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 14:30:55.076635 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545491 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 14:30:55.076635 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545494 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 14:30:55.077220 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545496 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 14:30:55.077220 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545499 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 14:30:55.077220 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545502 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 14:30:55.077220 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545505 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 14:30:55.077220 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545509 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 14:30:55.077220 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545512 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 14:30:55.077220 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545515 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 14:30:55.077220 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545518 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 14:30:55.077220 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545521 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 14:30:55.077220 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545524 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 14:30:55.077220 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545526 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 14:30:55.077220 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545529 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 14:30:55.077220 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545531 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 14:30:55.077220 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545534 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 14:30:55.077220 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545536 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 14:30:55.077220 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545539 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 14:30:55.077220 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545541 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 14:30:55.077220 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545544 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 14:30:55.077220 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545546 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 14:30:55.077220 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545549 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 10 14:30:55.077779 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545552 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 14:30:55.077779 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545554 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 14:30:55.077779 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545557 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 14:30:55.077779 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545559 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 14:30:55.077779 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545562 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 14:30:55.077779 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545564 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 14:30:55.077779 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545567 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 14:30:55.077779 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545569 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 14:30:55.077779 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545574 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 14:30:55.077779 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545578 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 14:30:55.077779 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545582 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 14:30:55.077779 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545585 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 14:30:55.077779 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545588 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 14:30:55.077779 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545591 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 14:30:55.077779 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545594 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 14:30:55.077779 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545597 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 14:30:55.077779 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545599 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 14:30:55.077779 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.545602 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 14:30:55.078298 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.545607 2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 14:30:55.078298 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.552554 2569 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 10 14:30:55.078298 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.552570 2569 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 10 14:30:55.078298 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.553300 2569 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 10 14:30:55.078298 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.553307 2569 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 10 14:30:55.078298 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553367 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 14:30:55.078298 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553372 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 14:30:55.078298 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553375 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 14:30:55.078298 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553378 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 14:30:55.078298 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553382 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 14:30:55.078298 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553385 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 14:30:55.078298 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553388 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 14:30:55.078298 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553391 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 14:30:55.078298 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553394 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 14:30:55.078298 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553397 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 14:30:55.078723 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553400 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 14:30:55.078723 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553403 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 14:30:55.078723 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553405 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 14:30:55.078723 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553408 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 14:30:55.078723 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553411 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 14:30:55.078723 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553413 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 14:30:55.078723 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553416 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 14:30:55.078723 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553419 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 14:30:55.078723 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553423 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 14:30:55.078723 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553426 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 14:30:55.078723 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553428 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 14:30:55.078723 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553431 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 14:30:55.078723 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553434 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 14:30:55.078723 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553437 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 14:30:55.078723 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553440 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 14:30:55.078723 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553443 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 14:30:55.078723 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553446 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 14:30:55.078723 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553449 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 14:30:55.078723 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553452 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 14:30:55.078723 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553460 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 14:30:55.079315 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553463 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 14:30:55.079315 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553466 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 14:30:55.079315 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553468 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 14:30:55.079315 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553471 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 14:30:55.079315 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553473 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 14:30:55.079315 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553475 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 14:30:55.079315 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553478 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 14:30:55.079315 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553481 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 14:30:55.079315 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553483 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 14:30:55.079315 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553486 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 14:30:55.079315 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553488 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 14:30:55.079315 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553491 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 14:30:55.079315 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553493 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 14:30:55.079315 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553496 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 14:30:55.079315 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553499 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 14:30:55.079315 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553501 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 14:30:55.079315 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553504 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 14:30:55.079315 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553506 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 14:30:55.079315 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553510 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 14:30:55.079315 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553514 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 14:30:55.079872 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553517 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 14:30:55.079872 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553519 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 14:30:55.079872 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553522 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 14:30:55.079872 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553524 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 14:30:55.079872 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553526 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 14:30:55.079872 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553529 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 14:30:55.079872 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553531 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 14:30:55.079872 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553534 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 14:30:55.079872 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553536 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 14:30:55.079872 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553539 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 14:30:55.079872 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553544 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 14:30:55.079872 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553547 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 14:30:55.079872 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553556 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 14:30:55.079872 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553559 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 14:30:55.079872 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553562 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 14:30:55.079872 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553564 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 14:30:55.079872 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553567 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 14:30:55.079872 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553569 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 14:30:55.079872 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553572 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 14:30:55.079872 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553574 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 14:30:55.080498 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553577 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 14:30:55.080498 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553579 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 10 14:30:55.080498 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553582 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 14:30:55.080498 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553585 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 14:30:55.080498 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553588 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 14:30:55.080498 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553590 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 14:30:55.080498 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553593 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 14:30:55.080498 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553597 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 14:30:55.080498 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553600 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 14:30:55.080498 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553603 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 14:30:55.080498 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553606 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 14:30:55.080498 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553609 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 14:30:55.080498 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553612 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 14:30:55.080498 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553615 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 14:30:55.080498 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553618 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 14:30:55.080498 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553621 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 14:30:55.080498 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553624 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 14:30:55.080498 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553626 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 14:30:55.080498 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553628 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 14:30:55.081024 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553631 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 14:30:55.081024 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553634 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 14:30:55.081024 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553636 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 14:30:55.081024 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553640 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 14:30:55.081024 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553642 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 14:30:55.081024 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553645 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 14:30:55.081024 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553653 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 14:30:55.081024 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553656 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 14:30:55.081024 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553658 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 14:30:55.081024 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553660 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 14:30:55.081024 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.553666 2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 14:30:55.081024 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553819 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 14:30:55.081024 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553825 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 14:30:55.081024 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553828 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 14:30:55.081024 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553831 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 14:30:55.081501 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553834 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 14:30:55.081501 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553837 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 14:30:55.081501 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553841 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 14:30:55.081501 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553844 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 14:30:55.081501 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553847 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 14:30:55.081501 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553850 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 14:30:55.081501 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553853 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 14:30:55.081501 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553856 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 14:30:55.081501 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553859 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 14:30:55.081501 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553863 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 14:30:55.081501 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553866 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 14:30:55.081501 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553869 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 14:30:55.081501 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553871 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 14:30:55.081501 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553873 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 14:30:55.081501 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553876 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 14:30:55.081501 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553878 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 14:30:55.081501 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553881 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 14:30:55.081501 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553883 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 14:30:55.081501 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553886 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 14:30:55.081501 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553889 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 14:30:55.082116 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553893 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 14:30:55.082116 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553895 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 14:30:55.082116 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553898 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 14:30:55.082116 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553900 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 14:30:55.082116 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553909 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 14:30:55.082116 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553912 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 14:30:55.082116 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553915 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 14:30:55.082116 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553917 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 14:30:55.082116 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553936 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 14:30:55.082116 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553940 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 14:30:55.082116 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553943 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 14:30:55.082116 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553946 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 14:30:55.082116 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553949 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 14:30:55.082116 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553952 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 14:30:55.082116 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553954 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 14:30:55.082116 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553957 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 14:30:55.082116 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553959 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 14:30:55.082116 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553962 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 14:30:55.082116 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553965 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 14:30:55.082116 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553968 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 14:30:55.082698 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553970 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 14:30:55.082698 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553975 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 14:30:55.082698 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553978 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 14:30:55.082698 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553981 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 14:30:55.082698 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553983 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 14:30:55.082698 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553986 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 14:30:55.082698 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553988 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 14:30:55.082698 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553991 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 14:30:55.082698 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553993 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 14:30:55.082698 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553996 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 14:30:55.082698 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.553998 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 14:30:55.082698 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.554001 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 14:30:55.082698 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.554005 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 14:30:55.082698 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.554007 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 14:30:55.082698 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.554010 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 14:30:55.082698 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.554012 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 14:30:55.082698 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.554015 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 14:30:55.082698 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.554023 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 14:30:55.082698 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.554026 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 14:30:55.082698 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.554028 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 14:30:55.083282 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.554031 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 14:30:55.083282 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.554034 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 14:30:55.083282 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.554037 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 14:30:55.083282 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.554039 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 14:30:55.083282 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.554042 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 14:30:55.083282 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.554044 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 14:30:55.083282 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.554047 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 14:30:55.083282 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.554050 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 14:30:55.083282 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.554053 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 14:30:55.083282 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.554056 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 14:30:55.083282 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.554059 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 14:30:55.083282 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.554062 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 14:30:55.083282 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.554065 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 14:30:55.083282 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.554069 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 14:30:55.083282 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.554071 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 14:30:55.083282 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.554074 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 14:30:55.083282 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.554077 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 14:30:55.083282 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.554079 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 10 14:30:55.083282 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.554083 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 14:30:55.083839 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.554087 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 14:30:55.083839 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.554089 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 14:30:55.083839 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.554092 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 14:30:55.083839 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.554094 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 14:30:55.083839 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.554097 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 14:30:55.083839 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.554100 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 14:30:55.083839 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.554103 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 14:30:55.083839 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.554106 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 14:30:55.083839 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.554108 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 14:30:55.083839 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.554111 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 14:30:55.083839 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.554114 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 14:30:55.083839 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.554123 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 14:30:55.083839 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.554125 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 14:30:55.083839 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.554128 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 14:30:55.083839 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.554130 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 14:30:55.083839 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:54.554133 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 14:30:55.084278 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.554137 2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 14:30:55.084278 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.554267 2569 server.go:962] "Client rotation is on, will bootstrap in background" Jul 10 14:30:55.084278 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.556420 2569 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 10 14:30:55.084278 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.557742 2569 server.go:1019] "Starting client certificate rotation" Jul 10 14:30:55.084278 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.557856 2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 14:30:55.084278 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.558488 2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 14:30:55.084278 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.560700 2569 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 14:30:55.084278 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.565899 2569 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 14:30:55.084278 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.577170 2569 log.go:25] "Validated CRI v1 runtime API" Jul 10 14:30:55.084278 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.579192 2569 log.go:25] "Validated CRI v1 image API" Jul 10 14:30:55.084278 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.580710 2569 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 10 14:30:55.084278 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.586824 2569 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8a3d87f8-8f32-439b-a71c-f4110b326173:/dev/nvme0n1p3 d8b31718-6f97-48fb-8512-4ad603daff05:/dev/nvme0n1p4] Jul 10 14:30:55.084656 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.586863 2569 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 10 14:30:55.084726 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.591934 2569 manager.go:216] Machine: {Timestamp:2026-07-10 14:30:54.591173561 +0000 UTC m=+0.342545250 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099578 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec223e5a04b7701fc8986f331d65dfc6 SystemUUID:ec223e5a-04b7-701f-c898-6f331d65dfc6 BootID:5b3c74d8-634c-4d6f-9a7d-02923fdaf851 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:a9:6c:f5:68:8b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:a9:6c:f5:68:8b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7e:e5:88:77:10:df Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 10 14:30:55.084726 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.592062 2569 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 10 14:30:55.084726 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.592210 2569 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 10 14:30:55.084726 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.593472 2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 14:30:55.084726 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.594555 2569 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 10 14:30:55.084726 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.594588 2569 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-35.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 10 14:30:55.084726 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.595443 2569 topology_manager.go:138] "Creating topology manager with none policy" Jul 10 14:30:55.085045 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.595455 2569 container_manager_linux.go:309] "Creating device plugin manager" Jul 10 14:30:55.085045 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.595463 2569 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 14:30:55.085045 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.595492 2569 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 14:30:55.085045 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.596959 2569 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 10 14:30:55.085045 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.600105 2569 state_mem.go:36] "Initialized new in-memory state store" Jul 10 14:30:55.085045 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.600238 2569 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 10 14:30:55.085045 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.600266 2569 kubelet.go:486] "Attempting to sync node with API server" Jul 10 14:30:55.085045 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.600275 2569 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 10 14:30:55.085045 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.600289 2569 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 10 14:30:55.085045 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.600299 2569 kubelet.go:398] "Adding apiserver pod source" Jul 10 14:30:55.085045 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.600308 2569 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 10 14:30:55.085045 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.600861 2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 14:30:55.085045 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.600889 2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 14:30:55.085045 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.604045 2569 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 10 14:30:55.085045 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.605942 2569 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 10 14:30:55.085045 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.605970 2569 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 10 14:30:55.085045 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.606154 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 10 14:30:55.085045 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.606166 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 10 14:30:55.085045 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.606172 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 10 14:30:55.085045 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.606177 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 10 14:30:55.085045 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.606185 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 10 14:30:55.085045 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.606190 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 10 14:30:55.085045 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.606636 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 10 14:30:55.085045 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.606643 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 10 14:30:55.085045 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.606651 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 10 14:30:55.085045 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.606656 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 10 14:30:55.085045 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.606665 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 10 14:30:55.085045 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.606673 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 10 14:30:55.085045 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.607545 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 10 14:30:55.085045 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.607557 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 10 14:30:55.085045 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.614704 2569 server.go:1268] "Started kubelet" Jul 10 14:30:55.085045 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.615408 2569 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 10 14:30:55.085045 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.615858 2569 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 10 14:30:55.085045 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.615943 2569 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 10 14:30:55.085045 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.617412 2569 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-35.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 10 14:30:55.085045 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:30:54.617425 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-35.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 10 14:30:55.085045 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.617588 2569 server.go:310] "Adding debug handlers to kubelet server" Jul 10 14:30:55.085045 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.617895 2569 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 10 14:30:55.085045 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:30:54.619052 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 10 14:30:55.086375 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:30:54.624039 2569 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 10 14:30:55.086375 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.624947 2569 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 10 14:30:55.086375 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.624955 2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 10 14:30:55.086375 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:30:54.625093 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-35.ec2.internal\" not found" Jul 10 14:30:55.086375 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.625558 2569 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 10 14:30:55.086375 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.625566 2569 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 10 14:30:55.086375 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.625595 2569 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 10 14:30:55.086375 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.625656 2569 reconstruct.go:97] "Volume reconstruction finished" Jul 10 14:30:55.086375 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.625666 2569 reconciler.go:29] "Reconciler: start to sync state" Jul 10 14:30:55.086375 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.625825 2569 factory.go:55] Registering systemd factory Jul 10 14:30:55.086375 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.625838 2569 factory.go:223] Registration of the systemd container factory successfully Jul 10 14:30:55.086375 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.626046 2569 factory.go:153] Registering CRI-O factory Jul 10 14:30:55.086375 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.626062 2569 factory.go:223] Registration of the crio container factory successfully Jul 10 14:30:55.086375 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.626111 2569 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 10 14:30:55.086375 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.626137 2569 factory.go:103] Registering Raw factory Jul 10 14:30:55.086375 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.626149 2569 manager.go:1196] Started watching for new ooms in manager Jul 10 14:30:55.086375 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.627111 2569 manager.go:349] Starting recovery of all containers Jul 10 14:30:55.086375 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.636865 2569 manager.go:354] Recovery completed Jul 10 14:30:55.086375 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:30:54.637740 2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-35.ec2.internal.18c0f41c4059b684 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-35.ec2.internal,UID:ip-10-0-140-35.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-35.ec2.internal,},FirstTimestamp:2026-07-10 14:30:54.6145993 +0000 UTC m=+0.365970982,LastTimestamp:2026-07-10 14:30:54.6145993 +0000 UTC m=+0.365970982,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-35.ec2.internal,}" Jul 10 14:30:55.086375 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:30:54.639181 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 10 14:30:55.086375 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:30:54.639625 2569 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-35.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 10 14:30:55.086375 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.642103 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 14:30:55.086375 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.644041 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-35.ec2.internal" event="NodeHasSufficientMemory" Jul 10 14:30:55.086375 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.644076 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-35.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 14:30:55.086375 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.644086 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-35.ec2.internal" event="NodeHasSufficientPID" Jul 10 14:30:55.086375 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.644605 2569 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 10 14:30:55.086375 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.644614 2569 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 10 14:30:55.086375 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.644619 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 14:30:55.086375 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.644629 2569 state_mem.go:36] "Initialized new in-memory state store" Jul 10 14:30:55.086375 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.645354 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-35.ec2.internal" event="NodeHasSufficientMemory" Jul 10 14:30:55.086375 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.645377 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-35.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 14:30:55.086375 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.645386 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-35.ec2.internal" event="NodeHasSufficientPID" Jul 10 14:30:55.086375 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.645843 2569 policy_none.go:49] "None policy: Start" Jul 10 14:30:55.087330 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.645855 2569 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 10 14:30:55.087330 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.645865 2569 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 10 14:30:55.087330 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:30:54.646362 2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-35.ec2.internal.18c0f41c421b420d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-35.ec2.internal,UID:ip-10-0-140-35.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-35.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-35.ec2.internal,},FirstTimestamp:2026-07-10 14:30:54.644060685 +0000 UTC m=+0.395432372,LastTimestamp:2026-07-10 14:30:54.644060685 +0000 UTC m=+0.395432372,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-35.ec2.internal,}" Jul 10 14:30:55.087330 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.646777 2569 policy_none.go:47] "Start" Jul 10 14:30:55.087330 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.651912 2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6sb7r" Jul 10 14:30:55.087330 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:30:54.656271 2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-35.ec2.internal.18c0f41c421b8f4e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-35.ec2.internal,UID:ip-10-0-140-35.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-35.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-35.ec2.internal,},FirstTimestamp:2026-07-10 14:30:54.644080462 +0000 UTC m=+0.395452149,LastTimestamp:2026-07-10 14:30:54.644080462 +0000 UTC m=+0.395452149,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-35.ec2.internal,}" Jul 10 14:30:55.087330 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.660472 2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6sb7r" Jul 10 14:30:55.087330 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:30:54.669105 2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-35.ec2.internal.18c0f41c421bb62b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-35.ec2.internal,UID:ip-10-0-140-35.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-140-35.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-140-35.ec2.internal,},FirstTimestamp:2026-07-10 14:30:54.644090411 +0000 UTC m=+0.395462098,LastTimestamp:2026-07-10 14:30:54.644090411 +0000 UTC m=+0.395462098,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-35.ec2.internal,}" Jul 10 14:30:55.087330 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:30:54.681012 2569 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-35.ec2.internal.18c0f41c421b420d\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-35.ec2.internal.18c0f41c421b420d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-35.ec2.internal,UID:ip-10-0-140-35.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-35.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-35.ec2.internal,},FirstTimestamp:2026-07-10 14:30:54.644060685 +0000 UTC m=+0.395432372,LastTimestamp:2026-07-10 14:30:54.645364236 +0000 UTC m=+0.396735926,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-35.ec2.internal,}" Jul 10 14:30:55.087330 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.688360 2569 manager.go:337] "Starting Device Plugin manager" Jul 10 14:30:55.087330 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:30:54.688565 2569 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 10 14:30:55.087330 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.688578 2569 server.go:87] "Starting device plugin registration server" Jul 10 14:30:55.087330 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.688853 2569 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 10 14:30:55.306033 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.688865 2569 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 10 14:30:55.306033 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.688945 2569 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 10 14:30:55.306033 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.689035 2569 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 10 14:30:55.306033 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.689044 2569 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 10 14:30:55.306033 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:30:54.689632 2569 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 10 14:30:55.306033 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:30:54.689690 2569 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-35.ec2.internal\" not found" Jul 10 14:30:55.306033 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:30:54.698549 2569 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-35.ec2.internal.18c0f41c421b8f4e\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-35.ec2.internal.18c0f41c421b8f4e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-35.ec2.internal,UID:ip-10-0-140-35.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-35.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-35.ec2.internal,},FirstTimestamp:2026-07-10 14:30:54.644080462 +0000 UTC m=+0.395452149,LastTimestamp:2026-07-10 14:30:54.64538179 +0000 UTC m=+0.396753477,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-35.ec2.internal,}" Jul 10 14:30:55.306033 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:30:54.707944 2569 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-35.ec2.internal.18c0f41c421bb62b\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-35.ec2.internal.18c0f41c421bb62b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-35.ec2.internal,UID:ip-10-0-140-35.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-140-35.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-140-35.ec2.internal,},FirstTimestamp:2026-07-10 14:30:54.644090411 +0000 UTC m=+0.395462098,LastTimestamp:2026-07-10 14:30:54.645390101 +0000 UTC m=+0.396761788,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-35.ec2.internal,}" Jul 10 14:30:55.306033 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:30:54.718035 2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-35.ec2.internal.18c0f41c45de913f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-35.ec2.internal,UID:ip-10-0-140-35.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-140-35.ec2.internal,},FirstTimestamp:2026-07-10 14:30:54.707192127 +0000 UTC m=+0.458563801,LastTimestamp:2026-07-10 14:30:54.707192127 +0000 UTC m=+0.458563801,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-35.ec2.internal,}" Jul 10 14:30:55.306033 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.789375 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 14:30:55.306033 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.790304 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-35.ec2.internal" event="NodeHasSufficientMemory" Jul 10 14:30:55.306033 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.790331 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-35.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 14:30:55.306033 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.790341 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-35.ec2.internal" event="NodeHasSufficientPID" Jul 10 14:30:55.306033 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.790366 2569 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-35.ec2.internal" Jul 10 14:30:55.306535 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:30:54.798192 2569 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-35.ec2.internal.18c0f41c421b420d\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-35.ec2.internal.18c0f41c421b420d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-35.ec2.internal,UID:ip-10-0-140-35.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-35.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-35.ec2.internal,},FirstTimestamp:2026-07-10 14:30:54.644060685 +0000 UTC m=+0.395432372,LastTimestamp:2026-07-10 14:30:54.790315958 +0000 UTC m=+0.541687648,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-35.ec2.internal,}" Jul 10 14:30:55.306535 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:30:54.810932 2569 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-140-35.ec2.internal\" not found" node="ip-10-0-140-35.ec2.internal" Jul 10 14:30:55.306535 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:30:54.846601 2569 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-140-35.ec2.internal\" not found" node="ip-10-0-140-35.ec2.internal" Jul 10 14:30:55.306535 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.923331 2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 10 14:30:55.306535 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.924999 2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 10 14:30:55.306535 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.925039 2569 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 10 14:30:55.306535 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.925064 2569 kubelet.go:2448] "Starting kubelet main sync loop" Jul 10 14:30:55.306535 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:30:54.925105 2569 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 10 14:30:55.306535 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:54.928026 2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 14:30:55.306535 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:55.011658 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 14:30:55.306535 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:55.012687 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-35.ec2.internal" event="NodeHasSufficientMemory" Jul 10 14:30:55.306535 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:55.012721 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-35.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 14:30:55.306535 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:55.012731 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-35.ec2.internal" event="NodeHasSufficientPID" Jul 10 14:30:55.306535 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:55.012772 2569 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-35.ec2.internal" Jul 10 14:30:55.306535 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:55.024080 2569 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-35.ec2.internal" Jul 10 14:30:55.306535 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:30:55.024103 2569 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-35.ec2.internal\": node \"ip-10-0-140-35.ec2.internal\" not found" Jul 10 14:30:55.306535 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:55.025275 2569 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-35.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-35.ec2.internal"] Jul 10 14:30:55.306535 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:55.025337 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 14:30:55.306535 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:55.026681 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/12fdb1aa4afb104872332bdd0f7db7e5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-35.ec2.internal\" (UID: \"12fdb1aa4afb104872332bdd0f7db7e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-35.ec2.internal" Jul 10 14:30:55.306535 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:55.026707 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/12fdb1aa4afb104872332bdd0f7db7e5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-35.ec2.internal\" (UID: \"12fdb1aa4afb104872332bdd0f7db7e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-35.ec2.internal" Jul 10 14:30:55.306535 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:55.026918 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-35.ec2.internal" event="NodeHasSufficientMemory" Jul 10 14:30:55.306535 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:55.026946 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-35.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 14:30:55.306535 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:55.026956 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-35.ec2.internal" event="NodeHasSufficientPID" Jul 10 14:30:55.306535 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:55.028187 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 14:30:55.307247 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:55.028281 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-35.ec2.internal" Jul 10 14:30:55.307247 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:55.028337 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 14:30:55.307247 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:55.028869 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-35.ec2.internal" event="NodeHasSufficientMemory" Jul 10 14:30:55.307247 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:55.028875 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-35.ec2.internal" event="NodeHasSufficientMemory" Jul 10 14:30:55.307247 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:55.028893 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-35.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 14:30:55.307247 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:55.028896 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-35.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 14:30:55.307247 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:55.028905 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-35.ec2.internal" event="NodeHasSufficientPID" Jul 10 14:30:55.307247 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:55.028911 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-35.ec2.internal" event="NodeHasSufficientPID" Jul 10 14:30:55.307247 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:55.030070 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-35.ec2.internal" Jul 10 14:30:55.307247 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:55.030093 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 14:30:55.307247 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:55.030654 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-35.ec2.internal" event="NodeHasSufficientMemory" Jul 10 14:30:55.307247 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:55.030673 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-35.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 14:30:55.307247 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:55.030683 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-35.ec2.internal" event="NodeHasSufficientPID" Jul 10 14:30:55.307247 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:30:55.046305 2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-35.ec2.internal\" not found" node="ip-10-0-140-35.ec2.internal" Jul 10 14:30:55.307247 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:30:55.049792 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-35.ec2.internal\" not found" Jul 10 14:30:55.307247 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:30:55.050108 2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-35.ec2.internal\" not found" node="ip-10-0-140-35.ec2.internal" Jul 10 14:30:55.307247 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:55.127573 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/12fdb1aa4afb104872332bdd0f7db7e5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-35.ec2.internal\" (UID: \"12fdb1aa4afb104872332bdd0f7db7e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-35.ec2.internal" Jul 10 14:30:55.307247 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:55.127601 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/12fdb1aa4afb104872332bdd0f7db7e5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-35.ec2.internal\" (UID: \"12fdb1aa4afb104872332bdd0f7db7e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-35.ec2.internal" Jul 10 14:30:55.307247 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:55.127620 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/486f4ab7baea4188ee524f2052e08a91-config\") pod \"kube-apiserver-proxy-ip-10-0-140-35.ec2.internal\" (UID: \"486f4ab7baea4188ee524f2052e08a91\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-35.ec2.internal" Jul 10 14:30:55.307247 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:55.127674 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/12fdb1aa4afb104872332bdd0f7db7e5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-35.ec2.internal\" (UID: \"12fdb1aa4afb104872332bdd0f7db7e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-35.ec2.internal" Jul 10 14:30:55.307247 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:55.127725 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/12fdb1aa4afb104872332bdd0f7db7e5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-35.ec2.internal\" (UID: \"12fdb1aa4afb104872332bdd0f7db7e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-35.ec2.internal" Jul 10 14:30:55.307247 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:30:55.149979 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-35.ec2.internal\" not found" Jul 10 14:30:55.307247 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:55.228363 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/486f4ab7baea4188ee524f2052e08a91-config\") pod \"kube-apiserver-proxy-ip-10-0-140-35.ec2.internal\" (UID: \"486f4ab7baea4188ee524f2052e08a91\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-35.ec2.internal" Jul 10 14:30:55.307924 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:55.228441 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/486f4ab7baea4188ee524f2052e08a91-config\") pod \"kube-apiserver-proxy-ip-10-0-140-35.ec2.internal\" (UID: \"486f4ab7baea4188ee524f2052e08a91\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-35.ec2.internal" Jul 10 14:30:55.307924 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:30:55.250479 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-35.ec2.internal\" not found" Jul 10 14:30:55.514717 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:55.347976 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-35.ec2.internal" Jul 10 14:30:55.514717 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:30:55.350647 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-35.ec2.internal\" not found" Jul 10 14:30:55.514717 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:55.352946 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-35.ec2.internal" Jul 10 14:30:55.514717 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:30:55.451238 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-35.ec2.internal\" not found" Jul 10 14:30:55.741352 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:55.520160 2569 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 14:30:55.741352 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:30:55.552815 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-35.ec2.internal\" not found" Jul 10 14:30:55.741352 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:55.557297 2569 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 10 14:30:55.741352 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:55.557431 2569 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 10 14:30:55.741352 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:55.557465 2569 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 10 14:30:55.741352 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:30:55.653644 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-35.ec2.internal\" not found" Jul 10 14:30:55.971789 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:55.745020 2569 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 14:30:55.971789 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:55.797438 2569 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 14:30:55.971789 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:55.804502 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-09 14:25:55 +0000 UTC" deadline="2028-01-06 08:12:44.61748299 +0000 UTC" Jul 10 14:30:55.971789 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:55.804523 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13073h41m48.812962703s" Jul 10 14:30:55.971789 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:55.825808 2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-35.ec2.internal" Jul 10 14:30:55.971789 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:55.839313 2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 14:30:55.971789 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:55.840459 2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-35.ec2.internal" Jul 10 14:30:55.971789 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:55.850973 2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 14:30:56.179592 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:56.062981 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12fdb1aa4afb104872332bdd0f7db7e5.slice/crio-eb9e8676b32ae764ada6925152ceb5fceb7663c6139db5255b35e40d68a70c09 WatchSource:0}: Error finding container eb9e8676b32ae764ada6925152ceb5fceb7663c6139db5255b35e40d68a70c09: Status 404 returned error can't find the container with id eb9e8676b32ae764ada6925152ceb5fceb7663c6139db5255b35e40d68a70c09 Jul 10 14:30:56.179592 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:56.063271 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod486f4ab7baea4188ee524f2052e08a91.slice/crio-1496d00d2a9a5103764ed0b9d33b7ff953ad0e82a6b7e75d21414404d9cb9183 WatchSource:0}: Error finding container 1496d00d2a9a5103764ed0b9d33b7ff953ad0e82a6b7e75d21414404d9cb9183: Status 404 returned error can't find the container with id 1496d00d2a9a5103764ed0b9d33b7ff953ad0e82a6b7e75d21414404d9cb9183 Jul 10 14:30:56.179592 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.068436 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 14:30:56.605201 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.603961 2569 apiserver.go:52] "Watching apiserver" Jul 10 14:30:56.615183 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.614488 2569 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 10 14:30:56.722945 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.615898 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-35.ec2.internal","openshift-multus/multus-r6rkz","openshift-ovn-kubernetes/ovnkube-node-kxvq2","kube-system/konnectivity-agent-j2g7m","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwv5j","openshift-multus/multus-additional-cni-plugins-2sjq2","openshift-multus/network-metrics-daemon-fdhw7","openshift-network-diagnostics/network-check-target-t9jl6","openshift-network-operator/iptables-alerter-lfrbz","kube-system/global-pull-secret-syncer-rxmgq","kube-system/kube-apiserver-proxy-ip-10-0-140-35.ec2.internal","openshift-cluster-node-tuning-operator/tuned-7jtx8","openshift-dns/node-resolver-842qb","openshift-image-registry/node-ca-w9ww7"] Jul 10 14:30:56.722945 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.617451 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fdhw7" Jul 10 14:30:56.722945 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:30:56.617590 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fdhw7" podUID="ff3c9cb2-1924-4ea9-9a93-282213b083db" Jul 10 14:30:56.722945 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.618517 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-842qb" Jul 10 14:30:56.722945 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.619788 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:30:56.722945 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.621430 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-m5w69\"" Jul 10 14:30:56.722945 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.621474 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 10 14:30:56.722945 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.621511 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 10 14:30:56.722945 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.622107 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 10 14:30:56.722945 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.622528 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-j2g7m" Jul 10 14:30:56.722945 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.623183 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 10 14:30:56.722945 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.624189 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 10 14:30:56.722945 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.624576 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 10 14:30:56.722945 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.624880 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 10 14:30:56.722945 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.624906 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 10 14:30:56.722945 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.624938 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-468w4\"" Jul 10 14:30:56.722945 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.626615 2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 10 14:30:56.722945 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.627779 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 10 14:30:56.722945 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.628077 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-gb6hm\"" Jul 10 14:30:56.722945 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.628306 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 10 14:30:56.722945 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.633719 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwv5j" Jul 10 14:30:56.722945 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.634981 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2sjq2" Jul 10 14:30:56.722945 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.636356 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lfrbz" Jul 10 14:30:56.722945 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.636778 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7fsgc\"" Jul 10 14:30:56.722945 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.636931 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 10 14:30:56.722945 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.637070 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 10 14:30:56.722945 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.637168 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 10 14:30:56.724452 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.637731 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 10 14:30:56.724452 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.637750 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 10 14:30:56.724452 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.637850 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-w9ww7" Jul 10 14:30:56.724452 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.637998 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 10 14:30:56.724452 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.638031 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 10 14:30:56.724452 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.638319 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5574c\"" Jul 10 14:30:56.724452 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.638319 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 10 14:30:56.724452 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.638578 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/270982ae-963c-42a2-a10f-06cd88f30f81-hosts-file\") pod \"node-resolver-842qb\" (UID: \"270982ae-963c-42a2-a10f-06cd88f30f81\") " pod="openshift-dns/node-resolver-842qb" Jul 10 14:30:56.724452 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.638605 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/016248a0-bbe2-44c9-8e51-5d4695ee6a52-run-openvswitch\") pod \"ovnkube-node-kxvq2\" (UID: \"016248a0-bbe2-44c9-8e51-5d4695ee6a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:30:56.724452 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.638624 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/016248a0-bbe2-44c9-8e51-5d4695ee6a52-run-ovn\") pod \"ovnkube-node-kxvq2\" (UID: \"016248a0-bbe2-44c9-8e51-5d4695ee6a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:30:56.724452 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.638639 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/016248a0-bbe2-44c9-8e51-5d4695ee6a52-host-run-ovn-kubernetes\") pod \"ovnkube-node-kxvq2\" (UID: \"016248a0-bbe2-44c9-8e51-5d4695ee6a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:30:56.724452 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.638662 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4ee5894d-0007-499f-a401-2f11724910ac-agent-certs\") pod \"konnectivity-agent-j2g7m\" (UID: \"4ee5894d-0007-499f-a401-2f11724910ac\") " pod="kube-system/konnectivity-agent-j2g7m" Jul 10 14:30:56.724452 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.638687 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/016248a0-bbe2-44c9-8e51-5d4695ee6a52-host-slash\") pod \"ovnkube-node-kxvq2\" (UID: \"016248a0-bbe2-44c9-8e51-5d4695ee6a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:30:56.724452 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.638694 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 10 14:30:56.724452 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.638742 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/016248a0-bbe2-44c9-8e51-5d4695ee6a52-host-run-netns\") pod \"ovnkube-node-kxvq2\" (UID: \"016248a0-bbe2-44c9-8e51-5d4695ee6a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:30:56.724452 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.638806 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/016248a0-bbe2-44c9-8e51-5d4695ee6a52-log-socket\") pod \"ovnkube-node-kxvq2\" (UID: \"016248a0-bbe2-44c9-8e51-5d4695ee6a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:30:56.724452 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.638831 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/016248a0-bbe2-44c9-8e51-5d4695ee6a52-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kxvq2\" (UID: \"016248a0-bbe2-44c9-8e51-5d4695ee6a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:30:56.724452 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.638870 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/270982ae-963c-42a2-a10f-06cd88f30f81-tmp-dir\") pod \"node-resolver-842qb\" (UID: \"270982ae-963c-42a2-a10f-06cd88f30f81\") " pod="openshift-dns/node-resolver-842qb" Jul 10 14:30:56.725572 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.638896 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4ee5894d-0007-499f-a401-2f11724910ac-konnectivity-ca\") pod \"konnectivity-agent-j2g7m\" (UID: \"4ee5894d-0007-499f-a401-2f11724910ac\") " pod="kube-system/konnectivity-agent-j2g7m" Jul 10 14:30:56.725572 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.638920 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/016248a0-bbe2-44c9-8e51-5d4695ee6a52-run-systemd\") pod \"ovnkube-node-kxvq2\" (UID: \"016248a0-bbe2-44c9-8e51-5d4695ee6a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:30:56.725572 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.638958 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/016248a0-bbe2-44c9-8e51-5d4695ee6a52-host-cni-bin\") pod \"ovnkube-node-kxvq2\" (UID: \"016248a0-bbe2-44c9-8e51-5d4695ee6a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:30:56.725572 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.638981 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/016248a0-bbe2-44c9-8e51-5d4695ee6a52-env-overrides\") pod \"ovnkube-node-kxvq2\" (UID: \"016248a0-bbe2-44c9-8e51-5d4695ee6a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:30:56.725572 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.639007 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/016248a0-bbe2-44c9-8e51-5d4695ee6a52-ovn-node-metrics-cert\") pod \"ovnkube-node-kxvq2\" (UID: \"016248a0-bbe2-44c9-8e51-5d4695ee6a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:30:56.725572 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.639033 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/016248a0-bbe2-44c9-8e51-5d4695ee6a52-ovnkube-script-lib\") pod \"ovnkube-node-kxvq2\" (UID: \"016248a0-bbe2-44c9-8e51-5d4695ee6a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:30:56.725572 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.639058 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqh6h\" (UniqueName: \"kubernetes.io/projected/016248a0-bbe2-44c9-8e51-5d4695ee6a52-kube-api-access-hqh6h\") pod \"ovnkube-node-kxvq2\" (UID: \"016248a0-bbe2-44c9-8e51-5d4695ee6a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:30:56.725572 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.639081 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8d9a6391-5821-4247-b85e-f6089b65f9f0-etc-selinux\") pod \"aws-ebs-csi-driver-node-xwv5j\" (UID: \"8d9a6391-5821-4247-b85e-f6089b65f9f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwv5j" Jul 10 14:30:56.725572 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.639105 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/016248a0-bbe2-44c9-8e51-5d4695ee6a52-node-log\") pod \"ovnkube-node-kxvq2\" (UID: \"016248a0-bbe2-44c9-8e51-5d4695ee6a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:30:56.725572 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.639131 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmsn7\" (UniqueName: \"kubernetes.io/projected/8d9a6391-5821-4247-b85e-f6089b65f9f0-kube-api-access-vmsn7\") pod \"aws-ebs-csi-driver-node-xwv5j\" (UID: \"8d9a6391-5821-4247-b85e-f6089b65f9f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwv5j" Jul 10 14:30:56.725572 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.639163 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgtfp\" (UniqueName: \"kubernetes.io/projected/270982ae-963c-42a2-a10f-06cd88f30f81-kube-api-access-kgtfp\") pod \"node-resolver-842qb\" (UID: \"270982ae-963c-42a2-a10f-06cd88f30f81\") " pod="openshift-dns/node-resolver-842qb" Jul 10 14:30:56.725572 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.639187 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/016248a0-bbe2-44c9-8e51-5d4695ee6a52-var-lib-openvswitch\") pod \"ovnkube-node-kxvq2\" (UID: \"016248a0-bbe2-44c9-8e51-5d4695ee6a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:30:56.725572 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.639210 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/016248a0-bbe2-44c9-8e51-5d4695ee6a52-host-cni-netd\") pod \"ovnkube-node-kxvq2\" (UID: \"016248a0-bbe2-44c9-8e51-5d4695ee6a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:30:56.726425 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.639248 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/016248a0-bbe2-44c9-8e51-5d4695ee6a52-ovnkube-config\") pod \"ovnkube-node-kxvq2\" (UID: \"016248a0-bbe2-44c9-8e51-5d4695ee6a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:30:56.726425 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.639278 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8d9a6391-5821-4247-b85e-f6089b65f9f0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xwv5j\" (UID: \"8d9a6391-5821-4247-b85e-f6089b65f9f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwv5j" Jul 10 14:30:56.726425 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.639302 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8d9a6391-5821-4247-b85e-f6089b65f9f0-sys-fs\") pod \"aws-ebs-csi-driver-node-xwv5j\" (UID: \"8d9a6391-5821-4247-b85e-f6089b65f9f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwv5j" Jul 10 14:30:56.726425 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.639325 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff3c9cb2-1924-4ea9-9a93-282213b083db-metrics-certs\") pod \"network-metrics-daemon-fdhw7\" (UID: \"ff3c9cb2-1924-4ea9-9a93-282213b083db\") " pod="openshift-multus/network-metrics-daemon-fdhw7" Jul 10 14:30:56.726425 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.639349 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/016248a0-bbe2-44c9-8e51-5d4695ee6a52-host-kubelet\") pod \"ovnkube-node-kxvq2\" (UID: \"016248a0-bbe2-44c9-8e51-5d4695ee6a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:30:56.726425 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.639372 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8d9a6391-5821-4247-b85e-f6089b65f9f0-socket-dir\") pod \"aws-ebs-csi-driver-node-xwv5j\" (UID: \"8d9a6391-5821-4247-b85e-f6089b65f9f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwv5j" Jul 10 14:30:56.726425 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.639397 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vhlx\" (UniqueName: \"kubernetes.io/projected/ff3c9cb2-1924-4ea9-9a93-282213b083db-kube-api-access-4vhlx\") pod \"network-metrics-daemon-fdhw7\" (UID: \"ff3c9cb2-1924-4ea9-9a93-282213b083db\") " pod="openshift-multus/network-metrics-daemon-fdhw7" Jul 10 14:30:56.726425 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.639424 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/016248a0-bbe2-44c9-8e51-5d4695ee6a52-systemd-units\") pod \"ovnkube-node-kxvq2\" (UID: \"016248a0-bbe2-44c9-8e51-5d4695ee6a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:30:56.726425 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.639448 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 10 14:30:56.726425 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.639452 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/016248a0-bbe2-44c9-8e51-5d4695ee6a52-etc-openvswitch\") pod \"ovnkube-node-kxvq2\" (UID: \"016248a0-bbe2-44c9-8e51-5d4695ee6a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:30:56.726425 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.639504 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8d9a6391-5821-4247-b85e-f6089b65f9f0-registration-dir\") pod \"aws-ebs-csi-driver-node-xwv5j\" (UID: \"8d9a6391-5821-4247-b85e-f6089b65f9f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwv5j" Jul 10 14:30:56.726425 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.639526 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8d9a6391-5821-4247-b85e-f6089b65f9f0-device-dir\") pod \"aws-ebs-csi-driver-node-xwv5j\" (UID: \"8d9a6391-5821-4247-b85e-f6089b65f9f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwv5j" Jul 10 14:30:56.726425 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.639596 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-r6rkz" Jul 10 14:30:56.726425 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.640123 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7pr2x\"" Jul 10 14:30:56.726425 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.640601 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 10 14:30:56.726425 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.640648 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 10 14:30:56.726425 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.640915 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rxmgq" Jul 10 14:30:56.727506 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:30:56.640988 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rxmgq" podUID="4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f" Jul 10 14:30:56.727506 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.641045 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 10 14:30:56.727506 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.641132 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 10 14:30:56.727506 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.641190 2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 14:30:56.727506 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.641340 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-m7nwc\"" Jul 10 14:30:56.727506 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.642373 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t9jl6" Jul 10 14:30:56.727506 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:30:56.642441 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t9jl6" podUID="ced07efc-4719-4439-b2be-b84dadef62d3" Jul 10 14:30:56.727506 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.642469 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 10 14:30:56.727506 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.642764 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-srpsq\"" Jul 10 14:30:56.727506 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.643928 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7jtx8" Jul 10 14:30:56.727506 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.646275 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 10 14:30:56.727506 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.646874 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 10 14:30:56.727506 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.647362 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rthdk\"" Jul 10 14:30:56.727506 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.687349 2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-hw9lf" Jul 10 14:30:56.727506 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.712578 2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-hw9lf" Jul 10 14:30:56.728424 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.727598 2569 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 10 14:30:56.915048 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.740101 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff3c9cb2-1924-4ea9-9a93-282213b083db-metrics-certs\") pod \"network-metrics-daemon-fdhw7\" (UID: \"ff3c9cb2-1924-4ea9-9a93-282213b083db\") " pod="openshift-multus/network-metrics-daemon-fdhw7" Jul 10 14:30:56.915048 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.740127 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bc2e14e7-0dd6-4e09-b724-948115817cb8-etc-sysconfig\") pod \"tuned-7jtx8\" (UID: \"bc2e14e7-0dd6-4e09-b724-948115817cb8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jtx8" Jul 10 14:30:56.915048 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.740145 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/016248a0-bbe2-44c9-8e51-5d4695ee6a52-host-kubelet\") pod \"ovnkube-node-kxvq2\" (UID: \"016248a0-bbe2-44c9-8e51-5d4695ee6a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:30:56.915048 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.740163 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8d9a6391-5821-4247-b85e-f6089b65f9f0-socket-dir\") pod \"aws-ebs-csi-driver-node-xwv5j\" (UID: \"8d9a6391-5821-4247-b85e-f6089b65f9f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwv5j" Jul 10 14:30:56.915048 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.740180 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f-kubelet-config\") pod \"global-pull-secret-syncer-rxmgq\" (UID: \"4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f\") " pod="kube-system/global-pull-secret-syncer-rxmgq" Jul 10 14:30:56.915048 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.740235 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/016248a0-bbe2-44c9-8e51-5d4695ee6a52-host-kubelet\") pod \"ovnkube-node-kxvq2\" (UID: \"016248a0-bbe2-44c9-8e51-5d4695ee6a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:30:56.915048 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.740274 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ba16bc50-891a-4477-8bd5-de3ef058d04f-system-cni-dir\") pod \"multus-additional-cni-plugins-2sjq2\" (UID: \"ba16bc50-891a-4477-8bd5-de3ef058d04f\") " pod="openshift-multus/multus-additional-cni-plugins-2sjq2" Jul 10 14:30:56.915048 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:30:56.740284 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 14:30:56.915048 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.740303 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ba16bc50-891a-4477-8bd5-de3ef058d04f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2sjq2\" (UID: \"ba16bc50-891a-4477-8bd5-de3ef058d04f\") " pod="openshift-multus/multus-additional-cni-plugins-2sjq2" Jul 10 14:30:56.915048 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.740325 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/016248a0-bbe2-44c9-8e51-5d4695ee6a52-systemd-units\") pod \"ovnkube-node-kxvq2\" (UID: \"016248a0-bbe2-44c9-8e51-5d4695ee6a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:30:56.915048 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.740282 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8d9a6391-5821-4247-b85e-f6089b65f9f0-socket-dir\") pod \"aws-ebs-csi-driver-node-xwv5j\" (UID: \"8d9a6391-5821-4247-b85e-f6089b65f9f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwv5j" Jul 10 14:30:56.915048 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.740366 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/016248a0-bbe2-44c9-8e51-5d4695ee6a52-systemd-units\") pod \"ovnkube-node-kxvq2\" (UID: \"016248a0-bbe2-44c9-8e51-5d4695ee6a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:30:56.915048 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:30:56.740353 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff3c9cb2-1924-4ea9-9a93-282213b083db-metrics-certs podName:ff3c9cb2-1924-4ea9-9a93-282213b083db nodeName:}" failed. No retries permitted until 2026-07-10 14:30:57.240332151 +0000 UTC m=+2.991703841 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ff3c9cb2-1924-4ea9-9a93-282213b083db-metrics-certs") pod "network-metrics-daemon-fdhw7" (UID: "ff3c9cb2-1924-4ea9-9a93-282213b083db") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 14:30:56.915048 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.740400 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc922b38-8f9b-4ec4-95c0-c132253a3158-host-var-lib-kubelet\") pod \"multus-r6rkz\" (UID: \"fc922b38-8f9b-4ec4-95c0-c132253a3158\") " pod="openshift-multus/multus-r6rkz" Jul 10 14:30:56.924215 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.740424 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bc2e14e7-0dd6-4e09-b724-948115817cb8-run\") pod \"tuned-7jtx8\" (UID: \"bc2e14e7-0dd6-4e09-b724-948115817cb8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jtx8" Jul 10 14:30:56.924215 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.740448 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/016248a0-bbe2-44c9-8e51-5d4695ee6a52-run-openvswitch\") pod \"ovnkube-node-kxvq2\" (UID: \"016248a0-bbe2-44c9-8e51-5d4695ee6a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:30:56.924215 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.740473 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/21295856-adfb-486d-a3dc-b9ff8aa3ff50-host\") pod \"node-ca-w9ww7\" (UID: \"21295856-adfb-486d-a3dc-b9ff8aa3ff50\") " pod="openshift-image-registry/node-ca-w9ww7" Jul 10 14:30:56.924215 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.740517 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fc922b38-8f9b-4ec4-95c0-c132253a3158-host-run-multus-certs\") pod \"multus-r6rkz\" (UID: \"fc922b38-8f9b-4ec4-95c0-c132253a3158\") " pod="openshift-multus/multus-r6rkz" Jul 10 14:30:56.924215 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.740532 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/016248a0-bbe2-44c9-8e51-5d4695ee6a52-run-openvswitch\") pod \"ovnkube-node-kxvq2\" (UID: \"016248a0-bbe2-44c9-8e51-5d4695ee6a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:30:56.924215 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.740543 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bc2e14e7-0dd6-4e09-b724-948115817cb8-etc-systemd\") pod \"tuned-7jtx8\" (UID: \"bc2e14e7-0dd6-4e09-b724-948115817cb8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jtx8" Jul 10 14:30:56.924215 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.740573 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bc2e14e7-0dd6-4e09-b724-948115817cb8-etc-tuned\") pod \"tuned-7jtx8\" (UID: \"bc2e14e7-0dd6-4e09-b724-948115817cb8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jtx8" Jul 10 14:30:56.924215 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.740612 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc2e14e7-0dd6-4e09-b724-948115817cb8-tmp\") pod \"tuned-7jtx8\" (UID: \"bc2e14e7-0dd6-4e09-b724-948115817cb8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jtx8" Jul 10 14:30:56.924215 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.740634 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ba16bc50-891a-4477-8bd5-de3ef058d04f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2sjq2\" (UID: \"ba16bc50-891a-4477-8bd5-de3ef058d04f\") " pod="openshift-multus/multus-additional-cni-plugins-2sjq2" Jul 10 14:30:56.924215 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.740655 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/016248a0-bbe2-44c9-8e51-5d4695ee6a52-host-run-netns\") pod \"ovnkube-node-kxvq2\" (UID: \"016248a0-bbe2-44c9-8e51-5d4695ee6a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:30:56.924215 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.740673 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/016248a0-bbe2-44c9-8e51-5d4695ee6a52-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kxvq2\" (UID: \"016248a0-bbe2-44c9-8e51-5d4695ee6a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:30:56.924215 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.740690 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4ee5894d-0007-499f-a401-2f11724910ac-konnectivity-ca\") pod \"konnectivity-agent-j2g7m\" (UID: \"4ee5894d-0007-499f-a401-2f11724910ac\") " pod="kube-system/konnectivity-agent-j2g7m" Jul 10 14:30:56.924215 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.740716 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fc922b38-8f9b-4ec4-95c0-c132253a3158-multus-socket-dir-parent\") pod \"multus-r6rkz\" (UID: \"fc922b38-8f9b-4ec4-95c0-c132253a3158\") " pod="openshift-multus/multus-r6rkz" Jul 10 14:30:56.924215 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.740743 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/016248a0-bbe2-44c9-8e51-5d4695ee6a52-host-run-netns\") pod \"ovnkube-node-kxvq2\" (UID: \"016248a0-bbe2-44c9-8e51-5d4695ee6a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:30:56.925143 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.740751 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/016248a0-bbe2-44c9-8e51-5d4695ee6a52-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kxvq2\" (UID: \"016248a0-bbe2-44c9-8e51-5d4695ee6a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:30:56.925143 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.740772 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fc922b38-8f9b-4ec4-95c0-c132253a3158-host-run-netns\") pod \"multus-r6rkz\" (UID: \"fc922b38-8f9b-4ec4-95c0-c132253a3158\") " pod="openshift-multus/multus-r6rkz" Jul 10 14:30:56.925143 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.740812 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fc922b38-8f9b-4ec4-95c0-c132253a3158-multus-conf-dir\") pod \"multus-r6rkz\" (UID: \"fc922b38-8f9b-4ec4-95c0-c132253a3158\") " pod="openshift-multus/multus-r6rkz" Jul 10 14:30:56.925143 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.740844 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f-dbus\") pod \"global-pull-secret-syncer-rxmgq\" (UID: \"4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f\") " pod="kube-system/global-pull-secret-syncer-rxmgq" Jul 10 14:30:56.925143 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.740875 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bc2e14e7-0dd6-4e09-b724-948115817cb8-sys\") pod \"tuned-7jtx8\" (UID: \"bc2e14e7-0dd6-4e09-b724-948115817cb8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jtx8" Jul 10 14:30:56.925143 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.740899 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/016248a0-bbe2-44c9-8e51-5d4695ee6a52-run-systemd\") pod \"ovnkube-node-kxvq2\" (UID: \"016248a0-bbe2-44c9-8e51-5d4695ee6a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:30:56.925143 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.740936 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/016248a0-bbe2-44c9-8e51-5d4695ee6a52-host-cni-bin\") pod \"ovnkube-node-kxvq2\" (UID: \"016248a0-bbe2-44c9-8e51-5d4695ee6a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:30:56.925143 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.740939 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/016248a0-bbe2-44c9-8e51-5d4695ee6a52-run-systemd\") pod \"ovnkube-node-kxvq2\" (UID: \"016248a0-bbe2-44c9-8e51-5d4695ee6a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:30:56.925143 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.740974 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/016248a0-bbe2-44c9-8e51-5d4695ee6a52-host-cni-bin\") pod \"ovnkube-node-kxvq2\" (UID: \"016248a0-bbe2-44c9-8e51-5d4695ee6a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:30:56.925143 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.740976 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqh6h\" (UniqueName: \"kubernetes.io/projected/016248a0-bbe2-44c9-8e51-5d4695ee6a52-kube-api-access-hqh6h\") pod \"ovnkube-node-kxvq2\" (UID: \"016248a0-bbe2-44c9-8e51-5d4695ee6a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:30:56.925143 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.741010 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8d9a6391-5821-4247-b85e-f6089b65f9f0-etc-selinux\") pod \"aws-ebs-csi-driver-node-xwv5j\" (UID: \"8d9a6391-5821-4247-b85e-f6089b65f9f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwv5j" Jul 10 14:30:56.925143 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.741085 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fc922b38-8f9b-4ec4-95c0-c132253a3158-cni-binary-copy\") pod \"multus-r6rkz\" (UID: \"fc922b38-8f9b-4ec4-95c0-c132253a3158\") " pod="openshift-multus/multus-r6rkz" Jul 10 14:30:56.925143 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.741123 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7vqc\" (UniqueName: \"kubernetes.io/projected/3a4169fa-a43d-4cf7-b104-b92cbfcdbcb3-kube-api-access-k7vqc\") pod \"iptables-alerter-lfrbz\" (UID: \"3a4169fa-a43d-4cf7-b104-b92cbfcdbcb3\") " pod="openshift-network-operator/iptables-alerter-lfrbz" Jul 10 14:30:56.925143 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.741155 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpsm2\" (UniqueName: \"kubernetes.io/projected/ced07efc-4719-4439-b2be-b84dadef62d3-kube-api-access-cpsm2\") pod \"network-check-target-t9jl6\" (UID: \"ced07efc-4719-4439-b2be-b84dadef62d3\") " pod="openshift-network-diagnostics/network-check-target-t9jl6" Jul 10 14:30:56.926560 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.741173 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8d9a6391-5821-4247-b85e-f6089b65f9f0-etc-selinux\") pod \"aws-ebs-csi-driver-node-xwv5j\" (UID: \"8d9a6391-5821-4247-b85e-f6089b65f9f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwv5j" Jul 10 14:30:56.926560 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.741178 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9hdw\" (UniqueName: \"kubernetes.io/projected/bc2e14e7-0dd6-4e09-b724-948115817cb8-kube-api-access-t9hdw\") pod \"tuned-7jtx8\" (UID: \"bc2e14e7-0dd6-4e09-b724-948115817cb8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jtx8" Jul 10 14:30:56.926560 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.741222 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ba16bc50-891a-4477-8bd5-de3ef058d04f-cni-binary-copy\") pod \"multus-additional-cni-plugins-2sjq2\" (UID: \"ba16bc50-891a-4477-8bd5-de3ef058d04f\") " pod="openshift-multus/multus-additional-cni-plugins-2sjq2" Jul 10 14:30:56.926560 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.741254 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/016248a0-bbe2-44c9-8e51-5d4695ee6a52-host-cni-netd\") pod \"ovnkube-node-kxvq2\" (UID: \"016248a0-bbe2-44c9-8e51-5d4695ee6a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:30:56.926560 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.741285 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/016248a0-bbe2-44c9-8e51-5d4695ee6a52-ovnkube-config\") pod \"ovnkube-node-kxvq2\" (UID: \"016248a0-bbe2-44c9-8e51-5d4695ee6a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:30:56.926560 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.741309 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8d9a6391-5821-4247-b85e-f6089b65f9f0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xwv5j\" (UID: \"8d9a6391-5821-4247-b85e-f6089b65f9f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwv5j" Jul 10 14:30:56.926560 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.741324 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/016248a0-bbe2-44c9-8e51-5d4695ee6a52-host-cni-netd\") pod \"ovnkube-node-kxvq2\" (UID: \"016248a0-bbe2-44c9-8e51-5d4695ee6a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:30:56.926560 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.741334 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fc922b38-8f9b-4ec4-95c0-c132253a3158-host-run-k8s-cni-cncf-io\") pod \"multus-r6rkz\" (UID: \"fc922b38-8f9b-4ec4-95c0-c132253a3158\") " pod="openshift-multus/multus-r6rkz" Jul 10 14:30:56.926560 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.741356 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bc2e14e7-0dd6-4e09-b724-948115817cb8-etc-sysctl-conf\") pod \"tuned-7jtx8\" (UID: \"bc2e14e7-0dd6-4e09-b724-948115817cb8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jtx8" Jul 10 14:30:56.926560 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.741364 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4ee5894d-0007-499f-a401-2f11724910ac-konnectivity-ca\") pod \"konnectivity-agent-j2g7m\" (UID: \"4ee5894d-0007-499f-a401-2f11724910ac\") " pod="kube-system/konnectivity-agent-j2g7m" Jul 10 14:30:56.926560 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.741379 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vhlx\" (UniqueName: \"kubernetes.io/projected/ff3c9cb2-1924-4ea9-9a93-282213b083db-kube-api-access-4vhlx\") pod \"network-metrics-daemon-fdhw7\" (UID: \"ff3c9cb2-1924-4ea9-9a93-282213b083db\") " pod="openshift-multus/network-metrics-daemon-fdhw7" Jul 10 14:30:56.926560 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.741848 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc922b38-8f9b-4ec4-95c0-c132253a3158-multus-cni-dir\") pod \"multus-r6rkz\" (UID: \"fc922b38-8f9b-4ec4-95c0-c132253a3158\") " pod="openshift-multus/multus-r6rkz" Jul 10 14:30:56.926560 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.741902 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fc922b38-8f9b-4ec4-95c0-c132253a3158-cnibin\") pod \"multus-r6rkz\" (UID: \"fc922b38-8f9b-4ec4-95c0-c132253a3158\") " pod="openshift-multus/multus-r6rkz" Jul 10 14:30:56.926560 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.741940 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fc922b38-8f9b-4ec4-95c0-c132253a3158-multus-daemon-config\") pod \"multus-r6rkz\" (UID: \"fc922b38-8f9b-4ec4-95c0-c132253a3158\") " pod="openshift-multus/multus-r6rkz" Jul 10 14:30:56.927494 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.741966 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bc2e14e7-0dd6-4e09-b724-948115817cb8-var-lib-kubelet\") pod \"tuned-7jtx8\" (UID: \"bc2e14e7-0dd6-4e09-b724-948115817cb8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jtx8" Jul 10 14:30:56.927494 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.741997 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ba16bc50-891a-4477-8bd5-de3ef058d04f-os-release\") pod \"multus-additional-cni-plugins-2sjq2\" (UID: \"ba16bc50-891a-4477-8bd5-de3ef058d04f\") " pod="openshift-multus/multus-additional-cni-plugins-2sjq2" Jul 10 14:30:56.927494 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.741999 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8d9a6391-5821-4247-b85e-f6089b65f9f0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xwv5j\" (UID: \"8d9a6391-5821-4247-b85e-f6089b65f9f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwv5j" Jul 10 14:30:56.927494 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.742031 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5j9p\" (UniqueName: \"kubernetes.io/projected/ba16bc50-891a-4477-8bd5-de3ef058d04f-kube-api-access-f5j9p\") pod \"multus-additional-cni-plugins-2sjq2\" (UID: \"ba16bc50-891a-4477-8bd5-de3ef058d04f\") " pod="openshift-multus/multus-additional-cni-plugins-2sjq2" Jul 10 14:30:56.927494 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.742070 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/016248a0-bbe2-44c9-8e51-5d4695ee6a52-etc-openvswitch\") pod \"ovnkube-node-kxvq2\" (UID: \"016248a0-bbe2-44c9-8e51-5d4695ee6a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:30:56.927494 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.742115 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8d9a6391-5821-4247-b85e-f6089b65f9f0-registration-dir\") pod \"aws-ebs-csi-driver-node-xwv5j\" (UID: \"8d9a6391-5821-4247-b85e-f6089b65f9f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwv5j" Jul 10 14:30:56.927494 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.742125 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/016248a0-bbe2-44c9-8e51-5d4695ee6a52-etc-openvswitch\") pod \"ovnkube-node-kxvq2\" (UID: \"016248a0-bbe2-44c9-8e51-5d4695ee6a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:30:56.927494 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.742149 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/016248a0-bbe2-44c9-8e51-5d4695ee6a52-ovnkube-config\") pod \"ovnkube-node-kxvq2\" (UID: \"016248a0-bbe2-44c9-8e51-5d4695ee6a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:30:56.927494 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.742244 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8d9a6391-5821-4247-b85e-f6089b65f9f0-device-dir\") pod \"aws-ebs-csi-driver-node-xwv5j\" (UID: \"8d9a6391-5821-4247-b85e-f6089b65f9f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwv5j" Jul 10 14:30:56.927494 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.742271 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8d9a6391-5821-4247-b85e-f6089b65f9f0-device-dir\") pod \"aws-ebs-csi-driver-node-xwv5j\" (UID: \"8d9a6391-5821-4247-b85e-f6089b65f9f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwv5j" Jul 10 14:30:56.927494 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.742286 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8d9a6391-5821-4247-b85e-f6089b65f9f0-registration-dir\") pod \"aws-ebs-csi-driver-node-xwv5j\" (UID: \"8d9a6391-5821-4247-b85e-f6089b65f9f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwv5j" Jul 10 14:30:56.927494 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.742310 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fc922b38-8f9b-4ec4-95c0-c132253a3158-hostroot\") pod \"multus-r6rkz\" (UID: \"fc922b38-8f9b-4ec4-95c0-c132253a3158\") " pod="openshift-multus/multus-r6rkz" Jul 10 14:30:56.927494 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.742346 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bc2e14e7-0dd6-4e09-b724-948115817cb8-etc-modprobe-d\") pod \"tuned-7jtx8\" (UID: \"bc2e14e7-0dd6-4e09-b724-948115817cb8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jtx8" Jul 10 14:30:56.927494 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.742382 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/270982ae-963c-42a2-a10f-06cd88f30f81-hosts-file\") pod \"node-resolver-842qb\" (UID: \"270982ae-963c-42a2-a10f-06cd88f30f81\") " pod="openshift-dns/node-resolver-842qb" Jul 10 14:30:56.929359 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.742414 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/016248a0-bbe2-44c9-8e51-5d4695ee6a52-run-ovn\") pod \"ovnkube-node-kxvq2\" (UID: \"016248a0-bbe2-44c9-8e51-5d4695ee6a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:30:56.929359 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.742467 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/016248a0-bbe2-44c9-8e51-5d4695ee6a52-host-run-ovn-kubernetes\") pod \"ovnkube-node-kxvq2\" (UID: \"016248a0-bbe2-44c9-8e51-5d4695ee6a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:30:56.929359 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.742478 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/270982ae-963c-42a2-a10f-06cd88f30f81-hosts-file\") pod \"node-resolver-842qb\" (UID: \"270982ae-963c-42a2-a10f-06cd88f30f81\") " pod="openshift-dns/node-resolver-842qb" Jul 10 14:30:56.929359 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.742499 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4ee5894d-0007-499f-a401-2f11724910ac-agent-certs\") pod \"konnectivity-agent-j2g7m\" (UID: \"4ee5894d-0007-499f-a401-2f11724910ac\") " pod="kube-system/konnectivity-agent-j2g7m" Jul 10 14:30:56.929359 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.742547 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/016248a0-bbe2-44c9-8e51-5d4695ee6a52-run-ovn\") pod \"ovnkube-node-kxvq2\" (UID: \"016248a0-bbe2-44c9-8e51-5d4695ee6a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:30:56.929359 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.742588 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bc2e14e7-0dd6-4e09-b724-948115817cb8-host\") pod \"tuned-7jtx8\" (UID: \"bc2e14e7-0dd6-4e09-b724-948115817cb8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jtx8" Jul 10 14:30:56.929359 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.742626 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ba16bc50-891a-4477-8bd5-de3ef058d04f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2sjq2\" (UID: \"ba16bc50-891a-4477-8bd5-de3ef058d04f\") " pod="openshift-multus/multus-additional-cni-plugins-2sjq2" Jul 10 14:30:56.929359 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.742666 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/016248a0-bbe2-44c9-8e51-5d4695ee6a52-host-slash\") pod \"ovnkube-node-kxvq2\" (UID: \"016248a0-bbe2-44c9-8e51-5d4695ee6a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:30:56.929359 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.742699 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/016248a0-bbe2-44c9-8e51-5d4695ee6a52-log-socket\") pod \"ovnkube-node-kxvq2\" (UID: \"016248a0-bbe2-44c9-8e51-5d4695ee6a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:30:56.929359 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.742735 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/21295856-adfb-486d-a3dc-b9ff8aa3ff50-serviceca\") pod \"node-ca-w9ww7\" (UID: \"21295856-adfb-486d-a3dc-b9ff8aa3ff50\") " pod="openshift-image-registry/node-ca-w9ww7" Jul 10 14:30:56.929359 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.742785 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2r2m\" (UniqueName: \"kubernetes.io/projected/21295856-adfb-486d-a3dc-b9ff8aa3ff50-kube-api-access-k2r2m\") pod \"node-ca-w9ww7\" (UID: \"21295856-adfb-486d-a3dc-b9ff8aa3ff50\") " pod="openshift-image-registry/node-ca-w9ww7" Jul 10 14:30:56.929359 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.742819 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fc922b38-8f9b-4ec4-95c0-c132253a3158-os-release\") pod \"multus-r6rkz\" (UID: \"fc922b38-8f9b-4ec4-95c0-c132253a3158\") " pod="openshift-multus/multus-r6rkz" Jul 10 14:30:56.929359 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.742851 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fc922b38-8f9b-4ec4-95c0-c132253a3158-host-var-lib-cni-multus\") pod \"multus-r6rkz\" (UID: \"fc922b38-8f9b-4ec4-95c0-c132253a3158\") " pod="openshift-multus/multus-r6rkz" Jul 10 14:30:56.929359 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.742865 2569 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 10 14:30:56.929359 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.742890 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc922b38-8f9b-4ec4-95c0-c132253a3158-etc-kubernetes\") pod \"multus-r6rkz\" (UID: \"fc922b38-8f9b-4ec4-95c0-c132253a3158\") " pod="openshift-multus/multus-r6rkz" Jul 10 14:30:56.930258 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.742918 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz992\" (UniqueName: \"kubernetes.io/projected/fc922b38-8f9b-4ec4-95c0-c132253a3158-kube-api-access-fz992\") pod \"multus-r6rkz\" (UID: \"fc922b38-8f9b-4ec4-95c0-c132253a3158\") " pod="openshift-multus/multus-r6rkz" Jul 10 14:30:56.930258 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.742953 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/270982ae-963c-42a2-a10f-06cd88f30f81-tmp-dir\") pod \"node-resolver-842qb\" (UID: \"270982ae-963c-42a2-a10f-06cd88f30f81\") " pod="openshift-dns/node-resolver-842qb" Jul 10 14:30:56.930258 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.742930 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/016248a0-bbe2-44c9-8e51-5d4695ee6a52-log-socket\") pod \"ovnkube-node-kxvq2\" (UID: \"016248a0-bbe2-44c9-8e51-5d4695ee6a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:30:56.930258 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.743008 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/016248a0-bbe2-44c9-8e51-5d4695ee6a52-host-slash\") pod \"ovnkube-node-kxvq2\" (UID: \"016248a0-bbe2-44c9-8e51-5d4695ee6a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:30:56.930258 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.743039 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/016248a0-bbe2-44c9-8e51-5d4695ee6a52-host-run-ovn-kubernetes\") pod \"ovnkube-node-kxvq2\" (UID: \"016248a0-bbe2-44c9-8e51-5d4695ee6a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:30:56.930258 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.743071 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fc922b38-8f9b-4ec4-95c0-c132253a3158-host-var-lib-cni-bin\") pod \"multus-r6rkz\" (UID: \"fc922b38-8f9b-4ec4-95c0-c132253a3158\") " pod="openshift-multus/multus-r6rkz" Jul 10 14:30:56.930258 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.743121 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc2e14e7-0dd6-4e09-b724-948115817cb8-etc-kubernetes\") pod \"tuned-7jtx8\" (UID: \"bc2e14e7-0dd6-4e09-b724-948115817cb8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jtx8" Jul 10 14:30:56.930258 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.743174 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3a4169fa-a43d-4cf7-b104-b92cbfcdbcb3-iptables-alerter-script\") pod \"iptables-alerter-lfrbz\" (UID: \"3a4169fa-a43d-4cf7-b104-b92cbfcdbcb3\") " pod="openshift-network-operator/iptables-alerter-lfrbz" Jul 10 14:30:56.930258 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.743209 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/016248a0-bbe2-44c9-8e51-5d4695ee6a52-env-overrides\") pod \"ovnkube-node-kxvq2\" (UID: \"016248a0-bbe2-44c9-8e51-5d4695ee6a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:30:56.930258 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.743244 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/016248a0-bbe2-44c9-8e51-5d4695ee6a52-ovn-node-metrics-cert\") pod \"ovnkube-node-kxvq2\" (UID: \"016248a0-bbe2-44c9-8e51-5d4695ee6a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:30:56.930258 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.743277 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/270982ae-963c-42a2-a10f-06cd88f30f81-tmp-dir\") pod \"node-resolver-842qb\" (UID: \"270982ae-963c-42a2-a10f-06cd88f30f81\") " pod="openshift-dns/node-resolver-842qb" Jul 10 14:30:56.930258 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.743287 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/016248a0-bbe2-44c9-8e51-5d4695ee6a52-ovnkube-script-lib\") pod \"ovnkube-node-kxvq2\" (UID: \"016248a0-bbe2-44c9-8e51-5d4695ee6a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:30:56.930258 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.743382 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc922b38-8f9b-4ec4-95c0-c132253a3158-system-cni-dir\") pod \"multus-r6rkz\" (UID: \"fc922b38-8f9b-4ec4-95c0-c132253a3158\") " pod="openshift-multus/multus-r6rkz" Jul 10 14:30:56.930258 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.743450 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f-original-pull-secret\") pod \"global-pull-secret-syncer-rxmgq\" (UID: \"4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f\") " pod="kube-system/global-pull-secret-syncer-rxmgq" Jul 10 14:30:56.932065 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.743484 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ba16bc50-891a-4477-8bd5-de3ef058d04f-cnibin\") pod \"multus-additional-cni-plugins-2sjq2\" (UID: \"ba16bc50-891a-4477-8bd5-de3ef058d04f\") " pod="openshift-multus/multus-additional-cni-plugins-2sjq2" Jul 10 14:30:56.932065 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.743516 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3a4169fa-a43d-4cf7-b104-b92cbfcdbcb3-host-slash\") pod \"iptables-alerter-lfrbz\" (UID: \"3a4169fa-a43d-4cf7-b104-b92cbfcdbcb3\") " pod="openshift-network-operator/iptables-alerter-lfrbz" Jul 10 14:30:56.932065 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.743545 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/016248a0-bbe2-44c9-8e51-5d4695ee6a52-node-log\") pod \"ovnkube-node-kxvq2\" (UID: \"016248a0-bbe2-44c9-8e51-5d4695ee6a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:30:56.932065 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.743581 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmsn7\" (UniqueName: \"kubernetes.io/projected/8d9a6391-5821-4247-b85e-f6089b65f9f0-kube-api-access-vmsn7\") pod \"aws-ebs-csi-driver-node-xwv5j\" (UID: \"8d9a6391-5821-4247-b85e-f6089b65f9f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwv5j" Jul 10 14:30:56.932065 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.743612 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bc2e14e7-0dd6-4e09-b724-948115817cb8-etc-sysctl-d\") pod \"tuned-7jtx8\" (UID: \"bc2e14e7-0dd6-4e09-b724-948115817cb8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jtx8" Jul 10 14:30:56.932065 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.743641 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bc2e14e7-0dd6-4e09-b724-948115817cb8-lib-modules\") pod \"tuned-7jtx8\" (UID: \"bc2e14e7-0dd6-4e09-b724-948115817cb8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jtx8" Jul 10 14:30:56.932065 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.743661 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/016248a0-bbe2-44c9-8e51-5d4695ee6a52-env-overrides\") pod \"ovnkube-node-kxvq2\" (UID: \"016248a0-bbe2-44c9-8e51-5d4695ee6a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:30:56.932065 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.743667 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgtfp\" (UniqueName: \"kubernetes.io/projected/270982ae-963c-42a2-a10f-06cd88f30f81-kube-api-access-kgtfp\") pod \"node-resolver-842qb\" (UID: \"270982ae-963c-42a2-a10f-06cd88f30f81\") " pod="openshift-dns/node-resolver-842qb" Jul 10 14:30:56.932065 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.743701 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/016248a0-bbe2-44c9-8e51-5d4695ee6a52-var-lib-openvswitch\") pod \"ovnkube-node-kxvq2\" (UID: \"016248a0-bbe2-44c9-8e51-5d4695ee6a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:30:56.932065 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.743733 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8d9a6391-5821-4247-b85e-f6089b65f9f0-sys-fs\") pod \"aws-ebs-csi-driver-node-xwv5j\" (UID: \"8d9a6391-5821-4247-b85e-f6089b65f9f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwv5j" Jul 10 14:30:56.932065 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.743734 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/016248a0-bbe2-44c9-8e51-5d4695ee6a52-node-log\") pod \"ovnkube-node-kxvq2\" (UID: \"016248a0-bbe2-44c9-8e51-5d4695ee6a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:30:56.932065 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.744105 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/016248a0-bbe2-44c9-8e51-5d4695ee6a52-ovnkube-script-lib\") pod \"ovnkube-node-kxvq2\" (UID: \"016248a0-bbe2-44c9-8e51-5d4695ee6a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:30:56.932065 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.744116 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/016248a0-bbe2-44c9-8e51-5d4695ee6a52-var-lib-openvswitch\") pod \"ovnkube-node-kxvq2\" (UID: \"016248a0-bbe2-44c9-8e51-5d4695ee6a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:30:56.932065 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.744330 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8d9a6391-5821-4247-b85e-f6089b65f9f0-sys-fs\") pod \"aws-ebs-csi-driver-node-xwv5j\" (UID: \"8d9a6391-5821-4247-b85e-f6089b65f9f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwv5j" Jul 10 14:30:56.932743 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.747722 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4ee5894d-0007-499f-a401-2f11724910ac-agent-certs\") pod \"konnectivity-agent-j2g7m\" (UID: \"4ee5894d-0007-499f-a401-2f11724910ac\") " pod="kube-system/konnectivity-agent-j2g7m" Jul 10 14:30:56.932743 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.747807 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/016248a0-bbe2-44c9-8e51-5d4695ee6a52-ovn-node-metrics-cert\") pod \"ovnkube-node-kxvq2\" (UID: \"016248a0-bbe2-44c9-8e51-5d4695ee6a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:30:56.932743 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.752452 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vhlx\" (UniqueName: \"kubernetes.io/projected/ff3c9cb2-1924-4ea9-9a93-282213b083db-kube-api-access-4vhlx\") pod \"network-metrics-daemon-fdhw7\" (UID: \"ff3c9cb2-1924-4ea9-9a93-282213b083db\") " pod="openshift-multus/network-metrics-daemon-fdhw7" Jul 10 14:30:56.932743 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.752876 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqh6h\" (UniqueName: \"kubernetes.io/projected/016248a0-bbe2-44c9-8e51-5d4695ee6a52-kube-api-access-hqh6h\") pod \"ovnkube-node-kxvq2\" (UID: \"016248a0-bbe2-44c9-8e51-5d4695ee6a52\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:30:56.932743 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.753819 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgtfp\" (UniqueName: \"kubernetes.io/projected/270982ae-963c-42a2-a10f-06cd88f30f81-kube-api-access-kgtfp\") pod \"node-resolver-842qb\" (UID: \"270982ae-963c-42a2-a10f-06cd88f30f81\") " pod="openshift-dns/node-resolver-842qb" Jul 10 14:30:56.932743 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.756127 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmsn7\" (UniqueName: \"kubernetes.io/projected/8d9a6391-5821-4247-b85e-f6089b65f9f0-kube-api-access-vmsn7\") pod \"aws-ebs-csi-driver-node-xwv5j\" (UID: \"8d9a6391-5821-4247-b85e-f6089b65f9f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwv5j" Jul 10 14:30:56.932743 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.763847 2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 14:30:56.932743 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.844135 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc922b38-8f9b-4ec4-95c0-c132253a3158-multus-cni-dir\") pod \"multus-r6rkz\" (UID: \"fc922b38-8f9b-4ec4-95c0-c132253a3158\") " pod="openshift-multus/multus-r6rkz" Jul 10 14:30:56.932743 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.844169 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fc922b38-8f9b-4ec4-95c0-c132253a3158-cnibin\") pod \"multus-r6rkz\" (UID: \"fc922b38-8f9b-4ec4-95c0-c132253a3158\") " pod="openshift-multus/multus-r6rkz" Jul 10 14:30:56.932743 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.844199 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fc922b38-8f9b-4ec4-95c0-c132253a3158-multus-daemon-config\") pod \"multus-r6rkz\" (UID: \"fc922b38-8f9b-4ec4-95c0-c132253a3158\") " pod="openshift-multus/multus-r6rkz" Jul 10 14:30:56.932743 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.844224 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bc2e14e7-0dd6-4e09-b724-948115817cb8-var-lib-kubelet\") pod \"tuned-7jtx8\" (UID: \"bc2e14e7-0dd6-4e09-b724-948115817cb8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jtx8" Jul 10 14:30:56.932743 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.844249 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ba16bc50-891a-4477-8bd5-de3ef058d04f-os-release\") pod \"multus-additional-cni-plugins-2sjq2\" (UID: \"ba16bc50-891a-4477-8bd5-de3ef058d04f\") " pod="openshift-multus/multus-additional-cni-plugins-2sjq2" Jul 10 14:30:56.932743 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.844276 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5j9p\" (UniqueName: \"kubernetes.io/projected/ba16bc50-891a-4477-8bd5-de3ef058d04f-kube-api-access-f5j9p\") pod \"multus-additional-cni-plugins-2sjq2\" (UID: \"ba16bc50-891a-4477-8bd5-de3ef058d04f\") " pod="openshift-multus/multus-additional-cni-plugins-2sjq2" Jul 10 14:30:56.932743 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.844302 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fc922b38-8f9b-4ec4-95c0-c132253a3158-hostroot\") pod \"multus-r6rkz\" (UID: \"fc922b38-8f9b-4ec4-95c0-c132253a3158\") " pod="openshift-multus/multus-r6rkz" Jul 10 14:30:56.932743 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.844328 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bc2e14e7-0dd6-4e09-b724-948115817cb8-etc-modprobe-d\") pod \"tuned-7jtx8\" (UID: \"bc2e14e7-0dd6-4e09-b724-948115817cb8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jtx8" Jul 10 14:30:56.935487 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.844354 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bc2e14e7-0dd6-4e09-b724-948115817cb8-host\") pod \"tuned-7jtx8\" (UID: \"bc2e14e7-0dd6-4e09-b724-948115817cb8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jtx8" Jul 10 14:30:56.935487 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.844399 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ba16bc50-891a-4477-8bd5-de3ef058d04f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2sjq2\" (UID: \"ba16bc50-891a-4477-8bd5-de3ef058d04f\") " pod="openshift-multus/multus-additional-cni-plugins-2sjq2" Jul 10 14:30:56.935487 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.844397 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bc2e14e7-0dd6-4e09-b724-948115817cb8-var-lib-kubelet\") pod \"tuned-7jtx8\" (UID: \"bc2e14e7-0dd6-4e09-b724-948115817cb8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jtx8" Jul 10 14:30:56.935487 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.844412 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fc922b38-8f9b-4ec4-95c0-c132253a3158-cnibin\") pod \"multus-r6rkz\" (UID: \"fc922b38-8f9b-4ec4-95c0-c132253a3158\") " pod="openshift-multus/multus-r6rkz" Jul 10 14:30:56.935487 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.844427 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/21295856-adfb-486d-a3dc-b9ff8aa3ff50-serviceca\") pod \"node-ca-w9ww7\" (UID: \"21295856-adfb-486d-a3dc-b9ff8aa3ff50\") " pod="openshift-image-registry/node-ca-w9ww7" Jul 10 14:30:56.935487 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.844431 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fc922b38-8f9b-4ec4-95c0-c132253a3158-hostroot\") pod \"multus-r6rkz\" (UID: \"fc922b38-8f9b-4ec4-95c0-c132253a3158\") " pod="openshift-multus/multus-r6rkz" Jul 10 14:30:56.935487 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.844454 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ba16bc50-891a-4477-8bd5-de3ef058d04f-os-release\") pod \"multus-additional-cni-plugins-2sjq2\" (UID: \"ba16bc50-891a-4477-8bd5-de3ef058d04f\") " pod="openshift-multus/multus-additional-cni-plugins-2sjq2" Jul 10 14:30:56.935487 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.844467 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2r2m\" (UniqueName: \"kubernetes.io/projected/21295856-adfb-486d-a3dc-b9ff8aa3ff50-kube-api-access-k2r2m\") pod \"node-ca-w9ww7\" (UID: \"21295856-adfb-486d-a3dc-b9ff8aa3ff50\") " pod="openshift-image-registry/node-ca-w9ww7" Jul 10 14:30:56.935487 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.844486 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bc2e14e7-0dd6-4e09-b724-948115817cb8-host\") pod \"tuned-7jtx8\" (UID: \"bc2e14e7-0dd6-4e09-b724-948115817cb8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jtx8" Jul 10 14:30:56.935487 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.844494 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fc922b38-8f9b-4ec4-95c0-c132253a3158-os-release\") pod \"multus-r6rkz\" (UID: \"fc922b38-8f9b-4ec4-95c0-c132253a3158\") " pod="openshift-multus/multus-r6rkz" Jul 10 14:30:56.935487 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.844517 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fc922b38-8f9b-4ec4-95c0-c132253a3158-host-var-lib-cni-multus\") pod \"multus-r6rkz\" (UID: \"fc922b38-8f9b-4ec4-95c0-c132253a3158\") " pod="openshift-multus/multus-r6rkz" Jul 10 14:30:56.935487 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.844561 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc922b38-8f9b-4ec4-95c0-c132253a3158-etc-kubernetes\") pod \"multus-r6rkz\" (UID: \"fc922b38-8f9b-4ec4-95c0-c132253a3158\") " pod="openshift-multus/multus-r6rkz" Jul 10 14:30:56.935487 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.844567 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bc2e14e7-0dd6-4e09-b724-948115817cb8-etc-modprobe-d\") pod \"tuned-7jtx8\" (UID: \"bc2e14e7-0dd6-4e09-b724-948115817cb8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jtx8" Jul 10 14:30:56.935487 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.844610 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc922b38-8f9b-4ec4-95c0-c132253a3158-etc-kubernetes\") pod \"multus-r6rkz\" (UID: \"fc922b38-8f9b-4ec4-95c0-c132253a3158\") " pod="openshift-multus/multus-r6rkz" Jul 10 14:30:56.935487 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.844621 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fc922b38-8f9b-4ec4-95c0-c132253a3158-os-release\") pod \"multus-r6rkz\" (UID: \"fc922b38-8f9b-4ec4-95c0-c132253a3158\") " pod="openshift-multus/multus-r6rkz" Jul 10 14:30:56.936477 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.844645 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz992\" (UniqueName: \"kubernetes.io/projected/fc922b38-8f9b-4ec4-95c0-c132253a3158-kube-api-access-fz992\") pod \"multus-r6rkz\" (UID: \"fc922b38-8f9b-4ec4-95c0-c132253a3158\") " pod="openshift-multus/multus-r6rkz" Jul 10 14:30:56.936477 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.844677 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fc922b38-8f9b-4ec4-95c0-c132253a3158-host-var-lib-cni-bin\") pod \"multus-r6rkz\" (UID: \"fc922b38-8f9b-4ec4-95c0-c132253a3158\") " pod="openshift-multus/multus-r6rkz" Jul 10 14:30:56.936477 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.844646 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ba16bc50-891a-4477-8bd5-de3ef058d04f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2sjq2\" (UID: \"ba16bc50-891a-4477-8bd5-de3ef058d04f\") " pod="openshift-multus/multus-additional-cni-plugins-2sjq2" Jul 10 14:30:56.936477 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.844710 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc2e14e7-0dd6-4e09-b724-948115817cb8-etc-kubernetes\") pod \"tuned-7jtx8\" (UID: \"bc2e14e7-0dd6-4e09-b724-948115817cb8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jtx8" Jul 10 14:30:56.936477 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.844724 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fc922b38-8f9b-4ec4-95c0-c132253a3158-host-var-lib-cni-bin\") pod \"multus-r6rkz\" (UID: \"fc922b38-8f9b-4ec4-95c0-c132253a3158\") " pod="openshift-multus/multus-r6rkz" Jul 10 14:30:56.936477 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.844738 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3a4169fa-a43d-4cf7-b104-b92cbfcdbcb3-iptables-alerter-script\") pod \"iptables-alerter-lfrbz\" (UID: \"3a4169fa-a43d-4cf7-b104-b92cbfcdbcb3\") " pod="openshift-network-operator/iptables-alerter-lfrbz" Jul 10 14:30:56.936477 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.844662 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fc922b38-8f9b-4ec4-95c0-c132253a3158-host-var-lib-cni-multus\") pod \"multus-r6rkz\" (UID: \"fc922b38-8f9b-4ec4-95c0-c132253a3158\") " pod="openshift-multus/multus-r6rkz" Jul 10 14:30:56.936477 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.844771 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc922b38-8f9b-4ec4-95c0-c132253a3158-multus-cni-dir\") pod \"multus-r6rkz\" (UID: \"fc922b38-8f9b-4ec4-95c0-c132253a3158\") " pod="openshift-multus/multus-r6rkz" Jul 10 14:30:56.936477 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.844773 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc922b38-8f9b-4ec4-95c0-c132253a3158-system-cni-dir\") pod \"multus-r6rkz\" (UID: \"fc922b38-8f9b-4ec4-95c0-c132253a3158\") " pod="openshift-multus/multus-r6rkz" Jul 10 14:30:56.936477 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.844823 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc2e14e7-0dd6-4e09-b724-948115817cb8-etc-kubernetes\") pod \"tuned-7jtx8\" (UID: \"bc2e14e7-0dd6-4e09-b724-948115817cb8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jtx8" Jul 10 14:30:56.936477 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.844831 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc922b38-8f9b-4ec4-95c0-c132253a3158-system-cni-dir\") pod \"multus-r6rkz\" (UID: \"fc922b38-8f9b-4ec4-95c0-c132253a3158\") " pod="openshift-multus/multus-r6rkz" Jul 10 14:30:56.936477 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.844836 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f-original-pull-secret\") pod \"global-pull-secret-syncer-rxmgq\" (UID: \"4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f\") " pod="kube-system/global-pull-secret-syncer-rxmgq" Jul 10 14:30:56.936477 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.844867 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ba16bc50-891a-4477-8bd5-de3ef058d04f-cnibin\") pod \"multus-additional-cni-plugins-2sjq2\" (UID: \"ba16bc50-891a-4477-8bd5-de3ef058d04f\") " pod="openshift-multus/multus-additional-cni-plugins-2sjq2" Jul 10 14:30:56.936477 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.844893 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3a4169fa-a43d-4cf7-b104-b92cbfcdbcb3-host-slash\") pod \"iptables-alerter-lfrbz\" (UID: \"3a4169fa-a43d-4cf7-b104-b92cbfcdbcb3\") " pod="openshift-network-operator/iptables-alerter-lfrbz" Jul 10 14:30:56.936477 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.844920 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bc2e14e7-0dd6-4e09-b724-948115817cb8-etc-sysctl-d\") pod \"tuned-7jtx8\" (UID: \"bc2e14e7-0dd6-4e09-b724-948115817cb8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jtx8" Jul 10 14:30:56.937796 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.844926 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/21295856-adfb-486d-a3dc-b9ff8aa3ff50-serviceca\") pod \"node-ca-w9ww7\" (UID: \"21295856-adfb-486d-a3dc-b9ff8aa3ff50\") " pod="openshift-image-registry/node-ca-w9ww7" Jul 10 14:30:56.937796 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:30:56.844936 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 14:30:56.937796 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.844946 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bc2e14e7-0dd6-4e09-b724-948115817cb8-lib-modules\") pod \"tuned-7jtx8\" (UID: \"bc2e14e7-0dd6-4e09-b724-948115817cb8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jtx8" Jul 10 14:30:56.937796 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.844980 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ba16bc50-891a-4477-8bd5-de3ef058d04f-cnibin\") pod \"multus-additional-cni-plugins-2sjq2\" (UID: \"ba16bc50-891a-4477-8bd5-de3ef058d04f\") " pod="openshift-multus/multus-additional-cni-plugins-2sjq2" Jul 10 14:30:56.937796 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:30:56.845003 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f-original-pull-secret podName:4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f nodeName:}" failed. No retries permitted until 2026-07-10 14:30:57.344982913 +0000 UTC m=+3.096354610 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f-original-pull-secret") pod "global-pull-secret-syncer-rxmgq" (UID: "4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f") : object "kube-system"/"original-pull-secret" not registered Jul 10 14:30:56.937796 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.845033 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fc922b38-8f9b-4ec4-95c0-c132253a3158-multus-daemon-config\") pod \"multus-r6rkz\" (UID: \"fc922b38-8f9b-4ec4-95c0-c132253a3158\") " pod="openshift-multus/multus-r6rkz" Jul 10 14:30:56.937796 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.845059 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3a4169fa-a43d-4cf7-b104-b92cbfcdbcb3-host-slash\") pod \"iptables-alerter-lfrbz\" (UID: \"3a4169fa-a43d-4cf7-b104-b92cbfcdbcb3\") " pod="openshift-network-operator/iptables-alerter-lfrbz" Jul 10 14:30:56.937796 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.845083 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bc2e14e7-0dd6-4e09-b724-948115817cb8-lib-modules\") pod \"tuned-7jtx8\" (UID: \"bc2e14e7-0dd6-4e09-b724-948115817cb8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jtx8" Jul 10 14:30:56.937796 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.845094 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bc2e14e7-0dd6-4e09-b724-948115817cb8-etc-sysctl-d\") pod \"tuned-7jtx8\" (UID: \"bc2e14e7-0dd6-4e09-b724-948115817cb8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jtx8" Jul 10 14:30:56.937796 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.845144 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bc2e14e7-0dd6-4e09-b724-948115817cb8-etc-sysconfig\") pod \"tuned-7jtx8\" (UID: \"bc2e14e7-0dd6-4e09-b724-948115817cb8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jtx8" Jul 10 14:30:56.937796 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.845185 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f-kubelet-config\") pod \"global-pull-secret-syncer-rxmgq\" (UID: \"4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f\") " pod="kube-system/global-pull-secret-syncer-rxmgq" Jul 10 14:30:56.937796 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.845213 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ba16bc50-891a-4477-8bd5-de3ef058d04f-system-cni-dir\") pod \"multus-additional-cni-plugins-2sjq2\" (UID: \"ba16bc50-891a-4477-8bd5-de3ef058d04f\") " pod="openshift-multus/multus-additional-cni-plugins-2sjq2" Jul 10 14:30:56.937796 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.845246 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ba16bc50-891a-4477-8bd5-de3ef058d04f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2sjq2\" (UID: \"ba16bc50-891a-4477-8bd5-de3ef058d04f\") " pod="openshift-multus/multus-additional-cni-plugins-2sjq2" Jul 10 14:30:56.937796 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.845275 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc922b38-8f9b-4ec4-95c0-c132253a3158-host-var-lib-kubelet\") pod \"multus-r6rkz\" (UID: \"fc922b38-8f9b-4ec4-95c0-c132253a3158\") " pod="openshift-multus/multus-r6rkz" Jul 10 14:30:56.938586 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.845287 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f-kubelet-config\") pod \"global-pull-secret-syncer-rxmgq\" (UID: \"4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f\") " pod="kube-system/global-pull-secret-syncer-rxmgq" Jul 10 14:30:56.938586 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.845301 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bc2e14e7-0dd6-4e09-b724-948115817cb8-run\") pod \"tuned-7jtx8\" (UID: \"bc2e14e7-0dd6-4e09-b724-948115817cb8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jtx8" Jul 10 14:30:56.938586 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.845307 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3a4169fa-a43d-4cf7-b104-b92cbfcdbcb3-iptables-alerter-script\") pod \"iptables-alerter-lfrbz\" (UID: \"3a4169fa-a43d-4cf7-b104-b92cbfcdbcb3\") " pod="openshift-network-operator/iptables-alerter-lfrbz" Jul 10 14:30:56.938586 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.845304 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bc2e14e7-0dd6-4e09-b724-948115817cb8-etc-sysconfig\") pod \"tuned-7jtx8\" (UID: \"bc2e14e7-0dd6-4e09-b724-948115817cb8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jtx8" Jul 10 14:30:56.938586 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.845344 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/21295856-adfb-486d-a3dc-b9ff8aa3ff50-host\") pod \"node-ca-w9ww7\" (UID: \"21295856-adfb-486d-a3dc-b9ff8aa3ff50\") " pod="openshift-image-registry/node-ca-w9ww7" Jul 10 14:30:56.938586 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.845353 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc922b38-8f9b-4ec4-95c0-c132253a3158-host-var-lib-kubelet\") pod \"multus-r6rkz\" (UID: \"fc922b38-8f9b-4ec4-95c0-c132253a3158\") " pod="openshift-multus/multus-r6rkz" Jul 10 14:30:56.938586 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.845373 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fc922b38-8f9b-4ec4-95c0-c132253a3158-host-run-multus-certs\") pod \"multus-r6rkz\" (UID: \"fc922b38-8f9b-4ec4-95c0-c132253a3158\") " pod="openshift-multus/multus-r6rkz" Jul 10 14:30:56.938586 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.845384 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bc2e14e7-0dd6-4e09-b724-948115817cb8-run\") pod \"tuned-7jtx8\" (UID: \"bc2e14e7-0dd6-4e09-b724-948115817cb8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jtx8" Jul 10 14:30:56.938586 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.845402 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bc2e14e7-0dd6-4e09-b724-948115817cb8-etc-systemd\") pod \"tuned-7jtx8\" (UID: \"bc2e14e7-0dd6-4e09-b724-948115817cb8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jtx8" Jul 10 14:30:56.938586 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.845423 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fc922b38-8f9b-4ec4-95c0-c132253a3158-host-run-multus-certs\") pod \"multus-r6rkz\" (UID: \"fc922b38-8f9b-4ec4-95c0-c132253a3158\") " pod="openshift-multus/multus-r6rkz" Jul 10 14:30:56.938586 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.845429 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bc2e14e7-0dd6-4e09-b724-948115817cb8-etc-tuned\") pod \"tuned-7jtx8\" (UID: \"bc2e14e7-0dd6-4e09-b724-948115817cb8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jtx8" Jul 10 14:30:56.938586 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.845464 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc2e14e7-0dd6-4e09-b724-948115817cb8-tmp\") pod \"tuned-7jtx8\" (UID: \"bc2e14e7-0dd6-4e09-b724-948115817cb8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jtx8" Jul 10 14:30:56.938586 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.845470 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bc2e14e7-0dd6-4e09-b724-948115817cb8-etc-systemd\") pod \"tuned-7jtx8\" (UID: \"bc2e14e7-0dd6-4e09-b724-948115817cb8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jtx8" Jul 10 14:30:56.938586 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.845404 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/21295856-adfb-486d-a3dc-b9ff8aa3ff50-host\") pod \"node-ca-w9ww7\" (UID: \"21295856-adfb-486d-a3dc-b9ff8aa3ff50\") " pod="openshift-image-registry/node-ca-w9ww7" Jul 10 14:30:56.938586 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.845434 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ba16bc50-891a-4477-8bd5-de3ef058d04f-system-cni-dir\") pod \"multus-additional-cni-plugins-2sjq2\" (UID: \"ba16bc50-891a-4477-8bd5-de3ef058d04f\") " pod="openshift-multus/multus-additional-cni-plugins-2sjq2" Jul 10 14:30:56.999244 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.845499 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ba16bc50-891a-4477-8bd5-de3ef058d04f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2sjq2\" (UID: \"ba16bc50-891a-4477-8bd5-de3ef058d04f\") " pod="openshift-multus/multus-additional-cni-plugins-2sjq2" Jul 10 14:30:56.999244 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.845531 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fc922b38-8f9b-4ec4-95c0-c132253a3158-multus-socket-dir-parent\") pod \"multus-r6rkz\" (UID: \"fc922b38-8f9b-4ec4-95c0-c132253a3158\") " pod="openshift-multus/multus-r6rkz" Jul 10 14:30:56.999244 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.845557 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fc922b38-8f9b-4ec4-95c0-c132253a3158-host-run-netns\") pod \"multus-r6rkz\" (UID: \"fc922b38-8f9b-4ec4-95c0-c132253a3158\") " pod="openshift-multus/multus-r6rkz" Jul 10 14:30:56.999244 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.845582 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fc922b38-8f9b-4ec4-95c0-c132253a3158-multus-conf-dir\") pod \"multus-r6rkz\" (UID: \"fc922b38-8f9b-4ec4-95c0-c132253a3158\") " pod="openshift-multus/multus-r6rkz" Jul 10 14:30:56.999244 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.845607 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f-dbus\") pod \"global-pull-secret-syncer-rxmgq\" (UID: \"4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f\") " pod="kube-system/global-pull-secret-syncer-rxmgq" Jul 10 14:30:56.999244 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.845628 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fc922b38-8f9b-4ec4-95c0-c132253a3158-multus-socket-dir-parent\") pod \"multus-r6rkz\" (UID: \"fc922b38-8f9b-4ec4-95c0-c132253a3158\") " pod="openshift-multus/multus-r6rkz" Jul 10 14:30:56.999244 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.845679 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fc922b38-8f9b-4ec4-95c0-c132253a3158-multus-conf-dir\") pod \"multus-r6rkz\" (UID: \"fc922b38-8f9b-4ec4-95c0-c132253a3158\") " pod="openshift-multus/multus-r6rkz" Jul 10 14:30:56.999244 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.845679 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bc2e14e7-0dd6-4e09-b724-948115817cb8-sys\") pod \"tuned-7jtx8\" (UID: \"bc2e14e7-0dd6-4e09-b724-948115817cb8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jtx8" Jul 10 14:30:56.999244 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.845707 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fc922b38-8f9b-4ec4-95c0-c132253a3158-host-run-netns\") pod \"multus-r6rkz\" (UID: \"fc922b38-8f9b-4ec4-95c0-c132253a3158\") " pod="openshift-multus/multus-r6rkz" Jul 10 14:30:56.999244 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.845631 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bc2e14e7-0dd6-4e09-b724-948115817cb8-sys\") pod \"tuned-7jtx8\" (UID: \"bc2e14e7-0dd6-4e09-b724-948115817cb8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jtx8" Jul 10 14:30:56.999244 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.845759 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fc922b38-8f9b-4ec4-95c0-c132253a3158-cni-binary-copy\") pod \"multus-r6rkz\" (UID: \"fc922b38-8f9b-4ec4-95c0-c132253a3158\") " pod="openshift-multus/multus-r6rkz" Jul 10 14:30:56.999244 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.845788 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f-dbus\") pod \"global-pull-secret-syncer-rxmgq\" (UID: \"4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f\") " pod="kube-system/global-pull-secret-syncer-rxmgq" Jul 10 14:30:56.999244 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.845788 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7vqc\" (UniqueName: \"kubernetes.io/projected/3a4169fa-a43d-4cf7-b104-b92cbfcdbcb3-kube-api-access-k7vqc\") pod \"iptables-alerter-lfrbz\" (UID: \"3a4169fa-a43d-4cf7-b104-b92cbfcdbcb3\") " pod="openshift-network-operator/iptables-alerter-lfrbz" Jul 10 14:30:56.999244 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.845833 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpsm2\" (UniqueName: \"kubernetes.io/projected/ced07efc-4719-4439-b2be-b84dadef62d3-kube-api-access-cpsm2\") pod \"network-check-target-t9jl6\" (UID: \"ced07efc-4719-4439-b2be-b84dadef62d3\") " pod="openshift-network-diagnostics/network-check-target-t9jl6" Jul 10 14:30:56.999244 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.845860 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9hdw\" (UniqueName: \"kubernetes.io/projected/bc2e14e7-0dd6-4e09-b724-948115817cb8-kube-api-access-t9hdw\") pod \"tuned-7jtx8\" (UID: \"bc2e14e7-0dd6-4e09-b724-948115817cb8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jtx8" Jul 10 14:30:57.002613 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.845894 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ba16bc50-891a-4477-8bd5-de3ef058d04f-cni-binary-copy\") pod \"multus-additional-cni-plugins-2sjq2\" (UID: \"ba16bc50-891a-4477-8bd5-de3ef058d04f\") " pod="openshift-multus/multus-additional-cni-plugins-2sjq2" Jul 10 14:30:57.002613 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.845911 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ba16bc50-891a-4477-8bd5-de3ef058d04f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2sjq2\" (UID: \"ba16bc50-891a-4477-8bd5-de3ef058d04f\") " pod="openshift-multus/multus-additional-cni-plugins-2sjq2" Jul 10 14:30:57.002613 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.845925 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fc922b38-8f9b-4ec4-95c0-c132253a3158-host-run-k8s-cni-cncf-io\") pod \"multus-r6rkz\" (UID: \"fc922b38-8f9b-4ec4-95c0-c132253a3158\") " pod="openshift-multus/multus-r6rkz" Jul 10 14:30:57.002613 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.845951 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bc2e14e7-0dd6-4e09-b724-948115817cb8-etc-sysctl-conf\") pod \"tuned-7jtx8\" (UID: \"bc2e14e7-0dd6-4e09-b724-948115817cb8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jtx8" Jul 10 14:30:57.002613 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.846067 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bc2e14e7-0dd6-4e09-b724-948115817cb8-etc-sysctl-conf\") pod \"tuned-7jtx8\" (UID: \"bc2e14e7-0dd6-4e09-b724-948115817cb8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jtx8" Jul 10 14:30:57.002613 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.846114 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fc922b38-8f9b-4ec4-95c0-c132253a3158-host-run-k8s-cni-cncf-io\") pod \"multus-r6rkz\" (UID: \"fc922b38-8f9b-4ec4-95c0-c132253a3158\") " pod="openshift-multus/multus-r6rkz" Jul 10 14:30:57.002613 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.846229 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ba16bc50-891a-4477-8bd5-de3ef058d04f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2sjq2\" (UID: \"ba16bc50-891a-4477-8bd5-de3ef058d04f\") " pod="openshift-multus/multus-additional-cni-plugins-2sjq2" Jul 10 14:30:57.002613 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.846278 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fc922b38-8f9b-4ec4-95c0-c132253a3158-cni-binary-copy\") pod \"multus-r6rkz\" (UID: \"fc922b38-8f9b-4ec4-95c0-c132253a3158\") " pod="openshift-multus/multus-r6rkz" Jul 10 14:30:57.002613 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.846457 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ba16bc50-891a-4477-8bd5-de3ef058d04f-cni-binary-copy\") pod \"multus-additional-cni-plugins-2sjq2\" (UID: \"ba16bc50-891a-4477-8bd5-de3ef058d04f\") " pod="openshift-multus/multus-additional-cni-plugins-2sjq2" Jul 10 14:30:57.002613 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.848032 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc2e14e7-0dd6-4e09-b724-948115817cb8-tmp\") pod \"tuned-7jtx8\" (UID: \"bc2e14e7-0dd6-4e09-b724-948115817cb8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jtx8" Jul 10 14:30:57.002613 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.848091 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bc2e14e7-0dd6-4e09-b724-948115817cb8-etc-tuned\") pod \"tuned-7jtx8\" (UID: \"bc2e14e7-0dd6-4e09-b724-948115817cb8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jtx8" Jul 10 14:30:57.002613 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:30:56.853142 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 14:30:57.002613 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:30:56.853161 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 14:30:57.002613 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:30:56.853174 2569 projected.go:196] Error preparing data for projected volume kube-api-access-cpsm2 for pod openshift-network-diagnostics/network-check-target-t9jl6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 14:30:57.006202 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:30:56.853240 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ced07efc-4719-4439-b2be-b84dadef62d3-kube-api-access-cpsm2 podName:ced07efc-4719-4439-b2be-b84dadef62d3 nodeName:}" failed. No retries permitted until 2026-07-10 14:30:57.353219893 +0000 UTC m=+3.104591589 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-cpsm2" (UniqueName: "kubernetes.io/projected/ced07efc-4719-4439-b2be-b84dadef62d3-kube-api-access-cpsm2") pod "network-check-target-t9jl6" (UID: "ced07efc-4719-4439-b2be-b84dadef62d3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 14:30:57.006202 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.854041 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2r2m\" (UniqueName: \"kubernetes.io/projected/21295856-adfb-486d-a3dc-b9ff8aa3ff50-kube-api-access-k2r2m\") pod \"node-ca-w9ww7\" (UID: \"21295856-adfb-486d-a3dc-b9ff8aa3ff50\") " pod="openshift-image-registry/node-ca-w9ww7" Jul 10 14:30:57.006202 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.854209 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5j9p\" (UniqueName: \"kubernetes.io/projected/ba16bc50-891a-4477-8bd5-de3ef058d04f-kube-api-access-f5j9p\") pod \"multus-additional-cni-plugins-2sjq2\" (UID: \"ba16bc50-891a-4477-8bd5-de3ef058d04f\") " pod="openshift-multus/multus-additional-cni-plugins-2sjq2" Jul 10 14:30:57.006202 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.854395 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz992\" (UniqueName: \"kubernetes.io/projected/fc922b38-8f9b-4ec4-95c0-c132253a3158-kube-api-access-fz992\") pod \"multus-r6rkz\" (UID: \"fc922b38-8f9b-4ec4-95c0-c132253a3158\") " pod="openshift-multus/multus-r6rkz" Jul 10 14:30:57.006202 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.855938 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9hdw\" (UniqueName: \"kubernetes.io/projected/bc2e14e7-0dd6-4e09-b724-948115817cb8-kube-api-access-t9hdw\") pod \"tuned-7jtx8\" (UID: \"bc2e14e7-0dd6-4e09-b724-948115817cb8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jtx8" Jul 10 14:30:57.006202 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.855975 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7vqc\" (UniqueName: \"kubernetes.io/projected/3a4169fa-a43d-4cf7-b104-b92cbfcdbcb3-kube-api-access-k7vqc\") pod \"iptables-alerter-lfrbz\" (UID: \"3a4169fa-a43d-4cf7-b104-b92cbfcdbcb3\") " pod="openshift-network-operator/iptables-alerter-lfrbz" Jul 10 14:30:57.006202 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.932270 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-842qb" Jul 10 14:30:57.006202 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.939148 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-35.ec2.internal" event={"ID":"12fdb1aa4afb104872332bdd0f7db7e5","Type":"ContainerStarted","Data":"eb9e8676b32ae764ada6925152ceb5fceb7663c6139db5255b35e40d68a70c09"} Jul 10 14:30:57.006202 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.942159 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-35.ec2.internal" event={"ID":"486f4ab7baea4188ee524f2052e08a91","Type":"ContainerStarted","Data":"1496d00d2a9a5103764ed0b9d33b7ff953ad0e82a6b7e75d21414404d9cb9183"} Jul 10 14:30:57.006202 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.944655 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:30:57.006202 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:56.946253 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod270982ae_963c_42a2_a10f_06cd88f30f81.slice/crio-b7f7669ed30176507708084864a05d470d9f0a8ba1bc654dabcd0885e16f123d WatchSource:0}: Error finding container b7f7669ed30176507708084864a05d470d9f0a8ba1bc654dabcd0885e16f123d: Status 404 returned error can't find the container with id b7f7669ed30176507708084864a05d470d9f0a8ba1bc654dabcd0885e16f123d Jul 10 14:30:57.006202 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.954207 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-j2g7m" Jul 10 14:30:57.006202 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:56.955154 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod016248a0_bbe2_44c9_8e51_5d4695ee6a52.slice/crio-f9f40f1e9b024dc6aeae1190780aaa384a5f1428fdd9ed25830066641eeae395 WatchSource:0}: Error finding container f9f40f1e9b024dc6aeae1190780aaa384a5f1428fdd9ed25830066641eeae395: Status 404 returned error can't find the container with id f9f40f1e9b024dc6aeae1190780aaa384a5f1428fdd9ed25830066641eeae395 Jul 10 14:30:57.006202 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.962480 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwv5j" Jul 10 14:30:57.006202 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.970988 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2sjq2" Jul 10 14:30:57.007564 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:56.976941 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d9a6391_5821_4247_b85e_f6089b65f9f0.slice/crio-8d38c9c47d1b19d00758c9afc0dd79ea582883f12411f2335d6bd2692c3bf229 WatchSource:0}: Error finding container 8d38c9c47d1b19d00758c9afc0dd79ea582883f12411f2335d6bd2692c3bf229: Status 404 returned error can't find the container with id 8d38c9c47d1b19d00758c9afc0dd79ea582883f12411f2335d6bd2692c3bf229 Jul 10 14:30:57.007564 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.983227 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lfrbz" Jul 10 14:30:57.007564 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:56.983415 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba16bc50_891a_4477_8bd5_de3ef058d04f.slice/crio-29cac3a544a71a58ec7d1c0758110ba2febb5c92917e7b3fa896700cf1ca8a5b WatchSource:0}: Error finding container 29cac3a544a71a58ec7d1c0758110ba2febb5c92917e7b3fa896700cf1ca8a5b: Status 404 returned error can't find the container with id 29cac3a544a71a58ec7d1c0758110ba2febb5c92917e7b3fa896700cf1ca8a5b Jul 10 14:30:57.007564 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:56.993651 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-w9ww7" Jul 10 14:30:57.007564 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:56.994421 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a4169fa_a43d_4cf7_b104_b92cbfcdbcb3.slice/crio-d53cac06eeff66d41f2e92f7edef0c369c44be9dec6a67c7c26bf67280fc2d1c WatchSource:0}: Error finding container d53cac06eeff66d41f2e92f7edef0c369c44be9dec6a67c7c26bf67280fc2d1c: Status 404 returned error can't find the container with id d53cac06eeff66d41f2e92f7edef0c369c44be9dec6a67c7c26bf67280fc2d1c Jul 10 14:30:57.007564 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:57.001973 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-r6rkz" Jul 10 14:30:57.007564 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:57.004973 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21295856_adfb_486d_a3dc_b9ff8aa3ff50.slice/crio-906d9926309e8d6e8c764089b9e8fba34ef8f8b0be33820e672e66b93e69ff7c WatchSource:0}: Error finding container 906d9926309e8d6e8c764089b9e8fba34ef8f8b0be33820e672e66b93e69ff7c: Status 404 returned error can't find the container with id 906d9926309e8d6e8c764089b9e8fba34ef8f8b0be33820e672e66b93e69ff7c Jul 10 14:30:57.011208 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:57.010813 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7jtx8" Jul 10 14:30:57.013086 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:57.011463 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc922b38_8f9b_4ec4_95c0_c132253a3158.slice/crio-11ccd2ee31d8f0aa56c5c681887fdcef87243f3c34ebdbe14f40bf7a26b64008 WatchSource:0}: Error finding container 11ccd2ee31d8f0aa56c5c681887fdcef87243f3c34ebdbe14f40bf7a26b64008: Status 404 returned error can't find the container with id 11ccd2ee31d8f0aa56c5c681887fdcef87243f3c34ebdbe14f40bf7a26b64008 Jul 10 14:30:57.021537 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:30:57.020556 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc2e14e7_0dd6_4e09_b724_948115817cb8.slice/crio-8bd0177c148116f1850004ad3cc1725dd34b7370e0d7188d373984a0af665809 WatchSource:0}: Error finding container 8bd0177c148116f1850004ad3cc1725dd34b7370e0d7188d373984a0af665809: Status 404 returned error can't find the container with id 8bd0177c148116f1850004ad3cc1725dd34b7370e0d7188d373984a0af665809 Jul 10 14:30:57.290354 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:57.248252 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff3c9cb2-1924-4ea9-9a93-282213b083db-metrics-certs\") pod \"network-metrics-daemon-fdhw7\" (UID: \"ff3c9cb2-1924-4ea9-9a93-282213b083db\") " pod="openshift-multus/network-metrics-daemon-fdhw7" Jul 10 14:30:57.290354 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:30:57.248401 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 14:30:57.290354 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:30:57.248467 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff3c9cb2-1924-4ea9-9a93-282213b083db-metrics-certs podName:ff3c9cb2-1924-4ea9-9a93-282213b083db nodeName:}" failed. No retries permitted until 2026-07-10 14:30:58.248450878 +0000 UTC m=+3.999822553 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ff3c9cb2-1924-4ea9-9a93-282213b083db-metrics-certs") pod "network-metrics-daemon-fdhw7" (UID: "ff3c9cb2-1924-4ea9-9a93-282213b083db") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 14:30:57.425446 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:57.349200 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f-original-pull-secret\") pod \"global-pull-secret-syncer-rxmgq\" (UID: \"4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f\") " pod="kube-system/global-pull-secret-syncer-rxmgq" Jul 10 14:30:57.425446 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:30:57.349342 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 14:30:57.425446 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:30:57.349435 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f-original-pull-secret podName:4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f nodeName:}" failed. No retries permitted until 2026-07-10 14:30:58.349393059 +0000 UTC m=+4.100764760 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f-original-pull-secret") pod "global-pull-secret-syncer-rxmgq" (UID: "4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f") : object "kube-system"/"original-pull-secret" not registered Jul 10 14:30:57.450609 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:57.450572 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpsm2\" (UniqueName: \"kubernetes.io/projected/ced07efc-4719-4439-b2be-b84dadef62d3-kube-api-access-cpsm2\") pod \"network-check-target-t9jl6\" (UID: \"ced07efc-4719-4439-b2be-b84dadef62d3\") " pod="openshift-network-diagnostics/network-check-target-t9jl6" Jul 10 14:30:57.450795 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:30:57.450753 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 14:30:57.450873 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:30:57.450798 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 14:30:57.450873 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:30:57.450815 2569 projected.go:196] Error preparing data for projected volume kube-api-access-cpsm2 for pod openshift-network-diagnostics/network-check-target-t9jl6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 14:30:57.450954 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:30:57.450879 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ced07efc-4719-4439-b2be-b84dadef62d3-kube-api-access-cpsm2 podName:ced07efc-4719-4439-b2be-b84dadef62d3 nodeName:}" failed. No retries permitted until 2026-07-10 14:30:58.450856603 +0000 UTC m=+4.202228300 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-cpsm2" (UniqueName: "kubernetes.io/projected/ced07efc-4719-4439-b2be-b84dadef62d3-kube-api-access-cpsm2") pod "network-check-target-t9jl6" (UID: "ced07efc-4719-4439-b2be-b84dadef62d3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 14:30:57.715698 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:57.715590 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 14:25:57 +0000 UTC" deadline="2028-03-27 11:35:07.084909372 +0000 UTC" Jul 10 14:30:57.715698 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:57.715646 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15021h4m9.369266405s" Jul 10 14:30:57.925295 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:57.925266 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fdhw7" Jul 10 14:30:57.925824 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:30:57.925398 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fdhw7" podUID="ff3c9cb2-1924-4ea9-9a93-282213b083db" Jul 10 14:30:57.945057 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:57.945026 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-r6rkz" event={"ID":"fc922b38-8f9b-4ec4-95c0-c132253a3158","Type":"ContainerStarted","Data":"11ccd2ee31d8f0aa56c5c681887fdcef87243f3c34ebdbe14f40bf7a26b64008"} Jul 10 14:30:57.946153 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:57.946126 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lfrbz" event={"ID":"3a4169fa-a43d-4cf7-b104-b92cbfcdbcb3","Type":"ContainerStarted","Data":"d53cac06eeff66d41f2e92f7edef0c369c44be9dec6a67c7c26bf67280fc2d1c"} Jul 10 14:30:57.947362 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:57.947341 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-842qb" event={"ID":"270982ae-963c-42a2-a10f-06cd88f30f81","Type":"ContainerStarted","Data":"b7f7669ed30176507708084864a05d470d9f0a8ba1bc654dabcd0885e16f123d"} Jul 10 14:30:57.948437 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:57.948412 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7jtx8" event={"ID":"bc2e14e7-0dd6-4e09-b724-948115817cb8","Type":"ContainerStarted","Data":"8bd0177c148116f1850004ad3cc1725dd34b7370e0d7188d373984a0af665809"} Jul 10 14:30:57.949536 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:57.949503 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-w9ww7" event={"ID":"21295856-adfb-486d-a3dc-b9ff8aa3ff50","Type":"ContainerStarted","Data":"906d9926309e8d6e8c764089b9e8fba34ef8f8b0be33820e672e66b93e69ff7c"} Jul 10 14:30:57.950612 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:57.950592 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2sjq2" event={"ID":"ba16bc50-891a-4477-8bd5-de3ef058d04f","Type":"ContainerStarted","Data":"29cac3a544a71a58ec7d1c0758110ba2febb5c92917e7b3fa896700cf1ca8a5b"} Jul 10 14:30:57.951713 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:57.951695 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwv5j" event={"ID":"8d9a6391-5821-4247-b85e-f6089b65f9f0","Type":"ContainerStarted","Data":"8d38c9c47d1b19d00758c9afc0dd79ea582883f12411f2335d6bd2692c3bf229"} Jul 10 14:30:57.952658 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:57.952637 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-j2g7m" event={"ID":"4ee5894d-0007-499f-a401-2f11724910ac","Type":"ContainerStarted","Data":"b26723e50239466b9ce33a7d688e3a83a4d8ac4669a08cfb9efb294f33cd7fc7"} Jul 10 14:30:57.953669 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:57.953650 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" event={"ID":"016248a0-bbe2-44c9-8e51-5d4695ee6a52","Type":"ContainerStarted","Data":"f9f40f1e9b024dc6aeae1190780aaa384a5f1428fdd9ed25830066641eeae395"} Jul 10 14:30:58.257802 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:58.257751 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff3c9cb2-1924-4ea9-9a93-282213b083db-metrics-certs\") pod \"network-metrics-daemon-fdhw7\" (UID: \"ff3c9cb2-1924-4ea9-9a93-282213b083db\") " pod="openshift-multus/network-metrics-daemon-fdhw7" Jul 10 14:30:58.258001 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:30:58.257974 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 14:30:58.258070 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:30:58.258043 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff3c9cb2-1924-4ea9-9a93-282213b083db-metrics-certs podName:ff3c9cb2-1924-4ea9-9a93-282213b083db nodeName:}" failed. No retries permitted until 2026-07-10 14:31:00.258021621 +0000 UTC m=+6.009393305 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ff3c9cb2-1924-4ea9-9a93-282213b083db-metrics-certs") pod "network-metrics-daemon-fdhw7" (UID: "ff3c9cb2-1924-4ea9-9a93-282213b083db") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 14:30:58.358684 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:58.358641 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f-original-pull-secret\") pod \"global-pull-secret-syncer-rxmgq\" (UID: \"4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f\") " pod="kube-system/global-pull-secret-syncer-rxmgq" Jul 10 14:30:58.358924 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:30:58.358834 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 14:30:58.358924 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:30:58.358917 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f-original-pull-secret podName:4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f nodeName:}" failed. No retries permitted until 2026-07-10 14:31:00.358894934 +0000 UTC m=+6.110266614 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f-original-pull-secret") pod "global-pull-secret-syncer-rxmgq" (UID: "4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f") : object "kube-system"/"original-pull-secret" not registered Jul 10 14:30:58.459139 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:58.459100 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpsm2\" (UniqueName: \"kubernetes.io/projected/ced07efc-4719-4439-b2be-b84dadef62d3-kube-api-access-cpsm2\") pod \"network-check-target-t9jl6\" (UID: \"ced07efc-4719-4439-b2be-b84dadef62d3\") " pod="openshift-network-diagnostics/network-check-target-t9jl6" Jul 10 14:30:58.459371 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:30:58.459331 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 14:30:58.459371 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:30:58.459359 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 14:30:58.459371 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:30:58.459373 2569 projected.go:196] Error preparing data for projected volume kube-api-access-cpsm2 for pod openshift-network-diagnostics/network-check-target-t9jl6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 14:30:58.459574 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:30:58.459436 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ced07efc-4719-4439-b2be-b84dadef62d3-kube-api-access-cpsm2 podName:ced07efc-4719-4439-b2be-b84dadef62d3 nodeName:}" failed. No retries permitted until 2026-07-10 14:31:00.459415791 +0000 UTC m=+6.210787492 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-cpsm2" (UniqueName: "kubernetes.io/projected/ced07efc-4719-4439-b2be-b84dadef62d3-kube-api-access-cpsm2") pod "network-check-target-t9jl6" (UID: "ced07efc-4719-4439-b2be-b84dadef62d3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 14:30:58.616511 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:58.616486 2569 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 14:30:58.716927 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:58.716856 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 14:25:57 +0000 UTC" deadline="2027-12-28 09:46:27.956741048 +0000 UTC" Jul 10 14:30:58.716927 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:58.716895 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12859h15m29.239851053s" Jul 10 14:30:58.927921 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:58.925678 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rxmgq" Jul 10 14:30:58.927921 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:30:58.925840 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rxmgq" podUID="4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f" Jul 10 14:30:58.927921 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:58.926367 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t9jl6" Jul 10 14:30:58.927921 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:30:58.926469 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t9jl6" podUID="ced07efc-4719-4439-b2be-b84dadef62d3" Jul 10 14:30:58.973711 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:58.973009 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-35.ec2.internal" event={"ID":"486f4ab7baea4188ee524f2052e08a91","Type":"ContainerStarted","Data":"433b166470c20b4ebcda9d1d2b39532c2f9f147cb13d6e0903dd8c6b1d65434c"} Jul 10 14:30:59.926031 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:59.925929 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fdhw7" Jul 10 14:30:59.926208 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:30:59.926172 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fdhw7" podUID="ff3c9cb2-1924-4ea9-9a93-282213b083db" Jul 10 14:30:59.984772 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:59.984725 2569 generic.go:356] "Generic (PLEG): container finished" podID="12fdb1aa4afb104872332bdd0f7db7e5" containerID="ff769a8e72b2db09fe5f7ec3c708d4ad930495a2fdf0e23c2bf6e6bba1dd64eb" exitCode=0 Jul 10 14:30:59.985717 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:30:59.985685 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-35.ec2.internal" event={"ID":"12fdb1aa4afb104872332bdd0f7db7e5","Type":"ContainerDied","Data":"ff769a8e72b2db09fe5f7ec3c708d4ad930495a2fdf0e23c2bf6e6bba1dd64eb"} Jul 10 14:31:00.005808 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:00.005753 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-35.ec2.internal" podStartSLOduration=4.005730741 podStartE2EDuration="4.005730741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 14:30:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:30:59.003896838 +0000 UTC m=+4.755268538" watchObservedRunningTime="2026-07-10 14:31:00.005730741 +0000 UTC m=+5.757102444" Jul 10 14:31:00.279904 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:00.278673 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff3c9cb2-1924-4ea9-9a93-282213b083db-metrics-certs\") pod \"network-metrics-daemon-fdhw7\" (UID: \"ff3c9cb2-1924-4ea9-9a93-282213b083db\") " pod="openshift-multus/network-metrics-daemon-fdhw7" Jul 10 14:31:00.279904 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:00.278853 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 14:31:00.279904 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:00.278911 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff3c9cb2-1924-4ea9-9a93-282213b083db-metrics-certs podName:ff3c9cb2-1924-4ea9-9a93-282213b083db nodeName:}" failed. No retries permitted until 2026-07-10 14:31:04.278892961 +0000 UTC m=+10.030264640 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ff3c9cb2-1924-4ea9-9a93-282213b083db-metrics-certs") pod "network-metrics-daemon-fdhw7" (UID: "ff3c9cb2-1924-4ea9-9a93-282213b083db") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 14:31:00.379533 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:00.379491 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f-original-pull-secret\") pod \"global-pull-secret-syncer-rxmgq\" (UID: \"4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f\") " pod="kube-system/global-pull-secret-syncer-rxmgq" Jul 10 14:31:00.379731 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:00.379698 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 14:31:00.379800 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:00.379771 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f-original-pull-secret podName:4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f nodeName:}" failed. No retries permitted until 2026-07-10 14:31:04.379751276 +0000 UTC m=+10.131122953 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f-original-pull-secret") pod "global-pull-secret-syncer-rxmgq" (UID: "4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f") : object "kube-system"/"original-pull-secret" not registered Jul 10 14:31:00.480365 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:00.480244 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpsm2\" (UniqueName: \"kubernetes.io/projected/ced07efc-4719-4439-b2be-b84dadef62d3-kube-api-access-cpsm2\") pod \"network-check-target-t9jl6\" (UID: \"ced07efc-4719-4439-b2be-b84dadef62d3\") " pod="openshift-network-diagnostics/network-check-target-t9jl6" Jul 10 14:31:00.480546 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:00.480461 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 14:31:00.480546 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:00.480480 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 14:31:00.480546 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:00.480494 2569 projected.go:196] Error preparing data for projected volume kube-api-access-cpsm2 for pod openshift-network-diagnostics/network-check-target-t9jl6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 14:31:00.480712 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:00.480597 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ced07efc-4719-4439-b2be-b84dadef62d3-kube-api-access-cpsm2 podName:ced07efc-4719-4439-b2be-b84dadef62d3 nodeName:}" failed. No retries permitted until 2026-07-10 14:31:04.480533441 +0000 UTC m=+10.231905120 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-cpsm2" (UniqueName: "kubernetes.io/projected/ced07efc-4719-4439-b2be-b84dadef62d3-kube-api-access-cpsm2") pod "network-check-target-t9jl6" (UID: "ced07efc-4719-4439-b2be-b84dadef62d3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 14:31:00.928627 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:00.928530 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rxmgq" Jul 10 14:31:00.928821 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:00.928672 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rxmgq" podUID="4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f" Jul 10 14:31:00.929152 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:00.929122 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t9jl6" Jul 10 14:31:00.929275 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:00.929233 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t9jl6" podUID="ced07efc-4719-4439-b2be-b84dadef62d3" Jul 10 14:31:00.999100 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:00.998345 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-35.ec2.internal" event={"ID":"12fdb1aa4afb104872332bdd0f7db7e5","Type":"ContainerStarted","Data":"bc92f124c2122f2810000ee87c5dc05723879a735e2df9bc2a2c933543d8adaa"} Jul 10 14:31:01.016118 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:01.016065 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-35.ec2.internal" podStartSLOduration=5.016043208 podStartE2EDuration="5.016043208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 14:30:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:31:01.015600925 +0000 UTC m=+6.766972626" watchObservedRunningTime="2026-07-10 14:31:01.016043208 +0000 UTC m=+6.767414910" Jul 10 14:31:01.925484 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:01.925447 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fdhw7" Jul 10 14:31:01.925671 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:01.925595 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fdhw7" podUID="ff3c9cb2-1924-4ea9-9a93-282213b083db" Jul 10 14:31:02.925782 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:02.925477 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rxmgq" Jul 10 14:31:02.925782 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:02.925614 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rxmgq" podUID="4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f" Jul 10 14:31:02.926315 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:02.925866 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t9jl6" Jul 10 14:31:02.926315 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:02.925972 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t9jl6" podUID="ced07efc-4719-4439-b2be-b84dadef62d3" Jul 10 14:31:03.925968 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:03.925932 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fdhw7" Jul 10 14:31:03.926423 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:03.926075 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fdhw7" podUID="ff3c9cb2-1924-4ea9-9a93-282213b083db" Jul 10 14:31:04.313691 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:04.313586 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff3c9cb2-1924-4ea9-9a93-282213b083db-metrics-certs\") pod \"network-metrics-daemon-fdhw7\" (UID: \"ff3c9cb2-1924-4ea9-9a93-282213b083db\") " pod="openshift-multus/network-metrics-daemon-fdhw7" Jul 10 14:31:04.313937 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:04.313725 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 14:31:04.313937 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:04.313811 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff3c9cb2-1924-4ea9-9a93-282213b083db-metrics-certs podName:ff3c9cb2-1924-4ea9-9a93-282213b083db nodeName:}" failed. No retries permitted until 2026-07-10 14:31:12.313788266 +0000 UTC m=+18.065159943 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ff3c9cb2-1924-4ea9-9a93-282213b083db-metrics-certs") pod "network-metrics-daemon-fdhw7" (UID: "ff3c9cb2-1924-4ea9-9a93-282213b083db") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 14:31:04.413936 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:04.413906 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f-original-pull-secret\") pod \"global-pull-secret-syncer-rxmgq\" (UID: \"4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f\") " pod="kube-system/global-pull-secret-syncer-rxmgq" Jul 10 14:31:04.414124 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:04.414081 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 14:31:04.414181 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:04.414143 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f-original-pull-secret podName:4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f nodeName:}" failed. No retries permitted until 2026-07-10 14:31:12.414122145 +0000 UTC m=+18.165493837 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f-original-pull-secret") pod "global-pull-secret-syncer-rxmgq" (UID: "4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f") : object "kube-system"/"original-pull-secret" not registered Jul 10 14:31:04.515154 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:04.515105 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpsm2\" (UniqueName: \"kubernetes.io/projected/ced07efc-4719-4439-b2be-b84dadef62d3-kube-api-access-cpsm2\") pod \"network-check-target-t9jl6\" (UID: \"ced07efc-4719-4439-b2be-b84dadef62d3\") " pod="openshift-network-diagnostics/network-check-target-t9jl6" Jul 10 14:31:04.515355 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:04.515272 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 14:31:04.515355 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:04.515292 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 14:31:04.515355 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:04.515304 2569 projected.go:196] Error preparing data for projected volume kube-api-access-cpsm2 for pod openshift-network-diagnostics/network-check-target-t9jl6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 14:31:04.515516 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:04.515366 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ced07efc-4719-4439-b2be-b84dadef62d3-kube-api-access-cpsm2 podName:ced07efc-4719-4439-b2be-b84dadef62d3 nodeName:}" failed. No retries permitted until 2026-07-10 14:31:12.515348164 +0000 UTC m=+18.266719855 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-cpsm2" (UniqueName: "kubernetes.io/projected/ced07efc-4719-4439-b2be-b84dadef62d3-kube-api-access-cpsm2") pod "network-check-target-t9jl6" (UID: "ced07efc-4719-4439-b2be-b84dadef62d3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 14:31:04.926640 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:04.926609 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rxmgq" Jul 10 14:31:04.927129 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:04.926727 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rxmgq" podUID="4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f" Jul 10 14:31:04.927129 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:04.926774 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t9jl6" Jul 10 14:31:04.927129 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:04.926849 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t9jl6" podUID="ced07efc-4719-4439-b2be-b84dadef62d3" Jul 10 14:31:05.926235 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:05.926190 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fdhw7" Jul 10 14:31:05.926404 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:05.926346 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fdhw7" podUID="ff3c9cb2-1924-4ea9-9a93-282213b083db" Jul 10 14:31:06.926343 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:06.926302 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t9jl6" Jul 10 14:31:06.926910 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:06.926459 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t9jl6" podUID="ced07efc-4719-4439-b2be-b84dadef62d3" Jul 10 14:31:06.926910 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:06.926647 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rxmgq" Jul 10 14:31:06.926910 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:06.926812 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rxmgq" podUID="4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f" Jul 10 14:31:07.926165 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:07.926133 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fdhw7" Jul 10 14:31:07.926321 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:07.926259 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fdhw7" podUID="ff3c9cb2-1924-4ea9-9a93-282213b083db" Jul 10 14:31:08.926410 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:08.925804 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rxmgq" Jul 10 14:31:08.926410 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:08.925925 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rxmgq" podUID="4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f" Jul 10 14:31:08.927131 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:08.927112 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t9jl6" Jul 10 14:31:08.927235 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:08.927212 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t9jl6" podUID="ced07efc-4719-4439-b2be-b84dadef62d3" Jul 10 14:31:09.029848 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:09.029811 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-w9ww7" event={"ID":"21295856-adfb-486d-a3dc-b9ff8aa3ff50","Type":"ContainerStarted","Data":"ad25c53e7aeee060ce5e6ed8658a5a729d24f799891e49467a2e40a01bd01daa"} Jul 10 14:31:09.032799 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:09.032744 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2sjq2" event={"ID":"ba16bc50-891a-4477-8bd5-de3ef058d04f","Type":"ContainerStarted","Data":"ab4ca31b187bf6747dcc1340ae83e90d247e370d24f8d625a757f28a1095b5dc"} Jul 10 14:31:09.073035 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:09.072335 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-w9ww7" podStartSLOduration=2.262581147 podStartE2EDuration="14.072312401s" totalImagesPullingTime="11.809731254s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 14:30:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:31:09.045851784 +0000 UTC m=+14.797223482" watchObservedRunningTime="2026-07-10 14:31:09.072312401 +0000 UTC m=+14.823684098" Jul 10 14:31:09.925800 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:09.925707 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fdhw7" Jul 10 14:31:09.925943 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:09.925857 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fdhw7" podUID="ff3c9cb2-1924-4ea9-9a93-282213b083db" Jul 10 14:31:10.040430 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:10.040394 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-842qb" event={"ID":"270982ae-963c-42a2-a10f-06cd88f30f81","Type":"ContainerStarted","Data":"21197ff5e7e9c2f5d4ce878c6810e17e187478703280718a4945a2846604439f"} Jul 10 14:31:10.042638 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:10.042603 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7jtx8" event={"ID":"bc2e14e7-0dd6-4e09-b724-948115817cb8","Type":"ContainerStarted","Data":"e5601210e4d20bc8bf6808512b2dace3e68d5a17e8629be67ebf0744b39c4d3e"} Jul 10 14:31:10.044794 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:10.044766 2569 generic.go:356] "Generic (PLEG): container finished" podID="ba16bc50-891a-4477-8bd5-de3ef058d04f" containerID="ab4ca31b187bf6747dcc1340ae83e90d247e370d24f8d625a757f28a1095b5dc" exitCode=0 Jul 10 14:31:10.044914 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:10.044859 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2sjq2" event={"ID":"ba16bc50-891a-4477-8bd5-de3ef058d04f","Type":"ContainerDied","Data":"ab4ca31b187bf6747dcc1340ae83e90d247e370d24f8d625a757f28a1095b5dc"} Jul 10 14:31:10.047396 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:10.047368 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwv5j" event={"ID":"8d9a6391-5821-4247-b85e-f6089b65f9f0","Type":"ContainerStarted","Data":"53b27b98b5363c8cbff48b76c49caba41a59cff66a602da70bd6a6854326dacf"} Jul 10 14:31:10.049209 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:10.049184 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-j2g7m" event={"ID":"4ee5894d-0007-499f-a401-2f11724910ac","Type":"ContainerStarted","Data":"b594d58bb8f720be8b7354fe5b66ac99ce4b8588850c7a5cf4f2fb139375097c"} Jul 10 14:31:10.056789 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:10.056727 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-842qb" podStartSLOduration=3.147250789 podStartE2EDuration="15.056705111s" totalImagesPullingTime="11.909454322s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 14:30:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:31:10.055996472 +0000 UTC m=+15.807368171" watchObservedRunningTime="2026-07-10 14:31:10.056705111 +0000 UTC m=+15.808076807" Jul 10 14:31:10.073455 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:10.073403 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7jtx8" podStartSLOduration=3.277083259 podStartE2EDuration="15.073389112s" totalImagesPullingTime="11.796305853s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 14:30:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:31:10.073313948 +0000 UTC m=+15.824685645" watchObservedRunningTime="2026-07-10 14:31:10.073389112 +0000 UTC m=+15.824760807" Jul 10 14:31:10.088228 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:10.088169 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-j2g7m" podStartSLOduration=3.237755904 podStartE2EDuration="15.08815444s" totalImagesPullingTime="11.850398536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 14:30:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:31:10.087640713 +0000 UTC m=+15.839012410" watchObservedRunningTime="2026-07-10 14:31:10.08815444 +0000 UTC m=+15.839526135" Jul 10 14:31:10.925508 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:10.925470 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rxmgq" Jul 10 14:31:10.925508 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:10.925504 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t9jl6" Jul 10 14:31:10.925810 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:10.925618 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rxmgq" podUID="4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f" Jul 10 14:31:10.925810 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:10.925674 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t9jl6" podUID="ced07efc-4719-4439-b2be-b84dadef62d3" Jul 10 14:31:11.052639 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:11.052607 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lfrbz" event={"ID":"3a4169fa-a43d-4cf7-b104-b92cbfcdbcb3","Type":"ContainerStarted","Data":"24217abccc7482a45ccf90b7a63d685bfe7d3055d6d189bdeefd0e5b87746ff0"} Jul 10 14:31:11.090916 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:11.090870 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-lfrbz" podStartSLOduration=4.268528995 podStartE2EDuration="16.090854504s" totalImagesPullingTime="11.822325509s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 14:30:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:31:11.089916449 +0000 UTC m=+16.841288146" watchObservedRunningTime="2026-07-10 14:31:11.090854504 +0000 UTC m=+16.842226200" Jul 10 14:31:11.925698 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:11.925666 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fdhw7" Jul 10 14:31:11.925931 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:11.925828 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fdhw7" podUID="ff3c9cb2-1924-4ea9-9a93-282213b083db" Jul 10 14:31:12.381125 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:12.381085 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff3c9cb2-1924-4ea9-9a93-282213b083db-metrics-certs\") pod \"network-metrics-daemon-fdhw7\" (UID: \"ff3c9cb2-1924-4ea9-9a93-282213b083db\") " pod="openshift-multus/network-metrics-daemon-fdhw7" Jul 10 14:31:12.381518 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:12.381205 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 14:31:12.381518 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:12.381274 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff3c9cb2-1924-4ea9-9a93-282213b083db-metrics-certs podName:ff3c9cb2-1924-4ea9-9a93-282213b083db nodeName:}" failed. No retries permitted until 2026-07-10 14:31:28.381253593 +0000 UTC m=+34.132625268 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ff3c9cb2-1924-4ea9-9a93-282213b083db-metrics-certs") pod "network-metrics-daemon-fdhw7" (UID: "ff3c9cb2-1924-4ea9-9a93-282213b083db") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 14:31:12.481849 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:12.481816 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f-original-pull-secret\") pod \"global-pull-secret-syncer-rxmgq\" (UID: \"4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f\") " pod="kube-system/global-pull-secret-syncer-rxmgq" Jul 10 14:31:12.482009 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:12.481976 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 14:31:12.482082 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:12.482049 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f-original-pull-secret podName:4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f nodeName:}" failed. No retries permitted until 2026-07-10 14:31:28.482028431 +0000 UTC m=+34.233400111 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f-original-pull-secret") pod "global-pull-secret-syncer-rxmgq" (UID: "4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f") : object "kube-system"/"original-pull-secret" not registered Jul 10 14:31:12.583209 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:12.583170 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpsm2\" (UniqueName: \"kubernetes.io/projected/ced07efc-4719-4439-b2be-b84dadef62d3-kube-api-access-cpsm2\") pod \"network-check-target-t9jl6\" (UID: \"ced07efc-4719-4439-b2be-b84dadef62d3\") " pod="openshift-network-diagnostics/network-check-target-t9jl6" Jul 10 14:31:12.583366 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:12.583300 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 14:31:12.583366 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:12.583314 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 14:31:12.583366 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:12.583325 2569 projected.go:196] Error preparing data for projected volume kube-api-access-cpsm2 for pod openshift-network-diagnostics/network-check-target-t9jl6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 14:31:12.583508 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:12.583390 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ced07efc-4719-4439-b2be-b84dadef62d3-kube-api-access-cpsm2 podName:ced07efc-4719-4439-b2be-b84dadef62d3 nodeName:}" failed. No retries permitted until 2026-07-10 14:31:28.583371372 +0000 UTC m=+34.334743067 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-cpsm2" (UniqueName: "kubernetes.io/projected/ced07efc-4719-4439-b2be-b84dadef62d3-kube-api-access-cpsm2") pod "network-check-target-t9jl6" (UID: "ced07efc-4719-4439-b2be-b84dadef62d3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 14:31:12.925453 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:12.925410 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rxmgq" Jul 10 14:31:12.925618 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:12.925417 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t9jl6" Jul 10 14:31:12.925618 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:12.925549 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rxmgq" podUID="4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f" Jul 10 14:31:12.925744 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:12.925634 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t9jl6" podUID="ced07efc-4719-4439-b2be-b84dadef62d3" Jul 10 14:31:13.926065 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:13.926030 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fdhw7" Jul 10 14:31:13.926660 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:13.926151 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fdhw7" podUID="ff3c9cb2-1924-4ea9-9a93-282213b083db" Jul 10 14:31:14.756535 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:14.756311 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-j2g7m" Jul 10 14:31:14.757188 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:14.757164 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-j2g7m" Jul 10 14:31:14.926218 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:14.926183 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rxmgq" Jul 10 14:31:14.926661 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:14.926286 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rxmgq" podUID="4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f" Jul 10 14:31:14.926661 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:14.926343 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t9jl6" Jul 10 14:31:14.926661 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:14.926453 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t9jl6" podUID="ced07efc-4719-4439-b2be-b84dadef62d3" Jul 10 14:31:15.060776 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:15.060665 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-j2g7m" Jul 10 14:31:15.061209 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:15.061183 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-j2g7m" Jul 10 14:31:15.925651 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:15.925626 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fdhw7" Jul 10 14:31:15.925807 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:15.925721 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fdhw7" podUID="ff3c9cb2-1924-4ea9-9a93-282213b083db" Jul 10 14:31:16.925787 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:16.925741 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rxmgq" Jul 10 14:31:16.926268 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:16.925794 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t9jl6" Jul 10 14:31:16.926268 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:16.925876 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rxmgq" podUID="4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f" Jul 10 14:31:16.926268 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:16.926017 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t9jl6" podUID="ced07efc-4719-4439-b2be-b84dadef62d3" Jul 10 14:31:17.925787 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:17.925745 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fdhw7" Jul 10 14:31:17.925920 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:17.925890 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fdhw7" podUID="ff3c9cb2-1924-4ea9-9a93-282213b083db" Jul 10 14:31:18.068821 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:18.068785 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-r6rkz" event={"ID":"fc922b38-8f9b-4ec4-95c0-c132253a3158","Type":"ContainerStarted","Data":"f064ea1abf09f3c004e2f68dd017cf029b6d5650201909f0a3b828d82969ffae"} Jul 10 14:31:18.074104 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:18.072935 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxvq2_016248a0-bbe2-44c9-8e51-5d4695ee6a52/ovn-acl-logging/0.log" Jul 10 14:31:18.074104 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:18.073396 2569 generic.go:356] "Generic (PLEG): container finished" podID="016248a0-bbe2-44c9-8e51-5d4695ee6a52" containerID="58e4ca33e6c4316dc9b27d2a68ae17836cd09a48fcca355c642bd52d10a6a843" exitCode=1 Jul 10 14:31:18.074104 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:18.073487 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" event={"ID":"016248a0-bbe2-44c9-8e51-5d4695ee6a52","Type":"ContainerStarted","Data":"314bb9c9e3f7edce586bf177c3c2dc7fe39891b6a0d90a7a6e46180845457b2b"} Jul 10 14:31:18.074429 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:18.074170 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" event={"ID":"016248a0-bbe2-44c9-8e51-5d4695ee6a52","Type":"ContainerDied","Data":"58e4ca33e6c4316dc9b27d2a68ae17836cd09a48fcca355c642bd52d10a6a843"} Jul 10 14:31:18.074429 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:18.074311 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" event={"ID":"016248a0-bbe2-44c9-8e51-5d4695ee6a52","Type":"ContainerStarted","Data":"e1374945e6cd827eba6ff7e841d2327598e85f164278406a86e22f4feff8975b"} Jul 10 14:31:18.609403 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:18.609233 2569 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 10 14:31:18.702957 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:18.702829 2569 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-10T14:31:18.609397568Z","UUID":"e62bf9c9-5a4d-48fa-8d85-b1b4e378597e","Handler":null,"Name":"","Endpoint":""} Jul 10 14:31:18.705233 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:18.705210 2569 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 10 14:31:18.705360 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:18.705249 2569 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 10 14:31:18.925869 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:18.925839 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t9jl6" Jul 10 14:31:18.925869 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:18.925871 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rxmgq" Jul 10 14:31:18.926431 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:18.925972 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t9jl6" podUID="ced07efc-4719-4439-b2be-b84dadef62d3" Jul 10 14:31:18.926431 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:18.926107 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rxmgq" podUID="4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f" Jul 10 14:31:19.079517 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:19.079494 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxvq2_016248a0-bbe2-44c9-8e51-5d4695ee6a52/ovn-acl-logging/0.log" Jul 10 14:31:19.079985 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:19.079952 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" event={"ID":"016248a0-bbe2-44c9-8e51-5d4695ee6a52","Type":"ContainerStarted","Data":"b41e3f5228e323162755d5ee544532a4750607de52f0b604865b597ab1a2fd12"} Jul 10 14:31:19.080131 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:19.080031 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" event={"ID":"016248a0-bbe2-44c9-8e51-5d4695ee6a52","Type":"ContainerStarted","Data":"71fb5037b1cb64fda08f24794d5e5ade96dcaa52c76cea120f12190c4efa2bcd"} Jul 10 14:31:19.080131 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:19.080080 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" event={"ID":"016248a0-bbe2-44c9-8e51-5d4695ee6a52","Type":"ContainerStarted","Data":"b432c1c0d7cd028769a5b65aecdc428bfe456aaf0564701bcdb4387bea573032"} Jul 10 14:31:19.081900 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:19.081875 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwv5j" event={"ID":"8d9a6391-5821-4247-b85e-f6089b65f9f0","Type":"ContainerStarted","Data":"b2111e2d89d108e834d3d64db3eb30a6687c5f1e113e5ea0cbf74205d233d8f3"} Jul 10 14:31:19.925823 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:19.925770 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fdhw7" Jul 10 14:31:19.926049 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:19.925878 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fdhw7" podUID="ff3c9cb2-1924-4ea9-9a93-282213b083db" Jul 10 14:31:20.925448 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:20.925407 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rxmgq" Jul 10 14:31:20.925448 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:20.925446 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t9jl6" Jul 10 14:31:20.926086 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:20.925525 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rxmgq" podUID="4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f" Jul 10 14:31:20.926086 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:20.925639 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t9jl6" podUID="ced07efc-4719-4439-b2be-b84dadef62d3" Jul 10 14:31:21.088253 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:21.088164 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwv5j" event={"ID":"8d9a6391-5821-4247-b85e-f6089b65f9f0","Type":"ContainerStarted","Data":"f4d055e1ab3b7190e39b77b589925b12f0850fd3906b611cf599270dacadb10a"} Jul 10 14:31:21.091522 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:21.091498 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxvq2_016248a0-bbe2-44c9-8e51-5d4695ee6a52/ovn-acl-logging/0.log" Jul 10 14:31:21.091901 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:21.091874 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" event={"ID":"016248a0-bbe2-44c9-8e51-5d4695ee6a52","Type":"ContainerStarted","Data":"4fc0d9dcb70455f3fd7a4ed115c79cc9f8155868688a03bf57a1d103794b282a"} Jul 10 14:31:21.108133 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:21.108096 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwv5j" podStartSLOduration=3.396060379 podStartE2EDuration="26.108084468s" totalImagesPullingTime="22.712024089s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 14:30:55 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:31:21.107070436 +0000 UTC m=+26.858442133" watchObservedRunningTime="2026-07-10 14:31:21.108084468 +0000 UTC m=+26.859456164" Jul 10 14:31:21.108532 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:21.108512 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-r6rkz" podStartSLOduration=5.519850315 podStartE2EDuration="26.108506871s" totalImagesPullingTime="20.588656556s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 14:30:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:31:18.086563165 +0000 UTC m=+23.837934862" watchObservedRunningTime="2026-07-10 14:31:21.108506871 +0000 UTC m=+26.859878566" Jul 10 14:31:21.925294 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:21.925269 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fdhw7" Jul 10 14:31:21.925485 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:21.925388 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fdhw7" podUID="ff3c9cb2-1924-4ea9-9a93-282213b083db" Jul 10 14:31:22.926157 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:22.926126 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rxmgq" Jul 10 14:31:22.926516 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:22.926129 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t9jl6" Jul 10 14:31:22.926516 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:22.926256 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rxmgq" podUID="4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f" Jul 10 14:31:22.926516 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:22.926299 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t9jl6" podUID="ced07efc-4719-4439-b2be-b84dadef62d3" Jul 10 14:31:23.925558 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:23.925366 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fdhw7" Jul 10 14:31:23.925731 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:23.925663 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fdhw7" podUID="ff3c9cb2-1924-4ea9-9a93-282213b083db" Jul 10 14:31:24.102157 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:24.102132 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxvq2_016248a0-bbe2-44c9-8e51-5d4695ee6a52/ovn-acl-logging/0.log" Jul 10 14:31:24.102707 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:24.102514 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" event={"ID":"016248a0-bbe2-44c9-8e51-5d4695ee6a52","Type":"ContainerStarted","Data":"febd31fa85309792ed092bc87cce4b15a4fd0ea5f22adca6066166e4e564fcb3"} Jul 10 14:31:24.102785 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:24.102747 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:31:24.102850 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:24.102827 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:31:24.102983 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:24.102963 2569 scope.go:117] "RemoveContainer" containerID="58e4ca33e6c4316dc9b27d2a68ae17836cd09a48fcca355c642bd52d10a6a843" Jul 10 14:31:24.104470 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:24.104444 2569 generic.go:356] "Generic (PLEG): container finished" podID="ba16bc50-891a-4477-8bd5-de3ef058d04f" containerID="511954a3e69a7d2599136434394563913d9481ce0915c2772ce40e580980b23c" exitCode=0 Jul 10 14:31:24.104568 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:24.104471 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2sjq2" event={"ID":"ba16bc50-891a-4477-8bd5-de3ef058d04f","Type":"ContainerDied","Data":"511954a3e69a7d2599136434394563913d9481ce0915c2772ce40e580980b23c"} Jul 10 14:31:24.119994 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:24.119972 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:31:24.926631 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:24.926608 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rxmgq" Jul 10 14:31:24.926773 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:24.926683 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t9jl6" Jul 10 14:31:24.926814 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:24.926773 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rxmgq" podUID="4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f" Jul 10 14:31:24.926849 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:24.926828 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t9jl6" podUID="ced07efc-4719-4439-b2be-b84dadef62d3" Jul 10 14:31:25.112713 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:25.112637 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxvq2_016248a0-bbe2-44c9-8e51-5d4695ee6a52/ovn-acl-logging/0.log" Jul 10 14:31:25.113179 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:25.113081 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" event={"ID":"016248a0-bbe2-44c9-8e51-5d4695ee6a52","Type":"ContainerStarted","Data":"b4b6a64223cb21c5e49d366f7be0df3937e33ba62462bfc36f407ba4948b508c"} Jul 10 14:31:25.113247 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:25.113194 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:31:25.115243 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:25.115205 2569 generic.go:356] "Generic (PLEG): container finished" podID="ba16bc50-891a-4477-8bd5-de3ef058d04f" containerID="8a43fd0e5e3623d221acbba83ff9ff73975b7530f062a8a2262bd2eb0c7134ed" exitCode=0 Jul 10 14:31:25.115243 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:25.115241 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2sjq2" event={"ID":"ba16bc50-891a-4477-8bd5-de3ef058d04f","Type":"ContainerDied","Data":"8a43fd0e5e3623d221acbba83ff9ff73975b7530f062a8a2262bd2eb0c7134ed"} Jul 10 14:31:25.127680 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:25.127660 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:31:25.141560 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:25.141522 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" podStartSLOduration=9.453639215 podStartE2EDuration="30.141508309s" totalImagesPullingTime="20.687869094s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 14:30:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:31:25.13992152 +0000 UTC m=+30.891293235" watchObservedRunningTime="2026-07-10 14:31:25.141508309 +0000 UTC m=+30.892880005" Jul 10 14:31:25.422746 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:25.422716 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-t9jl6"] Jul 10 14:31:25.422900 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:25.422848 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t9jl6" Jul 10 14:31:25.422946 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:25.422931 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t9jl6" podUID="ced07efc-4719-4439-b2be-b84dadef62d3" Jul 10 14:31:25.423230 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:25.423210 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rxmgq"] Jul 10 14:31:25.423307 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:25.423296 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rxmgq" Jul 10 14:31:25.423390 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:25.423372 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rxmgq" podUID="4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f" Jul 10 14:31:25.424232 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:25.424210 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fdhw7"] Jul 10 14:31:25.424340 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:25.424295 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fdhw7" Jul 10 14:31:25.424383 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:25.424369 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fdhw7" podUID="ff3c9cb2-1924-4ea9-9a93-282213b083db" Jul 10 14:31:26.119959 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:26.119929 2569 generic.go:356] "Generic (PLEG): container finished" podID="ba16bc50-891a-4477-8bd5-de3ef058d04f" containerID="c9eb1d828a07b42ae9040ba90ddf53af92655044603ce678e0079cc55b597008" exitCode=0 Jul 10 14:31:26.120270 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:26.120002 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2sjq2" event={"ID":"ba16bc50-891a-4477-8bd5-de3ef058d04f","Type":"ContainerDied","Data":"c9eb1d828a07b42ae9040ba90ddf53af92655044603ce678e0079cc55b597008"} Jul 10 14:31:26.926098 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:26.926053 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t9jl6" Jul 10 14:31:26.926098 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:26.926097 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fdhw7" Jul 10 14:31:26.926333 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:26.926052 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rxmgq" Jul 10 14:31:26.926333 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:26.926179 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t9jl6" podUID="ced07efc-4719-4439-b2be-b84dadef62d3" Jul 10 14:31:26.926333 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:26.926240 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rxmgq" podUID="4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f" Jul 10 14:31:26.926333 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:26.926304 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fdhw7" podUID="ff3c9cb2-1924-4ea9-9a93-282213b083db" Jul 10 14:31:28.405119 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:28.404936 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff3c9cb2-1924-4ea9-9a93-282213b083db-metrics-certs\") pod \"network-metrics-daemon-fdhw7\" (UID: \"ff3c9cb2-1924-4ea9-9a93-282213b083db\") " pod="openshift-multus/network-metrics-daemon-fdhw7" Jul 10 14:31:28.405517 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:28.405084 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 14:31:28.405517 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:28.405233 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff3c9cb2-1924-4ea9-9a93-282213b083db-metrics-certs podName:ff3c9cb2-1924-4ea9-9a93-282213b083db nodeName:}" failed. No retries permitted until 2026-07-10 14:32:00.405214741 +0000 UTC m=+66.156586415 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ff3c9cb2-1924-4ea9-9a93-282213b083db-metrics-certs") pod "network-metrics-daemon-fdhw7" (UID: "ff3c9cb2-1924-4ea9-9a93-282213b083db") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 14:31:28.505742 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:28.505713 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f-original-pull-secret\") pod \"global-pull-secret-syncer-rxmgq\" (UID: \"4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f\") " pod="kube-system/global-pull-secret-syncer-rxmgq" Jul 10 14:31:28.505903 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:28.505868 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 14:31:28.505974 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:28.505943 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f-original-pull-secret podName:4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f nodeName:}" failed. No retries permitted until 2026-07-10 14:32:00.505920876 +0000 UTC m=+66.257292555 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f-original-pull-secret") pod "global-pull-secret-syncer-rxmgq" (UID: "4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f") : object "kube-system"/"original-pull-secret" not registered Jul 10 14:31:28.607035 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:28.607001 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpsm2\" (UniqueName: \"kubernetes.io/projected/ced07efc-4719-4439-b2be-b84dadef62d3-kube-api-access-cpsm2\") pod \"network-check-target-t9jl6\" (UID: \"ced07efc-4719-4439-b2be-b84dadef62d3\") " pod="openshift-network-diagnostics/network-check-target-t9jl6" Jul 10 14:31:28.607206 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:28.607185 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 14:31:28.607261 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:28.607215 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 14:31:28.607261 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:28.607228 2569 projected.go:196] Error preparing data for projected volume kube-api-access-cpsm2 for pod openshift-network-diagnostics/network-check-target-t9jl6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 14:31:28.607334 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:28.607276 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ced07efc-4719-4439-b2be-b84dadef62d3-kube-api-access-cpsm2 podName:ced07efc-4719-4439-b2be-b84dadef62d3 nodeName:}" failed. No retries permitted until 2026-07-10 14:32:00.607263104 +0000 UTC m=+66.358634783 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-cpsm2" (UniqueName: "kubernetes.io/projected/ced07efc-4719-4439-b2be-b84dadef62d3-kube-api-access-cpsm2") pod "network-check-target-t9jl6" (UID: "ced07efc-4719-4439-b2be-b84dadef62d3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 14:31:28.925719 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:28.925688 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t9jl6" Jul 10 14:31:28.925902 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:28.925724 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fdhw7" Jul 10 14:31:28.925902 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:28.925688 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rxmgq" Jul 10 14:31:28.925902 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:28.925831 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t9jl6" podUID="ced07efc-4719-4439-b2be-b84dadef62d3" Jul 10 14:31:28.926062 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:28.925932 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rxmgq" podUID="4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f" Jul 10 14:31:28.926062 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:28.926019 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fdhw7" podUID="ff3c9cb2-1924-4ea9-9a93-282213b083db" Jul 10 14:31:30.528391 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.528365 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-35.ec2.internal" event="NodeReady" Jul 10 14:31:30.528945 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.528501 2569 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 10 14:31:30.577869 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.577827 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-8d46cc844-xf6cz"] Jul 10 14:31:30.595946 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.595918 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-795d96bbd4-2nql9"] Jul 10 14:31:30.596127 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.596056 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8d46cc844-xf6cz" Jul 10 14:31:30.610223 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.610198 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 10 14:31:30.610342 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.610311 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 10 14:31:30.610555 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.610533 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-ssv9d\"" Jul 10 14:31:30.612970 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.612956 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 10 14:31:30.614402 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.614379 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7866497494-n6pfx"] Jul 10 14:31:30.614581 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.614550 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-795d96bbd4-2nql9" Jul 10 14:31:30.618538 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.618515 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 10 14:31:30.619976 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.619953 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 10 14:31:30.628005 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.627985 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-tpvq9\"" Jul 10 14:31:30.631428 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.631407 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 10 14:31:30.631553 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.631463 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 10 14:31:30.631553 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.631492 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 10 14:31:30.631867 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.631843 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 10 14:31:30.634858 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.634820 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-bttbd"] Jul 10 14:31:30.635126 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.635108 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-n6pfx" Jul 10 14:31:30.650982 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.650962 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-c8ztx"] Jul 10 14:31:30.651096 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.651079 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-bttbd" Jul 10 14:31:30.652156 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.652030 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 10 14:31:30.652156 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.652055 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 10 14:31:30.652156 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.652154 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 10 14:31:30.652389 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.652175 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 10 14:31:30.654283 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.654255 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-mspgv\"" Jul 10 14:31:30.659927 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.659907 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-snhws\"" Jul 10 14:31:30.660273 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.660251 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 10 14:31:30.660365 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.660263 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 10 14:31:30.667409 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.667392 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-qtr8d"] Jul 10 14:31:30.667606 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.667584 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-c8ztx" Jul 10 14:31:30.681604 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.679439 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 10 14:31:30.681604 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.680100 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 10 14:31:30.681604 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.680284 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 10 14:31:30.681604 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.680680 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 10 14:31:30.681604 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.680904 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-sxgwf\"" Jul 10 14:31:30.688053 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.688032 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 10 14:31:30.691041 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.691022 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 10 14:31:30.692086 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.692066 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9gxrb"] Jul 10 14:31:30.692208 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.692184 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-qtr8d" Jul 10 14:31:30.695714 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.695691 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 10 14:31:30.696975 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.696958 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 10 14:31:30.700894 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.700459 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 10 14:31:30.700894 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.700461 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-cwxgr\"" Jul 10 14:31:30.708678 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.708661 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-f698x"] Jul 10 14:31:30.708678 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.708675 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9gxrb" Jul 10 14:31:30.712478 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.712447 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 10 14:31:30.716095 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.716055 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 10 14:31:30.716291 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.716196 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-c5njl\"" Jul 10 14:31:30.716372 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.716276 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 10 14:31:30.716734 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.716717 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 10 14:31:30.721143 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.721124 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 10 14:31:30.723153 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.723129 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c867eb4a-dae5-4edd-a157-7686c044be1e-registry-certificates\") pod \"image-registry-8d46cc844-xf6cz\" (UID: \"c867eb4a-dae5-4edd-a157-7686c044be1e\") " pod="openshift-image-registry/image-registry-8d46cc844-xf6cz" Jul 10 14:31:30.723248 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.723167 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c867eb4a-dae5-4edd-a157-7686c044be1e-trusted-ca\") pod \"image-registry-8d46cc844-xf6cz\" (UID: \"c867eb4a-dae5-4edd-a157-7686c044be1e\") " pod="openshift-image-registry/image-registry-8d46cc844-xf6cz" Jul 10 14:31:30.723248 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.723187 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c867eb4a-dae5-4edd-a157-7686c044be1e-bound-sa-token\") pod \"image-registry-8d46cc844-xf6cz\" (UID: \"c867eb4a-dae5-4edd-a157-7686c044be1e\") " pod="openshift-image-registry/image-registry-8d46cc844-xf6cz" Jul 10 14:31:30.723248 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.723245 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c867eb4a-dae5-4edd-a157-7686c044be1e-image-registry-private-configuration\") pod \"image-registry-8d46cc844-xf6cz\" (UID: \"c867eb4a-dae5-4edd-a157-7686c044be1e\") " pod="openshift-image-registry/image-registry-8d46cc844-xf6cz" Jul 10 14:31:30.723389 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.723267 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c867eb4a-dae5-4edd-a157-7686c044be1e-ca-trust-extracted\") pod \"image-registry-8d46cc844-xf6cz\" (UID: \"c867eb4a-dae5-4edd-a157-7686c044be1e\") " pod="openshift-image-registry/image-registry-8d46cc844-xf6cz" Jul 10 14:31:30.723389 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.723286 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c867eb4a-dae5-4edd-a157-7686c044be1e-installation-pull-secrets\") pod \"image-registry-8d46cc844-xf6cz\" (UID: \"c867eb4a-dae5-4edd-a157-7686c044be1e\") " pod="openshift-image-registry/image-registry-8d46cc844-xf6cz" Jul 10 14:31:30.723389 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.723302 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/296be890-64e9-4fb3-8c68-e4c2a7c8cc2c-config\") pod \"console-operator-7866497494-n6pfx\" (UID: \"296be890-64e9-4fb3-8c68-e4c2a7c8cc2c\") " pod="openshift-console-operator/console-operator-7866497494-n6pfx" Jul 10 14:31:30.723389 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.723325 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vvm7\" (UniqueName: \"kubernetes.io/projected/296be890-64e9-4fb3-8c68-e4c2a7c8cc2c-kube-api-access-7vvm7\") pod \"console-operator-7866497494-n6pfx\" (UID: \"296be890-64e9-4fb3-8c68-e4c2a7c8cc2c\") " pod="openshift-console-operator/console-operator-7866497494-n6pfx" Jul 10 14:31:30.723389 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.723362 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13a97abe-27b2-4b7b-ae18-8ae1ff61f043-metrics-certs\") pod \"router-default-795d96bbd4-2nql9\" (UID: \"13a97abe-27b2-4b7b-ae18-8ae1ff61f043\") " pod="openshift-ingress/router-default-795d96bbd4-2nql9" Jul 10 14:31:30.723613 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.723397 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khll2\" (UniqueName: \"kubernetes.io/projected/c867eb4a-dae5-4edd-a157-7686c044be1e-kube-api-access-khll2\") pod \"image-registry-8d46cc844-xf6cz\" (UID: \"c867eb4a-dae5-4edd-a157-7686c044be1e\") " pod="openshift-image-registry/image-registry-8d46cc844-xf6cz" Jul 10 14:31:30.723613 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.723419 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13a97abe-27b2-4b7b-ae18-8ae1ff61f043-service-ca-bundle\") pod \"router-default-795d96bbd4-2nql9\" (UID: \"13a97abe-27b2-4b7b-ae18-8ae1ff61f043\") " pod="openshift-ingress/router-default-795d96bbd4-2nql9" Jul 10 14:31:30.723613 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.723444 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c867eb4a-dae5-4edd-a157-7686c044be1e-registry-tls\") pod \"image-registry-8d46cc844-xf6cz\" (UID: \"c867eb4a-dae5-4edd-a157-7686c044be1e\") " pod="openshift-image-registry/image-registry-8d46cc844-xf6cz" Jul 10 14:31:30.723613 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.723459 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/296be890-64e9-4fb3-8c68-e4c2a7c8cc2c-trusted-ca\") pod \"console-operator-7866497494-n6pfx\" (UID: \"296be890-64e9-4fb3-8c68-e4c2a7c8cc2c\") " pod="openshift-console-operator/console-operator-7866497494-n6pfx" Jul 10 14:31:30.723613 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.723483 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/13a97abe-27b2-4b7b-ae18-8ae1ff61f043-default-certificate\") pod \"router-default-795d96bbd4-2nql9\" (UID: \"13a97abe-27b2-4b7b-ae18-8ae1ff61f043\") " pod="openshift-ingress/router-default-795d96bbd4-2nql9" Jul 10 14:31:30.723613 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.723501 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/13a97abe-27b2-4b7b-ae18-8ae1ff61f043-stats-auth\") pod \"router-default-795d96bbd4-2nql9\" (UID: \"13a97abe-27b2-4b7b-ae18-8ae1ff61f043\") " pod="openshift-ingress/router-default-795d96bbd4-2nql9" Jul 10 14:31:30.723613 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.723549 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhbbf\" (UniqueName: \"kubernetes.io/projected/13a97abe-27b2-4b7b-ae18-8ae1ff61f043-kube-api-access-rhbbf\") pod \"router-default-795d96bbd4-2nql9\" (UID: \"13a97abe-27b2-4b7b-ae18-8ae1ff61f043\") " pod="openshift-ingress/router-default-795d96bbd4-2nql9" Jul 10 14:31:30.723613 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.723605 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/296be890-64e9-4fb3-8c68-e4c2a7c8cc2c-serving-cert\") pod \"console-operator-7866497494-n6pfx\" (UID: \"296be890-64e9-4fb3-8c68-e4c2a7c8cc2c\") " pod="openshift-console-operator/console-operator-7866497494-n6pfx" Jul 10 14:31:30.725121 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.725100 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-tfq8b"] Jul 10 14:31:30.725256 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.725241 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-f698x" Jul 10 14:31:30.728179 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.728161 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 10 14:31:30.728179 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.728171 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2n5mh\"" Jul 10 14:31:30.728708 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.728680 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 10 14:31:30.739571 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.739552 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-gwwww"] Jul 10 14:31:30.739695 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.739677 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-tfq8b" Jul 10 14:31:30.743727 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.743706 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 10 14:31:30.743831 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.743790 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-pp7lp\"" Jul 10 14:31:30.743831 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.743809 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 10 14:31:30.755740 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.755722 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-wbbhv"] Jul 10 14:31:30.755846 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.755833 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-gwwww" Jul 10 14:31:30.759395 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.759373 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-pdzg4\"" Jul 10 14:31:30.759797 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.759720 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 10 14:31:30.759878 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.759800 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 10 14:31:30.759937 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.759720 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 10 14:31:30.759990 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.759959 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 10 14:31:30.769545 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.769526 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-d7kg8"] Jul 10 14:31:30.769664 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.769648 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wbbhv" Jul 10 14:31:30.774099 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.774081 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 10 14:31:30.774303 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.774281 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 10 14:31:30.774421 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.774402 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 10 14:31:30.774551 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.774311 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-g8rnp\"" Jul 10 14:31:30.786161 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.786107 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-c8ztx"] Jul 10 14:31:30.786161 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.786141 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8d46cc844-xf6cz"] Jul 10 14:31:30.786161 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.786162 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-bttbd"] Jul 10 14:31:30.786336 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.786179 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9gxrb"] Jul 10 14:31:30.786336 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.786203 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-795d96bbd4-2nql9"] Jul 10 14:31:30.786336 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.786223 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-qtr8d"] Jul 10 14:31:30.786336 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.786241 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-n6pfx"] Jul 10 14:31:30.786336 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.786259 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wbbhv"] Jul 10 14:31:30.786336 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.786279 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-tfq8b"] Jul 10 14:31:30.786336 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.786223 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-d7kg8" Jul 10 14:31:30.786653 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.786297 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-gwwww"] Jul 10 14:31:30.786653 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.786489 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-f698x"] Jul 10 14:31:30.786653 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.786523 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-d7kg8"] Jul 10 14:31:30.788837 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.788688 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2g7m_4ee5894d-0007-499f-a401-2f11724910ac/konnectivity-agent/0.log" limit=1024 Jul 10 14:31:30.789044 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.789026 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 10 14:31:30.789325 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.789305 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 10 14:31:30.789395 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.789331 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 10 14:31:30.789453 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.789430 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-8skpr\"" Jul 10 14:31:30.789512 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.789459 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 10 14:31:30.824491 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.824423 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c867eb4a-dae5-4edd-a157-7686c044be1e-registry-certificates\") pod \"image-registry-8d46cc844-xf6cz\" (UID: \"c867eb4a-dae5-4edd-a157-7686c044be1e\") " pod="openshift-image-registry/image-registry-8d46cc844-xf6cz" Jul 10 14:31:30.824491 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.824481 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7htb7\" (UniqueName: \"kubernetes.io/projected/cb41c5b6-dab8-4be4-b2be-56040d1659e7-kube-api-access-7htb7\") pod \"cluster-monitoring-operator-5f7f494dd4-c8ztx\" (UID: \"cb41c5b6-dab8-4be4-b2be-56040d1659e7\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-c8ztx" Jul 10 14:31:30.824671 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.824509 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbs2r\" (UniqueName: \"kubernetes.io/projected/5d6f7d4c-e2a0-40d0-9686-f016013d1504-kube-api-access-tbs2r\") pod \"cluster-samples-operator-65db858894-9gxrb\" (UID: \"5d6f7d4c-e2a0-40d0-9686-f016013d1504\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9gxrb" Jul 10 14:31:30.824671 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.824542 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc5a5ba3-52b4-4b58-923e-df7e716d9438-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-qtr8d\" (UID: \"dc5a5ba3-52b4-4b58-923e-df7e716d9438\") " pod="openshift-insights/insights-operator-68d9dd94cf-qtr8d" Jul 10 14:31:30.824671 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.824627 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c867eb4a-dae5-4edd-a157-7686c044be1e-ca-trust-extracted\") pod \"image-registry-8d46cc844-xf6cz\" (UID: \"c867eb4a-dae5-4edd-a157-7686c044be1e\") " pod="openshift-image-registry/image-registry-8d46cc844-xf6cz" Jul 10 14:31:30.824671 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.824661 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c867eb4a-dae5-4edd-a157-7686c044be1e-installation-pull-secrets\") pod \"image-registry-8d46cc844-xf6cz\" (UID: \"c867eb4a-dae5-4edd-a157-7686c044be1e\") " pod="openshift-image-registry/image-registry-8d46cc844-xf6cz" Jul 10 14:31:30.824898 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.824700 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dc5a5ba3-52b4-4b58-923e-df7e716d9438-serving-cert\") pod \"insights-operator-68d9dd94cf-qtr8d\" (UID: \"dc5a5ba3-52b4-4b58-923e-df7e716d9438\") " pod="openshift-insights/insights-operator-68d9dd94cf-qtr8d" Jul 10 14:31:30.824898 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.824729 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13a97abe-27b2-4b7b-ae18-8ae1ff61f043-metrics-certs\") pod \"router-default-795d96bbd4-2nql9\" (UID: \"13a97abe-27b2-4b7b-ae18-8ae1ff61f043\") " pod="openshift-ingress/router-default-795d96bbd4-2nql9" Jul 10 14:31:30.824898 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.824776 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cb41c5b6-dab8-4be4-b2be-56040d1659e7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-c8ztx\" (UID: \"cb41c5b6-dab8-4be4-b2be-56040d1659e7\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-c8ztx" Jul 10 14:31:30.824898 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.824829 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khll2\" (UniqueName: \"kubernetes.io/projected/c867eb4a-dae5-4edd-a157-7686c044be1e-kube-api-access-khll2\") pod \"image-registry-8d46cc844-xf6cz\" (UID: \"c867eb4a-dae5-4edd-a157-7686c044be1e\") " pod="openshift-image-registry/image-registry-8d46cc844-xf6cz" Jul 10 14:31:30.824898 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.824861 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13a97abe-27b2-4b7b-ae18-8ae1ff61f043-service-ca-bundle\") pod \"router-default-795d96bbd4-2nql9\" (UID: \"13a97abe-27b2-4b7b-ae18-8ae1ff61f043\") " pod="openshift-ingress/router-default-795d96bbd4-2nql9" Jul 10 14:31:30.824898 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:30.824881 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 10 14:31:30.824898 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.824894 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/13a97abe-27b2-4b7b-ae18-8ae1ff61f043-default-certificate\") pod \"router-default-795d96bbd4-2nql9\" (UID: \"13a97abe-27b2-4b7b-ae18-8ae1ff61f043\") " pod="openshift-ingress/router-default-795d96bbd4-2nql9" Jul 10 14:31:30.825233 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.824922 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/13a97abe-27b2-4b7b-ae18-8ae1ff61f043-stats-auth\") pod \"router-default-795d96bbd4-2nql9\" (UID: \"13a97abe-27b2-4b7b-ae18-8ae1ff61f043\") " pod="openshift-ingress/router-default-795d96bbd4-2nql9" Jul 10 14:31:30.825233 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.824950 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhbbf\" (UniqueName: \"kubernetes.io/projected/13a97abe-27b2-4b7b-ae18-8ae1ff61f043-kube-api-access-rhbbf\") pod \"router-default-795d96bbd4-2nql9\" (UID: \"13a97abe-27b2-4b7b-ae18-8ae1ff61f043\") " pod="openshift-ingress/router-default-795d96bbd4-2nql9" Jul 10 14:31:30.825233 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.824976 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5d6f7d4c-e2a0-40d0-9686-f016013d1504-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-9gxrb\" (UID: \"5d6f7d4c-e2a0-40d0-9686-f016013d1504\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9gxrb" Jul 10 14:31:30.825233 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.825009 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24c9w\" (UniqueName: \"kubernetes.io/projected/5b19b116-a765-40a7-9268-b45593b2e2d8-kube-api-access-24c9w\") pod \"network-check-source-869dcfcdcd-tfq8b\" (UID: \"5b19b116-a765-40a7-9268-b45593b2e2d8\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-tfq8b" Jul 10 14:31:30.825233 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.825033 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c867eb4a-dae5-4edd-a157-7686c044be1e-registry-certificates\") pod \"image-registry-8d46cc844-xf6cz\" (UID: \"c867eb4a-dae5-4edd-a157-7686c044be1e\") " pod="openshift-image-registry/image-registry-8d46cc844-xf6cz" Jul 10 14:31:30.825233 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.825038 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c867eb4a-dae5-4edd-a157-7686c044be1e-ca-trust-extracted\") pod \"image-registry-8d46cc844-xf6cz\" (UID: \"c867eb4a-dae5-4edd-a157-7686c044be1e\") " pod="openshift-image-registry/image-registry-8d46cc844-xf6cz" Jul 10 14:31:30.825233 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.825042 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt5wr\" (UniqueName: \"kubernetes.io/projected/68e91fda-6611-4951-806d-2514fe8bd1cb-kube-api-access-wt5wr\") pod \"dns-default-f698x\" (UID: \"68e91fda-6611-4951-806d-2514fe8bd1cb\") " pod="openshift-dns/dns-default-f698x" Jul 10 14:31:30.825233 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:30.825084 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13a97abe-27b2-4b7b-ae18-8ae1ff61f043-metrics-certs podName:13a97abe-27b2-4b7b-ae18-8ae1ff61f043 nodeName:}" failed. No retries permitted until 2026-07-10 14:31:31.325064791 +0000 UTC m=+37.076436469 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/13a97abe-27b2-4b7b-ae18-8ae1ff61f043-metrics-certs") pod "router-default-795d96bbd4-2nql9" (UID: "13a97abe-27b2-4b7b-ae18-8ae1ff61f043") : secret "router-metrics-certs-default" not found Jul 10 14:31:30.825233 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.825105 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7r4b\" (UniqueName: \"kubernetes.io/projected/dc5a5ba3-52b4-4b58-923e-df7e716d9438-kube-api-access-g7r4b\") pod \"insights-operator-68d9dd94cf-qtr8d\" (UID: \"dc5a5ba3-52b4-4b58-923e-df7e716d9438\") " pod="openshift-insights/insights-operator-68d9dd94cf-qtr8d" Jul 10 14:31:30.825233 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.825152 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c867eb4a-dae5-4edd-a157-7686c044be1e-trusted-ca\") pod \"image-registry-8d46cc844-xf6cz\" (UID: \"c867eb4a-dae5-4edd-a157-7686c044be1e\") " pod="openshift-image-registry/image-registry-8d46cc844-xf6cz" Jul 10 14:31:30.825233 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:30.825170 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/13a97abe-27b2-4b7b-ae18-8ae1ff61f043-service-ca-bundle podName:13a97abe-27b2-4b7b-ae18-8ae1ff61f043 nodeName:}" failed. No retries permitted until 2026-07-10 14:31:31.325158363 +0000 UTC m=+37.076530057 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/13a97abe-27b2-4b7b-ae18-8ae1ff61f043-service-ca-bundle") pod "router-default-795d96bbd4-2nql9" (UID: "13a97abe-27b2-4b7b-ae18-8ae1ff61f043") : configmap references non-existent config key: service-ca.crt Jul 10 14:31:30.825233 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.825197 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68e91fda-6611-4951-806d-2514fe8bd1cb-tmp-dir\") pod \"dns-default-f698x\" (UID: \"68e91fda-6611-4951-806d-2514fe8bd1cb\") " pod="openshift-dns/dns-default-f698x" Jul 10 14:31:30.825861 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.825230 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c867eb4a-dae5-4edd-a157-7686c044be1e-bound-sa-token\") pod \"image-registry-8d46cc844-xf6cz\" (UID: \"c867eb4a-dae5-4edd-a157-7686c044be1e\") " pod="openshift-image-registry/image-registry-8d46cc844-xf6cz" Jul 10 14:31:30.825861 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.825300 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/68e91fda-6611-4951-806d-2514fe8bd1cb-metrics-tls\") pod \"dns-default-f698x\" (UID: \"68e91fda-6611-4951-806d-2514fe8bd1cb\") " pod="openshift-dns/dns-default-f698x" Jul 10 14:31:30.825861 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.825354 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c867eb4a-dae5-4edd-a157-7686c044be1e-image-registry-private-configuration\") pod \"image-registry-8d46cc844-xf6cz\" (UID: \"c867eb4a-dae5-4edd-a157-7686c044be1e\") " pod="openshift-image-registry/image-registry-8d46cc844-xf6cz" Jul 10 14:31:30.825861 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.825388 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vvm7\" (UniqueName: \"kubernetes.io/projected/296be890-64e9-4fb3-8c68-e4c2a7c8cc2c-kube-api-access-7vvm7\") pod \"console-operator-7866497494-n6pfx\" (UID: \"296be890-64e9-4fb3-8c68-e4c2a7c8cc2c\") " pod="openshift-console-operator/console-operator-7866497494-n6pfx" Jul 10 14:31:30.825861 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.825418 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/296be890-64e9-4fb3-8c68-e4c2a7c8cc2c-config\") pod \"console-operator-7866497494-n6pfx\" (UID: \"296be890-64e9-4fb3-8c68-e4c2a7c8cc2c\") " pod="openshift-console-operator/console-operator-7866497494-n6pfx" Jul 10 14:31:30.825861 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.825447 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9sjj\" (UniqueName: \"kubernetes.io/projected/4d2d05db-87c6-4dff-9d05-599de4cfa9e4-kube-api-access-n9sjj\") pod \"volume-data-source-validator-78495597f6-bttbd\" (UID: \"4d2d05db-87c6-4dff-9d05-599de4cfa9e4\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-bttbd" Jul 10 14:31:30.825861 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.825474 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/dc5a5ba3-52b4-4b58-923e-df7e716d9438-snapshots\") pod \"insights-operator-68d9dd94cf-qtr8d\" (UID: \"dc5a5ba3-52b4-4b58-923e-df7e716d9438\") " pod="openshift-insights/insights-operator-68d9dd94cf-qtr8d" Jul 10 14:31:30.825861 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.825494 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/68e91fda-6611-4951-806d-2514fe8bd1cb-config-volume\") pod \"dns-default-f698x\" (UID: \"68e91fda-6611-4951-806d-2514fe8bd1cb\") " pod="openshift-dns/dns-default-f698x" Jul 10 14:31:30.825861 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.825529 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c867eb4a-dae5-4edd-a157-7686c044be1e-registry-tls\") pod \"image-registry-8d46cc844-xf6cz\" (UID: \"c867eb4a-dae5-4edd-a157-7686c044be1e\") " pod="openshift-image-registry/image-registry-8d46cc844-xf6cz" Jul 10 14:31:30.825861 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.825549 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/296be890-64e9-4fb3-8c68-e4c2a7c8cc2c-trusted-ca\") pod \"console-operator-7866497494-n6pfx\" (UID: \"296be890-64e9-4fb3-8c68-e4c2a7c8cc2c\") " pod="openshift-console-operator/console-operator-7866497494-n6pfx" Jul 10 14:31:30.825861 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:30.825634 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 14:31:30.825861 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:30.825644 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8d46cc844-xf6cz: secret "image-registry-tls" not found Jul 10 14:31:30.825861 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:30.825692 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c867eb4a-dae5-4edd-a157-7686c044be1e-registry-tls podName:c867eb4a-dae5-4edd-a157-7686c044be1e nodeName:}" failed. No retries permitted until 2026-07-10 14:31:31.325679042 +0000 UTC m=+37.077050717 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c867eb4a-dae5-4edd-a157-7686c044be1e-registry-tls") pod "image-registry-8d46cc844-xf6cz" (UID: "c867eb4a-dae5-4edd-a157-7686c044be1e") : secret "image-registry-tls" not found Jul 10 14:31:30.826470 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.825877 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/296be890-64e9-4fb3-8c68-e4c2a7c8cc2c-serving-cert\") pod \"console-operator-7866497494-n6pfx\" (UID: \"296be890-64e9-4fb3-8c68-e4c2a7c8cc2c\") " pod="openshift-console-operator/console-operator-7866497494-n6pfx" Jul 10 14:31:30.826470 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.825906 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc5a5ba3-52b4-4b58-923e-df7e716d9438-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-qtr8d\" (UID: \"dc5a5ba3-52b4-4b58-923e-df7e716d9438\") " pod="openshift-insights/insights-operator-68d9dd94cf-qtr8d" Jul 10 14:31:30.826470 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.825935 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc5a5ba3-52b4-4b58-923e-df7e716d9438-tmp\") pod \"insights-operator-68d9dd94cf-qtr8d\" (UID: \"dc5a5ba3-52b4-4b58-923e-df7e716d9438\") " pod="openshift-insights/insights-operator-68d9dd94cf-qtr8d" Jul 10 14:31:30.826470 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.825959 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/cb41c5b6-dab8-4be4-b2be-56040d1659e7-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-c8ztx\" (UID: \"cb41c5b6-dab8-4be4-b2be-56040d1659e7\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-c8ztx" Jul 10 14:31:30.826470 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.826016 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c867eb4a-dae5-4edd-a157-7686c044be1e-trusted-ca\") pod \"image-registry-8d46cc844-xf6cz\" (UID: \"c867eb4a-dae5-4edd-a157-7686c044be1e\") " pod="openshift-image-registry/image-registry-8d46cc844-xf6cz" Jul 10 14:31:30.826470 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.826437 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/296be890-64e9-4fb3-8c68-e4c2a7c8cc2c-trusted-ca\") pod \"console-operator-7866497494-n6pfx\" (UID: \"296be890-64e9-4fb3-8c68-e4c2a7c8cc2c\") " pod="openshift-console-operator/console-operator-7866497494-n6pfx" Jul 10 14:31:30.826737 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.826711 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/296be890-64e9-4fb3-8c68-e4c2a7c8cc2c-config\") pod \"console-operator-7866497494-n6pfx\" (UID: \"296be890-64e9-4fb3-8c68-e4c2a7c8cc2c\") " pod="openshift-console-operator/console-operator-7866497494-n6pfx" Jul 10 14:31:30.829849 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.829800 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/296be890-64e9-4fb3-8c68-e4c2a7c8cc2c-serving-cert\") pod \"console-operator-7866497494-n6pfx\" (UID: \"296be890-64e9-4fb3-8c68-e4c2a7c8cc2c\") " pod="openshift-console-operator/console-operator-7866497494-n6pfx" Jul 10 14:31:30.830050 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.830017 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/13a97abe-27b2-4b7b-ae18-8ae1ff61f043-stats-auth\") pod \"router-default-795d96bbd4-2nql9\" (UID: \"13a97abe-27b2-4b7b-ae18-8ae1ff61f043\") " pod="openshift-ingress/router-default-795d96bbd4-2nql9" Jul 10 14:31:30.830275 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.830229 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c867eb4a-dae5-4edd-a157-7686c044be1e-image-registry-private-configuration\") pod \"image-registry-8d46cc844-xf6cz\" (UID: \"c867eb4a-dae5-4edd-a157-7686c044be1e\") " pod="openshift-image-registry/image-registry-8d46cc844-xf6cz" Jul 10 14:31:30.830368 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.830313 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/13a97abe-27b2-4b7b-ae18-8ae1ff61f043-default-certificate\") pod \"router-default-795d96bbd4-2nql9\" (UID: \"13a97abe-27b2-4b7b-ae18-8ae1ff61f043\") " pod="openshift-ingress/router-default-795d96bbd4-2nql9" Jul 10 14:31:30.830690 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.830669 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c867eb4a-dae5-4edd-a157-7686c044be1e-installation-pull-secrets\") pod \"image-registry-8d46cc844-xf6cz\" (UID: \"c867eb4a-dae5-4edd-a157-7686c044be1e\") " pod="openshift-image-registry/image-registry-8d46cc844-xf6cz" Jul 10 14:31:30.841733 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.841691 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khll2\" (UniqueName: \"kubernetes.io/projected/c867eb4a-dae5-4edd-a157-7686c044be1e-kube-api-access-khll2\") pod \"image-registry-8d46cc844-xf6cz\" (UID: \"c867eb4a-dae5-4edd-a157-7686c044be1e\") " pod="openshift-image-registry/image-registry-8d46cc844-xf6cz" Jul 10 14:31:30.841917 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.841891 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c867eb4a-dae5-4edd-a157-7686c044be1e-bound-sa-token\") pod \"image-registry-8d46cc844-xf6cz\" (UID: \"c867eb4a-dae5-4edd-a157-7686c044be1e\") " pod="openshift-image-registry/image-registry-8d46cc844-xf6cz" Jul 10 14:31:30.842333 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.842279 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhbbf\" (UniqueName: \"kubernetes.io/projected/13a97abe-27b2-4b7b-ae18-8ae1ff61f043-kube-api-access-rhbbf\") pod \"router-default-795d96bbd4-2nql9\" (UID: \"13a97abe-27b2-4b7b-ae18-8ae1ff61f043\") " pod="openshift-ingress/router-default-795d96bbd4-2nql9" Jul 10 14:31:30.842629 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.842611 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vvm7\" (UniqueName: \"kubernetes.io/projected/296be890-64e9-4fb3-8c68-e4c2a7c8cc2c-kube-api-access-7vvm7\") pod \"console-operator-7866497494-n6pfx\" (UID: \"296be890-64e9-4fb3-8c68-e4c2a7c8cc2c\") " pod="openshift-console-operator/console-operator-7866497494-n6pfx" Jul 10 14:31:30.916841 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.916809 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2g7m_4ee5894d-0007-499f-a401-2f11724910ac/konnectivity-agent/0.log" limit=1024 Jul 10 14:31:30.925429 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.925402 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rxmgq" Jul 10 14:31:30.925429 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.925426 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fdhw7" Jul 10 14:31:30.925810 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.925782 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t9jl6" Jul 10 14:31:30.926284 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.926262 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/70c38fec-83c3-4e81-90c3-ab3f7204fe3c-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-d7kg8\" (UID: \"70c38fec-83c3-4e81-90c3-ab3f7204fe3c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-d7kg8" Jul 10 14:31:30.926388 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.926301 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc5a5ba3-52b4-4b58-923e-df7e716d9438-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-qtr8d\" (UID: \"dc5a5ba3-52b4-4b58-923e-df7e716d9438\") " pod="openshift-insights/insights-operator-68d9dd94cf-qtr8d" Jul 10 14:31:30.926388 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.926329 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc5a5ba3-52b4-4b58-923e-df7e716d9438-tmp\") pod \"insights-operator-68d9dd94cf-qtr8d\" (UID: \"dc5a5ba3-52b4-4b58-923e-df7e716d9438\") " pod="openshift-insights/insights-operator-68d9dd94cf-qtr8d" Jul 10 14:31:30.926388 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.926345 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/cb41c5b6-dab8-4be4-b2be-56040d1659e7-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-c8ztx\" (UID: \"cb41c5b6-dab8-4be4-b2be-56040d1659e7\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-c8ztx" Jul 10 14:31:30.926388 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.926367 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7htb7\" (UniqueName: \"kubernetes.io/projected/cb41c5b6-dab8-4be4-b2be-56040d1659e7-kube-api-access-7htb7\") pod \"cluster-monitoring-operator-5f7f494dd4-c8ztx\" (UID: \"cb41c5b6-dab8-4be4-b2be-56040d1659e7\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-c8ztx" Jul 10 14:31:30.926721 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.926699 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbs2r\" (UniqueName: \"kubernetes.io/projected/5d6f7d4c-e2a0-40d0-9686-f016013d1504-kube-api-access-tbs2r\") pod \"cluster-samples-operator-65db858894-9gxrb\" (UID: \"5d6f7d4c-e2a0-40d0-9686-f016013d1504\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9gxrb" Jul 10 14:31:30.926815 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.926743 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5ceab7ff-a2fc-4115-8dcc-1d9b014a8ffe-serving-cert\") pod \"service-ca-operator-7695fc6c8c-gwwww\" (UID: \"5ceab7ff-a2fc-4115-8dcc-1d9b014a8ffe\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-gwwww" Jul 10 14:31:30.926815 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.926785 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc5a5ba3-52b4-4b58-923e-df7e716d9438-tmp\") pod \"insights-operator-68d9dd94cf-qtr8d\" (UID: \"dc5a5ba3-52b4-4b58-923e-df7e716d9438\") " pod="openshift-insights/insights-operator-68d9dd94cf-qtr8d" Jul 10 14:31:30.926933 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.926877 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc5a5ba3-52b4-4b58-923e-df7e716d9438-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-qtr8d\" (UID: \"dc5a5ba3-52b4-4b58-923e-df7e716d9438\") " pod="openshift-insights/insights-operator-68d9dd94cf-qtr8d" Jul 10 14:31:30.926933 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.926915 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dc5a5ba3-52b4-4b58-923e-df7e716d9438-serving-cert\") pod \"insights-operator-68d9dd94cf-qtr8d\" (UID: \"dc5a5ba3-52b4-4b58-923e-df7e716d9438\") " pod="openshift-insights/insights-operator-68d9dd94cf-qtr8d" Jul 10 14:31:30.927027 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.926941 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5ceab7ff-a2fc-4115-8dcc-1d9b014a8ffe-config\") pod \"service-ca-operator-7695fc6c8c-gwwww\" (UID: \"5ceab7ff-a2fc-4115-8dcc-1d9b014a8ffe\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-gwwww" Jul 10 14:31:30.927027 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.926980 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cb41c5b6-dab8-4be4-b2be-56040d1659e7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-c8ztx\" (UID: \"cb41c5b6-dab8-4be4-b2be-56040d1659e7\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-c8ztx" Jul 10 14:31:30.927027 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.927006 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/70c38fec-83c3-4e81-90c3-ab3f7204fe3c-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-d7kg8\" (UID: \"70c38fec-83c3-4e81-90c3-ab3f7204fe3c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-d7kg8" Jul 10 14:31:30.927222 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.927043 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc5a5ba3-52b4-4b58-923e-df7e716d9438-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-qtr8d\" (UID: \"dc5a5ba3-52b4-4b58-923e-df7e716d9438\") " pod="openshift-insights/insights-operator-68d9dd94cf-qtr8d" Jul 10 14:31:30.927222 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.927051 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5qtn\" (UniqueName: \"kubernetes.io/projected/5ceab7ff-a2fc-4115-8dcc-1d9b014a8ffe-kube-api-access-r5qtn\") pod \"service-ca-operator-7695fc6c8c-gwwww\" (UID: \"5ceab7ff-a2fc-4115-8dcc-1d9b014a8ffe\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-gwwww" Jul 10 14:31:30.927222 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.927084 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5d6f7d4c-e2a0-40d0-9686-f016013d1504-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-9gxrb\" (UID: \"5d6f7d4c-e2a0-40d0-9686-f016013d1504\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9gxrb" Jul 10 14:31:30.927222 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.927116 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v58sc\" (UniqueName: \"kubernetes.io/projected/70c38fec-83c3-4e81-90c3-ab3f7204fe3c-kube-api-access-v58sc\") pod \"kube-storage-version-migrator-operator-799cb7f958-d7kg8\" (UID: \"70c38fec-83c3-4e81-90c3-ab3f7204fe3c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-d7kg8" Jul 10 14:31:30.927222 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.927146 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24c9w\" (UniqueName: \"kubernetes.io/projected/5b19b116-a765-40a7-9268-b45593b2e2d8-kube-api-access-24c9w\") pod \"network-check-source-869dcfcdcd-tfq8b\" (UID: \"5b19b116-a765-40a7-9268-b45593b2e2d8\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-tfq8b" Jul 10 14:31:30.927222 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:30.927166 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 10 14:31:30.927222 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.927176 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt5wr\" (UniqueName: \"kubernetes.io/projected/68e91fda-6611-4951-806d-2514fe8bd1cb-kube-api-access-wt5wr\") pod \"dns-default-f698x\" (UID: \"68e91fda-6611-4951-806d-2514fe8bd1cb\") " pod="openshift-dns/dns-default-f698x" Jul 10 14:31:30.927222 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.927203 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7r4b\" (UniqueName: \"kubernetes.io/projected/dc5a5ba3-52b4-4b58-923e-df7e716d9438-kube-api-access-g7r4b\") pod \"insights-operator-68d9dd94cf-qtr8d\" (UID: \"dc5a5ba3-52b4-4b58-923e-df7e716d9438\") " pod="openshift-insights/insights-operator-68d9dd94cf-qtr8d" Jul 10 14:31:30.927222 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:30.927221 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb41c5b6-dab8-4be4-b2be-56040d1659e7-cluster-monitoring-operator-tls podName:cb41c5b6-dab8-4be4-b2be-56040d1659e7 nodeName:}" failed. No retries permitted until 2026-07-10 14:31:31.427203943 +0000 UTC m=+37.178575630 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cb41c5b6-dab8-4be4-b2be-56040d1659e7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-c8ztx" (UID: "cb41c5b6-dab8-4be4-b2be-56040d1659e7") : secret "cluster-monitoring-operator-tls" not found Jul 10 14:31:30.927657 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.927288 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/cb41c5b6-dab8-4be4-b2be-56040d1659e7-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-c8ztx\" (UID: \"cb41c5b6-dab8-4be4-b2be-56040d1659e7\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-c8ztx" Jul 10 14:31:30.927657 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:30.927370 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 10 14:31:30.927657 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:30.927419 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d6f7d4c-e2a0-40d0-9686-f016013d1504-samples-operator-tls podName:5d6f7d4c-e2a0-40d0-9686-f016013d1504 nodeName:}" failed. No retries permitted until 2026-07-10 14:31:31.427401581 +0000 UTC m=+37.178773266 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5d6f7d4c-e2a0-40d0-9686-f016013d1504-samples-operator-tls") pod "cluster-samples-operator-65db858894-9gxrb" (UID: "5d6f7d4c-e2a0-40d0-9686-f016013d1504") : secret "samples-operator-tls" not found Jul 10 14:31:30.927657 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.927461 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crhmg\" (UniqueName: \"kubernetes.io/projected/80537ce8-cbd4-4a4c-8604-7377993cd5b1-kube-api-access-crhmg\") pod \"ingress-canary-wbbhv\" (UID: \"80537ce8-cbd4-4a4c-8604-7377993cd5b1\") " pod="openshift-ingress-canary/ingress-canary-wbbhv" Jul 10 14:31:30.927657 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.927508 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68e91fda-6611-4951-806d-2514fe8bd1cb-tmp-dir\") pod \"dns-default-f698x\" (UID: \"68e91fda-6611-4951-806d-2514fe8bd1cb\") " pod="openshift-dns/dns-default-f698x" Jul 10 14:31:30.927657 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.927535 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/80537ce8-cbd4-4a4c-8604-7377993cd5b1-cert\") pod \"ingress-canary-wbbhv\" (UID: \"80537ce8-cbd4-4a4c-8604-7377993cd5b1\") " pod="openshift-ingress-canary/ingress-canary-wbbhv" Jul 10 14:31:30.927657 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.927575 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/68e91fda-6611-4951-806d-2514fe8bd1cb-metrics-tls\") pod \"dns-default-f698x\" (UID: \"68e91fda-6611-4951-806d-2514fe8bd1cb\") " pod="openshift-dns/dns-default-f698x" Jul 10 14:31:30.927657 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.927620 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9sjj\" (UniqueName: \"kubernetes.io/projected/4d2d05db-87c6-4dff-9d05-599de4cfa9e4-kube-api-access-n9sjj\") pod \"volume-data-source-validator-78495597f6-bttbd\" (UID: \"4d2d05db-87c6-4dff-9d05-599de4cfa9e4\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-bttbd" Jul 10 14:31:30.927657 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.927650 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/dc5a5ba3-52b4-4b58-923e-df7e716d9438-snapshots\") pod \"insights-operator-68d9dd94cf-qtr8d\" (UID: \"dc5a5ba3-52b4-4b58-923e-df7e716d9438\") " pod="openshift-insights/insights-operator-68d9dd94cf-qtr8d" Jul 10 14:31:30.928098 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.927669 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/68e91fda-6611-4951-806d-2514fe8bd1cb-config-volume\") pod \"dns-default-f698x\" (UID: \"68e91fda-6611-4951-806d-2514fe8bd1cb\") " pod="openshift-dns/dns-default-f698x" Jul 10 14:31:30.928098 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:30.927698 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 14:31:30.928098 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:30.927745 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68e91fda-6611-4951-806d-2514fe8bd1cb-metrics-tls podName:68e91fda-6611-4951-806d-2514fe8bd1cb nodeName:}" failed. No retries permitted until 2026-07-10 14:31:31.427730216 +0000 UTC m=+37.179101890 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/68e91fda-6611-4951-806d-2514fe8bd1cb-metrics-tls") pod "dns-default-f698x" (UID: "68e91fda-6611-4951-806d-2514fe8bd1cb") : secret "dns-default-metrics-tls" not found Jul 10 14:31:30.928098 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.927816 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc5a5ba3-52b4-4b58-923e-df7e716d9438-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-qtr8d\" (UID: \"dc5a5ba3-52b4-4b58-923e-df7e716d9438\") " pod="openshift-insights/insights-operator-68d9dd94cf-qtr8d" Jul 10 14:31:30.928098 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.927853 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68e91fda-6611-4951-806d-2514fe8bd1cb-tmp-dir\") pod \"dns-default-f698x\" (UID: \"68e91fda-6611-4951-806d-2514fe8bd1cb\") " pod="openshift-dns/dns-default-f698x" Jul 10 14:31:30.928321 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.928298 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/dc5a5ba3-52b4-4b58-923e-df7e716d9438-snapshots\") pod \"insights-operator-68d9dd94cf-qtr8d\" (UID: \"dc5a5ba3-52b4-4b58-923e-df7e716d9438\") " pod="openshift-insights/insights-operator-68d9dd94cf-qtr8d" Jul 10 14:31:30.928407 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.928383 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/68e91fda-6611-4951-806d-2514fe8bd1cb-config-volume\") pod \"dns-default-f698x\" (UID: \"68e91fda-6611-4951-806d-2514fe8bd1cb\") " pod="openshift-dns/dns-default-f698x" Jul 10 14:31:30.928407 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.928394 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-42vlj\"" Jul 10 14:31:30.928509 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.928413 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 14:31:30.928509 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.928458 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 14:31:30.928584 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.928514 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pxtsm\"" Jul 10 14:31:30.929782 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.929592 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dc5a5ba3-52b4-4b58-923e-df7e716d9438-serving-cert\") pod \"insights-operator-68d9dd94cf-qtr8d\" (UID: \"dc5a5ba3-52b4-4b58-923e-df7e716d9438\") " pod="openshift-insights/insights-operator-68d9dd94cf-qtr8d" Jul 10 14:31:30.939356 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.939305 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbs2r\" (UniqueName: \"kubernetes.io/projected/5d6f7d4c-e2a0-40d0-9686-f016013d1504-kube-api-access-tbs2r\") pod \"cluster-samples-operator-65db858894-9gxrb\" (UID: \"5d6f7d4c-e2a0-40d0-9686-f016013d1504\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9gxrb" Jul 10 14:31:30.940951 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.940926 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9sjj\" (UniqueName: \"kubernetes.io/projected/4d2d05db-87c6-4dff-9d05-599de4cfa9e4-kube-api-access-n9sjj\") pod \"volume-data-source-validator-78495597f6-bttbd\" (UID: \"4d2d05db-87c6-4dff-9d05-599de4cfa9e4\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-bttbd" Jul 10 14:31:30.941094 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.941072 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7r4b\" (UniqueName: \"kubernetes.io/projected/dc5a5ba3-52b4-4b58-923e-df7e716d9438-kube-api-access-g7r4b\") pod \"insights-operator-68d9dd94cf-qtr8d\" (UID: \"dc5a5ba3-52b4-4b58-923e-df7e716d9438\") " pod="openshift-insights/insights-operator-68d9dd94cf-qtr8d" Jul 10 14:31:30.941388 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.941370 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24c9w\" (UniqueName: \"kubernetes.io/projected/5b19b116-a765-40a7-9268-b45593b2e2d8-kube-api-access-24c9w\") pod \"network-check-source-869dcfcdcd-tfq8b\" (UID: \"5b19b116-a765-40a7-9268-b45593b2e2d8\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-tfq8b" Jul 10 14:31:30.941876 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.941830 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt5wr\" (UniqueName: \"kubernetes.io/projected/68e91fda-6611-4951-806d-2514fe8bd1cb-kube-api-access-wt5wr\") pod \"dns-default-f698x\" (UID: \"68e91fda-6611-4951-806d-2514fe8bd1cb\") " pod="openshift-dns/dns-default-f698x" Jul 10 14:31:30.941876 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.941858 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7htb7\" (UniqueName: \"kubernetes.io/projected/cb41c5b6-dab8-4be4-b2be-56040d1659e7-kube-api-access-7htb7\") pod \"cluster-monitoring-operator-5f7f494dd4-c8ztx\" (UID: \"cb41c5b6-dab8-4be4-b2be-56040d1659e7\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-c8ztx" Jul 10 14:31:30.952703 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.952674 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-n6pfx" Jul 10 14:31:30.961815 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.961791 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-bttbd" Jul 10 14:31:30.990601 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:30.990571 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2g7m_4ee5894d-0007-499f-a401-2f11724910ac/konnectivity-agent/0.log" limit=1024 Jul 10 14:31:31.004105 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:31.004082 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-qtr8d" Jul 10 14:31:31.028498 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:31.028474 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/80537ce8-cbd4-4a4c-8604-7377993cd5b1-cert\") pod \"ingress-canary-wbbhv\" (UID: \"80537ce8-cbd4-4a4c-8604-7377993cd5b1\") " pod="openshift-ingress-canary/ingress-canary-wbbhv" Jul 10 14:31:31.028586 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:31.028560 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/70c38fec-83c3-4e81-90c3-ab3f7204fe3c-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-d7kg8\" (UID: \"70c38fec-83c3-4e81-90c3-ab3f7204fe3c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-d7kg8" Jul 10 14:31:31.028637 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:31.028605 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5ceab7ff-a2fc-4115-8dcc-1d9b014a8ffe-serving-cert\") pod \"service-ca-operator-7695fc6c8c-gwwww\" (UID: \"5ceab7ff-a2fc-4115-8dcc-1d9b014a8ffe\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-gwwww" Jul 10 14:31:31.028700 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:31.028638 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5ceab7ff-a2fc-4115-8dcc-1d9b014a8ffe-config\") pod \"service-ca-operator-7695fc6c8c-gwwww\" (UID: \"5ceab7ff-a2fc-4115-8dcc-1d9b014a8ffe\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-gwwww" Jul 10 14:31:31.028700 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:31.028680 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/70c38fec-83c3-4e81-90c3-ab3f7204fe3c-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-d7kg8\" (UID: \"70c38fec-83c3-4e81-90c3-ab3f7204fe3c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-d7kg8" Jul 10 14:31:31.028816 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:31.028732 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5qtn\" (UniqueName: \"kubernetes.io/projected/5ceab7ff-a2fc-4115-8dcc-1d9b014a8ffe-kube-api-access-r5qtn\") pod \"service-ca-operator-7695fc6c8c-gwwww\" (UID: \"5ceab7ff-a2fc-4115-8dcc-1d9b014a8ffe\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-gwwww" Jul 10 14:31:31.028816 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:31.028799 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v58sc\" (UniqueName: \"kubernetes.io/projected/70c38fec-83c3-4e81-90c3-ab3f7204fe3c-kube-api-access-v58sc\") pod \"kube-storage-version-migrator-operator-799cb7f958-d7kg8\" (UID: \"70c38fec-83c3-4e81-90c3-ab3f7204fe3c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-d7kg8" Jul 10 14:31:31.028896 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:31.028836 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crhmg\" (UniqueName: \"kubernetes.io/projected/80537ce8-cbd4-4a4c-8604-7377993cd5b1-kube-api-access-crhmg\") pod \"ingress-canary-wbbhv\" (UID: \"80537ce8-cbd4-4a4c-8604-7377993cd5b1\") " pod="openshift-ingress-canary/ingress-canary-wbbhv" Jul 10 14:31:31.029340 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:31.029309 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/70c38fec-83c3-4e81-90c3-ab3f7204fe3c-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-d7kg8\" (UID: \"70c38fec-83c3-4e81-90c3-ab3f7204fe3c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-d7kg8" Jul 10 14:31:31.029340 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:31.029328 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5ceab7ff-a2fc-4115-8dcc-1d9b014a8ffe-config\") pod \"service-ca-operator-7695fc6c8c-gwwww\" (UID: \"5ceab7ff-a2fc-4115-8dcc-1d9b014a8ffe\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-gwwww" Jul 10 14:31:31.031318 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:31.031291 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5ceab7ff-a2fc-4115-8dcc-1d9b014a8ffe-serving-cert\") pod \"service-ca-operator-7695fc6c8c-gwwww\" (UID: \"5ceab7ff-a2fc-4115-8dcc-1d9b014a8ffe\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-gwwww" Jul 10 14:31:31.031412 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:31.031330 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/80537ce8-cbd4-4a4c-8604-7377993cd5b1-cert\") pod \"ingress-canary-wbbhv\" (UID: \"80537ce8-cbd4-4a4c-8604-7377993cd5b1\") " pod="openshift-ingress-canary/ingress-canary-wbbhv" Jul 10 14:31:31.031412 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:31.031371 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/70c38fec-83c3-4e81-90c3-ab3f7204fe3c-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-d7kg8\" (UID: \"70c38fec-83c3-4e81-90c3-ab3f7204fe3c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-d7kg8" Jul 10 14:31:31.036988 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:31.036936 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v58sc\" (UniqueName: \"kubernetes.io/projected/70c38fec-83c3-4e81-90c3-ab3f7204fe3c-kube-api-access-v58sc\") pod \"kube-storage-version-migrator-operator-799cb7f958-d7kg8\" (UID: \"70c38fec-83c3-4e81-90c3-ab3f7204fe3c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-d7kg8" Jul 10 14:31:31.037120 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:31.037097 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5qtn\" (UniqueName: \"kubernetes.io/projected/5ceab7ff-a2fc-4115-8dcc-1d9b014a8ffe-kube-api-access-r5qtn\") pod \"service-ca-operator-7695fc6c8c-gwwww\" (UID: \"5ceab7ff-a2fc-4115-8dcc-1d9b014a8ffe\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-gwwww" Jul 10 14:31:31.037223 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:31.037168 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crhmg\" (UniqueName: \"kubernetes.io/projected/80537ce8-cbd4-4a4c-8604-7377993cd5b1-kube-api-access-crhmg\") pod \"ingress-canary-wbbhv\" (UID: \"80537ce8-cbd4-4a4c-8604-7377993cd5b1\") " pod="openshift-ingress-canary/ingress-canary-wbbhv" Jul 10 14:31:31.048018 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:31.047999 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-tfq8b" Jul 10 14:31:31.065337 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:31.065311 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-gwwww" Jul 10 14:31:31.079224 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:31.079198 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wbbhv" Jul 10 14:31:31.096617 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:31.096600 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-d7kg8" Jul 10 14:31:31.332017 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:31.331939 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13a97abe-27b2-4b7b-ae18-8ae1ff61f043-metrics-certs\") pod \"router-default-795d96bbd4-2nql9\" (UID: \"13a97abe-27b2-4b7b-ae18-8ae1ff61f043\") " pod="openshift-ingress/router-default-795d96bbd4-2nql9" Jul 10 14:31:31.332017 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:31.331996 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13a97abe-27b2-4b7b-ae18-8ae1ff61f043-service-ca-bundle\") pod \"router-default-795d96bbd4-2nql9\" (UID: \"13a97abe-27b2-4b7b-ae18-8ae1ff61f043\") " pod="openshift-ingress/router-default-795d96bbd4-2nql9" Jul 10 14:31:31.332241 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:31.332117 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 10 14:31:31.332241 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:31.332146 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c867eb4a-dae5-4edd-a157-7686c044be1e-registry-tls\") pod \"image-registry-8d46cc844-xf6cz\" (UID: \"c867eb4a-dae5-4edd-a157-7686c044be1e\") " pod="openshift-image-registry/image-registry-8d46cc844-xf6cz" Jul 10 14:31:31.332241 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:31.332193 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13a97abe-27b2-4b7b-ae18-8ae1ff61f043-metrics-certs podName:13a97abe-27b2-4b7b-ae18-8ae1ff61f043 nodeName:}" failed. No retries permitted until 2026-07-10 14:31:32.332171912 +0000 UTC m=+38.083543588 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/13a97abe-27b2-4b7b-ae18-8ae1ff61f043-metrics-certs") pod "router-default-795d96bbd4-2nql9" (UID: "13a97abe-27b2-4b7b-ae18-8ae1ff61f043") : secret "router-metrics-certs-default" not found Jul 10 14:31:31.332392 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:31.332258 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 14:31:31.332392 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:31.332276 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8d46cc844-xf6cz: secret "image-registry-tls" not found Jul 10 14:31:31.332392 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:31.332280 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/13a97abe-27b2-4b7b-ae18-8ae1ff61f043-service-ca-bundle podName:13a97abe-27b2-4b7b-ae18-8ae1ff61f043 nodeName:}" failed. No retries permitted until 2026-07-10 14:31:32.332241608 +0000 UTC m=+38.083613287 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/13a97abe-27b2-4b7b-ae18-8ae1ff61f043-service-ca-bundle") pod "router-default-795d96bbd4-2nql9" (UID: "13a97abe-27b2-4b7b-ae18-8ae1ff61f043") : configmap references non-existent config key: service-ca.crt Jul 10 14:31:31.332392 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:31.332323 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c867eb4a-dae5-4edd-a157-7686c044be1e-registry-tls podName:c867eb4a-dae5-4edd-a157-7686c044be1e nodeName:}" failed. No retries permitted until 2026-07-10 14:31:32.332308327 +0000 UTC m=+38.083680027 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c867eb4a-dae5-4edd-a157-7686c044be1e-registry-tls") pod "image-registry-8d46cc844-xf6cz" (UID: "c867eb4a-dae5-4edd-a157-7686c044be1e") : secret "image-registry-tls" not found Jul 10 14:31:31.433487 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:31.433441 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cb41c5b6-dab8-4be4-b2be-56040d1659e7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-c8ztx\" (UID: \"cb41c5b6-dab8-4be4-b2be-56040d1659e7\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-c8ztx" Jul 10 14:31:31.433648 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:31.433511 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5d6f7d4c-e2a0-40d0-9686-f016013d1504-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-9gxrb\" (UID: \"5d6f7d4c-e2a0-40d0-9686-f016013d1504\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9gxrb" Jul 10 14:31:31.433648 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:31.433562 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/68e91fda-6611-4951-806d-2514fe8bd1cb-metrics-tls\") pod \"dns-default-f698x\" (UID: \"68e91fda-6611-4951-806d-2514fe8bd1cb\") " pod="openshift-dns/dns-default-f698x" Jul 10 14:31:31.433648 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:31.433610 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 10 14:31:31.433835 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:31.433669 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 10 14:31:31.433835 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:31.433688 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb41c5b6-dab8-4be4-b2be-56040d1659e7-cluster-monitoring-operator-tls podName:cb41c5b6-dab8-4be4-b2be-56040d1659e7 nodeName:}" failed. No retries permitted until 2026-07-10 14:31:32.433666908 +0000 UTC m=+38.185038595 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cb41c5b6-dab8-4be4-b2be-56040d1659e7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-c8ztx" (UID: "cb41c5b6-dab8-4be4-b2be-56040d1659e7") : secret "cluster-monitoring-operator-tls" not found Jul 10 14:31:31.433835 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:31.433729 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d6f7d4c-e2a0-40d0-9686-f016013d1504-samples-operator-tls podName:5d6f7d4c-e2a0-40d0-9686-f016013d1504 nodeName:}" failed. No retries permitted until 2026-07-10 14:31:32.433712772 +0000 UTC m=+38.185084470 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5d6f7d4c-e2a0-40d0-9686-f016013d1504-samples-operator-tls") pod "cluster-samples-operator-65db858894-9gxrb" (UID: "5d6f7d4c-e2a0-40d0-9686-f016013d1504") : secret "samples-operator-tls" not found Jul 10 14:31:31.433835 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:31.433734 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 14:31:31.433835 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:31.433801 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68e91fda-6611-4951-806d-2514fe8bd1cb-metrics-tls podName:68e91fda-6611-4951-806d-2514fe8bd1cb nodeName:}" failed. No retries permitted until 2026-07-10 14:31:32.433785382 +0000 UTC m=+38.185157060 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/68e91fda-6611-4951-806d-2514fe8bd1cb-metrics-tls") pod "dns-default-f698x" (UID: "68e91fda-6611-4951-806d-2514fe8bd1cb") : secret "dns-default-metrics-tls" not found Jul 10 14:31:32.308078 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:32.307848 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-n6pfx"] Jul 10 14:31:32.310679 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:32.310655 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-tfq8b"] Jul 10 14:31:32.311864 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:32.311840 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wbbhv"] Jul 10 14:31:32.312161 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:31:32.312140 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod296be890_64e9_4fb3_8c68_e4c2a7c8cc2c.slice/crio-d11328ca3373c7f19e4eb4b1e7f45376926807dcc846c7f165f882195d047107 WatchSource:0}: Error finding container d11328ca3373c7f19e4eb4b1e7f45376926807dcc846c7f165f882195d047107: Status 404 returned error can't find the container with id d11328ca3373c7f19e4eb4b1e7f45376926807dcc846c7f165f882195d047107 Jul 10 14:31:32.315571 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:31:32.315549 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b19b116_a765_40a7_9268_b45593b2e2d8.slice/crio-8790cad61638275cff614ab22c8a36a99e0789819ebef40a7b04161e38f89215 WatchSource:0}: Error finding container 8790cad61638275cff614ab22c8a36a99e0789819ebef40a7b04161e38f89215: Status 404 returned error can't find the container with id 8790cad61638275cff614ab22c8a36a99e0789819ebef40a7b04161e38f89215 Jul 10 14:31:32.316243 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:31:32.316213 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80537ce8_cbd4_4a4c_8604_7377993cd5b1.slice/crio-80ea15ed5ea23bbaf240f5114e6d16a71c178fa26c6bbb5c4ce2888f7d7735d2 WatchSource:0}: Error finding container 80ea15ed5ea23bbaf240f5114e6d16a71c178fa26c6bbb5c4ce2888f7d7735d2: Status 404 returned error can't find the container with id 80ea15ed5ea23bbaf240f5114e6d16a71c178fa26c6bbb5c4ce2888f7d7735d2 Jul 10 14:31:32.327676 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:32.327656 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-gwwww"] Jul 10 14:31:32.331447 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:31:32.331415 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ceab7ff_a2fc_4115_8dcc_1d9b014a8ffe.slice/crio-157d6ba29b85fa326471a89aca2eed84245ba0c26dda78a25378663361afb6b6 WatchSource:0}: Error finding container 157d6ba29b85fa326471a89aca2eed84245ba0c26dda78a25378663361afb6b6: Status 404 returned error can't find the container with id 157d6ba29b85fa326471a89aca2eed84245ba0c26dda78a25378663361afb6b6 Jul 10 14:31:32.336417 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:32.336371 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-d7kg8"] Jul 10 14:31:32.337672 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:32.337155 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-qtr8d"] Jul 10 14:31:32.338341 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:32.338277 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-bttbd"] Jul 10 14:31:32.339105 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:31:32.339059 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc5a5ba3_52b4_4b58_923e_df7e716d9438.slice/crio-936d07003835b4a6c5f861883751da81b25c86fca378f9d16659e21b0efc7b3e WatchSource:0}: Error finding container 936d07003835b4a6c5f861883751da81b25c86fca378f9d16659e21b0efc7b3e: Status 404 returned error can't find the container with id 936d07003835b4a6c5f861883751da81b25c86fca378f9d16659e21b0efc7b3e Jul 10 14:31:32.341892 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:32.341728 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13a97abe-27b2-4b7b-ae18-8ae1ff61f043-service-ca-bundle\") pod \"router-default-795d96bbd4-2nql9\" (UID: \"13a97abe-27b2-4b7b-ae18-8ae1ff61f043\") " pod="openshift-ingress/router-default-795d96bbd4-2nql9" Jul 10 14:31:32.341892 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:32.341864 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c867eb4a-dae5-4edd-a157-7686c044be1e-registry-tls\") pod \"image-registry-8d46cc844-xf6cz\" (UID: \"c867eb4a-dae5-4edd-a157-7686c044be1e\") " pod="openshift-image-registry/image-registry-8d46cc844-xf6cz" Jul 10 14:31:32.341892 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:32.341880 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/13a97abe-27b2-4b7b-ae18-8ae1ff61f043-service-ca-bundle podName:13a97abe-27b2-4b7b-ae18-8ae1ff61f043 nodeName:}" failed. No retries permitted until 2026-07-10 14:31:34.341858727 +0000 UTC m=+40.093230402 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/13a97abe-27b2-4b7b-ae18-8ae1ff61f043-service-ca-bundle") pod "router-default-795d96bbd4-2nql9" (UID: "13a97abe-27b2-4b7b-ae18-8ae1ff61f043") : configmap references non-existent config key: service-ca.crt Jul 10 14:31:32.342146 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:32.341949 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 14:31:32.342146 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:32.341959 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8d46cc844-xf6cz: secret "image-registry-tls" not found Jul 10 14:31:32.342146 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:32.341956 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13a97abe-27b2-4b7b-ae18-8ae1ff61f043-metrics-certs\") pod \"router-default-795d96bbd4-2nql9\" (UID: \"13a97abe-27b2-4b7b-ae18-8ae1ff61f043\") " pod="openshift-ingress/router-default-795d96bbd4-2nql9" Jul 10 14:31:32.342146 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:32.341994 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c867eb4a-dae5-4edd-a157-7686c044be1e-registry-tls podName:c867eb4a-dae5-4edd-a157-7686c044be1e nodeName:}" failed. No retries permitted until 2026-07-10 14:31:34.34198289 +0000 UTC m=+40.093354564 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c867eb4a-dae5-4edd-a157-7686c044be1e-registry-tls") pod "image-registry-8d46cc844-xf6cz" (UID: "c867eb4a-dae5-4edd-a157-7686c044be1e") : secret "image-registry-tls" not found Jul 10 14:31:32.342146 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:32.342060 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 10 14:31:32.342146 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:32.342102 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13a97abe-27b2-4b7b-ae18-8ae1ff61f043-metrics-certs podName:13a97abe-27b2-4b7b-ae18-8ae1ff61f043 nodeName:}" failed. No retries permitted until 2026-07-10 14:31:34.342089669 +0000 UTC m=+40.093461357 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/13a97abe-27b2-4b7b-ae18-8ae1ff61f043-metrics-certs") pod "router-default-795d96bbd4-2nql9" (UID: "13a97abe-27b2-4b7b-ae18-8ae1ff61f043") : secret "router-metrics-certs-default" not found Jul 10 14:31:32.346637 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:31:32.346614 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70c38fec_83c3_4e81_90c3_ab3f7204fe3c.slice/crio-a00b66bf95da5dcc04fdd7143887cb245eb90f8b0ea9043ab7ce0d2a60d337ea WatchSource:0}: Error finding container a00b66bf95da5dcc04fdd7143887cb245eb90f8b0ea9043ab7ce0d2a60d337ea: Status 404 returned error can't find the container with id a00b66bf95da5dcc04fdd7143887cb245eb90f8b0ea9043ab7ce0d2a60d337ea Jul 10 14:31:32.347985 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:31:32.347818 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d2d05db_87c6_4dff_9d05_599de4cfa9e4.slice/crio-2a9eaed8fa64c3033cb7667551231d6dc404e9d0f51f6c7aefd5cf998c70b724 WatchSource:0}: Error finding container 2a9eaed8fa64c3033cb7667551231d6dc404e9d0f51f6c7aefd5cf998c70b724: Status 404 returned error can't find the container with id 2a9eaed8fa64c3033cb7667551231d6dc404e9d0f51f6c7aefd5cf998c70b724 Jul 10 14:31:32.442781 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:32.442731 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cb41c5b6-dab8-4be4-b2be-56040d1659e7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-c8ztx\" (UID: \"cb41c5b6-dab8-4be4-b2be-56040d1659e7\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-c8ztx" Jul 10 14:31:32.442921 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:32.442826 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5d6f7d4c-e2a0-40d0-9686-f016013d1504-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-9gxrb\" (UID: \"5d6f7d4c-e2a0-40d0-9686-f016013d1504\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9gxrb" Jul 10 14:31:32.442921 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:32.442880 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/68e91fda-6611-4951-806d-2514fe8bd1cb-metrics-tls\") pod \"dns-default-f698x\" (UID: \"68e91fda-6611-4951-806d-2514fe8bd1cb\") " pod="openshift-dns/dns-default-f698x" Jul 10 14:31:32.442921 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:32.442882 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 10 14:31:32.443071 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:32.442937 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb41c5b6-dab8-4be4-b2be-56040d1659e7-cluster-monitoring-operator-tls podName:cb41c5b6-dab8-4be4-b2be-56040d1659e7 nodeName:}" failed. No retries permitted until 2026-07-10 14:31:34.442922006 +0000 UTC m=+40.194293680 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cb41c5b6-dab8-4be4-b2be-56040d1659e7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-c8ztx" (UID: "cb41c5b6-dab8-4be4-b2be-56040d1659e7") : secret "cluster-monitoring-operator-tls" not found Jul 10 14:31:32.443071 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:32.442979 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 14:31:32.443071 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:32.442992 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 10 14:31:32.443071 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:32.443012 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68e91fda-6611-4951-806d-2514fe8bd1cb-metrics-tls podName:68e91fda-6611-4951-806d-2514fe8bd1cb nodeName:}" failed. No retries permitted until 2026-07-10 14:31:34.443002471 +0000 UTC m=+40.194374145 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/68e91fda-6611-4951-806d-2514fe8bd1cb-metrics-tls") pod "dns-default-f698x" (UID: "68e91fda-6611-4951-806d-2514fe8bd1cb") : secret "dns-default-metrics-tls" not found Jul 10 14:31:32.443071 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:32.443063 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d6f7d4c-e2a0-40d0-9686-f016013d1504-samples-operator-tls podName:5d6f7d4c-e2a0-40d0-9686-f016013d1504 nodeName:}" failed. No retries permitted until 2026-07-10 14:31:34.443046343 +0000 UTC m=+40.194418094 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5d6f7d4c-e2a0-40d0-9686-f016013d1504-samples-operator-tls") pod "cluster-samples-operator-65db858894-9gxrb" (UID: "5d6f7d4c-e2a0-40d0-9686-f016013d1504") : secret "samples-operator-tls" not found Jul 10 14:31:33.144350 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:33.143555 2569 generic.go:356] "Generic (PLEG): container finished" podID="ba16bc50-891a-4477-8bd5-de3ef058d04f" containerID="2acda6e9a4ac85cc46ddab6870930e7e3f963d2b6d5f28ea009589746d73f7d6" exitCode=0 Jul 10 14:31:33.144350 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:33.143664 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2sjq2" event={"ID":"ba16bc50-891a-4477-8bd5-de3ef058d04f","Type":"ContainerDied","Data":"2acda6e9a4ac85cc46ddab6870930e7e3f963d2b6d5f28ea009589746d73f7d6"} Jul 10 14:31:33.147299 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:33.147138 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-bttbd" event={"ID":"4d2d05db-87c6-4dff-9d05-599de4cfa9e4","Type":"ContainerStarted","Data":"2a9eaed8fa64c3033cb7667551231d6dc404e9d0f51f6c7aefd5cf998c70b724"} Jul 10 14:31:33.150455 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:33.150425 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-qtr8d" event={"ID":"dc5a5ba3-52b4-4b58-923e-df7e716d9438","Type":"ContainerStarted","Data":"936d07003835b4a6c5f861883751da81b25c86fca378f9d16659e21b0efc7b3e"} Jul 10 14:31:33.152349 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:33.152302 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-tfq8b" event={"ID":"5b19b116-a765-40a7-9268-b45593b2e2d8","Type":"ContainerStarted","Data":"8790cad61638275cff614ab22c8a36a99e0789819ebef40a7b04161e38f89215"} Jul 10 14:31:33.154857 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:33.154781 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-n6pfx" event={"ID":"296be890-64e9-4fb3-8c68-e4c2a7c8cc2c","Type":"ContainerStarted","Data":"d11328ca3373c7f19e4eb4b1e7f45376926807dcc846c7f165f882195d047107"} Jul 10 14:31:33.157866 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:33.157841 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wbbhv" event={"ID":"80537ce8-cbd4-4a4c-8604-7377993cd5b1","Type":"ContainerStarted","Data":"80ea15ed5ea23bbaf240f5114e6d16a71c178fa26c6bbb5c4ce2888f7d7735d2"} Jul 10 14:31:33.160564 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:33.160539 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-d7kg8" event={"ID":"70c38fec-83c3-4e81-90c3-ab3f7204fe3c","Type":"ContainerStarted","Data":"a00b66bf95da5dcc04fdd7143887cb245eb90f8b0ea9043ab7ce0d2a60d337ea"} Jul 10 14:31:33.162125 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:33.162082 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-gwwww" event={"ID":"5ceab7ff-a2fc-4115-8dcc-1d9b014a8ffe","Type":"ContainerStarted","Data":"157d6ba29b85fa326471a89aca2eed84245ba0c26dda78a25378663361afb6b6"} Jul 10 14:31:34.194166 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:34.193166 2569 generic.go:356] "Generic (PLEG): container finished" podID="ba16bc50-891a-4477-8bd5-de3ef058d04f" containerID="5e669fe3648394d43be24472fd2ac1f6482cd785e548ace9131dd95194374e50" exitCode=0 Jul 10 14:31:34.194166 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:34.193247 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2sjq2" event={"ID":"ba16bc50-891a-4477-8bd5-de3ef058d04f","Type":"ContainerDied","Data":"5e669fe3648394d43be24472fd2ac1f6482cd785e548ace9131dd95194374e50"} Jul 10 14:31:34.360979 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:34.359641 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13a97abe-27b2-4b7b-ae18-8ae1ff61f043-metrics-certs\") pod \"router-default-795d96bbd4-2nql9\" (UID: \"13a97abe-27b2-4b7b-ae18-8ae1ff61f043\") " pod="openshift-ingress/router-default-795d96bbd4-2nql9" Jul 10 14:31:34.360979 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:34.359901 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13a97abe-27b2-4b7b-ae18-8ae1ff61f043-service-ca-bundle\") pod \"router-default-795d96bbd4-2nql9\" (UID: \"13a97abe-27b2-4b7b-ae18-8ae1ff61f043\") " pod="openshift-ingress/router-default-795d96bbd4-2nql9" Jul 10 14:31:34.360979 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:34.360067 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c867eb4a-dae5-4edd-a157-7686c044be1e-registry-tls\") pod \"image-registry-8d46cc844-xf6cz\" (UID: \"c867eb4a-dae5-4edd-a157-7686c044be1e\") " pod="openshift-image-registry/image-registry-8d46cc844-xf6cz" Jul 10 14:31:34.360979 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:34.360203 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 14:31:34.360979 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:34.360223 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8d46cc844-xf6cz: secret "image-registry-tls" not found Jul 10 14:31:34.360979 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:34.360284 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c867eb4a-dae5-4edd-a157-7686c044be1e-registry-tls podName:c867eb4a-dae5-4edd-a157-7686c044be1e nodeName:}" failed. No retries permitted until 2026-07-10 14:31:38.360263246 +0000 UTC m=+44.111634942 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c867eb4a-dae5-4edd-a157-7686c044be1e-registry-tls") pod "image-registry-8d46cc844-xf6cz" (UID: "c867eb4a-dae5-4edd-a157-7686c044be1e") : secret "image-registry-tls" not found Jul 10 14:31:34.360979 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:34.360606 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 10 14:31:34.360979 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:34.360669 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13a97abe-27b2-4b7b-ae18-8ae1ff61f043-metrics-certs podName:13a97abe-27b2-4b7b-ae18-8ae1ff61f043 nodeName:}" failed. No retries permitted until 2026-07-10 14:31:38.36065191 +0000 UTC m=+44.112023586 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/13a97abe-27b2-4b7b-ae18-8ae1ff61f043-metrics-certs") pod "router-default-795d96bbd4-2nql9" (UID: "13a97abe-27b2-4b7b-ae18-8ae1ff61f043") : secret "router-metrics-certs-default" not found Jul 10 14:31:34.360979 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:34.360802 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/13a97abe-27b2-4b7b-ae18-8ae1ff61f043-service-ca-bundle podName:13a97abe-27b2-4b7b-ae18-8ae1ff61f043 nodeName:}" failed. No retries permitted until 2026-07-10 14:31:38.360785517 +0000 UTC m=+44.112157206 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/13a97abe-27b2-4b7b-ae18-8ae1ff61f043-service-ca-bundle") pod "router-default-795d96bbd4-2nql9" (UID: "13a97abe-27b2-4b7b-ae18-8ae1ff61f043") : configmap references non-existent config key: service-ca.crt Jul 10 14:31:34.460777 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:34.460672 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5d6f7d4c-e2a0-40d0-9686-f016013d1504-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-9gxrb\" (UID: \"5d6f7d4c-e2a0-40d0-9686-f016013d1504\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9gxrb" Jul 10 14:31:34.460936 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:34.460801 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/68e91fda-6611-4951-806d-2514fe8bd1cb-metrics-tls\") pod \"dns-default-f698x\" (UID: \"68e91fda-6611-4951-806d-2514fe8bd1cb\") " pod="openshift-dns/dns-default-f698x" Jul 10 14:31:34.460936 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:34.460919 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cb41c5b6-dab8-4be4-b2be-56040d1659e7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-c8ztx\" (UID: \"cb41c5b6-dab8-4be4-b2be-56040d1659e7\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-c8ztx" Jul 10 14:31:34.461127 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:34.461081 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 10 14:31:34.461207 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:34.461147 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb41c5b6-dab8-4be4-b2be-56040d1659e7-cluster-monitoring-operator-tls podName:cb41c5b6-dab8-4be4-b2be-56040d1659e7 nodeName:}" failed. No retries permitted until 2026-07-10 14:31:38.461128956 +0000 UTC m=+44.212500633 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cb41c5b6-dab8-4be4-b2be-56040d1659e7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-c8ztx" (UID: "cb41c5b6-dab8-4be4-b2be-56040d1659e7") : secret "cluster-monitoring-operator-tls" not found Jul 10 14:31:34.461638 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:34.461424 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 14:31:34.461638 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:34.461481 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68e91fda-6611-4951-806d-2514fe8bd1cb-metrics-tls podName:68e91fda-6611-4951-806d-2514fe8bd1cb nodeName:}" failed. No retries permitted until 2026-07-10 14:31:38.461465837 +0000 UTC m=+44.212837529 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/68e91fda-6611-4951-806d-2514fe8bd1cb-metrics-tls") pod "dns-default-f698x" (UID: "68e91fda-6611-4951-806d-2514fe8bd1cb") : secret "dns-default-metrics-tls" not found Jul 10 14:31:34.461638 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:34.461524 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 10 14:31:34.461638 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:34.461591 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d6f7d4c-e2a0-40d0-9686-f016013d1504-samples-operator-tls podName:5d6f7d4c-e2a0-40d0-9686-f016013d1504 nodeName:}" failed. No retries permitted until 2026-07-10 14:31:38.461572199 +0000 UTC m=+44.212943876 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5d6f7d4c-e2a0-40d0-9686-f016013d1504-samples-operator-tls") pod "cluster-samples-operator-65db858894-9gxrb" (UID: "5d6f7d4c-e2a0-40d0-9686-f016013d1504") : secret "samples-operator-tls" not found Jul 10 14:31:35.200606 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:35.200569 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2sjq2" event={"ID":"ba16bc50-891a-4477-8bd5-de3ef058d04f","Type":"ContainerStarted","Data":"881f5c1382d700408896c3b9d4b24d7a4dc6bdfb855e1133fb0aaab4aea2fe32"} Jul 10 14:31:35.226974 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:35.226908 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2sjq2" podStartSLOduration=7.611688523 podStartE2EDuration="40.226887615s" totalImagesPullingTime="32.615199092s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 14:30:55 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:31:35.225470967 +0000 UTC m=+40.976842694" watchObservedRunningTime="2026-07-10 14:31:35.226887615 +0000 UTC m=+40.978259312" Jul 10 14:31:35.358517 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:35.358484 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2g7m_4ee5894d-0007-499f-a401-2f11724910ac/konnectivity-agent/0.log" limit=1024 Jul 10 14:31:35.453679 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:35.453586 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2g7m_4ee5894d-0007-499f-a401-2f11724910ac/konnectivity-agent/0.log" limit=1024 Jul 10 14:31:38.401360 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:38.401320 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13a97abe-27b2-4b7b-ae18-8ae1ff61f043-metrics-certs\") pod \"router-default-795d96bbd4-2nql9\" (UID: \"13a97abe-27b2-4b7b-ae18-8ae1ff61f043\") " pod="openshift-ingress/router-default-795d96bbd4-2nql9" Jul 10 14:31:38.401698 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:38.401372 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13a97abe-27b2-4b7b-ae18-8ae1ff61f043-service-ca-bundle\") pod \"router-default-795d96bbd4-2nql9\" (UID: \"13a97abe-27b2-4b7b-ae18-8ae1ff61f043\") " pod="openshift-ingress/router-default-795d96bbd4-2nql9" Jul 10 14:31:38.401698 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:38.401481 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 10 14:31:38.401698 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:38.401522 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c867eb4a-dae5-4edd-a157-7686c044be1e-registry-tls\") pod \"image-registry-8d46cc844-xf6cz\" (UID: \"c867eb4a-dae5-4edd-a157-7686c044be1e\") " pod="openshift-image-registry/image-registry-8d46cc844-xf6cz" Jul 10 14:31:38.401698 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:38.401564 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13a97abe-27b2-4b7b-ae18-8ae1ff61f043-metrics-certs podName:13a97abe-27b2-4b7b-ae18-8ae1ff61f043 nodeName:}" failed. No retries permitted until 2026-07-10 14:31:46.401544894 +0000 UTC m=+52.152916571 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/13a97abe-27b2-4b7b-ae18-8ae1ff61f043-metrics-certs") pod "router-default-795d96bbd4-2nql9" (UID: "13a97abe-27b2-4b7b-ae18-8ae1ff61f043") : secret "router-metrics-certs-default" not found Jul 10 14:31:38.401698 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:38.401583 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/13a97abe-27b2-4b7b-ae18-8ae1ff61f043-service-ca-bundle podName:13a97abe-27b2-4b7b-ae18-8ae1ff61f043 nodeName:}" failed. No retries permitted until 2026-07-10 14:31:46.401574646 +0000 UTC m=+52.152946325 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/13a97abe-27b2-4b7b-ae18-8ae1ff61f043-service-ca-bundle") pod "router-default-795d96bbd4-2nql9" (UID: "13a97abe-27b2-4b7b-ae18-8ae1ff61f043") : configmap references non-existent config key: service-ca.crt Jul 10 14:31:38.401698 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:38.401615 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 14:31:38.401698 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:38.401630 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8d46cc844-xf6cz: secret "image-registry-tls" not found Jul 10 14:31:38.401698 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:38.401669 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c867eb4a-dae5-4edd-a157-7686c044be1e-registry-tls podName:c867eb4a-dae5-4edd-a157-7686c044be1e nodeName:}" failed. No retries permitted until 2026-07-10 14:31:46.401656048 +0000 UTC m=+52.153027731 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c867eb4a-dae5-4edd-a157-7686c044be1e-registry-tls") pod "image-registry-8d46cc844-xf6cz" (UID: "c867eb4a-dae5-4edd-a157-7686c044be1e") : secret "image-registry-tls" not found Jul 10 14:31:38.501936 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:38.501897 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/68e91fda-6611-4951-806d-2514fe8bd1cb-metrics-tls\") pod \"dns-default-f698x\" (UID: \"68e91fda-6611-4951-806d-2514fe8bd1cb\") " pod="openshift-dns/dns-default-f698x" Jul 10 14:31:38.502074 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:38.501984 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cb41c5b6-dab8-4be4-b2be-56040d1659e7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-c8ztx\" (UID: \"cb41c5b6-dab8-4be4-b2be-56040d1659e7\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-c8ztx" Jul 10 14:31:38.502074 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:38.502016 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5d6f7d4c-e2a0-40d0-9686-f016013d1504-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-9gxrb\" (UID: \"5d6f7d4c-e2a0-40d0-9686-f016013d1504\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9gxrb" Jul 10 14:31:38.502074 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:38.502043 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 14:31:38.502201 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:38.502109 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 10 14:31:38.502201 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:38.502113 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 10 14:31:38.502201 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:38.502111 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68e91fda-6611-4951-806d-2514fe8bd1cb-metrics-tls podName:68e91fda-6611-4951-806d-2514fe8bd1cb nodeName:}" failed. No retries permitted until 2026-07-10 14:31:46.502095343 +0000 UTC m=+52.253467017 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/68e91fda-6611-4951-806d-2514fe8bd1cb-metrics-tls") pod "dns-default-f698x" (UID: "68e91fda-6611-4951-806d-2514fe8bd1cb") : secret "dns-default-metrics-tls" not found Jul 10 14:31:38.502201 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:38.502155 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d6f7d4c-e2a0-40d0-9686-f016013d1504-samples-operator-tls podName:5d6f7d4c-e2a0-40d0-9686-f016013d1504 nodeName:}" failed. No retries permitted until 2026-07-10 14:31:46.502144698 +0000 UTC m=+52.253516371 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5d6f7d4c-e2a0-40d0-9686-f016013d1504-samples-operator-tls") pod "cluster-samples-operator-65db858894-9gxrb" (UID: "5d6f7d4c-e2a0-40d0-9686-f016013d1504") : secret "samples-operator-tls" not found Jul 10 14:31:38.502201 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:38.502165 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb41c5b6-dab8-4be4-b2be-56040d1659e7-cluster-monitoring-operator-tls podName:cb41c5b6-dab8-4be4-b2be-56040d1659e7 nodeName:}" failed. No retries permitted until 2026-07-10 14:31:46.502159774 +0000 UTC m=+52.253531448 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cb41c5b6-dab8-4be4-b2be-56040d1659e7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-c8ztx" (UID: "cb41c5b6-dab8-4be4-b2be-56040d1659e7") : secret "cluster-monitoring-operator-tls" not found Jul 10 14:31:40.216388 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:40.216351 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-bttbd" event={"ID":"4d2d05db-87c6-4dff-9d05-599de4cfa9e4","Type":"ContainerStarted","Data":"28286ffb6bb311412ffb8b5c0a3859a4b9c0eb2b5f39a1a700823f543755abd8"} Jul 10 14:31:40.217896 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:40.217868 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-qtr8d" event={"ID":"dc5a5ba3-52b4-4b58-923e-df7e716d9438","Type":"ContainerStarted","Data":"578b74a3c3bb275f6f2574d0dfefeb5de35af020107e205feb0c0aa51001b3ea"} Jul 10 14:31:40.219336 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:40.219314 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-gwwww" event={"ID":"5ceab7ff-a2fc-4115-8dcc-1d9b014a8ffe","Type":"ContainerStarted","Data":"40dd25b7b07f01621719f61a5c4da68e0dd482f6919eb427ec52a72325fc462e"} Jul 10 14:31:40.273574 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:40.273185 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-68d9dd94cf-qtr8d" podStartSLOduration=12.787829811 podStartE2EDuration="20.27316686s" totalImagesPullingTime="7.485337049s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 14:31:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:31:40.271526855 +0000 UTC m=+46.022898554" watchObservedRunningTime="2026-07-10 14:31:40.27316686 +0000 UTC m=+46.024538557" Jul 10 14:31:40.287211 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:40.287159 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-bttbd" podStartSLOduration=14.258267542 podStartE2EDuration="20.287143211s" totalImagesPullingTime="6.028875669s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 14:31:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:31:40.286958691 +0000 UTC m=+46.038330388" watchObservedRunningTime="2026-07-10 14:31:40.287143211 +0000 UTC m=+46.038514920" Jul 10 14:31:40.373256 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:40.373197 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2g7m_4ee5894d-0007-499f-a401-2f11724910ac/konnectivity-agent/0.log" limit=1024 Jul 10 14:31:40.436057 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:40.435979 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2g7m_4ee5894d-0007-499f-a401-2f11724910ac/konnectivity-agent/0.log" limit=1024 Jul 10 14:31:41.225326 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:41.225287 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wbbhv" event={"ID":"80537ce8-cbd4-4a4c-8604-7377993cd5b1","Type":"ContainerStarted","Data":"b61aa935fb6d78ebece4c4aecfb242387d46bf8ef51cb02efe7533f34a1e8699"} Jul 10 14:31:41.227054 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:41.227022 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-d7kg8" event={"ID":"70c38fec-83c3-4e81-90c3-ab3f7204fe3c","Type":"ContainerStarted","Data":"4aa661743253eff78c10a084a7df3925015bfeaf686627e1bb638aaea106fdc1"} Jul 10 14:31:41.228605 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:41.228580 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-tfq8b" event={"ID":"5b19b116-a765-40a7-9268-b45593b2e2d8","Type":"ContainerStarted","Data":"307258f5d8bc1c4a23627650da0bfb05b61d75984be91539ddc985b9c17c54e4"} Jul 10 14:31:41.233080 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:41.230741 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-n6pfx_296be890-64e9-4fb3-8c68-e4c2a7c8cc2c/console-operator/0.log" Jul 10 14:31:41.233080 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:41.230806 2569 generic.go:356] "Generic (PLEG): container finished" podID="296be890-64e9-4fb3-8c68-e4c2a7c8cc2c" containerID="cb4cc1dc9d862dda6252c02649c203efe2f3895f8061709ac0e3b912fdbd33ed" exitCode=255 Jul 10 14:31:41.233080 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:41.231733 2569 scope.go:117] "RemoveContainer" containerID="cb4cc1dc9d862dda6252c02649c203efe2f3895f8061709ac0e3b912fdbd33ed" Jul 10 14:31:41.233080 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:41.232145 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-n6pfx" event={"ID":"296be890-64e9-4fb3-8c68-e4c2a7c8cc2c","Type":"ContainerDied","Data":"cb4cc1dc9d862dda6252c02649c203efe2f3895f8061709ac0e3b912fdbd33ed"} Jul 10 14:31:41.242953 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:41.242629 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-wbbhv" podStartSLOduration=3.50825853 podStartE2EDuration="11.242613219s" totalImagesPullingTime="7.734354689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 14:31:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:31:41.242573514 +0000 UTC m=+46.993945211" watchObservedRunningTime="2026-07-10 14:31:41.242613219 +0000 UTC m=+46.993984915" Jul 10 14:31:41.261321 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:41.258713 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-tfq8b" podStartSLOduration=7.51381831 podStartE2EDuration="15.258682434s" totalImagesPullingTime="7.744864124s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 14:31:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:31:41.257364016 +0000 UTC m=+47.008735722" watchObservedRunningTime="2026-07-10 14:31:41.258682434 +0000 UTC m=+47.010054131" Jul 10 14:31:41.277311 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:41.277249 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-gwwww" podStartSLOduration=7.560976978 podStartE2EDuration="15.277230823s" totalImagesPullingTime="7.716253845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 14:31:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:31:41.275902478 +0000 UTC m=+47.027274179" watchObservedRunningTime="2026-07-10 14:31:41.277230823 +0000 UTC m=+47.028602524" Jul 10 14:31:41.295851 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:41.295794 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-d7kg8" podStartSLOduration=9.59241597 podStartE2EDuration="17.295743269s" totalImagesPullingTime="7.703327299s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 14:31:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:31:41.294671568 +0000 UTC m=+47.046043265" watchObservedRunningTime="2026-07-10 14:31:41.295743269 +0000 UTC m=+47.047114965" Jul 10 14:31:41.560519 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:41.560485 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2g7m_4ee5894d-0007-499f-a401-2f11724910ac/konnectivity-agent/0.log" limit=1024 Jul 10 14:31:41.596348 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:41.595948 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-9z7n5"] Jul 10 14:31:41.603377 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:41.603355 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-9z7n5" Jul 10 14:31:41.608322 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:41.608127 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 10 14:31:41.609009 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:41.608987 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-96rhn\"" Jul 10 14:31:41.609117 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:41.609062 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 10 14:31:41.620219 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:41.620179 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-9z7n5"] Jul 10 14:31:41.628968 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:41.628945 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfzld\" (UniqueName: \"kubernetes.io/projected/064387da-b900-4cda-9839-2b5c166bc246-kube-api-access-jfzld\") pod \"migrator-7f59b54f6c-9z7n5\" (UID: \"064387da-b900-4cda-9839-2b5c166bc246\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-9z7n5" Jul 10 14:31:41.682230 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:41.681973 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-q9tkw"] Jul 10 14:31:41.688003 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:41.687964 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2g7m_4ee5894d-0007-499f-a401-2f11724910ac/konnectivity-agent/0.log" limit=1024 Jul 10 14:31:41.691669 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:41.691644 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-q9tkw" Jul 10 14:31:41.694390 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:41.694367 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 10 14:31:41.694714 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:41.694676 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 10 14:31:41.694928 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:41.694902 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2nt82\"" Jul 10 14:31:41.701953 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:41.701883 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-q9tkw"] Jul 10 14:31:41.729509 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:41.729484 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0-data-volume\") pod \"insights-runtime-extractor-q9tkw\" (UID: \"1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0\") " pod="openshift-insights/insights-runtime-extractor-q9tkw" Jul 10 14:31:41.729627 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:41.729533 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q9tkw\" (UID: \"1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0\") " pod="openshift-insights/insights-runtime-extractor-q9tkw" Jul 10 14:31:41.729673 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:41.729630 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfzld\" (UniqueName: \"kubernetes.io/projected/064387da-b900-4cda-9839-2b5c166bc246-kube-api-access-jfzld\") pod \"migrator-7f59b54f6c-9z7n5\" (UID: \"064387da-b900-4cda-9839-2b5c166bc246\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-9z7n5" Jul 10 14:31:41.729673 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:41.729657 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0-crio-socket\") pod \"insights-runtime-extractor-q9tkw\" (UID: \"1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0\") " pod="openshift-insights/insights-runtime-extractor-q9tkw" Jul 10 14:31:41.729765 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:41.729737 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26r9b\" (UniqueName: \"kubernetes.io/projected/1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0-kube-api-access-26r9b\") pod \"insights-runtime-extractor-q9tkw\" (UID: \"1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0\") " pod="openshift-insights/insights-runtime-extractor-q9tkw" Jul 10 14:31:41.729807 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:41.729793 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q9tkw\" (UID: \"1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0\") " pod="openshift-insights/insights-runtime-extractor-q9tkw" Jul 10 14:31:41.749277 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:41.749177 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfzld\" (UniqueName: \"kubernetes.io/projected/064387da-b900-4cda-9839-2b5c166bc246-kube-api-access-jfzld\") pod \"migrator-7f59b54f6c-9z7n5\" (UID: \"064387da-b900-4cda-9839-2b5c166bc246\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-9z7n5" Jul 10 14:31:41.779508 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:41.779482 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2g7m_4ee5894d-0007-499f-a401-2f11724910ac/konnectivity-agent/0.log" limit=1024 Jul 10 14:31:41.830632 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:41.830603 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0-data-volume\") pod \"insights-runtime-extractor-q9tkw\" (UID: \"1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0\") " pod="openshift-insights/insights-runtime-extractor-q9tkw" Jul 10 14:31:41.830742 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:41.830672 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q9tkw\" (UID: \"1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0\") " pod="openshift-insights/insights-runtime-extractor-q9tkw" Jul 10 14:31:41.830821 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:41.830780 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0-crio-socket\") pod \"insights-runtime-extractor-q9tkw\" (UID: \"1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0\") " pod="openshift-insights/insights-runtime-extractor-q9tkw" Jul 10 14:31:41.830884 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:41.830834 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 14:31:41.830932 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:41.830884 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26r9b\" (UniqueName: \"kubernetes.io/projected/1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0-kube-api-access-26r9b\") pod \"insights-runtime-extractor-q9tkw\" (UID: \"1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0\") " pod="openshift-insights/insights-runtime-extractor-q9tkw" Jul 10 14:31:41.830932 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:41.830905 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0-kube-rbac-proxy-cm podName:1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0 nodeName:}" failed. No retries permitted until 2026-07-10 14:31:42.330883358 +0000 UTC m=+48.082255047 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q9tkw" (UID: "1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0") : configmap "kube-rbac-proxy" not found Jul 10 14:31:41.831030 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:41.830946 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q9tkw\" (UID: \"1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0\") " pod="openshift-insights/insights-runtime-extractor-q9tkw" Jul 10 14:31:41.831030 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:41.830972 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0-data-volume\") pod \"insights-runtime-extractor-q9tkw\" (UID: \"1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0\") " pod="openshift-insights/insights-runtime-extractor-q9tkw" Jul 10 14:31:41.831030 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:41.830994 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0-crio-socket\") pod \"insights-runtime-extractor-q9tkw\" (UID: \"1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0\") " pod="openshift-insights/insights-runtime-extractor-q9tkw" Jul 10 14:31:41.831168 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:41.831072 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 14:31:41.831168 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:41.831121 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0-insights-runtime-extractor-tls podName:1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0 nodeName:}" failed. No retries permitted until 2026-07-10 14:31:42.33110722 +0000 UTC m=+48.082478896 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q9tkw" (UID: "1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0") : secret "insights-runtime-extractor-tls" not found Jul 10 14:31:41.839644 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:41.839616 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26r9b\" (UniqueName: \"kubernetes.io/projected/1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0-kube-api-access-26r9b\") pod \"insights-runtime-extractor-q9tkw\" (UID: \"1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0\") " pod="openshift-insights/insights-runtime-extractor-q9tkw" Jul 10 14:31:41.915962 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:41.915939 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-9z7n5" Jul 10 14:31:42.022149 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:42.022041 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2g7m_4ee5894d-0007-499f-a401-2f11724910ac/konnectivity-agent/0.log" limit=1024 Jul 10 14:31:42.049494 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:42.049469 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-9z7n5"] Jul 10 14:31:42.052403 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:31:42.052379 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod064387da_b900_4cda_9839_2b5c166bc246.slice/crio-72885e222fac64c8b3d14918736559aad58974a1f58f072e3cea32b5768ca547 WatchSource:0}: Error finding container 72885e222fac64c8b3d14918736559aad58974a1f58f072e3cea32b5768ca547: Status 404 returned error can't find the container with id 72885e222fac64c8b3d14918736559aad58974a1f58f072e3cea32b5768ca547 Jul 10 14:31:42.122390 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:42.122360 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2g7m_4ee5894d-0007-499f-a401-2f11724910ac/konnectivity-agent/0.log" limit=1024 Jul 10 14:31:42.235313 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:42.235242 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-n6pfx_296be890-64e9-4fb3-8c68-e4c2a7c8cc2c/console-operator/1.log" Jul 10 14:31:42.235867 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:42.235639 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-n6pfx_296be890-64e9-4fb3-8c68-e4c2a7c8cc2c/console-operator/0.log" Jul 10 14:31:42.235867 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:42.235671 2569 generic.go:356] "Generic (PLEG): container finished" podID="296be890-64e9-4fb3-8c68-e4c2a7c8cc2c" containerID="6c736c15a8141007d963d66291fd332b9e3d7b278c9f6cf7743f8602ee434fa3" exitCode=255 Jul 10 14:31:42.235867 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:42.235770 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-n6pfx" event={"ID":"296be890-64e9-4fb3-8c68-e4c2a7c8cc2c","Type":"ContainerDied","Data":"6c736c15a8141007d963d66291fd332b9e3d7b278c9f6cf7743f8602ee434fa3"} Jul 10 14:31:42.235867 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:42.235827 2569 scope.go:117] "RemoveContainer" containerID="cb4cc1dc9d862dda6252c02649c203efe2f3895f8061709ac0e3b912fdbd33ed" Jul 10 14:31:42.236119 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:42.236103 2569 scope.go:117] "RemoveContainer" containerID="6c736c15a8141007d963d66291fd332b9e3d7b278c9f6cf7743f8602ee434fa3" Jul 10 14:31:42.236294 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:42.236273 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-n6pfx_openshift-console-operator(296be890-64e9-4fb3-8c68-e4c2a7c8cc2c)\"" pod="openshift-console-operator/console-operator-7866497494-n6pfx" podUID="296be890-64e9-4fb3-8c68-e4c2a7c8cc2c" Jul 10 14:31:42.237151 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:42.237018 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-9z7n5" event={"ID":"064387da-b900-4cda-9839-2b5c166bc246","Type":"ContainerStarted","Data":"72885e222fac64c8b3d14918736559aad58974a1f58f072e3cea32b5768ca547"} Jul 10 14:31:42.334952 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:42.334619 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q9tkw\" (UID: \"1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0\") " pod="openshift-insights/insights-runtime-extractor-q9tkw" Jul 10 14:31:42.334952 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:42.334729 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q9tkw\" (UID: \"1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0\") " pod="openshift-insights/insights-runtime-extractor-q9tkw" Jul 10 14:31:42.334952 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:42.334846 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 14:31:42.334952 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:42.334894 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 14:31:42.334952 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:42.334928 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0-kube-rbac-proxy-cm podName:1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0 nodeName:}" failed. No retries permitted until 2026-07-10 14:31:43.334909014 +0000 UTC m=+49.086280693 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q9tkw" (UID: "1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0") : configmap "kube-rbac-proxy" not found Jul 10 14:31:42.334952 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:42.334959 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0-insights-runtime-extractor-tls podName:1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0 nodeName:}" failed. No retries permitted until 2026-07-10 14:31:43.334938869 +0000 UTC m=+49.086310548 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q9tkw" (UID: "1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0") : secret "insights-runtime-extractor-tls" not found Jul 10 14:31:42.453814 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:42.453786 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2g7m_4ee5894d-0007-499f-a401-2f11724910ac/konnectivity-agent/0.log" limit=1024 Jul 10 14:31:42.555185 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:42.555107 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2g7m_4ee5894d-0007-499f-a401-2f11724910ac/konnectivity-agent/0.log" limit=1024 Jul 10 14:31:43.243378 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:43.243349 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-n6pfx_296be890-64e9-4fb3-8c68-e4c2a7c8cc2c/console-operator/1.log" Jul 10 14:31:43.244012 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:43.243804 2569 scope.go:117] "RemoveContainer" containerID="6c736c15a8141007d963d66291fd332b9e3d7b278c9f6cf7743f8602ee434fa3" Jul 10 14:31:43.244012 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:43.243982 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-n6pfx_openshift-console-operator(296be890-64e9-4fb3-8c68-e4c2a7c8cc2c)\"" pod="openshift-console-operator/console-operator-7866497494-n6pfx" podUID="296be890-64e9-4fb3-8c68-e4c2a7c8cc2c" Jul 10 14:31:43.343123 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:43.343087 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q9tkw\" (UID: \"1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0\") " pod="openshift-insights/insights-runtime-extractor-q9tkw" Jul 10 14:31:43.343286 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:43.343172 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q9tkw\" (UID: \"1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0\") " pod="openshift-insights/insights-runtime-extractor-q9tkw" Jul 10 14:31:43.343360 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:43.343324 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 14:31:43.343428 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:43.343413 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0-kube-rbac-proxy-cm podName:1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0 nodeName:}" failed. No retries permitted until 2026-07-10 14:31:45.343391933 +0000 UTC m=+51.094763608 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q9tkw" (UID: "1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0") : configmap "kube-rbac-proxy" not found Jul 10 14:31:43.343479 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:43.343326 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 14:31:43.343523 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:43.343492 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0-insights-runtime-extractor-tls podName:1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0 nodeName:}" failed. No retries permitted until 2026-07-10 14:31:45.343477612 +0000 UTC m=+51.094849286 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q9tkw" (UID: "1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0") : secret "insights-runtime-extractor-tls" not found Jul 10 14:31:43.492919 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:43.492894 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-842qb_270982ae-963c-42a2-a10f-06cd88f30f81/dns-node-resolver/0.log" Jul 10 14:31:44.050617 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:44.050553 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-6bdgp"] Jul 10 14:31:44.073435 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:44.073413 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-6bdgp"] Jul 10 14:31:44.073549 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:44.073499 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-6bdgp" Jul 10 14:31:44.076343 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:44.076318 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 10 14:31:44.076444 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:44.076373 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 10 14:31:44.076444 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:44.076388 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 10 14:31:44.076444 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:44.076388 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-qhrl9\"" Jul 10 14:31:44.076444 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:44.076373 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 10 14:31:44.150594 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:44.150567 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzxw9\" (UniqueName: \"kubernetes.io/projected/f9eeece2-af40-4159-af94-a45236505c5b-kube-api-access-xzxw9\") pod \"service-ca-79f68b9f4f-6bdgp\" (UID: \"f9eeece2-af40-4159-af94-a45236505c5b\") " pod="openshift-service-ca/service-ca-79f68b9f4f-6bdgp" Jul 10 14:31:44.150713 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:44.150618 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f9eeece2-af40-4159-af94-a45236505c5b-signing-cabundle\") pod \"service-ca-79f68b9f4f-6bdgp\" (UID: \"f9eeece2-af40-4159-af94-a45236505c5b\") " pod="openshift-service-ca/service-ca-79f68b9f4f-6bdgp" Jul 10 14:31:44.150769 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:44.150738 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f9eeece2-af40-4159-af94-a45236505c5b-signing-key\") pod \"service-ca-79f68b9f4f-6bdgp\" (UID: \"f9eeece2-af40-4159-af94-a45236505c5b\") " pod="openshift-service-ca/service-ca-79f68b9f4f-6bdgp" Jul 10 14:31:44.248898 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:44.248866 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-9z7n5" event={"ID":"064387da-b900-4cda-9839-2b5c166bc246","Type":"ContainerStarted","Data":"7b398327a4426365eaf3e41662d74287edb55999b4e4e3a85bc43077d100842b"} Jul 10 14:31:44.249292 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:44.248913 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-9z7n5" event={"ID":"064387da-b900-4cda-9839-2b5c166bc246","Type":"ContainerStarted","Data":"a1ef61a19b72dfeaed3d3af11c2c8c3a5e4e7d2005e45ad791001f7e9929c82a"} Jul 10 14:31:44.251451 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:44.251435 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzxw9\" (UniqueName: \"kubernetes.io/projected/f9eeece2-af40-4159-af94-a45236505c5b-kube-api-access-xzxw9\") pod \"service-ca-79f68b9f4f-6bdgp\" (UID: \"f9eeece2-af40-4159-af94-a45236505c5b\") " pod="openshift-service-ca/service-ca-79f68b9f4f-6bdgp" Jul 10 14:31:44.251524 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:44.251486 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f9eeece2-af40-4159-af94-a45236505c5b-signing-cabundle\") pod \"service-ca-79f68b9f4f-6bdgp\" (UID: \"f9eeece2-af40-4159-af94-a45236505c5b\") " pod="openshift-service-ca/service-ca-79f68b9f4f-6bdgp" Jul 10 14:31:44.251558 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:44.251552 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f9eeece2-af40-4159-af94-a45236505c5b-signing-key\") pod \"service-ca-79f68b9f4f-6bdgp\" (UID: \"f9eeece2-af40-4159-af94-a45236505c5b\") " pod="openshift-service-ca/service-ca-79f68b9f4f-6bdgp" Jul 10 14:31:44.252206 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:44.252184 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f9eeece2-af40-4159-af94-a45236505c5b-signing-cabundle\") pod \"service-ca-79f68b9f4f-6bdgp\" (UID: \"f9eeece2-af40-4159-af94-a45236505c5b\") " pod="openshift-service-ca/service-ca-79f68b9f4f-6bdgp" Jul 10 14:31:44.254662 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:44.254644 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f9eeece2-af40-4159-af94-a45236505c5b-signing-key\") pod \"service-ca-79f68b9f4f-6bdgp\" (UID: \"f9eeece2-af40-4159-af94-a45236505c5b\") " pod="openshift-service-ca/service-ca-79f68b9f4f-6bdgp" Jul 10 14:31:44.260202 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:44.260182 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzxw9\" (UniqueName: \"kubernetes.io/projected/f9eeece2-af40-4159-af94-a45236505c5b-kube-api-access-xzxw9\") pod \"service-ca-79f68b9f4f-6bdgp\" (UID: \"f9eeece2-af40-4159-af94-a45236505c5b\") " pod="openshift-service-ca/service-ca-79f68b9f4f-6bdgp" Jul 10 14:31:44.267366 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:44.267329 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-9z7n5" podStartSLOduration=1.6502342159999999 podStartE2EDuration="3.267313836s" totalImagesPullingTime="1.61707962s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 14:31:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:31:44.266318054 +0000 UTC m=+50.017689763" watchObservedRunningTime="2026-07-10 14:31:44.267313836 +0000 UTC m=+50.018685552" Jul 10 14:31:44.382153 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:44.382126 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-6bdgp" Jul 10 14:31:44.518186 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:44.518160 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-6bdgp"] Jul 10 14:31:44.521356 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:31:44.521331 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9eeece2_af40_4159_af94_a45236505c5b.slice/crio-6752016f68acd93fa9a0ea7bc7f2b67eaa8b5a1d3ae18398fd3ef8478169155b WatchSource:0}: Error finding container 6752016f68acd93fa9a0ea7bc7f2b67eaa8b5a1d3ae18398fd3ef8478169155b: Status 404 returned error can't find the container with id 6752016f68acd93fa9a0ea7bc7f2b67eaa8b5a1d3ae18398fd3ef8478169155b Jul 10 14:31:44.890936 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:44.890912 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-w9ww7_21295856-adfb-486d-a3dc-b9ff8aa3ff50/node-ca/0.log" Jul 10 14:31:45.253914 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:45.253817 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-6bdgp" event={"ID":"f9eeece2-af40-4159-af94-a45236505c5b","Type":"ContainerStarted","Data":"0ef4522cb3bf39b823bace416a43cdeb140ecaa18d53324006d15d78585c7aa8"} Jul 10 14:31:45.253914 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:45.253878 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-6bdgp" event={"ID":"f9eeece2-af40-4159-af94-a45236505c5b","Type":"ContainerStarted","Data":"6752016f68acd93fa9a0ea7bc7f2b67eaa8b5a1d3ae18398fd3ef8478169155b"} Jul 10 14:31:45.282036 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:45.281989 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-6bdgp" podStartSLOduration=1.281972767 podStartE2EDuration="1.281972767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 14:31:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:31:45.281588458 +0000 UTC m=+51.032960154" watchObservedRunningTime="2026-07-10 14:31:45.281972767 +0000 UTC m=+51.033344462" Jul 10 14:31:45.294021 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:45.293977 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wbbhv_80537ce8-cbd4-4a4c-8604-7377993cd5b1/serve-healthcheck-canary/0.log" Jul 10 14:31:45.359883 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:45.359851 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q9tkw\" (UID: \"1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0\") " pod="openshift-insights/insights-runtime-extractor-q9tkw" Jul 10 14:31:45.360061 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:45.359920 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q9tkw\" (UID: \"1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0\") " pod="openshift-insights/insights-runtime-extractor-q9tkw" Jul 10 14:31:45.360061 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:45.360017 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 14:31:45.360176 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:45.360073 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 14:31:45.360176 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:45.360087 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0-insights-runtime-extractor-tls podName:1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0 nodeName:}" failed. No retries permitted until 2026-07-10 14:31:49.360069133 +0000 UTC m=+55.111440808 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q9tkw" (UID: "1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0") : secret "insights-runtime-extractor-tls" not found Jul 10 14:31:45.360176 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:45.360138 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0-kube-rbac-proxy-cm podName:1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0 nodeName:}" failed. No retries permitted until 2026-07-10 14:31:49.360125676 +0000 UTC m=+55.111497349 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q9tkw" (UID: "1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0") : configmap "kube-rbac-proxy" not found Jul 10 14:31:45.492083 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:45.492054 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-9z7n5_064387da-b900-4cda-9839-2b5c166bc246/migrator/0.log" Jul 10 14:31:45.692362 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:45.692331 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-9z7n5_064387da-b900-4cda-9839-2b5c166bc246/graceful-termination/0.log" Jul 10 14:31:45.899338 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:45.899304 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-d7kg8_70c38fec-83c3-4e81-90c3-ab3f7204fe3c/kube-storage-version-migrator-operator/0.log" Jul 10 14:31:46.470782 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:46.470726 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c867eb4a-dae5-4edd-a157-7686c044be1e-registry-tls\") pod \"image-registry-8d46cc844-xf6cz\" (UID: \"c867eb4a-dae5-4edd-a157-7686c044be1e\") " pod="openshift-image-registry/image-registry-8d46cc844-xf6cz" Jul 10 14:31:46.471238 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:46.470833 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13a97abe-27b2-4b7b-ae18-8ae1ff61f043-metrics-certs\") pod \"router-default-795d96bbd4-2nql9\" (UID: \"13a97abe-27b2-4b7b-ae18-8ae1ff61f043\") " pod="openshift-ingress/router-default-795d96bbd4-2nql9" Jul 10 14:31:46.471238 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:46.470885 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13a97abe-27b2-4b7b-ae18-8ae1ff61f043-service-ca-bundle\") pod \"router-default-795d96bbd4-2nql9\" (UID: \"13a97abe-27b2-4b7b-ae18-8ae1ff61f043\") " pod="openshift-ingress/router-default-795d96bbd4-2nql9" Jul 10 14:31:46.471238 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:46.470886 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 14:31:46.471238 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:46.470908 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8d46cc844-xf6cz: secret "image-registry-tls" not found Jul 10 14:31:46.471238 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:46.470967 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c867eb4a-dae5-4edd-a157-7686c044be1e-registry-tls podName:c867eb4a-dae5-4edd-a157-7686c044be1e nodeName:}" failed. No retries permitted until 2026-07-10 14:32:02.470948241 +0000 UTC m=+68.222319925 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c867eb4a-dae5-4edd-a157-7686c044be1e-registry-tls") pod "image-registry-8d46cc844-xf6cz" (UID: "c867eb4a-dae5-4edd-a157-7686c044be1e") : secret "image-registry-tls" not found Jul 10 14:31:46.471238 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:46.470999 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/13a97abe-27b2-4b7b-ae18-8ae1ff61f043-service-ca-bundle podName:13a97abe-27b2-4b7b-ae18-8ae1ff61f043 nodeName:}" failed. No retries permitted until 2026-07-10 14:32:02.47098399 +0000 UTC m=+68.222355667 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/13a97abe-27b2-4b7b-ae18-8ae1ff61f043-service-ca-bundle") pod "router-default-795d96bbd4-2nql9" (UID: "13a97abe-27b2-4b7b-ae18-8ae1ff61f043") : configmap references non-existent config key: service-ca.crt Jul 10 14:31:46.471238 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:46.471064 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 10 14:31:46.471238 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:46.471139 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13a97abe-27b2-4b7b-ae18-8ae1ff61f043-metrics-certs podName:13a97abe-27b2-4b7b-ae18-8ae1ff61f043 nodeName:}" failed. No retries permitted until 2026-07-10 14:32:02.471124245 +0000 UTC m=+68.222495928 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/13a97abe-27b2-4b7b-ae18-8ae1ff61f043-metrics-certs") pod "router-default-795d96bbd4-2nql9" (UID: "13a97abe-27b2-4b7b-ae18-8ae1ff61f043") : secret "router-metrics-certs-default" not found Jul 10 14:31:46.572153 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:46.572120 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cb41c5b6-dab8-4be4-b2be-56040d1659e7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-c8ztx\" (UID: \"cb41c5b6-dab8-4be4-b2be-56040d1659e7\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-c8ztx" Jul 10 14:31:46.572271 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:46.572193 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5d6f7d4c-e2a0-40d0-9686-f016013d1504-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-9gxrb\" (UID: \"5d6f7d4c-e2a0-40d0-9686-f016013d1504\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9gxrb" Jul 10 14:31:46.572271 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:46.572258 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/68e91fda-6611-4951-806d-2514fe8bd1cb-metrics-tls\") pod \"dns-default-f698x\" (UID: \"68e91fda-6611-4951-806d-2514fe8bd1cb\") " pod="openshift-dns/dns-default-f698x" Jul 10 14:31:46.572271 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:46.572263 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 10 14:31:46.572445 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:46.572319 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb41c5b6-dab8-4be4-b2be-56040d1659e7-cluster-monitoring-operator-tls podName:cb41c5b6-dab8-4be4-b2be-56040d1659e7 nodeName:}" failed. No retries permitted until 2026-07-10 14:32:02.572304812 +0000 UTC m=+68.323676493 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cb41c5b6-dab8-4be4-b2be-56040d1659e7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-c8ztx" (UID: "cb41c5b6-dab8-4be4-b2be-56040d1659e7") : secret "cluster-monitoring-operator-tls" not found Jul 10 14:31:46.572445 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:46.572333 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 10 14:31:46.572445 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:46.572383 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d6f7d4c-e2a0-40d0-9686-f016013d1504-samples-operator-tls podName:5d6f7d4c-e2a0-40d0-9686-f016013d1504 nodeName:}" failed. No retries permitted until 2026-07-10 14:32:02.572368539 +0000 UTC m=+68.323740219 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5d6f7d4c-e2a0-40d0-9686-f016013d1504-samples-operator-tls") pod "cluster-samples-operator-65db858894-9gxrb" (UID: "5d6f7d4c-e2a0-40d0-9686-f016013d1504") : secret "samples-operator-tls" not found Jul 10 14:31:46.572445 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:46.572418 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 14:31:46.572591 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:46.572466 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68e91fda-6611-4951-806d-2514fe8bd1cb-metrics-tls podName:68e91fda-6611-4951-806d-2514fe8bd1cb nodeName:}" failed. No retries permitted until 2026-07-10 14:32:02.57245129 +0000 UTC m=+68.323822965 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/68e91fda-6611-4951-806d-2514fe8bd1cb-metrics-tls") pod "dns-default-f698x" (UID: "68e91fda-6611-4951-806d-2514fe8bd1cb") : secret "dns-default-metrics-tls" not found Jul 10 14:31:49.397849 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:49.397813 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q9tkw\" (UID: \"1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0\") " pod="openshift-insights/insights-runtime-extractor-q9tkw" Jul 10 14:31:49.398356 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:49.397873 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q9tkw\" (UID: \"1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0\") " pod="openshift-insights/insights-runtime-extractor-q9tkw" Jul 10 14:31:49.398356 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:49.397964 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 14:31:49.398356 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:49.397978 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 14:31:49.398356 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:49.398015 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0-kube-rbac-proxy-cm podName:1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0 nodeName:}" failed. No retries permitted until 2026-07-10 14:31:57.398003056 +0000 UTC m=+63.149374730 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q9tkw" (UID: "1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0") : configmap "kube-rbac-proxy" not found Jul 10 14:31:49.398356 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:49.398027 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0-insights-runtime-extractor-tls podName:1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0 nodeName:}" failed. No retries permitted until 2026-07-10 14:31:57.398021725 +0000 UTC m=+63.149393398 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q9tkw" (UID: "1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0") : secret "insights-runtime-extractor-tls" not found Jul 10 14:31:50.952919 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:50.952883 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-n6pfx" Jul 10 14:31:50.953295 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:50.952994 2569 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-n6pfx" Jul 10 14:31:50.953336 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:50.953304 2569 scope.go:117] "RemoveContainer" containerID="6c736c15a8141007d963d66291fd332b9e3d7b278c9f6cf7743f8602ee434fa3" Jul 10 14:31:50.953485 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:50.953465 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-n6pfx_openshift-console-operator(296be890-64e9-4fb3-8c68-e4c2a7c8cc2c)\"" pod="openshift-console-operator/console-operator-7866497494-n6pfx" podUID="296be890-64e9-4fb3-8c68-e4c2a7c8cc2c" Jul 10 14:31:51.275830 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:51.275737 2569 scope.go:117] "RemoveContainer" containerID="6c736c15a8141007d963d66291fd332b9e3d7b278c9f6cf7743f8602ee434fa3" Jul 10 14:31:51.275963 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:31:51.275945 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-n6pfx_openshift-console-operator(296be890-64e9-4fb3-8c68-e4c2a7c8cc2c)\"" pod="openshift-console-operator/console-operator-7866497494-n6pfx" podUID="296be890-64e9-4fb3-8c68-e4c2a7c8cc2c" Jul 10 14:31:57.133213 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:57.133184 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kxvq2" Jul 10 14:31:57.472566 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:57.472478 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q9tkw\" (UID: \"1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0\") " pod="openshift-insights/insights-runtime-extractor-q9tkw" Jul 10 14:31:57.472566 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:57.472544 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q9tkw\" (UID: \"1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0\") " pod="openshift-insights/insights-runtime-extractor-q9tkw" Jul 10 14:31:57.473124 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:57.473099 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q9tkw\" (UID: \"1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0\") " pod="openshift-insights/insights-runtime-extractor-q9tkw" Jul 10 14:31:57.474787 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:57.474745 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q9tkw\" (UID: \"1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0\") " pod="openshift-insights/insights-runtime-extractor-q9tkw" Jul 10 14:31:57.606909 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:57.606881 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2nt82\"" Jul 10 14:31:57.614805 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:57.614783 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-q9tkw" Jul 10 14:31:57.738816 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:57.738721 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-q9tkw"] Jul 10 14:31:57.741705 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:31:57.741677 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dd7a68f_7b1e_44c0_afbe_ab1bb7df83f0.slice/crio-97c39c4ee2eb0e2683521a53d8c3ea0257cad2c473f256bad23878725a1a01d2 WatchSource:0}: Error finding container 97c39c4ee2eb0e2683521a53d8c3ea0257cad2c473f256bad23878725a1a01d2: Status 404 returned error can't find the container with id 97c39c4ee2eb0e2683521a53d8c3ea0257cad2c473f256bad23878725a1a01d2 Jul 10 14:31:58.299736 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:31:58.299694 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q9tkw" event={"ID":"1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0","Type":"ContainerStarted","Data":"97c39c4ee2eb0e2683521a53d8c3ea0257cad2c473f256bad23878725a1a01d2"} Jul 10 14:32:00.501667 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:00.501624 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff3c9cb2-1924-4ea9-9a93-282213b083db-metrics-certs\") pod \"network-metrics-daemon-fdhw7\" (UID: \"ff3c9cb2-1924-4ea9-9a93-282213b083db\") " pod="openshift-multus/network-metrics-daemon-fdhw7" Jul 10 14:32:00.504504 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:00.504483 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 14:32:00.514583 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:00.514549 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff3c9cb2-1924-4ea9-9a93-282213b083db-metrics-certs\") pod \"network-metrics-daemon-fdhw7\" (UID: \"ff3c9cb2-1924-4ea9-9a93-282213b083db\") " pod="openshift-multus/network-metrics-daemon-fdhw7" Jul 10 14:32:00.602294 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:00.602246 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f-original-pull-secret\") pod \"global-pull-secret-syncer-rxmgq\" (UID: \"4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f\") " pod="kube-system/global-pull-secret-syncer-rxmgq" Jul 10 14:32:00.605538 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:00.605512 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 14:32:00.615374 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:00.615347 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f-original-pull-secret\") pod \"global-pull-secret-syncer-rxmgq\" (UID: \"4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f\") " pod="kube-system/global-pull-secret-syncer-rxmgq" Jul 10 14:32:00.643511 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:00.643477 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rxmgq" Jul 10 14:32:00.657571 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:00.657544 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pxtsm\"" Jul 10 14:32:00.664766 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:00.664713 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fdhw7" Jul 10 14:32:00.703604 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:00.703572 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpsm2\" (UniqueName: \"kubernetes.io/projected/ced07efc-4719-4439-b2be-b84dadef62d3-kube-api-access-cpsm2\") pod \"network-check-target-t9jl6\" (UID: \"ced07efc-4719-4439-b2be-b84dadef62d3\") " pod="openshift-network-diagnostics/network-check-target-t9jl6" Jul 10 14:32:00.707728 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:00.707141 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpsm2\" (UniqueName: \"kubernetes.io/projected/ced07efc-4719-4439-b2be-b84dadef62d3-kube-api-access-cpsm2\") pod \"network-check-target-t9jl6\" (UID: \"ced07efc-4719-4439-b2be-b84dadef62d3\") " pod="openshift-network-diagnostics/network-check-target-t9jl6" Jul 10 14:32:00.961795 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:00.961746 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-42vlj\"" Jul 10 14:32:00.969709 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:00.969683 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t9jl6" Jul 10 14:32:01.595018 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:01.594930 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fdhw7"] Jul 10 14:32:01.599278 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:32:01.599252 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff3c9cb2_1924_4ea9_9a93_282213b083db.slice/crio-861f6fbad88372051d00c7e6216a1ae0f97d63376d564916da95e8da09e1d364 WatchSource:0}: Error finding container 861f6fbad88372051d00c7e6216a1ae0f97d63376d564916da95e8da09e1d364: Status 404 returned error can't find the container with id 861f6fbad88372051d00c7e6216a1ae0f97d63376d564916da95e8da09e1d364 Jul 10 14:32:01.818216 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:01.818147 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-t9jl6"] Jul 10 14:32:01.820869 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:01.820845 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rxmgq"] Jul 10 14:32:01.821888 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:32:01.821863 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podced07efc_4719_4439_b2be_b84dadef62d3.slice/crio-acb4c767d1ec1707683ca0b628110dc7f3e55cc7b5498182768f33a27fc82720 WatchSource:0}: Error finding container acb4c767d1ec1707683ca0b628110dc7f3e55cc7b5498182768f33a27fc82720: Status 404 returned error can't find the container with id acb4c767d1ec1707683ca0b628110dc7f3e55cc7b5498182768f33a27fc82720 Jul 10 14:32:01.824591 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:32:01.824562 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bba05ad_b3f8_4bc4_acef_c5e573ea8e4f.slice/crio-6846bf3c2aa2fa252b8f6dc97c7ed4d02fbf3ffa8f112b85c4aae883e158869d WatchSource:0}: Error finding container 6846bf3c2aa2fa252b8f6dc97c7ed4d02fbf3ffa8f112b85c4aae883e158869d: Status 404 returned error can't find the container with id 6846bf3c2aa2fa252b8f6dc97c7ed4d02fbf3ffa8f112b85c4aae883e158869d Jul 10 14:32:02.313764 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:02.313708 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-t9jl6" event={"ID":"ced07efc-4719-4439-b2be-b84dadef62d3","Type":"ContainerStarted","Data":"ee6f4f5539d4de1ed3eca7f10ca1d6187cfd80c2fa2e4975e9ac3db826823a4b"} Jul 10 14:32:02.313948 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:02.313791 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-t9jl6" Jul 10 14:32:02.313948 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:02.313814 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-t9jl6" event={"ID":"ced07efc-4719-4439-b2be-b84dadef62d3","Type":"ContainerStarted","Data":"acb4c767d1ec1707683ca0b628110dc7f3e55cc7b5498182768f33a27fc82720"} Jul 10 14:32:02.315548 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:02.315485 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q9tkw" event={"ID":"1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0","Type":"ContainerStarted","Data":"2b53291afd2408fd949d73919d0e0e1ee142bb4c1d0ef6e36ad54cf0dbc16355"} Jul 10 14:32:02.316740 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:02.316713 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rxmgq" event={"ID":"4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f","Type":"ContainerStarted","Data":"6846bf3c2aa2fa252b8f6dc97c7ed4d02fbf3ffa8f112b85c4aae883e158869d"} Jul 10 14:32:02.317971 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:02.317948 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fdhw7" event={"ID":"ff3c9cb2-1924-4ea9-9a93-282213b083db","Type":"ContainerStarted","Data":"861f6fbad88372051d00c7e6216a1ae0f97d63376d564916da95e8da09e1d364"} Jul 10 14:32:02.335177 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:02.335122 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-t9jl6" podStartSLOduration=67.335103631 podStartE2EDuration="1m7.335103631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 14:30:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:32:02.334165486 +0000 UTC m=+68.085537184" watchObservedRunningTime="2026-07-10 14:32:02.335103631 +0000 UTC m=+68.086475365" Jul 10 14:32:02.517588 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:02.517544 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13a97abe-27b2-4b7b-ae18-8ae1ff61f043-metrics-certs\") pod \"router-default-795d96bbd4-2nql9\" (UID: \"13a97abe-27b2-4b7b-ae18-8ae1ff61f043\") " pod="openshift-ingress/router-default-795d96bbd4-2nql9" Jul 10 14:32:02.517849 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:02.517639 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13a97abe-27b2-4b7b-ae18-8ae1ff61f043-service-ca-bundle\") pod \"router-default-795d96bbd4-2nql9\" (UID: \"13a97abe-27b2-4b7b-ae18-8ae1ff61f043\") " pod="openshift-ingress/router-default-795d96bbd4-2nql9" Jul 10 14:32:02.517849 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:02.517746 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c867eb4a-dae5-4edd-a157-7686c044be1e-registry-tls\") pod \"image-registry-8d46cc844-xf6cz\" (UID: \"c867eb4a-dae5-4edd-a157-7686c044be1e\") " pod="openshift-image-registry/image-registry-8d46cc844-xf6cz" Jul 10 14:32:02.518506 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:02.518453 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13a97abe-27b2-4b7b-ae18-8ae1ff61f043-service-ca-bundle\") pod \"router-default-795d96bbd4-2nql9\" (UID: \"13a97abe-27b2-4b7b-ae18-8ae1ff61f043\") " pod="openshift-ingress/router-default-795d96bbd4-2nql9" Jul 10 14:32:02.520251 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:02.520228 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13a97abe-27b2-4b7b-ae18-8ae1ff61f043-metrics-certs\") pod \"router-default-795d96bbd4-2nql9\" (UID: \"13a97abe-27b2-4b7b-ae18-8ae1ff61f043\") " pod="openshift-ingress/router-default-795d96bbd4-2nql9" Jul 10 14:32:02.520369 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:02.520348 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c867eb4a-dae5-4edd-a157-7686c044be1e-registry-tls\") pod \"image-registry-8d46cc844-xf6cz\" (UID: \"c867eb4a-dae5-4edd-a157-7686c044be1e\") " pod="openshift-image-registry/image-registry-8d46cc844-xf6cz" Jul 10 14:32:02.618885 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:02.618848 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5d6f7d4c-e2a0-40d0-9686-f016013d1504-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-9gxrb\" (UID: \"5d6f7d4c-e2a0-40d0-9686-f016013d1504\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9gxrb" Jul 10 14:32:02.619226 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:02.618898 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/68e91fda-6611-4951-806d-2514fe8bd1cb-metrics-tls\") pod \"dns-default-f698x\" (UID: \"68e91fda-6611-4951-806d-2514fe8bd1cb\") " pod="openshift-dns/dns-default-f698x" Jul 10 14:32:02.619226 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:02.618947 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cb41c5b6-dab8-4be4-b2be-56040d1659e7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-c8ztx\" (UID: \"cb41c5b6-dab8-4be4-b2be-56040d1659e7\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-c8ztx" Jul 10 14:32:02.621296 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:02.621262 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cb41c5b6-dab8-4be4-b2be-56040d1659e7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-c8ztx\" (UID: \"cb41c5b6-dab8-4be4-b2be-56040d1659e7\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-c8ztx" Jul 10 14:32:02.621296 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:02.621269 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/68e91fda-6611-4951-806d-2514fe8bd1cb-metrics-tls\") pod \"dns-default-f698x\" (UID: \"68e91fda-6611-4951-806d-2514fe8bd1cb\") " pod="openshift-dns/dns-default-f698x" Jul 10 14:32:02.621444 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:02.621298 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5d6f7d4c-e2a0-40d0-9686-f016013d1504-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-9gxrb\" (UID: \"5d6f7d4c-e2a0-40d0-9686-f016013d1504\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9gxrb" Jul 10 14:32:02.709146 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:02.709104 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-ssv9d\"" Jul 10 14:32:02.717192 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:02.717164 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8d46cc844-xf6cz" Jul 10 14:32:02.728887 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:02.728858 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-tpvq9\"" Jul 10 14:32:02.737028 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:02.737003 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-795d96bbd4-2nql9" Jul 10 14:32:02.800429 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:02.800399 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-sxgwf\"" Jul 10 14:32:02.808539 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:02.808512 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-c8ztx" Jul 10 14:32:02.820325 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:02.820299 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-c5njl\"" Jul 10 14:32:02.828140 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:02.828115 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9gxrb" Jul 10 14:32:02.838616 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:02.838592 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2n5mh\"" Jul 10 14:32:02.847168 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:02.847142 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-f698x" Jul 10 14:32:03.152222 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:03.152191 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-c8ztx"] Jul 10 14:32:03.324191 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:03.324110 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q9tkw" event={"ID":"1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0","Type":"ContainerStarted","Data":"eaa7c2e0f858563228f6ba3a1961e46ec9c2cb3bb3ef25078236920f74667646"} Jul 10 14:32:03.341318 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:32:03.341288 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb41c5b6_dab8_4be4_b2be_56040d1659e7.slice/crio-b7127ba4067476041cde427e16bb13d1fc50431d2bc61dbd6d9de7b9b0e6b4ec WatchSource:0}: Error finding container b7127ba4067476041cde427e16bb13d1fc50431d2bc61dbd6d9de7b9b0e6b4ec: Status 404 returned error can't find the container with id b7127ba4067476041cde427e16bb13d1fc50431d2bc61dbd6d9de7b9b0e6b4ec Jul 10 14:32:03.378295 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:03.378237 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9gxrb"] Jul 10 14:32:03.380939 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:03.380911 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-f698x"] Jul 10 14:32:03.382419 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:03.382390 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-795d96bbd4-2nql9"] Jul 10 14:32:03.385149 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:03.384504 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8d46cc844-xf6cz"] Jul 10 14:32:03.385149 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:32:03.384775 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13a97abe_27b2_4b7b_ae18_8ae1ff61f043.slice/crio-56677c8b37b8bad116b304a045ec5a4296f87b35a9eeb9e4fea04533b1feee4e WatchSource:0}: Error finding container 56677c8b37b8bad116b304a045ec5a4296f87b35a9eeb9e4fea04533b1feee4e: Status 404 returned error can't find the container with id 56677c8b37b8bad116b304a045ec5a4296f87b35a9eeb9e4fea04533b1feee4e Jul 10 14:32:03.385940 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:32:03.385916 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68e91fda_6611_4951_806d_2514fe8bd1cb.slice/crio-6f3e721377a2a919677048a75aeef15b67eadb3e7ce8c8d6f8e962388876465e WatchSource:0}: Error finding container 6f3e721377a2a919677048a75aeef15b67eadb3e7ce8c8d6f8e962388876465e: Status 404 returned error can't find the container with id 6f3e721377a2a919677048a75aeef15b67eadb3e7ce8c8d6f8e962388876465e Jul 10 14:32:03.399637 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:32:03.399613 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc867eb4a_dae5_4edd_a157_7686c044be1e.slice/crio-0ba4faa564b856adbed2079e799efc584f9bb52147faf7639dd187baedb27781 WatchSource:0}: Error finding container 0ba4faa564b856adbed2079e799efc584f9bb52147faf7639dd187baedb27781: Status 404 returned error can't find the container with id 0ba4faa564b856adbed2079e799efc584f9bb52147faf7639dd187baedb27781 Jul 10 14:32:03.926615 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:03.926576 2569 scope.go:117] "RemoveContainer" containerID="6c736c15a8141007d963d66291fd332b9e3d7b278c9f6cf7743f8602ee434fa3" Jul 10 14:32:04.335064 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:04.334901 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-f698x" event={"ID":"68e91fda-6611-4951-806d-2514fe8bd1cb","Type":"ContainerStarted","Data":"6f3e721377a2a919677048a75aeef15b67eadb3e7ce8c8d6f8e962388876465e"} Jul 10 14:32:04.339813 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:04.339652 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-n6pfx_296be890-64e9-4fb3-8c68-e4c2a7c8cc2c/console-operator/1.log" Jul 10 14:32:04.339813 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:04.339790 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-n6pfx" event={"ID":"296be890-64e9-4fb3-8c68-e4c2a7c8cc2c","Type":"ContainerStarted","Data":"b053b27280db16878083ab6972b8f77a65f9ea50f05a6db9caacc8bc4cd092f7"} Jul 10 14:32:04.340161 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:04.340105 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-n6pfx" Jul 10 14:32:04.342106 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:04.342027 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9gxrb" event={"ID":"5d6f7d4c-e2a0-40d0-9686-f016013d1504","Type":"ContainerStarted","Data":"ce343591efca0015605c536f882dab441f63e7e64a1c2d5ce08092f393c4b03e"} Jul 10 14:32:04.344113 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:04.344049 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-c8ztx" event={"ID":"cb41c5b6-dab8-4be4-b2be-56040d1659e7","Type":"ContainerStarted","Data":"b7127ba4067476041cde427e16bb13d1fc50431d2bc61dbd6d9de7b9b0e6b4ec"} Jul 10 14:32:04.346698 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:04.346180 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-795d96bbd4-2nql9" event={"ID":"13a97abe-27b2-4b7b-ae18-8ae1ff61f043","Type":"ContainerStarted","Data":"a09aab8862621b96dda197ed5d5607271192331d368aeb1c1adf0a2caadbe424"} Jul 10 14:32:04.346698 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:04.346218 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-795d96bbd4-2nql9" event={"ID":"13a97abe-27b2-4b7b-ae18-8ae1ff61f043","Type":"ContainerStarted","Data":"56677c8b37b8bad116b304a045ec5a4296f87b35a9eeb9e4fea04533b1feee4e"} Jul 10 14:32:04.356243 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:04.354944 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8d46cc844-xf6cz" event={"ID":"c867eb4a-dae5-4edd-a157-7686c044be1e","Type":"ContainerStarted","Data":"3b922d1e3a5ae0e62636aa13d9704c361cc55eba6720901954ddcf3fbb04df3e"} Jul 10 14:32:04.356243 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:04.355023 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8d46cc844-xf6cz" event={"ID":"c867eb4a-dae5-4edd-a157-7686c044be1e","Type":"ContainerStarted","Data":"0ba4faa564b856adbed2079e799efc584f9bb52147faf7639dd187baedb27781"} Jul 10 14:32:04.356243 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:04.356068 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-8d46cc844-xf6cz" Jul 10 14:32:04.362462 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:04.359310 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fdhw7" event={"ID":"ff3c9cb2-1924-4ea9-9a93-282213b083db","Type":"ContainerStarted","Data":"ea6bcb7a55b3af2218a25198fb8144f63f7f8ce131750e0efefe41c6a4140101"} Jul 10 14:32:04.362462 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:04.359367 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fdhw7" event={"ID":"ff3c9cb2-1924-4ea9-9a93-282213b083db","Type":"ContainerStarted","Data":"ff3f0ad133d4b664731ab1ca8315c50ad518c07e440d8084496dee452631583a"} Jul 10 14:32:04.368791 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:04.368711 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7866497494-n6pfx" podStartSLOduration=36.630431223 podStartE2EDuration="44.368695341s" totalImagesPullingTime="7.738264118s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 14:31:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:32:04.367911895 +0000 UTC m=+70.119283603" watchObservedRunningTime="2026-07-10 14:32:04.368695341 +0000 UTC m=+70.120067038" Jul 10 14:32:04.390858 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:04.390085 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-795d96bbd4-2nql9" podStartSLOduration=44.390068193 podStartE2EDuration="44.390068193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 14:31:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:32:04.388945944 +0000 UTC m=+70.140317636" watchObservedRunningTime="2026-07-10 14:32:04.390068193 +0000 UTC m=+70.141439892" Jul 10 14:32:04.408472 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:04.407261 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-fdhw7" podStartSLOduration=67.633701414 podStartE2EDuration="1m9.407248171s" totalImagesPullingTime="1.773546757s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 14:30:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:32:04.406921623 +0000 UTC m=+70.158293321" watchObservedRunningTime="2026-07-10 14:32:04.407248171 +0000 UTC m=+70.158619867" Jul 10 14:32:04.428333 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:04.428289 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-8d46cc844-xf6cz" podStartSLOduration=69.428272003 podStartE2EDuration="1m9.428272003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 14:30:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:32:04.427029837 +0000 UTC m=+70.178401559" watchObservedRunningTime="2026-07-10 14:32:04.428272003 +0000 UTC m=+70.179643700" Jul 10 14:32:04.738913 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:04.737851 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-795d96bbd4-2nql9" Jul 10 14:32:04.740521 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:04.740494 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-795d96bbd4-2nql9" Jul 10 14:32:05.340483 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:05.340442 2569 patch_prober.go:28] interesting pod/console-operator-7866497494-n6pfx container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.8:8443/readyz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 10 14:32:05.341076 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:05.340516 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7866497494-n6pfx" podUID="296be890-64e9-4fb3-8c68-e4c2a7c8cc2c" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.8:8443/readyz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jul 10 14:32:05.367428 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:05.367355 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-795d96bbd4-2nql9" Jul 10 14:32:05.369147 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:05.368966 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-795d96bbd4-2nql9" Jul 10 14:32:05.894344 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:05.894313 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7866497494-n6pfx" Jul 10 14:32:06.023812 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.023163 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-95jwg"] Jul 10 14:32:06.026982 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.026681 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-95jwg" Jul 10 14:32:06.030281 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.030249 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 10 14:32:06.030413 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.030310 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-dkws9\"" Jul 10 14:32:06.031764 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.031724 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 10 14:32:06.032261 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.032161 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-857cfb796d-s468b"] Jul 10 14:32:06.040617 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.039563 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55c967b4fd-j8tkx"] Jul 10 14:32:06.040825 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.040734 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-857cfb796d-s468b" Jul 10 14:32:06.045948 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.045928 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-59c984c7cf-xhvzp"] Jul 10 14:32:06.046241 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.046220 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55c967b4fd-j8tkx" Jul 10 14:32:06.046731 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.046712 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 10 14:32:06.047596 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.046813 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 10 14:32:06.047596 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.046909 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 10 14:32:06.047596 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.047039 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-pf9pt\"" Jul 10 14:32:06.049264 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.048978 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59c984c7cf-xhvzp" Jul 10 14:32:06.051939 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.050959 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 10 14:32:06.051939 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.051181 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 10 14:32:06.051939 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.051411 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 10 14:32:06.051939 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.051622 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 10 14:32:06.051939 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.051872 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 10 14:32:06.052669 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.052480 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 10 14:32:06.054686 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.054668 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-95jwg"] Jul 10 14:32:06.058813 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.058794 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-857cfb796d-s468b"] Jul 10 14:32:06.060819 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.060467 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55c967b4fd-j8tkx"] Jul 10 14:32:06.083099 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.083077 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-59c984c7cf-xhvzp"] Jul 10 14:32:06.135246 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.135218 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-68c9cbb497-7jcj6"] Jul 10 14:32:06.137615 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.137568 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-7jcj6" Jul 10 14:32:06.140340 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.140323 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 10 14:32:06.140651 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.140409 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-pc2vg\"" Jul 10 14:32:06.140651 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.140643 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 10 14:32:06.153304 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.153275 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrkmw\" (UniqueName: \"kubernetes.io/projected/f2ded258-5977-4179-b653-d9f839f08b5d-kube-api-access-nrkmw\") pod \"cluster-proxy-proxy-agent-55c967b4fd-j8tkx\" (UID: \"f2ded258-5977-4179-b653-d9f839f08b5d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55c967b4fd-j8tkx" Jul 10 14:32:06.153427 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.153318 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t7jx\" (UniqueName: \"kubernetes.io/projected/3da3ef7c-814a-421e-8d47-29b65379e664-kube-api-access-2t7jx\") pod \"managed-serviceaccount-addon-agent-857cfb796d-s468b\" (UID: \"3da3ef7c-814a-421e-8d47-29b65379e664\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-857cfb796d-s468b" Jul 10 14:32:06.153427 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.153358 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2b94422c-c0a6-4731-a0c1-bb2e44fdf8ea-klusterlet-config\") pod \"klusterlet-addon-workmgr-59c984c7cf-xhvzp\" (UID: \"2b94422c-c0a6-4731-a0c1-bb2e44fdf8ea\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59c984c7cf-xhvzp" Jul 10 14:32:06.153427 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.153381 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4chk2\" (UniqueName: \"kubernetes.io/projected/2b94422c-c0a6-4731-a0c1-bb2e44fdf8ea-kube-api-access-4chk2\") pod \"klusterlet-addon-workmgr-59c984c7cf-xhvzp\" (UID: \"2b94422c-c0a6-4731-a0c1-bb2e44fdf8ea\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59c984c7cf-xhvzp" Jul 10 14:32:06.153427 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.153402 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bc08d681-36cd-4339-ad25-349ee4e4f916-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-95jwg\" (UID: \"bc08d681-36cd-4339-ad25-349ee4e4f916\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-95jwg" Jul 10 14:32:06.153427 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.153421 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3da3ef7c-814a-421e-8d47-29b65379e664-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-857cfb796d-s468b\" (UID: \"3da3ef7c-814a-421e-8d47-29b65379e664\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-857cfb796d-s468b" Jul 10 14:32:06.153690 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.153449 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f2ded258-5977-4179-b653-d9f839f08b5d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-55c967b4fd-j8tkx\" (UID: \"f2ded258-5977-4179-b653-d9f839f08b5d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55c967b4fd-j8tkx" Jul 10 14:32:06.153690 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.153485 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f2ded258-5977-4179-b653-d9f839f08b5d-ca\") pod \"cluster-proxy-proxy-agent-55c967b4fd-j8tkx\" (UID: \"f2ded258-5977-4179-b653-d9f839f08b5d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55c967b4fd-j8tkx" Jul 10 14:32:06.153690 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.153500 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b94422c-c0a6-4731-a0c1-bb2e44fdf8ea-tmp\") pod \"klusterlet-addon-workmgr-59c984c7cf-xhvzp\" (UID: \"2b94422c-c0a6-4731-a0c1-bb2e44fdf8ea\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59c984c7cf-xhvzp" Jul 10 14:32:06.153690 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.153519 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f2ded258-5977-4179-b653-d9f839f08b5d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-55c967b4fd-j8tkx\" (UID: \"f2ded258-5977-4179-b653-d9f839f08b5d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55c967b4fd-j8tkx" Jul 10 14:32:06.153690 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.153536 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2ded258-5977-4179-b653-d9f839f08b5d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-55c967b4fd-j8tkx\" (UID: \"f2ded258-5977-4179-b653-d9f839f08b5d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55c967b4fd-j8tkx" Jul 10 14:32:06.153690 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.153558 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bc08d681-36cd-4339-ad25-349ee4e4f916-nginx-conf\") pod \"networking-console-plugin-6cf856758c-95jwg\" (UID: \"bc08d681-36cd-4339-ad25-349ee4e4f916\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-95jwg" Jul 10 14:32:06.153690 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.153576 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f2ded258-5977-4179-b653-d9f839f08b5d-hub\") pod \"cluster-proxy-proxy-agent-55c967b4fd-j8tkx\" (UID: \"f2ded258-5977-4179-b653-d9f839f08b5d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55c967b4fd-j8tkx" Jul 10 14:32:06.158870 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.158798 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-7jcj6"] Jul 10 14:32:06.254192 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.254154 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2b94422c-c0a6-4731-a0c1-bb2e44fdf8ea-klusterlet-config\") pod \"klusterlet-addon-workmgr-59c984c7cf-xhvzp\" (UID: \"2b94422c-c0a6-4731-a0c1-bb2e44fdf8ea\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59c984c7cf-xhvzp" Jul 10 14:32:06.254426 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.254213 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4chk2\" (UniqueName: \"kubernetes.io/projected/2b94422c-c0a6-4731-a0c1-bb2e44fdf8ea-kube-api-access-4chk2\") pod \"klusterlet-addon-workmgr-59c984c7cf-xhvzp\" (UID: \"2b94422c-c0a6-4731-a0c1-bb2e44fdf8ea\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59c984c7cf-xhvzp" Jul 10 14:32:06.254426 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.254252 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bc08d681-36cd-4339-ad25-349ee4e4f916-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-95jwg\" (UID: \"bc08d681-36cd-4339-ad25-349ee4e4f916\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-95jwg" Jul 10 14:32:06.254426 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.254272 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3da3ef7c-814a-421e-8d47-29b65379e664-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-857cfb796d-s468b\" (UID: \"3da3ef7c-814a-421e-8d47-29b65379e664\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-857cfb796d-s468b" Jul 10 14:32:06.254426 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.254300 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f2ded258-5977-4179-b653-d9f839f08b5d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-55c967b4fd-j8tkx\" (UID: \"f2ded258-5977-4179-b653-d9f839f08b5d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55c967b4fd-j8tkx" Jul 10 14:32:06.254426 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.254347 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f2ded258-5977-4179-b653-d9f839f08b5d-ca\") pod \"cluster-proxy-proxy-agent-55c967b4fd-j8tkx\" (UID: \"f2ded258-5977-4179-b653-d9f839f08b5d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55c967b4fd-j8tkx" Jul 10 14:32:06.254426 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.254370 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b94422c-c0a6-4731-a0c1-bb2e44fdf8ea-tmp\") pod \"klusterlet-addon-workmgr-59c984c7cf-xhvzp\" (UID: \"2b94422c-c0a6-4731-a0c1-bb2e44fdf8ea\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59c984c7cf-xhvzp" Jul 10 14:32:06.254426 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.254399 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f2ded258-5977-4179-b653-d9f839f08b5d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-55c967b4fd-j8tkx\" (UID: \"f2ded258-5977-4179-b653-d9f839f08b5d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55c967b4fd-j8tkx" Jul 10 14:32:06.254426 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.254427 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2ded258-5977-4179-b653-d9f839f08b5d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-55c967b4fd-j8tkx\" (UID: \"f2ded258-5977-4179-b653-d9f839f08b5d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55c967b4fd-j8tkx" Jul 10 14:32:06.254901 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.254459 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bc08d681-36cd-4339-ad25-349ee4e4f916-nginx-conf\") pod \"networking-console-plugin-6cf856758c-95jwg\" (UID: \"bc08d681-36cd-4339-ad25-349ee4e4f916\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-95jwg" Jul 10 14:32:06.254901 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.254488 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p878\" (UniqueName: \"kubernetes.io/projected/ae441373-d195-4c90-b77a-d189ae1951af-kube-api-access-6p878\") pod \"downloads-68c9cbb497-7jcj6\" (UID: \"ae441373-d195-4c90-b77a-d189ae1951af\") " pod="openshift-console/downloads-68c9cbb497-7jcj6" Jul 10 14:32:06.254901 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.254515 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f2ded258-5977-4179-b653-d9f839f08b5d-hub\") pod \"cluster-proxy-proxy-agent-55c967b4fd-j8tkx\" (UID: \"f2ded258-5977-4179-b653-d9f839f08b5d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55c967b4fd-j8tkx" Jul 10 14:32:06.254901 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.254558 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrkmw\" (UniqueName: \"kubernetes.io/projected/f2ded258-5977-4179-b653-d9f839f08b5d-kube-api-access-nrkmw\") pod \"cluster-proxy-proxy-agent-55c967b4fd-j8tkx\" (UID: \"f2ded258-5977-4179-b653-d9f839f08b5d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55c967b4fd-j8tkx" Jul 10 14:32:06.254901 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.254589 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t7jx\" (UniqueName: \"kubernetes.io/projected/3da3ef7c-814a-421e-8d47-29b65379e664-kube-api-access-2t7jx\") pod \"managed-serviceaccount-addon-agent-857cfb796d-s468b\" (UID: \"3da3ef7c-814a-421e-8d47-29b65379e664\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-857cfb796d-s468b" Jul 10 14:32:06.256604 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.256217 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bc08d681-36cd-4339-ad25-349ee4e4f916-nginx-conf\") pod \"networking-console-plugin-6cf856758c-95jwg\" (UID: \"bc08d681-36cd-4339-ad25-349ee4e4f916\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-95jwg" Jul 10 14:32:06.256802 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.256779 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2ded258-5977-4179-b653-d9f839f08b5d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-55c967b4fd-j8tkx\" (UID: \"f2ded258-5977-4179-b653-d9f839f08b5d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55c967b4fd-j8tkx" Jul 10 14:32:06.256802 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.256791 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b94422c-c0a6-4731-a0c1-bb2e44fdf8ea-tmp\") pod \"klusterlet-addon-workmgr-59c984c7cf-xhvzp\" (UID: \"2b94422c-c0a6-4731-a0c1-bb2e44fdf8ea\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59c984c7cf-xhvzp" Jul 10 14:32:06.258463 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.258404 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f2ded258-5977-4179-b653-d9f839f08b5d-hub\") pod \"cluster-proxy-proxy-agent-55c967b4fd-j8tkx\" (UID: \"f2ded258-5977-4179-b653-d9f839f08b5d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55c967b4fd-j8tkx" Jul 10 14:32:06.259065 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.259042 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f2ded258-5977-4179-b653-d9f839f08b5d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-55c967b4fd-j8tkx\" (UID: \"f2ded258-5977-4179-b653-d9f839f08b5d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55c967b4fd-j8tkx" Jul 10 14:32:06.259262 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.259204 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f2ded258-5977-4179-b653-d9f839f08b5d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-55c967b4fd-j8tkx\" (UID: \"f2ded258-5977-4179-b653-d9f839f08b5d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55c967b4fd-j8tkx" Jul 10 14:32:06.260174 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.259971 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bc08d681-36cd-4339-ad25-349ee4e4f916-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-95jwg\" (UID: \"bc08d681-36cd-4339-ad25-349ee4e4f916\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-95jwg" Jul 10 14:32:06.260174 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.259973 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2b94422c-c0a6-4731-a0c1-bb2e44fdf8ea-klusterlet-config\") pod \"klusterlet-addon-workmgr-59c984c7cf-xhvzp\" (UID: \"2b94422c-c0a6-4731-a0c1-bb2e44fdf8ea\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59c984c7cf-xhvzp" Jul 10 14:32:06.260516 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.260494 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3da3ef7c-814a-421e-8d47-29b65379e664-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-857cfb796d-s468b\" (UID: \"3da3ef7c-814a-421e-8d47-29b65379e664\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-857cfb796d-s468b" Jul 10 14:32:06.260795 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.260745 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f2ded258-5977-4179-b653-d9f839f08b5d-ca\") pod \"cluster-proxy-proxy-agent-55c967b4fd-j8tkx\" (UID: \"f2ded258-5977-4179-b653-d9f839f08b5d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55c967b4fd-j8tkx" Jul 10 14:32:06.266870 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.266846 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t7jx\" (UniqueName: \"kubernetes.io/projected/3da3ef7c-814a-421e-8d47-29b65379e664-kube-api-access-2t7jx\") pod \"managed-serviceaccount-addon-agent-857cfb796d-s468b\" (UID: \"3da3ef7c-814a-421e-8d47-29b65379e664\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-857cfb796d-s468b" Jul 10 14:32:06.270036 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.270008 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4chk2\" (UniqueName: \"kubernetes.io/projected/2b94422c-c0a6-4731-a0c1-bb2e44fdf8ea-kube-api-access-4chk2\") pod \"klusterlet-addon-workmgr-59c984c7cf-xhvzp\" (UID: \"2b94422c-c0a6-4731-a0c1-bb2e44fdf8ea\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59c984c7cf-xhvzp" Jul 10 14:32:06.270375 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.270330 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrkmw\" (UniqueName: \"kubernetes.io/projected/f2ded258-5977-4179-b653-d9f839f08b5d-kube-api-access-nrkmw\") pod \"cluster-proxy-proxy-agent-55c967b4fd-j8tkx\" (UID: \"f2ded258-5977-4179-b653-d9f839f08b5d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55c967b4fd-j8tkx" Jul 10 14:32:06.351375 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.351337 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-95jwg" Jul 10 14:32:06.355473 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.355449 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p878\" (UniqueName: \"kubernetes.io/projected/ae441373-d195-4c90-b77a-d189ae1951af-kube-api-access-6p878\") pod \"downloads-68c9cbb497-7jcj6\" (UID: \"ae441373-d195-4c90-b77a-d189ae1951af\") " pod="openshift-console/downloads-68c9cbb497-7jcj6" Jul 10 14:32:06.365779 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.365726 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p878\" (UniqueName: \"kubernetes.io/projected/ae441373-d195-4c90-b77a-d189ae1951af-kube-api-access-6p878\") pod \"downloads-68c9cbb497-7jcj6\" (UID: \"ae441373-d195-4c90-b77a-d189ae1951af\") " pod="openshift-console/downloads-68c9cbb497-7jcj6" Jul 10 14:32:06.376223 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.376197 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-857cfb796d-s468b" Jul 10 14:32:06.397089 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.397025 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55c967b4fd-j8tkx" Jul 10 14:32:06.403200 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.403180 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59c984c7cf-xhvzp" Jul 10 14:32:06.449912 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:06.449874 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-7jcj6" Jul 10 14:32:10.147241 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:10.147209 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-7jcj6"] Jul 10 14:32:10.157058 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:32:10.157007 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae441373_d195_4c90_b77a_d189ae1951af.slice/crio-18d3cfd2b632ac4017eca91b489abd9f440cdc1a334276f8920c7ae64382c3d2 WatchSource:0}: Error finding container 18d3cfd2b632ac4017eca91b489abd9f440cdc1a334276f8920c7ae64382c3d2: Status 404 returned error can't find the container with id 18d3cfd2b632ac4017eca91b489abd9f440cdc1a334276f8920c7ae64382c3d2 Jul 10 14:32:10.384021 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:10.383954 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55c967b4fd-j8tkx"] Jul 10 14:32:10.386940 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:10.386909 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-857cfb796d-s468b"] Jul 10 14:32:10.387938 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:10.387910 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-7jcj6" event={"ID":"ae441373-d195-4c90-b77a-d189ae1951af","Type":"ContainerStarted","Data":"18d3cfd2b632ac4017eca91b489abd9f440cdc1a334276f8920c7ae64382c3d2"} Jul 10 14:32:10.392382 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:10.391539 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-59c984c7cf-xhvzp"] Jul 10 14:32:10.395234 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:10.394671 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-95jwg"] Jul 10 14:32:10.395234 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:32:10.394805 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2ded258_5977_4179_b653_d9f839f08b5d.slice/crio-db3eaebede08b3c59ab6517bb3e141702d906f2f376eeed7d7eb875f48ed4da5 WatchSource:0}: Error finding container db3eaebede08b3c59ab6517bb3e141702d906f2f376eeed7d7eb875f48ed4da5: Status 404 returned error can't find the container with id db3eaebede08b3c59ab6517bb3e141702d906f2f376eeed7d7eb875f48ed4da5 Jul 10 14:32:10.397462 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:10.396847 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q9tkw" event={"ID":"1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0","Type":"ContainerStarted","Data":"ed7f1e86c206cfc7901066c545f6aa9f5771b0713cfe2119843a00a96719b262"} Jul 10 14:32:10.406655 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:10.406139 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-f698x" event={"ID":"68e91fda-6611-4951-806d-2514fe8bd1cb","Type":"ContainerStarted","Data":"c76fa699514fd33d83e37f34ebbf7e862d48aac1cdafedeca2c42ca85911596f"} Jul 10 14:32:10.410211 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:10.410185 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rxmgq" event={"ID":"4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f","Type":"ContainerStarted","Data":"9f267925be95082ec1056b1805b90f17f80537a7865c92d9c60ed9d8c20632cf"} Jul 10 14:32:10.413801 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:10.413693 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9gxrb" event={"ID":"5d6f7d4c-e2a0-40d0-9686-f016013d1504","Type":"ContainerStarted","Data":"21fa1e6cbf931e4f0e11461ef0a50b4091aec6fcd121d14f2ef0a68a490d0770"} Jul 10 14:32:10.413801 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:10.413740 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9gxrb" event={"ID":"5d6f7d4c-e2a0-40d0-9686-f016013d1504","Type":"ContainerStarted","Data":"06d820338084324d0f603e398cd8d213385a10b5ea4a07b71d2bab2e5e2ac5fb"} Jul 10 14:32:10.415610 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:10.415583 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-c8ztx" event={"ID":"cb41c5b6-dab8-4be4-b2be-56040d1659e7","Type":"ContainerStarted","Data":"52aa789a209baa2b2a32c93340e15caa1643733361e3c466687a5daba24ecae2"} Jul 10 14:32:10.418119 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:10.418071 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-q9tkw" podStartSLOduration=17.492523523 podStartE2EDuration="29.41805287s" totalImagesPullingTime="11.925529347s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 14:31:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:32:10.417415904 +0000 UTC m=+76.168787612" watchObservedRunningTime="2026-07-10 14:32:10.41805287 +0000 UTC m=+76.169424568" Jul 10 14:32:10.439990 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:10.439937 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-rxmgq" podStartSLOduration=67.296728755 podStartE2EDuration="1m15.439917277s" totalImagesPullingTime="8.143188522s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 14:30:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:32:10.439607877 +0000 UTC m=+76.190979573" watchObservedRunningTime="2026-07-10 14:32:10.439917277 +0000 UTC m=+76.191288975" Jul 10 14:32:10.466726 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:10.466623 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-9gxrb" podStartSLOduration=43.951435972 podStartE2EDuration="50.466609039s" totalImagesPullingTime="6.515173067s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 14:31:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:32:10.465400392 +0000 UTC m=+76.216772089" watchObservedRunningTime="2026-07-10 14:32:10.466609039 +0000 UTC m=+76.217980736" Jul 10 14:32:10.485219 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:10.485165 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-c8ztx" podStartSLOduration=43.903945004 podStartE2EDuration="50.485144693s" totalImagesPullingTime="6.581199689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 14:31:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:32:10.48417292 +0000 UTC m=+76.235544617" watchObservedRunningTime="2026-07-10 14:32:10.485144693 +0000 UTC m=+76.236516386" Jul 10 14:32:10.683725 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:10.683686 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-tbg8l"] Jul 10 14:32:10.700102 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:10.699134 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-tbg8l"] Jul 10 14:32:10.700102 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:10.699251 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-tbg8l" Jul 10 14:32:10.702350 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:10.702324 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 10 14:32:10.702569 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:10.702437 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-clkdf\"" Jul 10 14:32:10.800792 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:10.800695 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7558a80d-ed5d-4625-9297-995bfd84b9de-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-tbg8l\" (UID: \"7558a80d-ed5d-4625-9297-995bfd84b9de\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-tbg8l" Jul 10 14:32:10.902153 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:10.902113 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7558a80d-ed5d-4625-9297-995bfd84b9de-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-tbg8l\" (UID: \"7558a80d-ed5d-4625-9297-995bfd84b9de\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-tbg8l" Jul 10 14:32:10.904875 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:10.904849 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7558a80d-ed5d-4625-9297-995bfd84b9de-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-tbg8l\" (UID: \"7558a80d-ed5d-4625-9297-995bfd84b9de\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-tbg8l" Jul 10 14:32:11.012975 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:11.012911 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-tbg8l" Jul 10 14:32:11.206152 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:11.206067 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-tbg8l"] Jul 10 14:32:11.430630 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:11.429535 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-f698x" event={"ID":"68e91fda-6611-4951-806d-2514fe8bd1cb","Type":"ContainerStarted","Data":"dcf314dae9ca71bfd0942433f72b76251d80cfd11564b634a6b27f5dbf6468b4"} Jul 10 14:32:11.430630 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:11.430526 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-f698x" Jul 10 14:32:11.432381 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:11.432313 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-tbg8l" event={"ID":"7558a80d-ed5d-4625-9297-995bfd84b9de","Type":"ContainerStarted","Data":"54e9a56bbd22656acede40149608cc4eac5061106757f496217b1ff729fb58b9"} Jul 10 14:32:11.434817 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:11.434472 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55c967b4fd-j8tkx" event={"ID":"f2ded258-5977-4179-b653-d9f839f08b5d","Type":"ContainerStarted","Data":"db3eaebede08b3c59ab6517bb3e141702d906f2f376eeed7d7eb875f48ed4da5"} Jul 10 14:32:11.437187 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:11.437063 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-95jwg" event={"ID":"bc08d681-36cd-4339-ad25-349ee4e4f916","Type":"ContainerStarted","Data":"38d5063c6959e96e241d547278dd094f9b56579042eec12f8bbc643a6a7e9416"} Jul 10 14:32:11.439295 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:11.439153 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59c984c7cf-xhvzp" event={"ID":"2b94422c-c0a6-4731-a0c1-bb2e44fdf8ea","Type":"ContainerStarted","Data":"4be0e7d5bda2eae38c8564959d83a520fd538a378cccee572935ee319a3fe0fe"} Jul 10 14:32:11.441556 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:11.441383 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-857cfb796d-s468b" event={"ID":"3da3ef7c-814a-421e-8d47-29b65379e664","Type":"ContainerStarted","Data":"3d99a508ea322560d91a04744b9d1ad0e950a58d3167220b57d8fe5fa56baac9"} Jul 10 14:32:11.451770 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:11.451422 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-f698x" podStartSLOduration=34.885740377 podStartE2EDuration="41.451404469s" totalImagesPullingTime="6.565664092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 14:31:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:32:11.450981237 +0000 UTC m=+77.202352935" watchObservedRunningTime="2026-07-10 14:32:11.451404469 +0000 UTC m=+77.202776166" Jul 10 14:32:13.458529 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:13.458447 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-95jwg" event={"ID":"bc08d681-36cd-4339-ad25-349ee4e4f916","Type":"ContainerStarted","Data":"8079b9d5df0d86adfbcd06ad31532657eca34ed46103d1a7ab4086578e989420"} Jul 10 14:32:18.485190 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:18.485153 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-tbg8l" event={"ID":"7558a80d-ed5d-4625-9297-995bfd84b9de","Type":"ContainerStarted","Data":"9126314fbaaac36b3e8856792d11b5e2ff6cac5364007830b0438e0f17a008dc"} Jul 10 14:32:18.485673 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:18.485390 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-tbg8l" Jul 10 14:32:18.487814 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:18.487773 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55c967b4fd-j8tkx" event={"ID":"f2ded258-5977-4179-b653-d9f839f08b5d","Type":"ContainerStarted","Data":"d366a7cd4293c33dc570ccf0497ff4bc53018a71318c374dea8f85c6ef5ca22e"} Jul 10 14:32:18.487814 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:18.487812 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55c967b4fd-j8tkx" event={"ID":"f2ded258-5977-4179-b653-d9f839f08b5d","Type":"ContainerStarted","Data":"d17ea872ea40d2a2805d67a7bb965060315929d1a44d28a9dfda928101c2d7e7"} Jul 10 14:32:18.488005 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:18.487824 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55c967b4fd-j8tkx" event={"ID":"f2ded258-5977-4179-b653-d9f839f08b5d","Type":"ContainerStarted","Data":"6f8e63995249bd723e69eb1cedb4c3f5f4035df409f45dd3e180b8da41903026"} Jul 10 14:32:18.489970 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:18.489923 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59c984c7cf-xhvzp" event={"ID":"2b94422c-c0a6-4731-a0c1-bb2e44fdf8ea","Type":"ContainerStarted","Data":"d11ce0e25be81667f1a916f528f104df4e3a43454de6a01e081af570652f217d"} Jul 10 14:32:18.490077 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:18.489975 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59c984c7cf-xhvzp" Jul 10 14:32:18.491696 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:18.491680 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59c984c7cf-xhvzp" Jul 10 14:32:18.491824 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:18.491793 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-tbg8l" Jul 10 14:32:18.491930 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:18.491908 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-857cfb796d-s468b" event={"ID":"3da3ef7c-814a-421e-8d47-29b65379e664","Type":"ContainerStarted","Data":"dfd2267f5b0fa032f3407e9a1488815101437af1a95f12a70d08e50ba9c47371"} Jul 10 14:32:18.502326 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:18.502278 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cf856758c-95jwg" podStartSLOduration=10.111280688 podStartE2EDuration="12.502265179s" totalImagesPullingTime="2.390984491s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 14:32:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:32:13.473688325 +0000 UTC m=+79.225060020" watchObservedRunningTime="2026-07-10 14:32:18.502265179 +0000 UTC m=+84.253636875" Jul 10 14:32:18.503106 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:18.503072 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-tbg8l" podStartSLOduration=1.559208822 podStartE2EDuration="8.503063046s" totalImagesPullingTime="6.943854224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 14:32:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:32:18.501255454 +0000 UTC m=+84.252627171" watchObservedRunningTime="2026-07-10 14:32:18.503063046 +0000 UTC m=+84.254434756" Jul 10 14:32:18.554862 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:18.554814 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55c967b4fd-j8tkx" podStartSLOduration=4.778110465 podStartE2EDuration="12.554795373s" totalImagesPullingTime="7.776684908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 14:32:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:32:18.553273707 +0000 UTC m=+84.304645404" watchObservedRunningTime="2026-07-10 14:32:18.554795373 +0000 UTC m=+84.306167069" Jul 10 14:32:18.555235 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:18.555204 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-857cfb796d-s468b" podStartSLOduration=4.805136093 podStartE2EDuration="12.555193333s" totalImagesPullingTime="7.75005724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 14:32:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:32:18.532108766 +0000 UTC m=+84.283480464" watchObservedRunningTime="2026-07-10 14:32:18.555193333 +0000 UTC m=+84.306565033" Jul 10 14:32:18.572885 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:18.572832 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59c984c7cf-xhvzp" podStartSLOduration=4.800722238 podStartE2EDuration="12.572812926s" totalImagesPullingTime="7.772090688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 14:32:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:32:18.571424808 +0000 UTC m=+84.322796521" watchObservedRunningTime="2026-07-10 14:32:18.572812926 +0000 UTC m=+84.324184625" Jul 10 14:32:22.460891 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:22.460858 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-f698x" Jul 10 14:32:22.722599 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:22.722505 2569 patch_prober.go:28] interesting pod/image-registry-8d46cc844-xf6cz container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 10 14:32:22.722599 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:22.722581 2569 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-8d46cc844-xf6cz" podUID="c867eb4a-dae5-4edd-a157-7686c044be1e" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 14:32:26.103097 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:26.102770 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7r8x9"] Jul 10 14:32:26.267532 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:26.267443 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7r8x9" Jul 10 14:32:26.270596 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:26.270229 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 10 14:32:26.270596 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:26.270295 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-nmp5h\"" Jul 10 14:32:26.270820 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:26.270613 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 10 14:32:26.271700 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:26.271484 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 10 14:32:26.271700 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:26.271518 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 10 14:32:26.345034 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:26.344915 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c84e238b-1e80-4718-af8c-e6fdf5c92b03-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7r8x9\" (UID: \"c84e238b-1e80-4718-af8c-e6fdf5c92b03\") " pod="openshift-monitoring/node-exporter-7r8x9" Jul 10 14:32:26.345034 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:26.344963 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c84e238b-1e80-4718-af8c-e6fdf5c92b03-metrics-client-ca\") pod \"node-exporter-7r8x9\" (UID: \"c84e238b-1e80-4718-af8c-e6fdf5c92b03\") " pod="openshift-monitoring/node-exporter-7r8x9" Jul 10 14:32:26.345034 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:26.344999 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c84e238b-1e80-4718-af8c-e6fdf5c92b03-sys\") pod \"node-exporter-7r8x9\" (UID: \"c84e238b-1e80-4718-af8c-e6fdf5c92b03\") " pod="openshift-monitoring/node-exporter-7r8x9" Jul 10 14:32:26.345314 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:26.345058 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c84e238b-1e80-4718-af8c-e6fdf5c92b03-node-exporter-wtmp\") pod \"node-exporter-7r8x9\" (UID: \"c84e238b-1e80-4718-af8c-e6fdf5c92b03\") " pod="openshift-monitoring/node-exporter-7r8x9" Jul 10 14:32:26.345314 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:26.345108 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcgmq\" (UniqueName: \"kubernetes.io/projected/c84e238b-1e80-4718-af8c-e6fdf5c92b03-kube-api-access-tcgmq\") pod \"node-exporter-7r8x9\" (UID: \"c84e238b-1e80-4718-af8c-e6fdf5c92b03\") " pod="openshift-monitoring/node-exporter-7r8x9" Jul 10 14:32:26.345314 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:26.345150 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c84e238b-1e80-4718-af8c-e6fdf5c92b03-node-exporter-tls\") pod \"node-exporter-7r8x9\" (UID: \"c84e238b-1e80-4718-af8c-e6fdf5c92b03\") " pod="openshift-monitoring/node-exporter-7r8x9" Jul 10 14:32:26.345314 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:26.345177 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c84e238b-1e80-4718-af8c-e6fdf5c92b03-node-exporter-accelerators-collector-config\") pod \"node-exporter-7r8x9\" (UID: \"c84e238b-1e80-4718-af8c-e6fdf5c92b03\") " pod="openshift-monitoring/node-exporter-7r8x9" Jul 10 14:32:26.345515 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:26.345230 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c84e238b-1e80-4718-af8c-e6fdf5c92b03-node-exporter-textfile\") pod \"node-exporter-7r8x9\" (UID: \"c84e238b-1e80-4718-af8c-e6fdf5c92b03\") " pod="openshift-monitoring/node-exporter-7r8x9" Jul 10 14:32:26.345869 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:26.345607 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c84e238b-1e80-4718-af8c-e6fdf5c92b03-root\") pod \"node-exporter-7r8x9\" (UID: \"c84e238b-1e80-4718-af8c-e6fdf5c92b03\") " pod="openshift-monitoring/node-exporter-7r8x9" Jul 10 14:32:26.373365 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:26.373344 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-8d46cc844-xf6cz" Jul 10 14:32:26.446907 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:26.446864 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c84e238b-1e80-4718-af8c-e6fdf5c92b03-node-exporter-wtmp\") pod \"node-exporter-7r8x9\" (UID: \"c84e238b-1e80-4718-af8c-e6fdf5c92b03\") " pod="openshift-monitoring/node-exporter-7r8x9" Jul 10 14:32:26.446907 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:26.446908 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcgmq\" (UniqueName: \"kubernetes.io/projected/c84e238b-1e80-4718-af8c-e6fdf5c92b03-kube-api-access-tcgmq\") pod \"node-exporter-7r8x9\" (UID: \"c84e238b-1e80-4718-af8c-e6fdf5c92b03\") " pod="openshift-monitoring/node-exporter-7r8x9" Jul 10 14:32:26.447127 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:26.446947 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c84e238b-1e80-4718-af8c-e6fdf5c92b03-node-exporter-tls\") pod \"node-exporter-7r8x9\" (UID: \"c84e238b-1e80-4718-af8c-e6fdf5c92b03\") " pod="openshift-monitoring/node-exporter-7r8x9" Jul 10 14:32:26.447127 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:26.446976 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c84e238b-1e80-4718-af8c-e6fdf5c92b03-node-exporter-accelerators-collector-config\") pod \"node-exporter-7r8x9\" (UID: \"c84e238b-1e80-4718-af8c-e6fdf5c92b03\") " pod="openshift-monitoring/node-exporter-7r8x9" Jul 10 14:32:26.447127 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:26.447010 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c84e238b-1e80-4718-af8c-e6fdf5c92b03-node-exporter-textfile\") pod \"node-exporter-7r8x9\" (UID: \"c84e238b-1e80-4718-af8c-e6fdf5c92b03\") " pod="openshift-monitoring/node-exporter-7r8x9" Jul 10 14:32:26.447127 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:26.447011 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c84e238b-1e80-4718-af8c-e6fdf5c92b03-node-exporter-wtmp\") pod \"node-exporter-7r8x9\" (UID: \"c84e238b-1e80-4718-af8c-e6fdf5c92b03\") " pod="openshift-monitoring/node-exporter-7r8x9" Jul 10 14:32:26.447127 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:26.447059 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c84e238b-1e80-4718-af8c-e6fdf5c92b03-root\") pod \"node-exporter-7r8x9\" (UID: \"c84e238b-1e80-4718-af8c-e6fdf5c92b03\") " pod="openshift-monitoring/node-exporter-7r8x9" Jul 10 14:32:26.447127 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:26.447112 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c84e238b-1e80-4718-af8c-e6fdf5c92b03-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7r8x9\" (UID: \"c84e238b-1e80-4718-af8c-e6fdf5c92b03\") " pod="openshift-monitoring/node-exporter-7r8x9" Jul 10 14:32:26.447438 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:26.447186 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c84e238b-1e80-4718-af8c-e6fdf5c92b03-metrics-client-ca\") pod \"node-exporter-7r8x9\" (UID: \"c84e238b-1e80-4718-af8c-e6fdf5c92b03\") " pod="openshift-monitoring/node-exporter-7r8x9" Jul 10 14:32:26.447438 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:26.447238 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c84e238b-1e80-4718-af8c-e6fdf5c92b03-sys\") pod \"node-exporter-7r8x9\" (UID: \"c84e238b-1e80-4718-af8c-e6fdf5c92b03\") " pod="openshift-monitoring/node-exporter-7r8x9" Jul 10 14:32:26.447575 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:26.447553 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c84e238b-1e80-4718-af8c-e6fdf5c92b03-sys\") pod \"node-exporter-7r8x9\" (UID: \"c84e238b-1e80-4718-af8c-e6fdf5c92b03\") " pod="openshift-monitoring/node-exporter-7r8x9" Jul 10 14:32:26.447962 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:26.447941 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c84e238b-1e80-4718-af8c-e6fdf5c92b03-metrics-client-ca\") pod \"node-exporter-7r8x9\" (UID: \"c84e238b-1e80-4718-af8c-e6fdf5c92b03\") " pod="openshift-monitoring/node-exporter-7r8x9" Jul 10 14:32:26.448622 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:26.448404 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c84e238b-1e80-4718-af8c-e6fdf5c92b03-root\") pod \"node-exporter-7r8x9\" (UID: \"c84e238b-1e80-4718-af8c-e6fdf5c92b03\") " pod="openshift-monitoring/node-exporter-7r8x9" Jul 10 14:32:26.448740 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:26.448696 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c84e238b-1e80-4718-af8c-e6fdf5c92b03-node-exporter-textfile\") pod \"node-exporter-7r8x9\" (UID: \"c84e238b-1e80-4718-af8c-e6fdf5c92b03\") " pod="openshift-monitoring/node-exporter-7r8x9" Jul 10 14:32:26.448740 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:26.448730 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c84e238b-1e80-4718-af8c-e6fdf5c92b03-node-exporter-accelerators-collector-config\") pod \"node-exporter-7r8x9\" (UID: \"c84e238b-1e80-4718-af8c-e6fdf5c92b03\") " pod="openshift-monitoring/node-exporter-7r8x9" Jul 10 14:32:26.450718 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:26.450693 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c84e238b-1e80-4718-af8c-e6fdf5c92b03-node-exporter-tls\") pod \"node-exporter-7r8x9\" (UID: \"c84e238b-1e80-4718-af8c-e6fdf5c92b03\") " pod="openshift-monitoring/node-exporter-7r8x9" Jul 10 14:32:26.451417 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:26.451393 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c84e238b-1e80-4718-af8c-e6fdf5c92b03-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7r8x9\" (UID: \"c84e238b-1e80-4718-af8c-e6fdf5c92b03\") " pod="openshift-monitoring/node-exporter-7r8x9" Jul 10 14:32:26.456511 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:26.456490 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcgmq\" (UniqueName: \"kubernetes.io/projected/c84e238b-1e80-4718-af8c-e6fdf5c92b03-kube-api-access-tcgmq\") pod \"node-exporter-7r8x9\" (UID: \"c84e238b-1e80-4718-af8c-e6fdf5c92b03\") " pod="openshift-monitoring/node-exporter-7r8x9" Jul 10 14:32:26.580369 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:26.580335 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7r8x9" Jul 10 14:32:29.392513 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:32:29.392481 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc84e238b_1e80_4718_af8c_e6fdf5c92b03.slice/crio-d9a5de09bfa6fae9de12cd82e0a8bc32f9b77edd09a7aad9a5e22a861f3c0a00 WatchSource:0}: Error finding container d9a5de09bfa6fae9de12cd82e0a8bc32f9b77edd09a7aad9a5e22a861f3c0a00: Status 404 returned error can't find the container with id d9a5de09bfa6fae9de12cd82e0a8bc32f9b77edd09a7aad9a5e22a861f3c0a00 Jul 10 14:32:29.543267 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:29.543230 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7r8x9" event={"ID":"c84e238b-1e80-4718-af8c-e6fdf5c92b03","Type":"ContainerStarted","Data":"d9a5de09bfa6fae9de12cd82e0a8bc32f9b77edd09a7aad9a5e22a861f3c0a00"} Jul 10 14:32:30.549663 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:30.549612 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-7jcj6" event={"ID":"ae441373-d195-4c90-b77a-d189ae1951af","Type":"ContainerStarted","Data":"d5d98f52bcee75570b4e87128bfee240b55c73cc4ee71c287a97bd74508dff3b"} Jul 10 14:32:30.550175 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:30.549821 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-68c9cbb497-7jcj6" Jul 10 14:32:30.562307 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:30.562279 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-68c9cbb497-7jcj6" Jul 10 14:32:30.569964 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:30.569909 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-68c9cbb497-7jcj6" podStartSLOduration=4.848194915 podStartE2EDuration="24.569894132s" totalImagesPullingTime="19.721699217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 14:32:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:32:30.56774625 +0000 UTC m=+96.319117956" watchObservedRunningTime="2026-07-10 14:32:30.569894132 +0000 UTC m=+96.321265827" Jul 10 14:32:31.557275 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:31.557228 2569 generic.go:356] "Generic (PLEG): container finished" podID="c84e238b-1e80-4718-af8c-e6fdf5c92b03" containerID="f75a65329116c26cafad46c3668960432252b815045df7f8d5c86c3229ab4844" exitCode=0 Jul 10 14:32:31.557928 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:31.557891 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7r8x9" event={"ID":"c84e238b-1e80-4718-af8c-e6fdf5c92b03","Type":"ContainerDied","Data":"f75a65329116c26cafad46c3668960432252b815045df7f8d5c86c3229ab4844"} Jul 10 14:32:32.565106 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:32.565065 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7r8x9" event={"ID":"c84e238b-1e80-4718-af8c-e6fdf5c92b03","Type":"ContainerStarted","Data":"a1f5bf39a4f4c6754d786be4086afddd95b610e8f3de802031d45ed2e2bf9cc1"} Jul 10 14:32:32.565570 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:32.565134 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7r8x9" event={"ID":"c84e238b-1e80-4718-af8c-e6fdf5c92b03","Type":"ContainerStarted","Data":"f131b5540738b8f5a73b904c707c57d965fb22117c09eac1908ace8b82597311"} Jul 10 14:32:32.584847 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:32.584791 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7r8x9" podStartSLOduration=5.199673231 podStartE2EDuration="6.584774411s" totalImagesPullingTime="1.259372526s" totalInitContainerRuntime="125.728654ms" isStatefulPod=true podCreationTimestamp="2026-07-10 14:32:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:32:32.583435432 +0000 UTC m=+98.334807131" watchObservedRunningTime="2026-07-10 14:32:32.584774411 +0000 UTC m=+98.336146105" Jul 10 14:32:33.327587 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:33.327554 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-t9jl6" Jul 10 14:32:33.827641 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:33.827605 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-8d46cc844-xf6cz"] Jul 10 14:32:51.647654 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:51.647620 2569 generic.go:356] "Generic (PLEG): container finished" podID="dc5a5ba3-52b4-4b58-923e-df7e716d9438" containerID="578b74a3c3bb275f6f2574d0dfefeb5de35af020107e205feb0c0aa51001b3ea" exitCode=0 Jul 10 14:32:51.648106 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:51.647692 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-qtr8d" event={"ID":"dc5a5ba3-52b4-4b58-923e-df7e716d9438","Type":"ContainerDied","Data":"578b74a3c3bb275f6f2574d0dfefeb5de35af020107e205feb0c0aa51001b3ea"} Jul 10 14:32:51.648106 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:51.648046 2569 scope.go:117] "RemoveContainer" containerID="578b74a3c3bb275f6f2574d0dfefeb5de35af020107e205feb0c0aa51001b3ea" Jul 10 14:32:51.649273 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:51.649249 2569 generic.go:356] "Generic (PLEG): container finished" podID="5ceab7ff-a2fc-4115-8dcc-1d9b014a8ffe" containerID="40dd25b7b07f01621719f61a5c4da68e0dd482f6919eb427ec52a72325fc462e" exitCode=0 Jul 10 14:32:51.649367 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:51.649279 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-gwwww" event={"ID":"5ceab7ff-a2fc-4115-8dcc-1d9b014a8ffe","Type":"ContainerDied","Data":"40dd25b7b07f01621719f61a5c4da68e0dd482f6919eb427ec52a72325fc462e"} Jul 10 14:32:51.649583 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:51.649567 2569 scope.go:117] "RemoveContainer" containerID="40dd25b7b07f01621719f61a5c4da68e0dd482f6919eb427ec52a72325fc462e" Jul 10 14:32:52.655372 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:52.655335 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-qtr8d" event={"ID":"dc5a5ba3-52b4-4b58-923e-df7e716d9438","Type":"ContainerStarted","Data":"4dc481084562a7e16d807a7a55380d57007f2ab4ec54a0f6649643f209a928d0"} Jul 10 14:32:52.657230 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:52.657202 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-gwwww" event={"ID":"5ceab7ff-a2fc-4115-8dcc-1d9b014a8ffe","Type":"ContainerStarted","Data":"18c901ea53630d01031d24871a5494c05efb72c07e1fbcf2dd195e2a92f3c6ab"} Jul 10 14:32:58.855007 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:58.854950 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-8d46cc844-xf6cz" podUID="c867eb4a-dae5-4edd-a157-7686c044be1e" containerName="registry" containerID="cri-o://3b922d1e3a5ae0e62636aa13d9704c361cc55eba6720901954ddcf3fbb04df3e" gracePeriod=30 Jul 10 14:32:59.142381 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:59.142348 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8d46cc844-xf6cz" Jul 10 14:32:59.327853 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:59.327807 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c867eb4a-dae5-4edd-a157-7686c044be1e-image-registry-private-configuration\") pod \"c867eb4a-dae5-4edd-a157-7686c044be1e\" (UID: \"c867eb4a-dae5-4edd-a157-7686c044be1e\") " Jul 10 14:32:59.328066 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:59.327869 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c867eb4a-dae5-4edd-a157-7686c044be1e-registry-tls\") pod \"c867eb4a-dae5-4edd-a157-7686c044be1e\" (UID: \"c867eb4a-dae5-4edd-a157-7686c044be1e\") " Jul 10 14:32:59.328066 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:59.327906 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c867eb4a-dae5-4edd-a157-7686c044be1e-installation-pull-secrets\") pod \"c867eb4a-dae5-4edd-a157-7686c044be1e\" (UID: \"c867eb4a-dae5-4edd-a157-7686c044be1e\") " Jul 10 14:32:59.328066 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:59.327951 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c867eb4a-dae5-4edd-a157-7686c044be1e-registry-certificates\") pod \"c867eb4a-dae5-4edd-a157-7686c044be1e\" (UID: \"c867eb4a-dae5-4edd-a157-7686c044be1e\") " Jul 10 14:32:59.328066 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:59.328004 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-khll2\" (UniqueName: \"kubernetes.io/projected/c867eb4a-dae5-4edd-a157-7686c044be1e-kube-api-access-khll2\") pod \"c867eb4a-dae5-4edd-a157-7686c044be1e\" (UID: \"c867eb4a-dae5-4edd-a157-7686c044be1e\") " Jul 10 14:32:59.328066 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:59.328030 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c867eb4a-dae5-4edd-a157-7686c044be1e-bound-sa-token\") pod \"c867eb4a-dae5-4edd-a157-7686c044be1e\" (UID: \"c867eb4a-dae5-4edd-a157-7686c044be1e\") " Jul 10 14:32:59.328302 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:59.328077 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c867eb4a-dae5-4edd-a157-7686c044be1e-trusted-ca\") pod \"c867eb4a-dae5-4edd-a157-7686c044be1e\" (UID: \"c867eb4a-dae5-4edd-a157-7686c044be1e\") " Jul 10 14:32:59.328302 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:59.328116 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c867eb4a-dae5-4edd-a157-7686c044be1e-ca-trust-extracted\") pod \"c867eb4a-dae5-4edd-a157-7686c044be1e\" (UID: \"c867eb4a-dae5-4edd-a157-7686c044be1e\") " Jul 10 14:32:59.328527 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:59.328464 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c867eb4a-dae5-4edd-a157-7686c044be1e-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "c867eb4a-dae5-4edd-a157-7686c044be1e" (UID: "c867eb4a-dae5-4edd-a157-7686c044be1e"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 14:32:59.328924 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:59.328892 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c867eb4a-dae5-4edd-a157-7686c044be1e-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "c867eb4a-dae5-4edd-a157-7686c044be1e" (UID: "c867eb4a-dae5-4edd-a157-7686c044be1e"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 14:32:59.330504 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:59.330476 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c867eb4a-dae5-4edd-a157-7686c044be1e-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "c867eb4a-dae5-4edd-a157-7686c044be1e" (UID: "c867eb4a-dae5-4edd-a157-7686c044be1e"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 14:32:59.330648 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:59.330575 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c867eb4a-dae5-4edd-a157-7686c044be1e-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "c867eb4a-dae5-4edd-a157-7686c044be1e" (UID: "c867eb4a-dae5-4edd-a157-7686c044be1e"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 14:32:59.330710 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:59.330657 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c867eb4a-dae5-4edd-a157-7686c044be1e-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "c867eb4a-dae5-4edd-a157-7686c044be1e" (UID: "c867eb4a-dae5-4edd-a157-7686c044be1e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 14:32:59.330972 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:59.330940 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c867eb4a-dae5-4edd-a157-7686c044be1e-kube-api-access-khll2" (OuterVolumeSpecName: "kube-api-access-khll2") pod "c867eb4a-dae5-4edd-a157-7686c044be1e" (UID: "c867eb4a-dae5-4edd-a157-7686c044be1e"). InnerVolumeSpecName "kube-api-access-khll2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 14:32:59.331078 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:59.331045 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c867eb4a-dae5-4edd-a157-7686c044be1e-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "c867eb4a-dae5-4edd-a157-7686c044be1e" (UID: "c867eb4a-dae5-4edd-a157-7686c044be1e"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 14:32:59.337095 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:59.337063 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c867eb4a-dae5-4edd-a157-7686c044be1e-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "c867eb4a-dae5-4edd-a157-7686c044be1e" (UID: "c867eb4a-dae5-4edd-a157-7686c044be1e"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:32:59.429504 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:59.429472 2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c867eb4a-dae5-4edd-a157-7686c044be1e-trusted-ca\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 14:32:59.429504 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:59.429501 2569 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c867eb4a-dae5-4edd-a157-7686c044be1e-ca-trust-extracted\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 14:32:59.429705 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:59.429517 2569 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c867eb4a-dae5-4edd-a157-7686c044be1e-image-registry-private-configuration\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 14:32:59.429705 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:59.429530 2569 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c867eb4a-dae5-4edd-a157-7686c044be1e-registry-tls\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 14:32:59.429705 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:59.429544 2569 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c867eb4a-dae5-4edd-a157-7686c044be1e-installation-pull-secrets\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 14:32:59.429705 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:59.429554 2569 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c867eb4a-dae5-4edd-a157-7686c044be1e-registry-certificates\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 14:32:59.429705 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:59.429566 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khll2\" (UniqueName: \"kubernetes.io/projected/c867eb4a-dae5-4edd-a157-7686c044be1e-kube-api-access-khll2\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 14:32:59.429705 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:59.429577 2569 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c867eb4a-dae5-4edd-a157-7686c044be1e-bound-sa-token\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 14:32:59.684825 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:59.684721 2569 generic.go:356] "Generic (PLEG): container finished" podID="c867eb4a-dae5-4edd-a157-7686c044be1e" containerID="3b922d1e3a5ae0e62636aa13d9704c361cc55eba6720901954ddcf3fbb04df3e" exitCode=0 Jul 10 14:32:59.684825 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:59.684784 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8d46cc844-xf6cz" Jul 10 14:32:59.685004 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:59.684783 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8d46cc844-xf6cz" event={"ID":"c867eb4a-dae5-4edd-a157-7686c044be1e","Type":"ContainerDied","Data":"3b922d1e3a5ae0e62636aa13d9704c361cc55eba6720901954ddcf3fbb04df3e"} Jul 10 14:32:59.685004 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:59.684893 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8d46cc844-xf6cz" event={"ID":"c867eb4a-dae5-4edd-a157-7686c044be1e","Type":"ContainerDied","Data":"0ba4faa564b856adbed2079e799efc584f9bb52147faf7639dd187baedb27781"} Jul 10 14:32:59.685004 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:59.684915 2569 scope.go:117] "RemoveContainer" containerID="3b922d1e3a5ae0e62636aa13d9704c361cc55eba6720901954ddcf3fbb04df3e" Jul 10 14:32:59.693821 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:59.693803 2569 scope.go:117] "RemoveContainer" containerID="3b922d1e3a5ae0e62636aa13d9704c361cc55eba6720901954ddcf3fbb04df3e" Jul 10 14:32:59.694097 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:32:59.694073 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b922d1e3a5ae0e62636aa13d9704c361cc55eba6720901954ddcf3fbb04df3e\": container with ID starting with 3b922d1e3a5ae0e62636aa13d9704c361cc55eba6720901954ddcf3fbb04df3e not found: ID does not exist" containerID="3b922d1e3a5ae0e62636aa13d9704c361cc55eba6720901954ddcf3fbb04df3e" Jul 10 14:32:59.694171 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:59.694107 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b922d1e3a5ae0e62636aa13d9704c361cc55eba6720901954ddcf3fbb04df3e"} err="failed to get container status \"3b922d1e3a5ae0e62636aa13d9704c361cc55eba6720901954ddcf3fbb04df3e\": rpc error: code = NotFound desc = could not find container \"3b922d1e3a5ae0e62636aa13d9704c361cc55eba6720901954ddcf3fbb04df3e\": container with ID starting with 3b922d1e3a5ae0e62636aa13d9704c361cc55eba6720901954ddcf3fbb04df3e not found: ID does not exist" Jul 10 14:32:59.706641 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:59.706619 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-8d46cc844-xf6cz"] Jul 10 14:32:59.710250 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:32:59.710227 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-8d46cc844-xf6cz"] Jul 10 14:33:00.930809 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:33:00.930777 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c867eb4a-dae5-4edd-a157-7686c044be1e" path="/var/lib/kubelet/pods/c867eb4a-dae5-4edd-a157-7686c044be1e/volumes" Jul 10 14:33:15.751724 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:33:15.751689 2569 generic.go:356] "Generic (PLEG): container finished" podID="70c38fec-83c3-4e81-90c3-ab3f7204fe3c" containerID="4aa661743253eff78c10a084a7df3925015bfeaf686627e1bb638aaea106fdc1" exitCode=0 Jul 10 14:33:15.752091 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:33:15.751779 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-d7kg8" event={"ID":"70c38fec-83c3-4e81-90c3-ab3f7204fe3c","Type":"ContainerDied","Data":"4aa661743253eff78c10a084a7df3925015bfeaf686627e1bb638aaea106fdc1"} Jul 10 14:33:15.752184 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:33:15.752167 2569 scope.go:117] "RemoveContainer" containerID="4aa661743253eff78c10a084a7df3925015bfeaf686627e1bb638aaea106fdc1" Jul 10 14:33:16.757921 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:33:16.757884 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-d7kg8" event={"ID":"70c38fec-83c3-4e81-90c3-ab3f7204fe3c","Type":"ContainerStarted","Data":"04e4c0473d53e67f38da0b5cbe14f5e7ec7de3370fce12e59e71a44d87fd5e71"} Jul 10 14:35:54.640120 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:35:54.640093 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-n6pfx_296be890-64e9-4fb3-8c68-e4c2a7c8cc2c/console-operator/1.log" Jul 10 14:35:54.641011 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:35:54.640985 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-n6pfx_296be890-64e9-4fb3-8c68-e4c2a7c8cc2c/console-operator/1.log" Jul 10 14:35:54.647558 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:35:54.647538 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxvq2_016248a0-bbe2-44c9-8e51-5d4695ee6a52/ovn-acl-logging/0.log" Jul 10 14:35:54.648268 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:35:54.648251 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxvq2_016248a0-bbe2-44c9-8e51-5d4695ee6a52/ovn-acl-logging/0.log" Jul 10 14:35:54.651223 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:35:54.651206 2569 kubelet.go:1648] "Image garbage collection succeeded" Jul 10 14:37:12.434721 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:12.434687 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-547957cd78-s4l9r"] Jul 10 14:37:12.435341 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:12.435127 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c867eb4a-dae5-4edd-a157-7686c044be1e" containerName="registry" Jul 10 14:37:12.435341 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:12.435146 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c867eb4a-dae5-4edd-a157-7686c044be1e" containerName="registry" Jul 10 14:37:12.435341 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:12.435251 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c867eb4a-dae5-4edd-a157-7686c044be1e" containerName="registry" Jul 10 14:37:12.437537 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:12.437514 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-547957cd78-s4l9r" Jul 10 14:37:12.441732 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:12.441706 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 10 14:37:12.441997 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:12.441982 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 10 14:37:12.443010 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:12.442987 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 10 14:37:12.443122 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:12.442987 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-6fv7g\"" Jul 10 14:37:12.443122 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:12.442997 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 10 14:37:12.463540 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:12.463513 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-547957cd78-s4l9r"] Jul 10 14:37:12.492812 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:12.492780 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5wlv\" (UniqueName: \"kubernetes.io/projected/1b90b50f-1257-447e-97cc-0b89ace148c5-kube-api-access-n5wlv\") pod \"opendatahub-operator-controller-manager-547957cd78-s4l9r\" (UID: \"1b90b50f-1257-447e-97cc-0b89ace148c5\") " pod="opendatahub/opendatahub-operator-controller-manager-547957cd78-s4l9r" Jul 10 14:37:12.492973 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:12.492824 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1b90b50f-1257-447e-97cc-0b89ace148c5-manifests\") pod \"opendatahub-operator-controller-manager-547957cd78-s4l9r\" (UID: \"1b90b50f-1257-447e-97cc-0b89ace148c5\") " pod="opendatahub/opendatahub-operator-controller-manager-547957cd78-s4l9r" Jul 10 14:37:12.492973 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:12.492851 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1b90b50f-1257-447e-97cc-0b89ace148c5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-547957cd78-s4l9r\" (UID: \"1b90b50f-1257-447e-97cc-0b89ace148c5\") " pod="opendatahub/opendatahub-operator-controller-manager-547957cd78-s4l9r" Jul 10 14:37:12.492973 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:12.492880 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b90b50f-1257-447e-97cc-0b89ace148c5-tmp\") pod \"opendatahub-operator-controller-manager-547957cd78-s4l9r\" (UID: \"1b90b50f-1257-447e-97cc-0b89ace148c5\") " pod="opendatahub/opendatahub-operator-controller-manager-547957cd78-s4l9r" Jul 10 14:37:12.492973 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:12.492936 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1b90b50f-1257-447e-97cc-0b89ace148c5-webhook-cert\") pod \"opendatahub-operator-controller-manager-547957cd78-s4l9r\" (UID: \"1b90b50f-1257-447e-97cc-0b89ace148c5\") " pod="opendatahub/opendatahub-operator-controller-manager-547957cd78-s4l9r" Jul 10 14:37:12.594151 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:12.594116 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b90b50f-1257-447e-97cc-0b89ace148c5-tmp\") pod \"opendatahub-operator-controller-manager-547957cd78-s4l9r\" (UID: \"1b90b50f-1257-447e-97cc-0b89ace148c5\") " pod="opendatahub/opendatahub-operator-controller-manager-547957cd78-s4l9r" Jul 10 14:37:12.594297 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:12.594160 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1b90b50f-1257-447e-97cc-0b89ace148c5-webhook-cert\") pod \"opendatahub-operator-controller-manager-547957cd78-s4l9r\" (UID: \"1b90b50f-1257-447e-97cc-0b89ace148c5\") " pod="opendatahub/opendatahub-operator-controller-manager-547957cd78-s4l9r" Jul 10 14:37:12.594297 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:12.594195 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5wlv\" (UniqueName: \"kubernetes.io/projected/1b90b50f-1257-447e-97cc-0b89ace148c5-kube-api-access-n5wlv\") pod \"opendatahub-operator-controller-manager-547957cd78-s4l9r\" (UID: \"1b90b50f-1257-447e-97cc-0b89ace148c5\") " pod="opendatahub/opendatahub-operator-controller-manager-547957cd78-s4l9r" Jul 10 14:37:12.594297 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:12.594217 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1b90b50f-1257-447e-97cc-0b89ace148c5-manifests\") pod \"opendatahub-operator-controller-manager-547957cd78-s4l9r\" (UID: \"1b90b50f-1257-447e-97cc-0b89ace148c5\") " pod="opendatahub/opendatahub-operator-controller-manager-547957cd78-s4l9r" Jul 10 14:37:12.594297 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:12.594235 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1b90b50f-1257-447e-97cc-0b89ace148c5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-547957cd78-s4l9r\" (UID: \"1b90b50f-1257-447e-97cc-0b89ace148c5\") " pod="opendatahub/opendatahub-operator-controller-manager-547957cd78-s4l9r" Jul 10 14:37:12.594640 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:12.594606 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1b90b50f-1257-447e-97cc-0b89ace148c5-manifests\") pod \"opendatahub-operator-controller-manager-547957cd78-s4l9r\" (UID: \"1b90b50f-1257-447e-97cc-0b89ace148c5\") " pod="opendatahub/opendatahub-operator-controller-manager-547957cd78-s4l9r" Jul 10 14:37:12.596534 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:12.596508 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b90b50f-1257-447e-97cc-0b89ace148c5-tmp\") pod \"opendatahub-operator-controller-manager-547957cd78-s4l9r\" (UID: \"1b90b50f-1257-447e-97cc-0b89ace148c5\") " pod="opendatahub/opendatahub-operator-controller-manager-547957cd78-s4l9r" Jul 10 14:37:12.596673 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:12.596655 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1b90b50f-1257-447e-97cc-0b89ace148c5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-547957cd78-s4l9r\" (UID: \"1b90b50f-1257-447e-97cc-0b89ace148c5\") " pod="opendatahub/opendatahub-operator-controller-manager-547957cd78-s4l9r" Jul 10 14:37:12.596725 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:12.596657 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1b90b50f-1257-447e-97cc-0b89ace148c5-webhook-cert\") pod \"opendatahub-operator-controller-manager-547957cd78-s4l9r\" (UID: \"1b90b50f-1257-447e-97cc-0b89ace148c5\") " pod="opendatahub/opendatahub-operator-controller-manager-547957cd78-s4l9r" Jul 10 14:37:12.608696 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:12.608676 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5wlv\" (UniqueName: \"kubernetes.io/projected/1b90b50f-1257-447e-97cc-0b89ace148c5-kube-api-access-n5wlv\") pod \"opendatahub-operator-controller-manager-547957cd78-s4l9r\" (UID: \"1b90b50f-1257-447e-97cc-0b89ace148c5\") " pod="opendatahub/opendatahub-operator-controller-manager-547957cd78-s4l9r" Jul 10 14:37:12.748226 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:12.748141 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-547957cd78-s4l9r" Jul 10 14:37:12.886158 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:12.886134 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-547957cd78-s4l9r"] Jul 10 14:37:12.888889 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:37:12.888866 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b90b50f_1257_447e_97cc_0b89ace148c5.slice/crio-fa09adb6fb3c5f6a42e4d23c58942551a6867f0ba8b96a1f31bcb5d0005e8bd6 WatchSource:0}: Error finding container fa09adb6fb3c5f6a42e4d23c58942551a6867f0ba8b96a1f31bcb5d0005e8bd6: Status 404 returned error can't find the container with id fa09adb6fb3c5f6a42e4d23c58942551a6867f0ba8b96a1f31bcb5d0005e8bd6 Jul 10 14:37:12.890680 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:12.890660 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 14:37:13.703372 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:13.703335 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-547957cd78-s4l9r" event={"ID":"1b90b50f-1257-447e-97cc-0b89ace148c5","Type":"ContainerStarted","Data":"fa09adb6fb3c5f6a42e4d23c58942551a6867f0ba8b96a1f31bcb5d0005e8bd6"} Jul 10 14:37:16.719533 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:16.719494 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-547957cd78-s4l9r" event={"ID":"1b90b50f-1257-447e-97cc-0b89ace148c5","Type":"ContainerStarted","Data":"d0ce3eeb19dbbd7368bb5ff0476e27ccc84546dc4a00a0614c3cc6c7f7ddf19c"} Jul 10 14:37:17.725520 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:17.725488 2569 generic.go:356] "Generic (PLEG): container finished" podID="1b90b50f-1257-447e-97cc-0b89ace148c5" containerID="d0ce3eeb19dbbd7368bb5ff0476e27ccc84546dc4a00a0614c3cc6c7f7ddf19c" exitCode=0 Jul 10 14:37:17.726000 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:17.725534 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-547957cd78-s4l9r" event={"ID":"1b90b50f-1257-447e-97cc-0b89ace148c5","Type":"ContainerDied","Data":"d0ce3eeb19dbbd7368bb5ff0476e27ccc84546dc4a00a0614c3cc6c7f7ddf19c"} Jul 10 14:37:18.733220 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:18.733185 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-547957cd78-s4l9r" event={"ID":"1b90b50f-1257-447e-97cc-0b89ace148c5","Type":"ContainerStarted","Data":"e6628666226f09049d15de0efed58c4d2a0b03c1bed4c9003df13195925cae82"} Jul 10 14:37:18.733650 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:18.733281 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-547957cd78-s4l9r" Jul 10 14:37:18.756718 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:18.756668 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-547957cd78-s4l9r" podStartSLOduration=2.374715591 podStartE2EDuration="6.756652204s" totalImagesPullingTime="3.363854769s" totalInitContainerRuntime="1.018081844s" isStatefulPod=false podCreationTimestamp="2026-07-10 14:37:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:37:18.755090563 +0000 UTC m=+384.506462276" watchObservedRunningTime="2026-07-10 14:37:18.756652204 +0000 UTC m=+384.508023904" Jul 10 14:37:29.739937 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:29.739905 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-547957cd78-s4l9r" Jul 10 14:37:45.824614 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:45.824578 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf427n9s"] Jul 10 14:37:45.837989 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:45.837958 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf427n9s" Jul 10 14:37:45.841332 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:45.841310 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 10 14:37:45.841332 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:45.841324 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-fkndb\"" Jul 10 14:37:45.866287 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:45.866261 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9c6d657-9f43-4734-bb7f-a28dcdfa2704-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf427n9s\" (UID: \"b9c6d657-9f43-4734-bb7f-a28dcdfa2704\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf427n9s" Jul 10 14:37:45.866401 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:45.866294 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9c6d657-9f43-4734-bb7f-a28dcdfa2704-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf427n9s\" (UID: \"b9c6d657-9f43-4734-bb7f-a28dcdfa2704\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf427n9s" Jul 10 14:37:45.866401 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:45.866324 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9c6d657-9f43-4734-bb7f-a28dcdfa2704-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf427n9s\" (UID: \"b9c6d657-9f43-4734-bb7f-a28dcdfa2704\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf427n9s" Jul 10 14:37:45.866401 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:45.866350 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9c6d657-9f43-4734-bb7f-a28dcdfa2704-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf427n9s\" (UID: \"b9c6d657-9f43-4734-bb7f-a28dcdfa2704\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf427n9s" Jul 10 14:37:45.866526 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:45.866440 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l4jx\" (UniqueName: \"kubernetes.io/projected/b9c6d657-9f43-4734-bb7f-a28dcdfa2704-kube-api-access-8l4jx\") pod \"data-science-gateway-data-science-gateway-class-6576859cf427n9s\" (UID: \"b9c6d657-9f43-4734-bb7f-a28dcdfa2704\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf427n9s" Jul 10 14:37:45.866526 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:45.866472 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9c6d657-9f43-4734-bb7f-a28dcdfa2704-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf427n9s\" (UID: \"b9c6d657-9f43-4734-bb7f-a28dcdfa2704\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf427n9s" Jul 10 14:37:45.866600 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:45.866526 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9c6d657-9f43-4734-bb7f-a28dcdfa2704-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf427n9s\" (UID: \"b9c6d657-9f43-4734-bb7f-a28dcdfa2704\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf427n9s" Jul 10 14:37:45.866600 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:45.866573 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9c6d657-9f43-4734-bb7f-a28dcdfa2704-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf427n9s\" (UID: \"b9c6d657-9f43-4734-bb7f-a28dcdfa2704\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf427n9s" Jul 10 14:37:45.866600 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:45.866595 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9c6d657-9f43-4734-bb7f-a28dcdfa2704-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf427n9s\" (UID: \"b9c6d657-9f43-4734-bb7f-a28dcdfa2704\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf427n9s" Jul 10 14:37:45.927635 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:45.927607 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf427n9s"] Jul 10 14:37:45.967099 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:45.967048 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9c6d657-9f43-4734-bb7f-a28dcdfa2704-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf427n9s\" (UID: \"b9c6d657-9f43-4734-bb7f-a28dcdfa2704\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf427n9s" Jul 10 14:37:45.967099 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:45.967107 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9c6d657-9f43-4734-bb7f-a28dcdfa2704-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf427n9s\" (UID: \"b9c6d657-9f43-4734-bb7f-a28dcdfa2704\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf427n9s" Jul 10 14:37:45.967411 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:45.967134 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9c6d657-9f43-4734-bb7f-a28dcdfa2704-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf427n9s\" (UID: \"b9c6d657-9f43-4734-bb7f-a28dcdfa2704\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf427n9s" Jul 10 14:37:45.967411 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:45.967161 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9c6d657-9f43-4734-bb7f-a28dcdfa2704-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf427n9s\" (UID: \"b9c6d657-9f43-4734-bb7f-a28dcdfa2704\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf427n9s" Jul 10 14:37:45.967411 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:45.967186 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9c6d657-9f43-4734-bb7f-a28dcdfa2704-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf427n9s\" (UID: \"b9c6d657-9f43-4734-bb7f-a28dcdfa2704\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf427n9s" Jul 10 14:37:45.967571 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:45.967230 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9c6d657-9f43-4734-bb7f-a28dcdfa2704-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf427n9s\" (UID: \"b9c6d657-9f43-4734-bb7f-a28dcdfa2704\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf427n9s" Jul 10 14:37:45.967626 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:45.967516 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9c6d657-9f43-4734-bb7f-a28dcdfa2704-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf427n9s\" (UID: \"b9c6d657-9f43-4734-bb7f-a28dcdfa2704\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf427n9s" Jul 10 14:37:45.967626 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:45.967578 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9c6d657-9f43-4734-bb7f-a28dcdfa2704-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf427n9s\" (UID: \"b9c6d657-9f43-4734-bb7f-a28dcdfa2704\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf427n9s" Jul 10 14:37:45.967626 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:45.967529 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9c6d657-9f43-4734-bb7f-a28dcdfa2704-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf427n9s\" (UID: \"b9c6d657-9f43-4734-bb7f-a28dcdfa2704\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf427n9s" Jul 10 14:37:45.967795 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:45.967691 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l4jx\" (UniqueName: \"kubernetes.io/projected/b9c6d657-9f43-4734-bb7f-a28dcdfa2704-kube-api-access-8l4jx\") pod \"data-science-gateway-data-science-gateway-class-6576859cf427n9s\" (UID: \"b9c6d657-9f43-4734-bb7f-a28dcdfa2704\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf427n9s" Jul 10 14:37:45.967795 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:45.967726 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9c6d657-9f43-4734-bb7f-a28dcdfa2704-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf427n9s\" (UID: \"b9c6d657-9f43-4734-bb7f-a28dcdfa2704\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf427n9s" Jul 10 14:37:45.967920 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:45.967820 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9c6d657-9f43-4734-bb7f-a28dcdfa2704-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf427n9s\" (UID: \"b9c6d657-9f43-4734-bb7f-a28dcdfa2704\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf427n9s" Jul 10 14:37:45.967920 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:45.967864 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9c6d657-9f43-4734-bb7f-a28dcdfa2704-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf427n9s\" (UID: \"b9c6d657-9f43-4734-bb7f-a28dcdfa2704\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf427n9s" Jul 10 14:37:45.968018 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:45.967923 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9c6d657-9f43-4734-bb7f-a28dcdfa2704-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf427n9s\" (UID: \"b9c6d657-9f43-4734-bb7f-a28dcdfa2704\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf427n9s" Jul 10 14:37:45.969693 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:45.969663 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9c6d657-9f43-4734-bb7f-a28dcdfa2704-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf427n9s\" (UID: \"b9c6d657-9f43-4734-bb7f-a28dcdfa2704\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf427n9s" Jul 10 14:37:45.969847 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:45.969811 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9c6d657-9f43-4734-bb7f-a28dcdfa2704-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf427n9s\" (UID: \"b9c6d657-9f43-4734-bb7f-a28dcdfa2704\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf427n9s" Jul 10 14:37:45.986713 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:45.986685 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9c6d657-9f43-4734-bb7f-a28dcdfa2704-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf427n9s\" (UID: \"b9c6d657-9f43-4734-bb7f-a28dcdfa2704\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf427n9s" Jul 10 14:37:45.987378 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:45.987352 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l4jx\" (UniqueName: \"kubernetes.io/projected/b9c6d657-9f43-4734-bb7f-a28dcdfa2704-kube-api-access-8l4jx\") pod \"data-science-gateway-data-science-gateway-class-6576859cf427n9s\" (UID: \"b9c6d657-9f43-4734-bb7f-a28dcdfa2704\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf427n9s" Jul 10 14:37:46.148820 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:46.148785 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf427n9s" Jul 10 14:37:46.311102 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:46.311072 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf427n9s"] Jul 10 14:37:46.313872 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:37:46.313844 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9c6d657_9f43_4734_bb7f_a28dcdfa2704.slice/crio-948c34451865d8fe875b8312dc1201f1cedb38f0d418ea81cbdab0d30c204f7c WatchSource:0}: Error finding container 948c34451865d8fe875b8312dc1201f1cedb38f0d418ea81cbdab0d30c204f7c: Status 404 returned error can't find the container with id 948c34451865d8fe875b8312dc1201f1cedb38f0d418ea81cbdab0d30c204f7c Jul 10 14:37:46.851643 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:46.851600 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf427n9s" event={"ID":"b9c6d657-9f43-4734-bb7f-a28dcdfa2704","Type":"ContainerStarted","Data":"948c34451865d8fe875b8312dc1201f1cedb38f0d418ea81cbdab0d30c204f7c"} Jul 10 14:37:48.203524 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:48.203487 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-d76757ddd-kt87g"] Jul 10 14:37:48.211145 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:48.211119 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-d76757ddd-kt87g" Jul 10 14:37:48.218645 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:48.218618 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 10 14:37:48.218811 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:48.218622 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 10 14:37:48.218811 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:48.218656 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 10 14:37:48.219303 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:48.219115 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-4sp6c\"" Jul 10 14:37:48.219303 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:48.219234 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 10 14:37:48.219420 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:48.219376 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 10 14:37:48.243030 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:48.242993 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-d76757ddd-kt87g"] Jul 10 14:37:48.285966 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:48.285911 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b4e00a7-48d5-4566-8e92-83cbd3bfea97-cert\") pod \"lws-controller-manager-d76757ddd-kt87g\" (UID: \"5b4e00a7-48d5-4566-8e92-83cbd3bfea97\") " pod="openshift-lws-operator/lws-controller-manager-d76757ddd-kt87g" Jul 10 14:37:48.286180 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:48.286141 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjd99\" (UniqueName: \"kubernetes.io/projected/5b4e00a7-48d5-4566-8e92-83cbd3bfea97-kube-api-access-jjd99\") pod \"lws-controller-manager-d76757ddd-kt87g\" (UID: \"5b4e00a7-48d5-4566-8e92-83cbd3bfea97\") " pod="openshift-lws-operator/lws-controller-manager-d76757ddd-kt87g" Jul 10 14:37:48.286313 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:48.286241 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5b4e00a7-48d5-4566-8e92-83cbd3bfea97-manager-config\") pod \"lws-controller-manager-d76757ddd-kt87g\" (UID: \"5b4e00a7-48d5-4566-8e92-83cbd3bfea97\") " pod="openshift-lws-operator/lws-controller-manager-d76757ddd-kt87g" Jul 10 14:37:48.286313 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:48.286272 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5b4e00a7-48d5-4566-8e92-83cbd3bfea97-metrics-cert\") pod \"lws-controller-manager-d76757ddd-kt87g\" (UID: \"5b4e00a7-48d5-4566-8e92-83cbd3bfea97\") " pod="openshift-lws-operator/lws-controller-manager-d76757ddd-kt87g" Jul 10 14:37:48.386921 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:48.386882 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjd99\" (UniqueName: \"kubernetes.io/projected/5b4e00a7-48d5-4566-8e92-83cbd3bfea97-kube-api-access-jjd99\") pod \"lws-controller-manager-d76757ddd-kt87g\" (UID: \"5b4e00a7-48d5-4566-8e92-83cbd3bfea97\") " pod="openshift-lws-operator/lws-controller-manager-d76757ddd-kt87g" Jul 10 14:37:48.387107 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:48.386945 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5b4e00a7-48d5-4566-8e92-83cbd3bfea97-manager-config\") pod \"lws-controller-manager-d76757ddd-kt87g\" (UID: \"5b4e00a7-48d5-4566-8e92-83cbd3bfea97\") " pod="openshift-lws-operator/lws-controller-manager-d76757ddd-kt87g" Jul 10 14:37:48.387107 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:48.386971 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5b4e00a7-48d5-4566-8e92-83cbd3bfea97-metrics-cert\") pod \"lws-controller-manager-d76757ddd-kt87g\" (UID: \"5b4e00a7-48d5-4566-8e92-83cbd3bfea97\") " pod="openshift-lws-operator/lws-controller-manager-d76757ddd-kt87g" Jul 10 14:37:48.387107 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:48.387047 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b4e00a7-48d5-4566-8e92-83cbd3bfea97-cert\") pod \"lws-controller-manager-d76757ddd-kt87g\" (UID: \"5b4e00a7-48d5-4566-8e92-83cbd3bfea97\") " pod="openshift-lws-operator/lws-controller-manager-d76757ddd-kt87g" Jul 10 14:37:48.387807 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:48.387739 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5b4e00a7-48d5-4566-8e92-83cbd3bfea97-manager-config\") pod \"lws-controller-manager-d76757ddd-kt87g\" (UID: \"5b4e00a7-48d5-4566-8e92-83cbd3bfea97\") " pod="openshift-lws-operator/lws-controller-manager-d76757ddd-kt87g" Jul 10 14:37:48.389936 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:48.389909 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b4e00a7-48d5-4566-8e92-83cbd3bfea97-cert\") pod \"lws-controller-manager-d76757ddd-kt87g\" (UID: \"5b4e00a7-48d5-4566-8e92-83cbd3bfea97\") " pod="openshift-lws-operator/lws-controller-manager-d76757ddd-kt87g" Jul 10 14:37:48.390205 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:48.390163 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5b4e00a7-48d5-4566-8e92-83cbd3bfea97-metrics-cert\") pod \"lws-controller-manager-d76757ddd-kt87g\" (UID: \"5b4e00a7-48d5-4566-8e92-83cbd3bfea97\") " pod="openshift-lws-operator/lws-controller-manager-d76757ddd-kt87g" Jul 10 14:37:48.408979 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:48.408919 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjd99\" (UniqueName: \"kubernetes.io/projected/5b4e00a7-48d5-4566-8e92-83cbd3bfea97-kube-api-access-jjd99\") pod \"lws-controller-manager-d76757ddd-kt87g\" (UID: \"5b4e00a7-48d5-4566-8e92-83cbd3bfea97\") " pod="openshift-lws-operator/lws-controller-manager-d76757ddd-kt87g" Jul 10 14:37:48.522364 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:48.522283 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-d76757ddd-kt87g" Jul 10 14:37:48.924070 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:48.924028 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 14:37:48.924193 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:48.924104 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 14:37:48.924193 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:48.924139 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 14:37:48.997673 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:48.997647 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-d76757ddd-kt87g"] Jul 10 14:37:48.999780 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:37:48.999741 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b4e00a7_48d5_4566_8e92_83cbd3bfea97.slice/crio-7193bf2f06fc0ded785791dbabe24dd29c0f5cd234e3511c8c1fe6c75ce28eee WatchSource:0}: Error finding container 7193bf2f06fc0ded785791dbabe24dd29c0f5cd234e3511c8c1fe6c75ce28eee: Status 404 returned error can't find the container with id 7193bf2f06fc0ded785791dbabe24dd29c0f5cd234e3511c8c1fe6c75ce28eee Jul 10 14:37:49.869826 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:49.869634 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf427n9s" event={"ID":"b9c6d657-9f43-4734-bb7f-a28dcdfa2704","Type":"ContainerStarted","Data":"419414ba27b6404b3f041a40abb5971fe0eee8536bb6ea327553aa41d7cf63e1"} Jul 10 14:37:49.871945 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:49.871886 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-d76757ddd-kt87g" event={"ID":"5b4e00a7-48d5-4566-8e92-83cbd3bfea97","Type":"ContainerStarted","Data":"7193bf2f06fc0ded785791dbabe24dd29c0f5cd234e3511c8c1fe6c75ce28eee"} Jul 10 14:37:49.953666 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:49.953213 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf427n9s" podStartSLOduration=2.346758703 podStartE2EDuration="4.953187647s" totalImagesPullingTime="2.606428944s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 14:37:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:37:49.951047381 +0000 UTC m=+415.702419097" watchObservedRunningTime="2026-07-10 14:37:49.953187647 +0000 UTC m=+415.704559346" Jul 10 14:37:50.150013 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:50.149921 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf427n9s" Jul 10 14:37:50.155146 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:50.155122 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf427n9s" Jul 10 14:37:50.878727 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:50.878688 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf427n9s" Jul 10 14:37:50.879698 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:50.879670 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf427n9s" Jul 10 14:37:51.883923 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:51.883886 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-d76757ddd-kt87g" event={"ID":"5b4e00a7-48d5-4566-8e92-83cbd3bfea97","Type":"ContainerStarted","Data":"73e4d6a8325d87d4973327f2b72cf167266a058b05e1e757ca8d570aa7eca907"} Jul 10 14:37:51.884314 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:51.883980 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-d76757ddd-kt87g" Jul 10 14:37:51.903555 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:37:51.903507 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-d76757ddd-kt87g" podStartSLOduration=1.574523481 podStartE2EDuration="3.903489925s" totalImagesPullingTime="2.328966444s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 14:37:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:37:51.902817293 +0000 UTC m=+417.654189007" watchObservedRunningTime="2026-07-10 14:37:51.903489925 +0000 UTC m=+417.654861624" Jul 10 14:38:02.894610 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:38:02.894576 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-d76757ddd-kt87g" Jul 10 14:39:16.674596 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:39:16.674562 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-n872j"] Jul 10 14:39:16.677869 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:39:16.677847 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-n872j" Jul 10 14:39:16.680593 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:39:16.680562 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 10 14:39:16.680784 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:39:16.680597 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 10 14:39:16.681488 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:39:16.681471 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 10 14:39:16.681537 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:39:16.681471 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-xjhjc\"" Jul 10 14:39:16.686056 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:39:16.686033 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-n872j"] Jul 10 14:39:16.827850 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:39:16.827815 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrrgg\" (UniqueName: \"kubernetes.io/projected/1bb64e24-930b-41c3-8bfd-b1d54cfef5c3-kube-api-access-lrrgg\") pod \"dns-operator-controller-manager-864dd78c4c-n872j\" (UID: \"1bb64e24-930b-41c3-8bfd-b1d54cfef5c3\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-n872j" Jul 10 14:39:16.928852 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:39:16.928777 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrrgg\" (UniqueName: \"kubernetes.io/projected/1bb64e24-930b-41c3-8bfd-b1d54cfef5c3-kube-api-access-lrrgg\") pod \"dns-operator-controller-manager-864dd78c4c-n872j\" (UID: \"1bb64e24-930b-41c3-8bfd-b1d54cfef5c3\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-n872j" Jul 10 14:39:16.936382 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:39:16.936353 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrrgg\" (UniqueName: \"kubernetes.io/projected/1bb64e24-930b-41c3-8bfd-b1d54cfef5c3-kube-api-access-lrrgg\") pod \"dns-operator-controller-manager-864dd78c4c-n872j\" (UID: \"1bb64e24-930b-41c3-8bfd-b1d54cfef5c3\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-n872j" Jul 10 14:39:16.989429 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:39:16.989393 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-n872j" Jul 10 14:39:17.118123 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:39:17.118091 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-n872j"] Jul 10 14:39:17.121315 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:39:17.121289 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bb64e24_930b_41c3_8bfd_b1d54cfef5c3.slice/crio-5e5b650c6ee2c172e8035ae82aeecfda8295737079150595d3e18ab78c4543a7 WatchSource:0}: Error finding container 5e5b650c6ee2c172e8035ae82aeecfda8295737079150595d3e18ab78c4543a7: Status 404 returned error can't find the container with id 5e5b650c6ee2c172e8035ae82aeecfda8295737079150595d3e18ab78c4543a7 Jul 10 14:39:17.250709 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:39:17.250632 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-n872j" event={"ID":"1bb64e24-930b-41c3-8bfd-b1d54cfef5c3","Type":"ContainerStarted","Data":"5e5b650c6ee2c172e8035ae82aeecfda8295737079150595d3e18ab78c4543a7"} Jul 10 14:39:20.265889 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:39:20.265848 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-n872j" event={"ID":"1bb64e24-930b-41c3-8bfd-b1d54cfef5c3","Type":"ContainerStarted","Data":"cfc87e1a6850926fae16699e36fe94862a1ff2348764535d80c1853e524c6178"} Jul 10 14:39:20.266271 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:39:20.266001 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-n872j" Jul 10 14:39:20.284484 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:39:20.284430 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-n872j" podStartSLOduration=1.643087008 podStartE2EDuration="4.284417563s" totalImagesPullingTime="2.641330555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 14:39:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:39:20.283510644 +0000 UTC m=+506.034882339" watchObservedRunningTime="2026-07-10 14:39:20.284417563 +0000 UTC m=+506.035789259" Jul 10 14:39:30.724249 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:39:30.724216 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-97b9r"] Jul 10 14:39:30.728456 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:39:30.728434 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-97b9r" Jul 10 14:39:30.732104 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:39:30.732080 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-tpm6j\"" Jul 10 14:39:30.740381 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:39:30.740360 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-97b9r"] Jul 10 14:39:30.849806 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:39:30.849773 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q47m\" (UniqueName: \"kubernetes.io/projected/400f05b5-cf78-464e-9e82-898cee7e117b-kube-api-access-9q47m\") pod \"limitador-operator-controller-manager-859d7b78bd-97b9r\" (UID: \"400f05b5-cf78-464e-9e82-898cee7e117b\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-97b9r" Jul 10 14:39:30.950532 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:39:30.950499 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q47m\" (UniqueName: \"kubernetes.io/projected/400f05b5-cf78-464e-9e82-898cee7e117b-kube-api-access-9q47m\") pod \"limitador-operator-controller-manager-859d7b78bd-97b9r\" (UID: \"400f05b5-cf78-464e-9e82-898cee7e117b\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-97b9r" Jul 10 14:39:30.962370 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:39:30.962343 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q47m\" (UniqueName: \"kubernetes.io/projected/400f05b5-cf78-464e-9e82-898cee7e117b-kube-api-access-9q47m\") pod \"limitador-operator-controller-manager-859d7b78bd-97b9r\" (UID: \"400f05b5-cf78-464e-9e82-898cee7e117b\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-97b9r" Jul 10 14:39:31.039398 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:39:31.039340 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-97b9r" Jul 10 14:39:31.182715 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:39:31.182688 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-97b9r"] Jul 10 14:39:31.188792 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:39:31.188760 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod400f05b5_cf78_464e_9e82_898cee7e117b.slice/crio-3f4587c5378c84707552cac7a3b21550c234424f9955669c3279cc052e71d76b WatchSource:0}: Error finding container 3f4587c5378c84707552cac7a3b21550c234424f9955669c3279cc052e71d76b: Status 404 returned error can't find the container with id 3f4587c5378c84707552cac7a3b21550c234424f9955669c3279cc052e71d76b Jul 10 14:39:31.273299 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:39:31.273271 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-n872j" Jul 10 14:39:31.314354 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:39:31.314273 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-97b9r" event={"ID":"400f05b5-cf78-464e-9e82-898cee7e117b","Type":"ContainerStarted","Data":"3f4587c5378c84707552cac7a3b21550c234424f9955669c3279cc052e71d76b"} Jul 10 14:39:33.326099 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:39:33.326005 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-97b9r" event={"ID":"400f05b5-cf78-464e-9e82-898cee7e117b","Type":"ContainerStarted","Data":"9b82b10f50d81ba66ed84c2bef5fdae5e3facb57354176d4292fa668c76b2c96"} Jul 10 14:39:33.326099 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:39:33.326097 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-97b9r" Jul 10 14:39:33.346288 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:39:33.346235 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-97b9r" podStartSLOduration=1.592592802 podStartE2EDuration="3.346220443s" totalImagesPullingTime="1.753627641s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 14:39:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:39:33.344043752 +0000 UTC m=+519.095415467" watchObservedRunningTime="2026-07-10 14:39:33.346220443 +0000 UTC m=+519.097592190" Jul 10 14:39:44.334621 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:39:44.334591 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-97b9r" Jul 10 14:40:26.777640 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:26.777604 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-4qb8j"] Jul 10 14:40:26.781193 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:26.781166 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-4qb8j" Jul 10 14:40:26.783567 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:26.783543 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-zmhrc\"" Jul 10 14:40:26.788089 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:26.788064 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-4qb8j"] Jul 10 14:40:26.913678 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:26.913642 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mss7b\" (UniqueName: \"kubernetes.io/projected/732c2426-fef8-4712-b0d8-b2964aa0def6-kube-api-access-mss7b\") pod \"authorino-58cd955675-4qb8j\" (UID: \"732c2426-fef8-4712-b0d8-b2964aa0def6\") " pod="kuadrant-system/authorino-58cd955675-4qb8j" Jul 10 14:40:27.014639 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:27.014606 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mss7b\" (UniqueName: \"kubernetes.io/projected/732c2426-fef8-4712-b0d8-b2964aa0def6-kube-api-access-mss7b\") pod \"authorino-58cd955675-4qb8j\" (UID: \"732c2426-fef8-4712-b0d8-b2964aa0def6\") " pod="kuadrant-system/authorino-58cd955675-4qb8j" Jul 10 14:40:27.022584 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:27.022559 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mss7b\" (UniqueName: \"kubernetes.io/projected/732c2426-fef8-4712-b0d8-b2964aa0def6-kube-api-access-mss7b\") pod \"authorino-58cd955675-4qb8j\" (UID: \"732c2426-fef8-4712-b0d8-b2964aa0def6\") " pod="kuadrant-system/authorino-58cd955675-4qb8j" Jul 10 14:40:27.091201 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:27.091147 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-4qb8j" Jul 10 14:40:27.216389 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:27.216360 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-4qb8j"] Jul 10 14:40:27.219571 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:40:27.219545 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod732c2426_fef8_4712_b0d8_b2964aa0def6.slice/crio-9858a768d66eb92742fec5165c2704f2a58d641bac89361f57ba9e88bfe5df68 WatchSource:0}: Error finding container 9858a768d66eb92742fec5165c2704f2a58d641bac89361f57ba9e88bfe5df68: Status 404 returned error can't find the container with id 9858a768d66eb92742fec5165c2704f2a58d641bac89361f57ba9e88bfe5df68 Jul 10 14:40:27.568482 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:27.568446 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-4qb8j" event={"ID":"732c2426-fef8-4712-b0d8-b2964aa0def6","Type":"ContainerStarted","Data":"9858a768d66eb92742fec5165c2704f2a58d641bac89361f57ba9e88bfe5df68"} Jul 10 14:40:31.594484 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:31.594443 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-4qb8j" event={"ID":"732c2426-fef8-4712-b0d8-b2964aa0def6","Type":"ContainerStarted","Data":"57889987394af923346960aa89af25cb7c0c3eb40692161dea5d83af6c6b9baf"} Jul 10 14:40:31.610804 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:31.610741 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-4qb8j" podStartSLOduration=2.038222236 podStartE2EDuration="5.610727173s" totalImagesPullingTime="3.572504937s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 14:40:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:40:31.608899556 +0000 UTC m=+577.360271251" watchObservedRunningTime="2026-07-10 14:40:31.610727173 +0000 UTC m=+577.362098868" Jul 10 14:40:32.134887 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:32.134849 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-4qb8j"] Jul 10 14:40:33.604740 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:33.604690 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-4qb8j" podUID="732c2426-fef8-4712-b0d8-b2964aa0def6" containerName="authorino" containerID="cri-o://57889987394af923346960aa89af25cb7c0c3eb40692161dea5d83af6c6b9baf" gracePeriod=30 Jul 10 14:40:33.850510 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:33.850484 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-4qb8j" Jul 10 14:40:33.973040 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:33.973014 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mss7b\" (UniqueName: \"kubernetes.io/projected/732c2426-fef8-4712-b0d8-b2964aa0def6-kube-api-access-mss7b\") pod \"732c2426-fef8-4712-b0d8-b2964aa0def6\" (UID: \"732c2426-fef8-4712-b0d8-b2964aa0def6\") " Jul 10 14:40:33.975109 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:33.975076 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/732c2426-fef8-4712-b0d8-b2964aa0def6-kube-api-access-mss7b" (OuterVolumeSpecName: "kube-api-access-mss7b") pod "732c2426-fef8-4712-b0d8-b2964aa0def6" (UID: "732c2426-fef8-4712-b0d8-b2964aa0def6"). InnerVolumeSpecName "kube-api-access-mss7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 14:40:34.074343 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:34.074319 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mss7b\" (UniqueName: \"kubernetes.io/projected/732c2426-fef8-4712-b0d8-b2964aa0def6-kube-api-access-mss7b\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 14:40:34.611027 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:34.610996 2569 generic.go:356] "Generic (PLEG): container finished" podID="732c2426-fef8-4712-b0d8-b2964aa0def6" containerID="57889987394af923346960aa89af25cb7c0c3eb40692161dea5d83af6c6b9baf" exitCode=0 Jul 10 14:40:34.611546 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:34.611052 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-4qb8j" Jul 10 14:40:34.611546 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:34.611096 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-4qb8j" event={"ID":"732c2426-fef8-4712-b0d8-b2964aa0def6","Type":"ContainerDied","Data":"57889987394af923346960aa89af25cb7c0c3eb40692161dea5d83af6c6b9baf"} Jul 10 14:40:34.611546 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:34.611156 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-4qb8j" event={"ID":"732c2426-fef8-4712-b0d8-b2964aa0def6","Type":"ContainerDied","Data":"9858a768d66eb92742fec5165c2704f2a58d641bac89361f57ba9e88bfe5df68"} Jul 10 14:40:34.611546 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:34.611187 2569 scope.go:117] "RemoveContainer" containerID="57889987394af923346960aa89af25cb7c0c3eb40692161dea5d83af6c6b9baf" Jul 10 14:40:34.620984 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:34.620967 2569 scope.go:117] "RemoveContainer" containerID="57889987394af923346960aa89af25cb7c0c3eb40692161dea5d83af6c6b9baf" Jul 10 14:40:34.621279 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:40:34.621259 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57889987394af923346960aa89af25cb7c0c3eb40692161dea5d83af6c6b9baf\": container with ID starting with 57889987394af923346960aa89af25cb7c0c3eb40692161dea5d83af6c6b9baf not found: ID does not exist" containerID="57889987394af923346960aa89af25cb7c0c3eb40692161dea5d83af6c6b9baf" Jul 10 14:40:34.621355 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:34.621292 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57889987394af923346960aa89af25cb7c0c3eb40692161dea5d83af6c6b9baf"} err="failed to get container status \"57889987394af923346960aa89af25cb7c0c3eb40692161dea5d83af6c6b9baf\": rpc error: code = NotFound desc = could not find container \"57889987394af923346960aa89af25cb7c0c3eb40692161dea5d83af6c6b9baf\": container with ID starting with 57889987394af923346960aa89af25cb7c0c3eb40692161dea5d83af6c6b9baf not found: ID does not exist" Jul 10 14:40:34.631032 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:34.631001 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-4qb8j"] Jul 10 14:40:34.633432 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:34.633411 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-4qb8j"] Jul 10 14:40:34.931241 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:34.931216 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="732c2426-fef8-4712-b0d8-b2964aa0def6" path="/var/lib/kubelet/pods/732c2426-fef8-4712-b0d8-b2964aa0def6/volumes" Jul 10 14:40:45.086973 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:45.086941 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-6kvml"] Jul 10 14:40:45.087313 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:45.087255 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="732c2426-fef8-4712-b0d8-b2964aa0def6" containerName="authorino" Jul 10 14:40:45.087313 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:45.087266 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="732c2426-fef8-4712-b0d8-b2964aa0def6" containerName="authorino" Jul 10 14:40:45.087386 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:45.087323 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="732c2426-fef8-4712-b0d8-b2964aa0def6" containerName="authorino" Jul 10 14:40:45.091315 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:45.091297 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-6kvml" Jul 10 14:40:45.094015 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:45.093993 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 10 14:40:45.094123 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:45.094013 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-5dhvv\"" Jul 10 14:40:45.094840 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:45.094824 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 10 14:40:45.094925 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:45.094831 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 10 14:40:45.099880 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:45.099861 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-6kvml"] Jul 10 14:40:45.157013 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:45.156990 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9e667200-d548-4673-9cb8-d923f2eba1fc-data\") pod \"postgres-69fb4594cf-6kvml\" (UID: \"9e667200-d548-4673-9cb8-d923f2eba1fc\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-6kvml" Jul 10 14:40:45.157124 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:45.157019 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdrkj\" (UniqueName: \"kubernetes.io/projected/9e667200-d548-4673-9cb8-d923f2eba1fc-kube-api-access-hdrkj\") pod \"postgres-69fb4594cf-6kvml\" (UID: \"9e667200-d548-4673-9cb8-d923f2eba1fc\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-6kvml" Jul 10 14:40:45.258289 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:45.258259 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9e667200-d548-4673-9cb8-d923f2eba1fc-data\") pod \"postgres-69fb4594cf-6kvml\" (UID: \"9e667200-d548-4673-9cb8-d923f2eba1fc\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-6kvml" Jul 10 14:40:45.258407 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:45.258295 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdrkj\" (UniqueName: \"kubernetes.io/projected/9e667200-d548-4673-9cb8-d923f2eba1fc-kube-api-access-hdrkj\") pod \"postgres-69fb4594cf-6kvml\" (UID: \"9e667200-d548-4673-9cb8-d923f2eba1fc\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-6kvml" Jul 10 14:40:45.258645 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:45.258625 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9e667200-d548-4673-9cb8-d923f2eba1fc-data\") pod \"postgres-69fb4594cf-6kvml\" (UID: \"9e667200-d548-4673-9cb8-d923f2eba1fc\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-6kvml" Jul 10 14:40:45.266381 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:45.266360 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdrkj\" (UniqueName: \"kubernetes.io/projected/9e667200-d548-4673-9cb8-d923f2eba1fc-kube-api-access-hdrkj\") pod \"postgres-69fb4594cf-6kvml\" (UID: \"9e667200-d548-4673-9cb8-d923f2eba1fc\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-6kvml" Jul 10 14:40:45.402105 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:45.402083 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-6kvml" Jul 10 14:40:45.531862 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:45.531830 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-6kvml"] Jul 10 14:40:45.534380 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:40:45.534353 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e667200_d548_4673_9cb8_d923f2eba1fc.slice/crio-a670bf0f8b89a780bddad71304ebe47614fa1bcdb69b607a6acfa9dc1ee80a60 WatchSource:0}: Error finding container a670bf0f8b89a780bddad71304ebe47614fa1bcdb69b607a6acfa9dc1ee80a60: Status 404 returned error can't find the container with id a670bf0f8b89a780bddad71304ebe47614fa1bcdb69b607a6acfa9dc1ee80a60 Jul 10 14:40:45.662178 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:45.662119 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-6kvml" event={"ID":"9e667200-d548-4673-9cb8-d923f2eba1fc","Type":"ContainerStarted","Data":"a670bf0f8b89a780bddad71304ebe47614fa1bcdb69b607a6acfa9dc1ee80a60"} Jul 10 14:40:50.690643 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:50.690608 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-6kvml" event={"ID":"9e667200-d548-4673-9cb8-d923f2eba1fc","Type":"ContainerStarted","Data":"f435872d6fa89cc85e2d05eda81526ce39c332b71872b4f3e7b0db67c393f523"} Jul 10 14:40:50.691023 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:50.690706 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-6kvml" Jul 10 14:40:50.711678 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:50.711637 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-6kvml" podStartSLOduration=1.1459538 podStartE2EDuration="5.711624973s" totalImagesPullingTime="4.565671173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 14:40:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:40:50.709672314 +0000 UTC m=+596.461044029" watchObservedRunningTime="2026-07-10 14:40:50.711624973 +0000 UTC m=+596.462996669" Jul 10 14:40:54.678027 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:54.677995 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-n6pfx_296be890-64e9-4fb3-8c68-e4c2a7c8cc2c/console-operator/1.log" Jul 10 14:40:54.678722 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:54.678696 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-n6pfx_296be890-64e9-4fb3-8c68-e4c2a7c8cc2c/console-operator/1.log" Jul 10 14:40:54.684301 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:54.684273 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxvq2_016248a0-bbe2-44c9-8e51-5d4695ee6a52/ovn-acl-logging/0.log" Jul 10 14:40:54.684898 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:54.684880 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxvq2_016248a0-bbe2-44c9-8e51-5d4695ee6a52/ovn-acl-logging/0.log" Jul 10 14:40:56.725246 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:56.725219 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-6kvml" Jul 10 14:40:57.752033 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:57.752003 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7ff75d46-l5ddg"] Jul 10 14:40:57.757994 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:57.757971 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7ff75d46-l5ddg" Jul 10 14:40:57.760941 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:57.760903 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-zmhrc\"" Jul 10 14:40:57.761826 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:57.761804 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 10 14:40:57.765662 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:57.765641 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7ff75d46-l5ddg"] Jul 10 14:40:57.867661 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:57.867630 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7e47d2a1-b90f-4942-b9c3-7aa9a0f8293c-tls-cert\") pod \"authorino-7ff75d46-l5ddg\" (UID: \"7e47d2a1-b90f-4942-b9c3-7aa9a0f8293c\") " pod="kuadrant-system/authorino-7ff75d46-l5ddg" Jul 10 14:40:57.867822 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:57.867666 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b78b4\" (UniqueName: \"kubernetes.io/projected/7e47d2a1-b90f-4942-b9c3-7aa9a0f8293c-kube-api-access-b78b4\") pod \"authorino-7ff75d46-l5ddg\" (UID: \"7e47d2a1-b90f-4942-b9c3-7aa9a0f8293c\") " pod="kuadrant-system/authorino-7ff75d46-l5ddg" Jul 10 14:40:57.968592 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:57.968560 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7e47d2a1-b90f-4942-b9c3-7aa9a0f8293c-tls-cert\") pod \"authorino-7ff75d46-l5ddg\" (UID: \"7e47d2a1-b90f-4942-b9c3-7aa9a0f8293c\") " pod="kuadrant-system/authorino-7ff75d46-l5ddg" Jul 10 14:40:57.968592 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:57.968594 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b78b4\" (UniqueName: \"kubernetes.io/projected/7e47d2a1-b90f-4942-b9c3-7aa9a0f8293c-kube-api-access-b78b4\") pod \"authorino-7ff75d46-l5ddg\" (UID: \"7e47d2a1-b90f-4942-b9c3-7aa9a0f8293c\") " pod="kuadrant-system/authorino-7ff75d46-l5ddg" Jul 10 14:40:57.971152 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:57.971129 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7e47d2a1-b90f-4942-b9c3-7aa9a0f8293c-tls-cert\") pod \"authorino-7ff75d46-l5ddg\" (UID: \"7e47d2a1-b90f-4942-b9c3-7aa9a0f8293c\") " pod="kuadrant-system/authorino-7ff75d46-l5ddg" Jul 10 14:40:57.977440 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:57.977417 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b78b4\" (UniqueName: \"kubernetes.io/projected/7e47d2a1-b90f-4942-b9c3-7aa9a0f8293c-kube-api-access-b78b4\") pod \"authorino-7ff75d46-l5ddg\" (UID: \"7e47d2a1-b90f-4942-b9c3-7aa9a0f8293c\") " pod="kuadrant-system/authorino-7ff75d46-l5ddg" Jul 10 14:40:58.067200 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:58.067134 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7ff75d46-l5ddg" Jul 10 14:40:58.203568 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:58.200597 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7ff75d46-l5ddg"] Jul 10 14:40:58.205364 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:40:58.205336 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e47d2a1_b90f_4942_b9c3_7aa9a0f8293c.slice/crio-6a110104d36f23ec62736f8c8cf67099e5c085ac58b49dfd91d38bffd0fe7043 WatchSource:0}: Error finding container 6a110104d36f23ec62736f8c8cf67099e5c085ac58b49dfd91d38bffd0fe7043: Status 404 returned error can't find the container with id 6a110104d36f23ec62736f8c8cf67099e5c085ac58b49dfd91d38bffd0fe7043 Jul 10 14:40:58.726602 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:58.726558 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7ff75d46-l5ddg" event={"ID":"7e47d2a1-b90f-4942-b9c3-7aa9a0f8293c","Type":"ContainerStarted","Data":"6a110104d36f23ec62736f8c8cf67099e5c085ac58b49dfd91d38bffd0fe7043"} Jul 10 14:40:59.733477 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:59.733439 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7ff75d46-l5ddg" event={"ID":"7e47d2a1-b90f-4942-b9c3-7aa9a0f8293c","Type":"ContainerStarted","Data":"de58546f6bd9666a59f2fd983ee24894ac08f39256be7f20946ce03d3dbd38ff"} Jul 10 14:40:59.753611 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:40:59.753561 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7ff75d46-l5ddg" podStartSLOduration=2.124882637 podStartE2EDuration="2.75354576s" totalImagesPullingTime="628.663123ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 14:40:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:40:59.751922862 +0000 UTC m=+605.503294568" watchObservedRunningTime="2026-07-10 14:40:59.75354576 +0000 UTC m=+605.504917456" Jul 10 14:41:43.479864 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:41:43.479827 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-7f974"] Jul 10 14:41:43.483267 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:41:43.483249 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-7f974" Jul 10 14:41:43.485643 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:41:43.485612 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 10 14:41:43.485643 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:41:43.485616 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-d5tr9\"" Jul 10 14:41:43.491916 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:41:43.491893 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-7f974"] Jul 10 14:41:43.521457 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:41:43.521427 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/992df02a-bd7d-4cbb-92f8-35c7d92035a7-plugins-config-volume\") pod \"payload-pre-processing-7964457696-7f974\" (UID: \"992df02a-bd7d-4cbb-92f8-35c7d92035a7\") " pod="openshift-ingress/payload-pre-processing-7964457696-7f974" Jul 10 14:41:43.521457 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:41:43.521460 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vpjh\" (UniqueName: \"kubernetes.io/projected/992df02a-bd7d-4cbb-92f8-35c7d92035a7-kube-api-access-7vpjh\") pod \"payload-pre-processing-7964457696-7f974\" (UID: \"992df02a-bd7d-4cbb-92f8-35c7d92035a7\") " pod="openshift-ingress/payload-pre-processing-7964457696-7f974" Jul 10 14:41:43.622637 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:41:43.622601 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vpjh\" (UniqueName: \"kubernetes.io/projected/992df02a-bd7d-4cbb-92f8-35c7d92035a7-kube-api-access-7vpjh\") pod \"payload-pre-processing-7964457696-7f974\" (UID: \"992df02a-bd7d-4cbb-92f8-35c7d92035a7\") " pod="openshift-ingress/payload-pre-processing-7964457696-7f974" Jul 10 14:41:43.622848 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:41:43.622703 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/992df02a-bd7d-4cbb-92f8-35c7d92035a7-plugins-config-volume\") pod \"payload-pre-processing-7964457696-7f974\" (UID: \"992df02a-bd7d-4cbb-92f8-35c7d92035a7\") " pod="openshift-ingress/payload-pre-processing-7964457696-7f974" Jul 10 14:41:43.623351 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:41:43.623333 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/992df02a-bd7d-4cbb-92f8-35c7d92035a7-plugins-config-volume\") pod \"payload-pre-processing-7964457696-7f974\" (UID: \"992df02a-bd7d-4cbb-92f8-35c7d92035a7\") " pod="openshift-ingress/payload-pre-processing-7964457696-7f974" Jul 10 14:41:43.633603 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:41:43.633570 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vpjh\" (UniqueName: \"kubernetes.io/projected/992df02a-bd7d-4cbb-92f8-35c7d92035a7-kube-api-access-7vpjh\") pod \"payload-pre-processing-7964457696-7f974\" (UID: \"992df02a-bd7d-4cbb-92f8-35c7d92035a7\") " pod="openshift-ingress/payload-pre-processing-7964457696-7f974" Jul 10 14:41:43.795487 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:41:43.795410 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-7f974" Jul 10 14:41:43.926364 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:41:43.926341 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-7f974"] Jul 10 14:41:43.928380 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:41:43.928353 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod992df02a_bd7d_4cbb_92f8_35c7d92035a7.slice/crio-62298f786a9eea03bac31d07a01365cd39ac48069cdcd5d5f4fdecb4660b250b WatchSource:0}: Error finding container 62298f786a9eea03bac31d07a01365cd39ac48069cdcd5d5f4fdecb4660b250b: Status 404 returned error can't find the container with id 62298f786a9eea03bac31d07a01365cd39ac48069cdcd5d5f4fdecb4660b250b Jul 10 14:41:43.939920 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:41:43.939896 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-7f974" event={"ID":"992df02a-bd7d-4cbb-92f8-35c7d92035a7","Type":"ContainerStarted","Data":"62298f786a9eea03bac31d07a01365cd39ac48069cdcd5d5f4fdecb4660b250b"} Jul 10 14:41:45.951037 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:41:45.951003 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-7f974" event={"ID":"992df02a-bd7d-4cbb-92f8-35c7d92035a7","Type":"ContainerStarted","Data":"850471f5d487fde228c1ddd65ca6b279b995317d80619d566a62f19fc88a939d"} Jul 10 14:41:45.951398 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:41:45.951077 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7964457696-7f974" Jul 10 14:41:45.969079 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:41:45.969035 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7964457696-7f974" podStartSLOduration=1.537762487 podStartE2EDuration="2.969019912s" totalImagesPullingTime="1.431257425s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 14:41:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:41:45.967199769 +0000 UTC m=+651.718571486" watchObservedRunningTime="2026-07-10 14:41:45.969019912 +0000 UTC m=+651.720391607" Jul 10 14:41:56.957612 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:41:56.957574 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-7964457696-7f974" podUID="992df02a-bd7d-4cbb-92f8-35c7d92035a7" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.35:9004: connect: connection refused" Jul 10 14:42:06.956476 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:42:06.956439 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7964457696-7f974" Jul 10 14:42:27.483768 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:42:27.483715 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-hl72k"] Jul 10 14:42:27.487572 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:42:27.487549 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hl72k" Jul 10 14:42:27.490409 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:42:27.490387 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 10 14:42:27.490510 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:42:27.490425 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 10 14:42:27.491482 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:42:27.491461 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 10 14:42:27.491550 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:42:27.491490 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-2dj76\"" Jul 10 14:42:27.497942 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:42:27.497923 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-hl72k"] Jul 10 14:42:27.563236 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:42:27.563208 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-785tt\" (UniqueName: \"kubernetes.io/projected/ea6fe6ba-3aaf-4967-8653-26de92c01e33-kube-api-access-785tt\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hl72k\" (UID: \"ea6fe6ba-3aaf-4967-8653-26de92c01e33\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hl72k" Jul 10 14:42:27.563346 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:42:27.563258 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea6fe6ba-3aaf-4967-8653-26de92c01e33-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hl72k\" (UID: \"ea6fe6ba-3aaf-4967-8653-26de92c01e33\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hl72k" Jul 10 14:42:27.563346 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:42:27.563284 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea6fe6ba-3aaf-4967-8653-26de92c01e33-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hl72k\" (UID: \"ea6fe6ba-3aaf-4967-8653-26de92c01e33\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hl72k" Jul 10 14:42:27.563346 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:42:27.563307 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea6fe6ba-3aaf-4967-8653-26de92c01e33-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hl72k\" (UID: \"ea6fe6ba-3aaf-4967-8653-26de92c01e33\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hl72k" Jul 10 14:42:27.563346 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:42:27.563327 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea6fe6ba-3aaf-4967-8653-26de92c01e33-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hl72k\" (UID: \"ea6fe6ba-3aaf-4967-8653-26de92c01e33\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hl72k" Jul 10 14:42:27.563478 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:42:27.563346 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea6fe6ba-3aaf-4967-8653-26de92c01e33-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hl72k\" (UID: \"ea6fe6ba-3aaf-4967-8653-26de92c01e33\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hl72k" Jul 10 14:42:27.663997 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:42:27.663969 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-785tt\" (UniqueName: \"kubernetes.io/projected/ea6fe6ba-3aaf-4967-8653-26de92c01e33-kube-api-access-785tt\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hl72k\" (UID: \"ea6fe6ba-3aaf-4967-8653-26de92c01e33\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hl72k" Jul 10 14:42:27.664145 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:42:27.664023 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea6fe6ba-3aaf-4967-8653-26de92c01e33-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hl72k\" (UID: \"ea6fe6ba-3aaf-4967-8653-26de92c01e33\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hl72k" Jul 10 14:42:27.664145 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:42:27.664053 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea6fe6ba-3aaf-4967-8653-26de92c01e33-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hl72k\" (UID: \"ea6fe6ba-3aaf-4967-8653-26de92c01e33\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hl72k" Jul 10 14:42:27.664145 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:42:27.664077 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea6fe6ba-3aaf-4967-8653-26de92c01e33-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hl72k\" (UID: \"ea6fe6ba-3aaf-4967-8653-26de92c01e33\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hl72k" Jul 10 14:42:27.664145 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:42:27.664107 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea6fe6ba-3aaf-4967-8653-26de92c01e33-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hl72k\" (UID: \"ea6fe6ba-3aaf-4967-8653-26de92c01e33\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hl72k" Jul 10 14:42:27.664311 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:42:27.664179 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea6fe6ba-3aaf-4967-8653-26de92c01e33-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hl72k\" (UID: \"ea6fe6ba-3aaf-4967-8653-26de92c01e33\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hl72k" Jul 10 14:42:27.664536 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:42:27.664503 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea6fe6ba-3aaf-4967-8653-26de92c01e33-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hl72k\" (UID: \"ea6fe6ba-3aaf-4967-8653-26de92c01e33\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hl72k" Jul 10 14:42:27.664645 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:42:27.664569 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea6fe6ba-3aaf-4967-8653-26de92c01e33-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hl72k\" (UID: \"ea6fe6ba-3aaf-4967-8653-26de92c01e33\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hl72k" Jul 10 14:42:27.664645 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:42:27.664598 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea6fe6ba-3aaf-4967-8653-26de92c01e33-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hl72k\" (UID: \"ea6fe6ba-3aaf-4967-8653-26de92c01e33\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hl72k" Jul 10 14:42:27.666241 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:42:27.666221 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea6fe6ba-3aaf-4967-8653-26de92c01e33-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hl72k\" (UID: \"ea6fe6ba-3aaf-4967-8653-26de92c01e33\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hl72k" Jul 10 14:42:27.666612 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:42:27.666593 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea6fe6ba-3aaf-4967-8653-26de92c01e33-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hl72k\" (UID: \"ea6fe6ba-3aaf-4967-8653-26de92c01e33\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hl72k" Jul 10 14:42:27.673080 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:42:27.673057 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-785tt\" (UniqueName: \"kubernetes.io/projected/ea6fe6ba-3aaf-4967-8653-26de92c01e33-kube-api-access-785tt\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hl72k\" (UID: \"ea6fe6ba-3aaf-4967-8653-26de92c01e33\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hl72k" Jul 10 14:42:27.798501 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:42:27.798401 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hl72k" Jul 10 14:42:27.933822 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:42:27.933796 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-hl72k"] Jul 10 14:42:27.936328 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:42:27.936302 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea6fe6ba_3aaf_4967_8653_26de92c01e33.slice/crio-3dbb20693c2e4d701b6e74a11df43b48075414816838fd1aaded0b8dfe32933d WatchSource:0}: Error finding container 3dbb20693c2e4d701b6e74a11df43b48075414816838fd1aaded0b8dfe32933d: Status 404 returned error can't find the container with id 3dbb20693c2e4d701b6e74a11df43b48075414816838fd1aaded0b8dfe32933d Jul 10 14:42:27.937853 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:42:27.937831 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 14:42:28.142820 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:42:28.142792 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hl72k" event={"ID":"ea6fe6ba-3aaf-4967-8653-26de92c01e33","Type":"ContainerStarted","Data":"3dbb20693c2e4d701b6e74a11df43b48075414816838fd1aaded0b8dfe32933d"} Jul 10 14:42:30.152931 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:42:30.152900 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hl72k" event={"ID":"ea6fe6ba-3aaf-4967-8653-26de92c01e33","Type":"ContainerStarted","Data":"6d996874d1580c2efaf2b3ebeaff9587d5b929dd44b232c2ae543cf3229efa98"} Jul 10 14:42:30.172729 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:42:30.172684 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hl72k" podStartSLOduration=2.010051826 podStartE2EDuration="3.172660452s" totalImagesPullingTime="1.162608626s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 14:42:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:42:30.171387995 +0000 UTC m=+695.922759691" watchObservedRunningTime="2026-07-10 14:42:30.172660452 +0000 UTC m=+695.924032148" Jul 10 14:42:37.799115 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:42:37.799069 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hl72k" Jul 10 14:42:37.799537 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:42:37.799143 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hl72k" Jul 10 14:42:37.812987 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:42:37.812963 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hl72k" Jul 10 14:42:38.200458 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:42:38.200419 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hl72k" Jul 10 14:42:52.300546 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:42:52.300464 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfqgxb"] Jul 10 14:42:52.304004 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:42:52.303983 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfqgxb" Jul 10 14:42:52.306605 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:42:52.306586 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 10 14:42:52.312545 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:42:52.312525 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfqgxb"] Jul 10 14:42:52.473850 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:42:52.473815 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4568e4d3-668b-4eed-86d7-e1b49c357b36-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfqgxb\" (UID: \"4568e4d3-668b-4eed-86d7-e1b49c357b36\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfqgxb" Jul 10 14:42:52.473850 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:42:52.473852 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrvk8\" (UniqueName: \"kubernetes.io/projected/4568e4d3-668b-4eed-86d7-e1b49c357b36-kube-api-access-qrvk8\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfqgxb\" (UID: \"4568e4d3-668b-4eed-86d7-e1b49c357b36\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfqgxb" Jul 10 14:42:52.474059 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:42:52.473902 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4568e4d3-668b-4eed-86d7-e1b49c357b36-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfqgxb\" (UID: \"4568e4d3-668b-4eed-86d7-e1b49c357b36\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfqgxb" Jul 10 14:42:52.474059 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:42:52.473921 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4568e4d3-668b-4eed-86d7-e1b49c357b36-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfqgxb\" (UID: \"4568e4d3-668b-4eed-86d7-e1b49c357b36\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfqgxb" Jul 10 14:42:52.474059 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:42:52.473941 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4568e4d3-668b-4eed-86d7-e1b49c357b36-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfqgxb\" (UID: \"4568e4d3-668b-4eed-86d7-e1b49c357b36\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfqgxb" Jul 10 14:42:52.474059 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:42:52.473967 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4568e4d3-668b-4eed-86d7-e1b49c357b36-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfqgxb\" (UID: \"4568e4d3-668b-4eed-86d7-e1b49c357b36\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfqgxb" Jul 10 14:42:52.574853 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:42:52.574732 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4568e4d3-668b-4eed-86d7-e1b49c357b36-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfqgxb\" (UID: \"4568e4d3-668b-4eed-86d7-e1b49c357b36\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfqgxb" Jul 10 14:42:52.574853 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:42:52.574802 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrvk8\" (UniqueName: \"kubernetes.io/projected/4568e4d3-668b-4eed-86d7-e1b49c357b36-kube-api-access-qrvk8\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfqgxb\" (UID: \"4568e4d3-668b-4eed-86d7-e1b49c357b36\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfqgxb" Jul 10 14:42:52.574853 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:42:52.574861 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4568e4d3-668b-4eed-86d7-e1b49c357b36-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfqgxb\" (UID: \"4568e4d3-668b-4eed-86d7-e1b49c357b36\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfqgxb" Jul 10 14:42:52.575188 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:42:52.574975 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4568e4d3-668b-4eed-86d7-e1b49c357b36-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfqgxb\" (UID: \"4568e4d3-668b-4eed-86d7-e1b49c357b36\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfqgxb" Jul 10 14:42:52.575188 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:42:52.575033 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4568e4d3-668b-4eed-86d7-e1b49c357b36-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfqgxb\" (UID: \"4568e4d3-668b-4eed-86d7-e1b49c357b36\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfqgxb" Jul 10 14:42:52.575188 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:42:52.575066 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4568e4d3-668b-4eed-86d7-e1b49c357b36-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfqgxb\" (UID: \"4568e4d3-668b-4eed-86d7-e1b49c357b36\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfqgxb" Jul 10 14:42:52.575322 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:42:52.575227 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4568e4d3-668b-4eed-86d7-e1b49c357b36-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfqgxb\" (UID: \"4568e4d3-668b-4eed-86d7-e1b49c357b36\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfqgxb" Jul 10 14:42:52.575385 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:42:52.575354 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4568e4d3-668b-4eed-86d7-e1b49c357b36-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfqgxb\" (UID: \"4568e4d3-668b-4eed-86d7-e1b49c357b36\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfqgxb" Jul 10 14:42:52.575447 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:42:52.575428 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4568e4d3-668b-4eed-86d7-e1b49c357b36-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfqgxb\" (UID: \"4568e4d3-668b-4eed-86d7-e1b49c357b36\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfqgxb" Jul 10 14:42:52.577135 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:42:52.577114 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4568e4d3-668b-4eed-86d7-e1b49c357b36-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfqgxb\" (UID: \"4568e4d3-668b-4eed-86d7-e1b49c357b36\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfqgxb" Jul 10 14:42:52.577447 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:42:52.577428 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4568e4d3-668b-4eed-86d7-e1b49c357b36-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfqgxb\" (UID: \"4568e4d3-668b-4eed-86d7-e1b49c357b36\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfqgxb" Jul 10 14:42:52.582295 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:42:52.582272 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrvk8\" (UniqueName: \"kubernetes.io/projected/4568e4d3-668b-4eed-86d7-e1b49c357b36-kube-api-access-qrvk8\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfqgxb\" (UID: \"4568e4d3-668b-4eed-86d7-e1b49c357b36\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfqgxb" Jul 10 14:42:52.614442 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:42:52.614419 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfqgxb" Jul 10 14:42:52.753098 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:42:52.753066 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfqgxb"] Jul 10 14:42:52.757873 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:42:52.757844 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4568e4d3_668b_4eed_86d7_e1b49c357b36.slice/crio-db789d9d872a3c807185ece03397eedb19d4929ae56b0916e6c3a20c14587c88 WatchSource:0}: Error finding container db789d9d872a3c807185ece03397eedb19d4929ae56b0916e6c3a20c14587c88: Status 404 returned error can't find the container with id db789d9d872a3c807185ece03397eedb19d4929ae56b0916e6c3a20c14587c88 Jul 10 14:42:53.257361 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:42:53.257320 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfqgxb" event={"ID":"4568e4d3-668b-4eed-86d7-e1b49c357b36","Type":"ContainerStarted","Data":"33eeb53af31e00b7ecff34f25bddce08a094650c0b841a7bd476585295305cc1"} Jul 10 14:42:53.257514 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:42:53.257377 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfqgxb" event={"ID":"4568e4d3-668b-4eed-86d7-e1b49c357b36","Type":"ContainerStarted","Data":"db789d9d872a3c807185ece03397eedb19d4929ae56b0916e6c3a20c14587c88"} Jul 10 14:42:53.282043 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:42:53.279195 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfqgxb" podStartSLOduration=0.984469104 podStartE2EDuration="1.279175321s" totalImagesPullingTime="294.706217ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 14:42:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:42:53.275671953 +0000 UTC m=+719.027043668" watchObservedRunningTime="2026-07-10 14:42:53.279175321 +0000 UTC m=+719.030547020" Jul 10 14:43:01.594152 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:43:01.594123 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-xfvgk"] Jul 10 14:43:01.600448 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:43:01.600428 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xfvgk" Jul 10 14:43:01.603093 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:43:01.603073 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 10 14:43:01.606414 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:43:01.606392 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-xfvgk"] Jul 10 14:43:01.655365 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:43:01.655340 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce819691-846a-49b3-a923-5b053cfe39d0-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xfvgk\" (UID: \"ce819691-846a-49b3-a923-5b053cfe39d0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xfvgk" Jul 10 14:43:01.655483 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:43:01.655397 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dvvb\" (UniqueName: \"kubernetes.io/projected/ce819691-846a-49b3-a923-5b053cfe39d0-kube-api-access-9dvvb\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xfvgk\" (UID: \"ce819691-846a-49b3-a923-5b053cfe39d0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xfvgk" Jul 10 14:43:01.655483 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:43:01.655444 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce819691-846a-49b3-a923-5b053cfe39d0-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xfvgk\" (UID: \"ce819691-846a-49b3-a923-5b053cfe39d0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xfvgk" Jul 10 14:43:01.655553 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:43:01.655512 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce819691-846a-49b3-a923-5b053cfe39d0-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xfvgk\" (UID: \"ce819691-846a-49b3-a923-5b053cfe39d0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xfvgk" Jul 10 14:43:01.655553 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:43:01.655536 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce819691-846a-49b3-a923-5b053cfe39d0-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xfvgk\" (UID: \"ce819691-846a-49b3-a923-5b053cfe39d0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xfvgk" Jul 10 14:43:01.655618 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:43:01.655580 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce819691-846a-49b3-a923-5b053cfe39d0-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xfvgk\" (UID: \"ce819691-846a-49b3-a923-5b053cfe39d0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xfvgk" Jul 10 14:43:01.756430 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:43:01.756398 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce819691-846a-49b3-a923-5b053cfe39d0-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xfvgk\" (UID: \"ce819691-846a-49b3-a923-5b053cfe39d0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xfvgk" Jul 10 14:43:01.756430 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:43:01.756434 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce819691-846a-49b3-a923-5b053cfe39d0-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xfvgk\" (UID: \"ce819691-846a-49b3-a923-5b053cfe39d0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xfvgk" Jul 10 14:43:01.756624 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:43:01.756464 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce819691-846a-49b3-a923-5b053cfe39d0-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xfvgk\" (UID: \"ce819691-846a-49b3-a923-5b053cfe39d0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xfvgk" Jul 10 14:43:01.756624 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:43:01.756504 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce819691-846a-49b3-a923-5b053cfe39d0-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xfvgk\" (UID: \"ce819691-846a-49b3-a923-5b053cfe39d0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xfvgk" Jul 10 14:43:01.756624 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:43:01.756570 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dvvb\" (UniqueName: \"kubernetes.io/projected/ce819691-846a-49b3-a923-5b053cfe39d0-kube-api-access-9dvvb\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xfvgk\" (UID: \"ce819691-846a-49b3-a923-5b053cfe39d0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xfvgk" Jul 10 14:43:01.756624 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:43:01.756597 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce819691-846a-49b3-a923-5b053cfe39d0-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xfvgk\" (UID: \"ce819691-846a-49b3-a923-5b053cfe39d0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xfvgk" Jul 10 14:43:01.756886 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:43:01.756858 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce819691-846a-49b3-a923-5b053cfe39d0-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xfvgk\" (UID: \"ce819691-846a-49b3-a923-5b053cfe39d0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xfvgk" Jul 10 14:43:01.757040 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:43:01.756920 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce819691-846a-49b3-a923-5b053cfe39d0-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xfvgk\" (UID: \"ce819691-846a-49b3-a923-5b053cfe39d0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xfvgk" Jul 10 14:43:01.757040 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:43:01.756935 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce819691-846a-49b3-a923-5b053cfe39d0-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xfvgk\" (UID: \"ce819691-846a-49b3-a923-5b053cfe39d0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xfvgk" Jul 10 14:43:01.758640 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:43:01.758621 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce819691-846a-49b3-a923-5b053cfe39d0-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xfvgk\" (UID: \"ce819691-846a-49b3-a923-5b053cfe39d0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xfvgk" Jul 10 14:43:01.758968 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:43:01.758952 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce819691-846a-49b3-a923-5b053cfe39d0-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xfvgk\" (UID: \"ce819691-846a-49b3-a923-5b053cfe39d0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xfvgk" Jul 10 14:43:01.764485 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:43:01.764465 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dvvb\" (UniqueName: \"kubernetes.io/projected/ce819691-846a-49b3-a923-5b053cfe39d0-kube-api-access-9dvvb\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xfvgk\" (UID: \"ce819691-846a-49b3-a923-5b053cfe39d0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xfvgk" Jul 10 14:43:01.910841 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:43:01.910808 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xfvgk" Jul 10 14:43:02.052820 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:43:02.051798 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-xfvgk"] Jul 10 14:43:02.300933 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:43:02.300903 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xfvgk" event={"ID":"ce819691-846a-49b3-a923-5b053cfe39d0","Type":"ContainerStarted","Data":"c4b63d6ba9d978615817317088b7904421640c0933af47038c724cd42719e88c"} Jul 10 14:43:02.614872 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:43:02.614778 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfqgxb" Jul 10 14:43:02.614872 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:43:02.614862 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfqgxb" Jul 10 14:43:02.626948 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:43:02.626923 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfqgxb" Jul 10 14:43:03.306901 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:43:03.306858 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xfvgk" event={"ID":"ce819691-846a-49b3-a923-5b053cfe39d0","Type":"ContainerStarted","Data":"3c8106f6820a1dbb54857c200dbaa655bdb8560ebda59e3d0ca9c9bf152dfeb3"} Jul 10 14:43:03.318011 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:43:03.317984 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfqgxb" Jul 10 14:43:03.324605 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:43:03.324568 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xfvgk" podStartSLOduration=2.135575546 podStartE2EDuration="2.324555901s" totalImagesPullingTime="188.980355ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 14:43:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:43:03.322997613 +0000 UTC m=+729.074369309" watchObservedRunningTime="2026-07-10 14:43:03.324555901 +0000 UTC m=+729.075927596" Jul 10 14:43:11.911738 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:43:11.911686 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xfvgk" Jul 10 14:43:11.912173 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:43:11.911785 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xfvgk" Jul 10 14:43:11.924489 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:43:11.924464 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xfvgk" Jul 10 14:43:12.366530 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:43:12.366506 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xfvgk" Jul 10 14:44:33.437252 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:44:33.437167 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7ff75d46-l5ddg"] Jul 10 14:44:33.437774 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:44:33.437386 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7ff75d46-l5ddg" podUID="7e47d2a1-b90f-4942-b9c3-7aa9a0f8293c" containerName="authorino" containerID="cri-o://de58546f6bd9666a59f2fd983ee24894ac08f39256be7f20946ce03d3dbd38ff" gracePeriod=30 Jul 10 14:44:33.693824 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:44:33.693768 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7ff75d46-l5ddg" Jul 10 14:44:33.737050 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:44:33.737016 2569 generic.go:356] "Generic (PLEG): container finished" podID="7e47d2a1-b90f-4942-b9c3-7aa9a0f8293c" containerID="de58546f6bd9666a59f2fd983ee24894ac08f39256be7f20946ce03d3dbd38ff" exitCode=0 Jul 10 14:44:33.737220 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:44:33.737076 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7ff75d46-l5ddg" Jul 10 14:44:33.737220 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:44:33.737120 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7ff75d46-l5ddg" event={"ID":"7e47d2a1-b90f-4942-b9c3-7aa9a0f8293c","Type":"ContainerDied","Data":"de58546f6bd9666a59f2fd983ee24894ac08f39256be7f20946ce03d3dbd38ff"} Jul 10 14:44:33.737220 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:44:33.737167 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7ff75d46-l5ddg" event={"ID":"7e47d2a1-b90f-4942-b9c3-7aa9a0f8293c","Type":"ContainerDied","Data":"6a110104d36f23ec62736f8c8cf67099e5c085ac58b49dfd91d38bffd0fe7043"} Jul 10 14:44:33.737220 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:44:33.737193 2569 scope.go:117] "RemoveContainer" containerID="de58546f6bd9666a59f2fd983ee24894ac08f39256be7f20946ce03d3dbd38ff" Jul 10 14:44:33.745780 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:44:33.745760 2569 scope.go:117] "RemoveContainer" containerID="de58546f6bd9666a59f2fd983ee24894ac08f39256be7f20946ce03d3dbd38ff" Jul 10 14:44:33.746049 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:44:33.746032 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de58546f6bd9666a59f2fd983ee24894ac08f39256be7f20946ce03d3dbd38ff\": container with ID starting with de58546f6bd9666a59f2fd983ee24894ac08f39256be7f20946ce03d3dbd38ff not found: ID does not exist" containerID="de58546f6bd9666a59f2fd983ee24894ac08f39256be7f20946ce03d3dbd38ff" Jul 10 14:44:33.746115 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:44:33.746057 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de58546f6bd9666a59f2fd983ee24894ac08f39256be7f20946ce03d3dbd38ff"} err="failed to get container status \"de58546f6bd9666a59f2fd983ee24894ac08f39256be7f20946ce03d3dbd38ff\": rpc error: code = NotFound desc = could not find container \"de58546f6bd9666a59f2fd983ee24894ac08f39256be7f20946ce03d3dbd38ff\": container with ID starting with de58546f6bd9666a59f2fd983ee24894ac08f39256be7f20946ce03d3dbd38ff not found: ID does not exist" Jul 10 14:44:33.757373 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:44:33.757352 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7e47d2a1-b90f-4942-b9c3-7aa9a0f8293c-tls-cert\") pod \"7e47d2a1-b90f-4942-b9c3-7aa9a0f8293c\" (UID: \"7e47d2a1-b90f-4942-b9c3-7aa9a0f8293c\") " Jul 10 14:44:33.757451 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:44:33.757390 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b78b4\" (UniqueName: \"kubernetes.io/projected/7e47d2a1-b90f-4942-b9c3-7aa9a0f8293c-kube-api-access-b78b4\") pod \"7e47d2a1-b90f-4942-b9c3-7aa9a0f8293c\" (UID: \"7e47d2a1-b90f-4942-b9c3-7aa9a0f8293c\") " Jul 10 14:44:33.759271 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:44:33.759248 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e47d2a1-b90f-4942-b9c3-7aa9a0f8293c-kube-api-access-b78b4" (OuterVolumeSpecName: "kube-api-access-b78b4") pod "7e47d2a1-b90f-4942-b9c3-7aa9a0f8293c" (UID: "7e47d2a1-b90f-4942-b9c3-7aa9a0f8293c"). InnerVolumeSpecName "kube-api-access-b78b4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 14:44:33.769788 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:44:33.769761 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e47d2a1-b90f-4942-b9c3-7aa9a0f8293c-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "7e47d2a1-b90f-4942-b9c3-7aa9a0f8293c" (UID: "7e47d2a1-b90f-4942-b9c3-7aa9a0f8293c"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 14:44:33.858840 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:44:33.858799 2569 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7e47d2a1-b90f-4942-b9c3-7aa9a0f8293c-tls-cert\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 14:44:33.858840 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:44:33.858835 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b78b4\" (UniqueName: \"kubernetes.io/projected/7e47d2a1-b90f-4942-b9c3-7aa9a0f8293c-kube-api-access-b78b4\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 14:44:34.064517 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:44:34.064488 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7ff75d46-l5ddg"] Jul 10 14:44:34.071946 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:44:34.071919 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7ff75d46-l5ddg"] Jul 10 14:44:34.936378 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:44:34.936344 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e47d2a1-b90f-4942-b9c3-7aa9a0f8293c" path="/var/lib/kubelet/pods/7e47d2a1-b90f-4942-b9c3-7aa9a0f8293c/volumes" Jul 10 14:45:54.707204 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:45:54.707122 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-n6pfx_296be890-64e9-4fb3-8c68-e4c2a7c8cc2c/console-operator/1.log" Jul 10 14:45:54.709146 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:45:54.709118 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-n6pfx_296be890-64e9-4fb3-8c68-e4c2a7c8cc2c/console-operator/1.log" Jul 10 14:45:54.713531 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:45:54.713506 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxvq2_016248a0-bbe2-44c9-8e51-5d4695ee6a52/ovn-acl-logging/0.log" Jul 10 14:45:54.715418 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:45:54.715400 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxvq2_016248a0-bbe2-44c9-8e51-5d4695ee6a52/ovn-acl-logging/0.log" Jul 10 14:50:54.736571 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:50:54.736487 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-n6pfx_296be890-64e9-4fb3-8c68-e4c2a7c8cc2c/console-operator/1.log" Jul 10 14:50:54.737896 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:50:54.737873 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-n6pfx_296be890-64e9-4fb3-8c68-e4c2a7c8cc2c/console-operator/1.log" Jul 10 14:50:54.742783 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:50:54.742746 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxvq2_016248a0-bbe2-44c9-8e51-5d4695ee6a52/ovn-acl-logging/0.log" Jul 10 14:50:54.743887 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:50:54.743865 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxvq2_016248a0-bbe2-44c9-8e51-5d4695ee6a52/ovn-acl-logging/0.log" Jul 10 14:55:54.764395 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:55:54.764366 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-n6pfx_296be890-64e9-4fb3-8c68-e4c2a7c8cc2c/console-operator/1.log" Jul 10 14:55:54.766186 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:55:54.766166 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-n6pfx_296be890-64e9-4fb3-8c68-e4c2a7c8cc2c/console-operator/1.log" Jul 10 14:55:54.770480 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:55:54.770458 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxvq2_016248a0-bbe2-44c9-8e51-5d4695ee6a52/ovn-acl-logging/0.log" Jul 10 14:55:54.772102 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:55:54.772078 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxvq2_016248a0-bbe2-44c9-8e51-5d4695ee6a52/ovn-acl-logging/0.log" Jul 10 14:58:02.968104 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:02.968021 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh"] Jul 10 14:58:02.968549 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:02.968391 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e47d2a1-b90f-4942-b9c3-7aa9a0f8293c" containerName="authorino" Jul 10 14:58:02.968549 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:02.968402 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e47d2a1-b90f-4942-b9c3-7aa9a0f8293c" containerName="authorino" Jul 10 14:58:02.968549 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:02.968460 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e47d2a1-b90f-4942-b9c3-7aa9a0f8293c" containerName="authorino" Jul 10 14:58:02.971308 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:02.971290 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh" Jul 10 14:58:02.973944 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:02.973922 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-eec6a163-openshift-default-dockercfg-pl64c\"" Jul 10 14:58:02.993049 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:02.993020 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh"] Jul 10 14:58:03.091117 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:03.091083 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7e6e6be-6352-4ecb-861e-cdf3da7aaafb-credential-socket\") pod \"e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh\" (UID: \"a7e6e6be-6352-4ecb-861e-cdf3da7aaafb\") " pod="openshift-ingress/e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh" Jul 10 14:58:03.091295 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:03.091133 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7e6e6be-6352-4ecb-861e-cdf3da7aaafb-istio-data\") pod \"e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh\" (UID: \"a7e6e6be-6352-4ecb-861e-cdf3da7aaafb\") " pod="openshift-ingress/e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh" Jul 10 14:58:03.091295 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:03.091159 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7e6e6be-6352-4ecb-861e-cdf3da7aaafb-istio-token\") pod \"e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh\" (UID: \"a7e6e6be-6352-4ecb-861e-cdf3da7aaafb\") " pod="openshift-ingress/e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh" Jul 10 14:58:03.091295 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:03.091233 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7e6e6be-6352-4ecb-861e-cdf3da7aaafb-istio-podinfo\") pod \"e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh\" (UID: \"a7e6e6be-6352-4ecb-861e-cdf3da7aaafb\") " pod="openshift-ingress/e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh" Jul 10 14:58:03.091295 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:03.091260 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7e6e6be-6352-4ecb-861e-cdf3da7aaafb-workload-certs\") pod \"e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh\" (UID: \"a7e6e6be-6352-4ecb-861e-cdf3da7aaafb\") " pod="openshift-ingress/e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh" Jul 10 14:58:03.091295 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:03.091296 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7e6e6be-6352-4ecb-861e-cdf3da7aaafb-istiod-ca-cert\") pod \"e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh\" (UID: \"a7e6e6be-6352-4ecb-861e-cdf3da7aaafb\") " pod="openshift-ingress/e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh" Jul 10 14:58:03.091603 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:03.091395 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7e6e6be-6352-4ecb-861e-cdf3da7aaafb-workload-socket\") pod \"e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh\" (UID: \"a7e6e6be-6352-4ecb-861e-cdf3da7aaafb\") " pod="openshift-ingress/e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh" Jul 10 14:58:03.091603 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:03.091431 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7e6e6be-6352-4ecb-861e-cdf3da7aaafb-istio-envoy\") pod \"e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh\" (UID: \"a7e6e6be-6352-4ecb-861e-cdf3da7aaafb\") " pod="openshift-ingress/e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh" Jul 10 14:58:03.091603 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:03.091486 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zdmz\" (UniqueName: \"kubernetes.io/projected/a7e6e6be-6352-4ecb-861e-cdf3da7aaafb-kube-api-access-8zdmz\") pod \"e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh\" (UID: \"a7e6e6be-6352-4ecb-861e-cdf3da7aaafb\") " pod="openshift-ingress/e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh" Jul 10 14:58:03.192385 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:03.192350 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7e6e6be-6352-4ecb-861e-cdf3da7aaafb-istio-data\") pod \"e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh\" (UID: \"a7e6e6be-6352-4ecb-861e-cdf3da7aaafb\") " pod="openshift-ingress/e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh" Jul 10 14:58:03.192385 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:03.192384 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7e6e6be-6352-4ecb-861e-cdf3da7aaafb-istio-token\") pod \"e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh\" (UID: \"a7e6e6be-6352-4ecb-861e-cdf3da7aaafb\") " pod="openshift-ingress/e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh" Jul 10 14:58:03.192613 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:03.192405 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7e6e6be-6352-4ecb-861e-cdf3da7aaafb-istio-podinfo\") pod \"e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh\" (UID: \"a7e6e6be-6352-4ecb-861e-cdf3da7aaafb\") " pod="openshift-ingress/e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh" Jul 10 14:58:03.192613 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:03.192423 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7e6e6be-6352-4ecb-861e-cdf3da7aaafb-workload-certs\") pod \"e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh\" (UID: \"a7e6e6be-6352-4ecb-861e-cdf3da7aaafb\") " pod="openshift-ingress/e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh" Jul 10 14:58:03.192613 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:03.192449 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7e6e6be-6352-4ecb-861e-cdf3da7aaafb-istiod-ca-cert\") pod \"e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh\" (UID: \"a7e6e6be-6352-4ecb-861e-cdf3da7aaafb\") " pod="openshift-ingress/e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh" Jul 10 14:58:03.192613 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:03.192476 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7e6e6be-6352-4ecb-861e-cdf3da7aaafb-workload-socket\") pod \"e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh\" (UID: \"a7e6e6be-6352-4ecb-861e-cdf3da7aaafb\") " pod="openshift-ingress/e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh" Jul 10 14:58:03.192613 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:03.192494 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7e6e6be-6352-4ecb-861e-cdf3da7aaafb-istio-envoy\") pod \"e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh\" (UID: \"a7e6e6be-6352-4ecb-861e-cdf3da7aaafb\") " pod="openshift-ingress/e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh" Jul 10 14:58:03.192613 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:03.192524 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zdmz\" (UniqueName: \"kubernetes.io/projected/a7e6e6be-6352-4ecb-861e-cdf3da7aaafb-kube-api-access-8zdmz\") pod \"e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh\" (UID: \"a7e6e6be-6352-4ecb-861e-cdf3da7aaafb\") " pod="openshift-ingress/e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh" Jul 10 14:58:03.192613 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:03.192564 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7e6e6be-6352-4ecb-861e-cdf3da7aaafb-credential-socket\") pod \"e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh\" (UID: \"a7e6e6be-6352-4ecb-861e-cdf3da7aaafb\") " pod="openshift-ingress/e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh" Jul 10 14:58:03.192999 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:03.192746 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7e6e6be-6352-4ecb-861e-cdf3da7aaafb-istio-data\") pod \"e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh\" (UID: \"a7e6e6be-6352-4ecb-861e-cdf3da7aaafb\") " pod="openshift-ingress/e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh" Jul 10 14:58:03.192999 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:03.192874 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7e6e6be-6352-4ecb-861e-cdf3da7aaafb-workload-certs\") pod \"e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh\" (UID: \"a7e6e6be-6352-4ecb-861e-cdf3da7aaafb\") " pod="openshift-ingress/e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh" Jul 10 14:58:03.192999 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:03.192917 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7e6e6be-6352-4ecb-861e-cdf3da7aaafb-credential-socket\") pod \"e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh\" (UID: \"a7e6e6be-6352-4ecb-861e-cdf3da7aaafb\") " pod="openshift-ingress/e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh" Jul 10 14:58:03.193148 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:03.193069 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7e6e6be-6352-4ecb-861e-cdf3da7aaafb-workload-socket\") pod \"e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh\" (UID: \"a7e6e6be-6352-4ecb-861e-cdf3da7aaafb\") " pod="openshift-ingress/e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh" Jul 10 14:58:03.193264 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:03.193242 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7e6e6be-6352-4ecb-861e-cdf3da7aaafb-istiod-ca-cert\") pod \"e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh\" (UID: \"a7e6e6be-6352-4ecb-861e-cdf3da7aaafb\") " pod="openshift-ingress/e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh" Jul 10 14:58:03.194984 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:03.194957 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7e6e6be-6352-4ecb-861e-cdf3da7aaafb-istio-envoy\") pod \"e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh\" (UID: \"a7e6e6be-6352-4ecb-861e-cdf3da7aaafb\") " pod="openshift-ingress/e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh" Jul 10 14:58:03.195095 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:03.195003 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7e6e6be-6352-4ecb-861e-cdf3da7aaafb-istio-podinfo\") pod \"e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh\" (UID: \"a7e6e6be-6352-4ecb-861e-cdf3da7aaafb\") " pod="openshift-ingress/e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh" Jul 10 14:58:03.200177 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:03.200153 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7e6e6be-6352-4ecb-861e-cdf3da7aaafb-istio-token\") pod \"e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh\" (UID: \"a7e6e6be-6352-4ecb-861e-cdf3da7aaafb\") " pod="openshift-ingress/e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh" Jul 10 14:58:03.200844 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:03.200825 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zdmz\" (UniqueName: \"kubernetes.io/projected/a7e6e6be-6352-4ecb-861e-cdf3da7aaafb-kube-api-access-8zdmz\") pod \"e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh\" (UID: \"a7e6e6be-6352-4ecb-861e-cdf3da7aaafb\") " pod="openshift-ingress/e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh" Jul 10 14:58:03.281440 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:03.281380 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh" Jul 10 14:58:03.422807 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:03.422783 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh"] Jul 10 14:58:03.425050 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:58:03.425020 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7e6e6be_6352_4ecb_861e_cdf3da7aaafb.slice/crio-e28771019fea9857e547cdb12e127c2cdc26d43905bfd3fb5c70856fd15d4437 WatchSource:0}: Error finding container e28771019fea9857e547cdb12e127c2cdc26d43905bfd3fb5c70856fd15d4437: Status 404 returned error can't find the container with id e28771019fea9857e547cdb12e127c2cdc26d43905bfd3fb5c70856fd15d4437 Jul 10 14:58:03.427286 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:03.427256 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 14:58:03.427349 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:03.427320 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 14:58:03.427394 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:03.427348 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 14:58:03.530944 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:03.530905 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh" event={"ID":"a7e6e6be-6352-4ecb-861e-cdf3da7aaafb","Type":"ContainerStarted","Data":"f70967b96af70b4c7bc90f8e385b208d003fd145208fd392016e4c08c6dee584"} Jul 10 14:58:03.531097 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:03.530975 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh" event={"ID":"a7e6e6be-6352-4ecb-861e-cdf3da7aaafb","Type":"ContainerStarted","Data":"e28771019fea9857e547cdb12e127c2cdc26d43905bfd3fb5c70856fd15d4437"} Jul 10 14:58:03.551283 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:03.551188 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh" podStartSLOduration=1.551168896 podStartE2EDuration="1.551168896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 14:58:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:58:03.549050307 +0000 UTC m=+1629.300422029" watchObservedRunningTime="2026-07-10 14:58:03.551168896 +0000 UTC m=+1629.302540594" Jul 10 14:58:04.282359 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:04.282328 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh" Jul 10 14:58:04.286878 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:04.286855 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh" Jul 10 14:58:04.535882 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:04.535800 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh" Jul 10 14:58:04.536631 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:04.536615 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh" Jul 10 14:58:09.509103 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:09.509068 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh"] Jul 10 14:58:09.509488 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:09.509267 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh" podUID="a7e6e6be-6352-4ecb-861e-cdf3da7aaafb" containerName="istio-proxy" containerID="cri-o://f70967b96af70b4c7bc90f8e385b208d003fd145208fd392016e4c08c6dee584" gracePeriod=30 Jul 10 14:58:14.767014 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:14.766988 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh" Jul 10 14:58:14.897334 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:14.897305 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7e6e6be-6352-4ecb-861e-cdf3da7aaafb-istio-token\") pod \"a7e6e6be-6352-4ecb-861e-cdf3da7aaafb\" (UID: \"a7e6e6be-6352-4ecb-861e-cdf3da7aaafb\") " Jul 10 14:58:14.897505 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:14.897348 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7e6e6be-6352-4ecb-861e-cdf3da7aaafb-credential-socket\") pod \"a7e6e6be-6352-4ecb-861e-cdf3da7aaafb\" (UID: \"a7e6e6be-6352-4ecb-861e-cdf3da7aaafb\") " Jul 10 14:58:14.897505 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:14.897366 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8zdmz\" (UniqueName: \"kubernetes.io/projected/a7e6e6be-6352-4ecb-861e-cdf3da7aaafb-kube-api-access-8zdmz\") pod \"a7e6e6be-6352-4ecb-861e-cdf3da7aaafb\" (UID: \"a7e6e6be-6352-4ecb-861e-cdf3da7aaafb\") " Jul 10 14:58:14.897505 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:14.897393 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7e6e6be-6352-4ecb-861e-cdf3da7aaafb-istiod-ca-cert\") pod \"a7e6e6be-6352-4ecb-861e-cdf3da7aaafb\" (UID: \"a7e6e6be-6352-4ecb-861e-cdf3da7aaafb\") " Jul 10 14:58:14.897505 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:14.897415 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7e6e6be-6352-4ecb-861e-cdf3da7aaafb-istio-envoy\") pod \"a7e6e6be-6352-4ecb-861e-cdf3da7aaafb\" (UID: \"a7e6e6be-6352-4ecb-861e-cdf3da7aaafb\") " Jul 10 14:58:14.897645 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:14.897512 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7e6e6be-6352-4ecb-861e-cdf3da7aaafb-workload-certs\") pod \"a7e6e6be-6352-4ecb-861e-cdf3da7aaafb\" (UID: \"a7e6e6be-6352-4ecb-861e-cdf3da7aaafb\") " Jul 10 14:58:14.897645 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:14.897553 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7e6e6be-6352-4ecb-861e-cdf3da7aaafb-istio-podinfo\") pod \"a7e6e6be-6352-4ecb-861e-cdf3da7aaafb\" (UID: \"a7e6e6be-6352-4ecb-861e-cdf3da7aaafb\") " Jul 10 14:58:14.897645 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:14.897587 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7e6e6be-6352-4ecb-861e-cdf3da7aaafb-workload-socket\") pod \"a7e6e6be-6352-4ecb-861e-cdf3da7aaafb\" (UID: \"a7e6e6be-6352-4ecb-861e-cdf3da7aaafb\") " Jul 10 14:58:14.897645 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:14.897610 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7e6e6be-6352-4ecb-861e-cdf3da7aaafb-istio-data\") pod \"a7e6e6be-6352-4ecb-861e-cdf3da7aaafb\" (UID: \"a7e6e6be-6352-4ecb-861e-cdf3da7aaafb\") " Jul 10 14:58:14.901664 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:14.901642 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7e6e6be-6352-4ecb-861e-cdf3da7aaafb-istio-token" (OuterVolumeSpecName: "istio-token") pod "a7e6e6be-6352-4ecb-861e-cdf3da7aaafb" (UID: "a7e6e6be-6352-4ecb-861e-cdf3da7aaafb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 14:58:14.901945 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:14.901913 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7e6e6be-6352-4ecb-861e-cdf3da7aaafb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a7e6e6be-6352-4ecb-861e-cdf3da7aaafb" (UID: "a7e6e6be-6352-4ecb-861e-cdf3da7aaafb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:58:14.902111 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:14.902039 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7e6e6be-6352-4ecb-861e-cdf3da7aaafb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a7e6e6be-6352-4ecb-861e-cdf3da7aaafb" (UID: "a7e6e6be-6352-4ecb-861e-cdf3da7aaafb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:58:14.902197 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:14.902051 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7e6e6be-6352-4ecb-861e-cdf3da7aaafb-istio-data" (OuterVolumeSpecName: "istio-data") pod "a7e6e6be-6352-4ecb-861e-cdf3da7aaafb" (UID: "a7e6e6be-6352-4ecb-861e-cdf3da7aaafb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:58:14.902197 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:14.902175 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7e6e6be-6352-4ecb-861e-cdf3da7aaafb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a7e6e6be-6352-4ecb-861e-cdf3da7aaafb" (UID: "a7e6e6be-6352-4ecb-861e-cdf3da7aaafb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:58:14.902318 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:14.902198 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7e6e6be-6352-4ecb-861e-cdf3da7aaafb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a7e6e6be-6352-4ecb-861e-cdf3da7aaafb" (UID: "a7e6e6be-6352-4ecb-861e-cdf3da7aaafb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 14:58:14.903700 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:14.903674 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7e6e6be-6352-4ecb-861e-cdf3da7aaafb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a7e6e6be-6352-4ecb-861e-cdf3da7aaafb" (UID: "a7e6e6be-6352-4ecb-861e-cdf3da7aaafb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:58:14.903830 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:14.903807 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7e6e6be-6352-4ecb-861e-cdf3da7aaafb-kube-api-access-8zdmz" (OuterVolumeSpecName: "kube-api-access-8zdmz") pod "a7e6e6be-6352-4ecb-861e-cdf3da7aaafb" (UID: "a7e6e6be-6352-4ecb-861e-cdf3da7aaafb"). InnerVolumeSpecName "kube-api-access-8zdmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 14:58:14.903890 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:14.903873 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7e6e6be-6352-4ecb-861e-cdf3da7aaafb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a7e6e6be-6352-4ecb-861e-cdf3da7aaafb" (UID: "a7e6e6be-6352-4ecb-861e-cdf3da7aaafb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 14:58:14.998157 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:14.998128 2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7e6e6be-6352-4ecb-861e-cdf3da7aaafb-istio-token\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 14:58:14.998157 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:14.998151 2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7e6e6be-6352-4ecb-861e-cdf3da7aaafb-credential-socket\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 14:58:14.998157 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:14.998160 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zdmz\" (UniqueName: \"kubernetes.io/projected/a7e6e6be-6352-4ecb-861e-cdf3da7aaafb-kube-api-access-8zdmz\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 14:58:14.998445 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:14.998170 2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7e6e6be-6352-4ecb-861e-cdf3da7aaafb-istiod-ca-cert\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 14:58:14.998445 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:14.998179 2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7e6e6be-6352-4ecb-861e-cdf3da7aaafb-istio-envoy\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 14:58:14.998445 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:14.998186 2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7e6e6be-6352-4ecb-861e-cdf3da7aaafb-workload-certs\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 14:58:14.998445 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:14.998194 2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7e6e6be-6352-4ecb-861e-cdf3da7aaafb-istio-podinfo\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 14:58:14.998445 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:14.998202 2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7e6e6be-6352-4ecb-861e-cdf3da7aaafb-workload-socket\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 14:58:14.998445 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:14.998209 2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7e6e6be-6352-4ecb-861e-cdf3da7aaafb-istio-data\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 14:58:15.591655 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:15.591626 2569 generic.go:356] "Generic (PLEG): container finished" podID="a7e6e6be-6352-4ecb-861e-cdf3da7aaafb" containerID="f70967b96af70b4c7bc90f8e385b208d003fd145208fd392016e4c08c6dee584" exitCode=0 Jul 10 14:58:15.591850 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:15.591677 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh" Jul 10 14:58:15.591850 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:15.591714 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh" event={"ID":"a7e6e6be-6352-4ecb-861e-cdf3da7aaafb","Type":"ContainerDied","Data":"f70967b96af70b4c7bc90f8e385b208d003fd145208fd392016e4c08c6dee584"} Jul 10 14:58:15.591850 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:15.591785 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh" event={"ID":"a7e6e6be-6352-4ecb-861e-cdf3da7aaafb","Type":"ContainerDied","Data":"e28771019fea9857e547cdb12e127c2cdc26d43905bfd3fb5c70856fd15d4437"} Jul 10 14:58:15.591850 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:15.591810 2569 scope.go:117] "RemoveContainer" containerID="f70967b96af70b4c7bc90f8e385b208d003fd145208fd392016e4c08c6dee584" Jul 10 14:58:15.600527 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:15.600492 2569 scope.go:117] "RemoveContainer" containerID="f70967b96af70b4c7bc90f8e385b208d003fd145208fd392016e4c08c6dee584" Jul 10 14:58:15.600807 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:58:15.600785 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f70967b96af70b4c7bc90f8e385b208d003fd145208fd392016e4c08c6dee584\": container with ID starting with f70967b96af70b4c7bc90f8e385b208d003fd145208fd392016e4c08c6dee584 not found: ID does not exist" containerID="f70967b96af70b4c7bc90f8e385b208d003fd145208fd392016e4c08c6dee584" Jul 10 14:58:15.600879 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:15.600815 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f70967b96af70b4c7bc90f8e385b208d003fd145208fd392016e4c08c6dee584"} err="failed to get container status \"f70967b96af70b4c7bc90f8e385b208d003fd145208fd392016e4c08c6dee584\": rpc error: code = NotFound desc = could not find container \"f70967b96af70b4c7bc90f8e385b208d003fd145208fd392016e4c08c6dee584\": container with ID starting with f70967b96af70b4c7bc90f8e385b208d003fd145208fd392016e4c08c6dee584 not found: ID does not exist" Jul 10 14:58:15.611869 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:15.611847 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh"] Jul 10 14:58:15.615217 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:15.615199 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-eec6a163-openshift-default-f9bb6d84c-6ptvh"] Jul 10 14:58:16.931977 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:16.931943 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7e6e6be-6352-4ecb-861e-cdf3da7aaafb" path="/var/lib/kubelet/pods/a7e6e6be-6352-4ecb-861e-cdf3da7aaafb/volumes" Jul 10 14:58:34.180381 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:34.180348 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s"] Jul 10 14:58:34.180872 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:34.180743 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7e6e6be-6352-4ecb-861e-cdf3da7aaafb" containerName="istio-proxy" Jul 10 14:58:34.180872 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:34.180772 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7e6e6be-6352-4ecb-861e-cdf3da7aaafb" containerName="istio-proxy" Jul 10 14:58:34.180872 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:34.180837 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7e6e6be-6352-4ecb-861e-cdf3da7aaafb" containerName="istio-proxy" Jul 10 14:58:34.183732 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:34.183710 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s" Jul 10 14:58:34.187824 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:34.187804 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-5d3267f0-openshift-default-dockercfg-gt4fg\"" Jul 10 14:58:34.195843 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:34.195821 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s"] Jul 10 14:58:34.264599 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:34.264575 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba021c11-13f6-4ccc-8796-48dc67ee546c-credential-socket\") pod \"e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s\" (UID: \"ba021c11-13f6-4ccc-8796-48dc67ee546c\") " pod="openshift-ingress/e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s" Jul 10 14:58:34.264696 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:34.264606 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba021c11-13f6-4ccc-8796-48dc67ee546c-workload-socket\") pod \"e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s\" (UID: \"ba021c11-13f6-4ccc-8796-48dc67ee546c\") " pod="openshift-ingress/e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s" Jul 10 14:58:34.264696 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:34.264624 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgxv4\" (UniqueName: \"kubernetes.io/projected/ba021c11-13f6-4ccc-8796-48dc67ee546c-kube-api-access-qgxv4\") pod \"e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s\" (UID: \"ba021c11-13f6-4ccc-8796-48dc67ee546c\") " pod="openshift-ingress/e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s" Jul 10 14:58:34.264696 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:34.264679 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba021c11-13f6-4ccc-8796-48dc67ee546c-istiod-ca-cert\") pod \"e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s\" (UID: \"ba021c11-13f6-4ccc-8796-48dc67ee546c\") " pod="openshift-ingress/e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s" Jul 10 14:58:34.264844 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:34.264730 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba021c11-13f6-4ccc-8796-48dc67ee546c-istio-envoy\") pod \"e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s\" (UID: \"ba021c11-13f6-4ccc-8796-48dc67ee546c\") " pod="openshift-ingress/e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s" Jul 10 14:58:34.264844 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:34.264779 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba021c11-13f6-4ccc-8796-48dc67ee546c-istio-data\") pod \"e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s\" (UID: \"ba021c11-13f6-4ccc-8796-48dc67ee546c\") " pod="openshift-ingress/e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s" Jul 10 14:58:34.264844 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:34.264836 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba021c11-13f6-4ccc-8796-48dc67ee546c-istio-token\") pod \"e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s\" (UID: \"ba021c11-13f6-4ccc-8796-48dc67ee546c\") " pod="openshift-ingress/e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s" Jul 10 14:58:34.264939 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:34.264854 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba021c11-13f6-4ccc-8796-48dc67ee546c-workload-certs\") pod \"e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s\" (UID: \"ba021c11-13f6-4ccc-8796-48dc67ee546c\") " pod="openshift-ingress/e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s" Jul 10 14:58:34.264939 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:34.264874 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba021c11-13f6-4ccc-8796-48dc67ee546c-istio-podinfo\") pod \"e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s\" (UID: \"ba021c11-13f6-4ccc-8796-48dc67ee546c\") " pod="openshift-ingress/e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s" Jul 10 14:58:34.365537 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:34.365502 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba021c11-13f6-4ccc-8796-48dc67ee546c-credential-socket\") pod \"e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s\" (UID: \"ba021c11-13f6-4ccc-8796-48dc67ee546c\") " pod="openshift-ingress/e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s" Jul 10 14:58:34.365537 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:34.365540 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba021c11-13f6-4ccc-8796-48dc67ee546c-workload-socket\") pod \"e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s\" (UID: \"ba021c11-13f6-4ccc-8796-48dc67ee546c\") " pod="openshift-ingress/e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s" Jul 10 14:58:34.365805 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:34.365556 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgxv4\" (UniqueName: \"kubernetes.io/projected/ba021c11-13f6-4ccc-8796-48dc67ee546c-kube-api-access-qgxv4\") pod \"e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s\" (UID: \"ba021c11-13f6-4ccc-8796-48dc67ee546c\") " pod="openshift-ingress/e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s" Jul 10 14:58:34.365805 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:34.365588 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba021c11-13f6-4ccc-8796-48dc67ee546c-istiod-ca-cert\") pod \"e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s\" (UID: \"ba021c11-13f6-4ccc-8796-48dc67ee546c\") " pod="openshift-ingress/e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s" Jul 10 14:58:34.365805 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:34.365614 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba021c11-13f6-4ccc-8796-48dc67ee546c-istio-envoy\") pod \"e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s\" (UID: \"ba021c11-13f6-4ccc-8796-48dc67ee546c\") " pod="openshift-ingress/e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s" Jul 10 14:58:34.365805 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:34.365631 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba021c11-13f6-4ccc-8796-48dc67ee546c-istio-data\") pod \"e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s\" (UID: \"ba021c11-13f6-4ccc-8796-48dc67ee546c\") " pod="openshift-ingress/e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s" Jul 10 14:58:34.365805 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:34.365660 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba021c11-13f6-4ccc-8796-48dc67ee546c-istio-token\") pod \"e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s\" (UID: \"ba021c11-13f6-4ccc-8796-48dc67ee546c\") " pod="openshift-ingress/e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s" Jul 10 14:58:34.365805 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:34.365678 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba021c11-13f6-4ccc-8796-48dc67ee546c-workload-certs\") pod \"e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s\" (UID: \"ba021c11-13f6-4ccc-8796-48dc67ee546c\") " pod="openshift-ingress/e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s" Jul 10 14:58:34.365805 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:34.365702 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba021c11-13f6-4ccc-8796-48dc67ee546c-istio-podinfo\") pod \"e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s\" (UID: \"ba021c11-13f6-4ccc-8796-48dc67ee546c\") " pod="openshift-ingress/e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s" Jul 10 14:58:34.366236 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:34.365972 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba021c11-13f6-4ccc-8796-48dc67ee546c-credential-socket\") pod \"e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s\" (UID: \"ba021c11-13f6-4ccc-8796-48dc67ee546c\") " pod="openshift-ingress/e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s" Jul 10 14:58:34.366236 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:34.366103 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba021c11-13f6-4ccc-8796-48dc67ee546c-istio-data\") pod \"e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s\" (UID: \"ba021c11-13f6-4ccc-8796-48dc67ee546c\") " pod="openshift-ingress/e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s" Jul 10 14:58:34.366236 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:34.366198 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba021c11-13f6-4ccc-8796-48dc67ee546c-workload-certs\") pod \"e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s\" (UID: \"ba021c11-13f6-4ccc-8796-48dc67ee546c\") " pod="openshift-ingress/e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s" Jul 10 14:58:34.366379 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:34.366302 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba021c11-13f6-4ccc-8796-48dc67ee546c-workload-socket\") pod \"e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s\" (UID: \"ba021c11-13f6-4ccc-8796-48dc67ee546c\") " pod="openshift-ingress/e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s" Jul 10 14:58:34.366511 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:34.366490 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba021c11-13f6-4ccc-8796-48dc67ee546c-istiod-ca-cert\") pod \"e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s\" (UID: \"ba021c11-13f6-4ccc-8796-48dc67ee546c\") " pod="openshift-ingress/e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s" Jul 10 14:58:34.367990 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:34.367968 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba021c11-13f6-4ccc-8796-48dc67ee546c-istio-envoy\") pod \"e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s\" (UID: \"ba021c11-13f6-4ccc-8796-48dc67ee546c\") " pod="openshift-ingress/e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s" Jul 10 14:58:34.368258 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:34.368239 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba021c11-13f6-4ccc-8796-48dc67ee546c-istio-podinfo\") pod \"e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s\" (UID: \"ba021c11-13f6-4ccc-8796-48dc67ee546c\") " pod="openshift-ingress/e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s" Jul 10 14:58:34.373975 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:34.373953 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba021c11-13f6-4ccc-8796-48dc67ee546c-istio-token\") pod \"e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s\" (UID: \"ba021c11-13f6-4ccc-8796-48dc67ee546c\") " pod="openshift-ingress/e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s" Jul 10 14:58:34.374173 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:34.374153 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgxv4\" (UniqueName: \"kubernetes.io/projected/ba021c11-13f6-4ccc-8796-48dc67ee546c-kube-api-access-qgxv4\") pod \"e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s\" (UID: \"ba021c11-13f6-4ccc-8796-48dc67ee546c\") " pod="openshift-ingress/e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s" Jul 10 14:58:34.496204 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:34.496102 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s" Jul 10 14:58:34.638036 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:34.638008 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s"] Jul 10 14:58:34.639828 ip-10-0-140-35 kubenswrapper[2569]: W0710 14:58:34.639804 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba021c11_13f6_4ccc_8796_48dc67ee546c.slice/crio-510408ccef7fa228905a9c882b5cd56d5bac213f56f4c125243e7898a36c2494 WatchSource:0}: Error finding container 510408ccef7fa228905a9c882b5cd56d5bac213f56f4c125243e7898a36c2494: Status 404 returned error can't find the container with id 510408ccef7fa228905a9c882b5cd56d5bac213f56f4c125243e7898a36c2494 Jul 10 14:58:34.642011 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:34.641980 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 14:58:34.642088 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:34.642048 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 14:58:34.642088 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:34.642079 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 14:58:34.684955 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:34.684920 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s" event={"ID":"ba021c11-13f6-4ccc-8796-48dc67ee546c","Type":"ContainerStarted","Data":"510408ccef7fa228905a9c882b5cd56d5bac213f56f4c125243e7898a36c2494"} Jul 10 14:58:35.691543 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:35.691503 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s" event={"ID":"ba021c11-13f6-4ccc-8796-48dc67ee546c","Type":"ContainerStarted","Data":"5a0cfba539ba9e6e83de347edee502e847008987dfae85cf88317a67bc8a7cb6"} Jul 10 14:58:35.715972 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:35.715920 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s" podStartSLOduration=1.71590125 podStartE2EDuration="1.71590125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 14:58:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 14:58:35.714786248 +0000 UTC m=+1661.466157943" watchObservedRunningTime="2026-07-10 14:58:35.71590125 +0000 UTC m=+1661.467272948" Jul 10 14:58:36.496530 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:36.496492 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s" Jul 10 14:58:36.501576 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:36.501552 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s" Jul 10 14:58:36.696973 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:36.696939 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s" Jul 10 14:58:36.697987 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:36.697955 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s" Jul 10 14:58:39.972689 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:39.972645 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s"] Jul 10 14:58:39.973182 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:39.972854 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s" podUID="ba021c11-13f6-4ccc-8796-48dc67ee546c" containerName="istio-proxy" containerID="cri-o://5a0cfba539ba9e6e83de347edee502e847008987dfae85cf88317a67bc8a7cb6" gracePeriod=30 Jul 10 14:58:45.229159 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:45.229126 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s" Jul 10 14:58:45.365931 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:45.365842 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba021c11-13f6-4ccc-8796-48dc67ee546c-credential-socket\") pod \"ba021c11-13f6-4ccc-8796-48dc67ee546c\" (UID: \"ba021c11-13f6-4ccc-8796-48dc67ee546c\") " Jul 10 14:58:45.365931 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:45.365887 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba021c11-13f6-4ccc-8796-48dc67ee546c-istiod-ca-cert\") pod \"ba021c11-13f6-4ccc-8796-48dc67ee546c\" (UID: \"ba021c11-13f6-4ccc-8796-48dc67ee546c\") " Jul 10 14:58:45.365931 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:45.365920 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba021c11-13f6-4ccc-8796-48dc67ee546c-workload-socket\") pod \"ba021c11-13f6-4ccc-8796-48dc67ee546c\" (UID: \"ba021c11-13f6-4ccc-8796-48dc67ee546c\") " Jul 10 14:58:45.366204 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:45.365986 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba021c11-13f6-4ccc-8796-48dc67ee546c-istio-data\") pod \"ba021c11-13f6-4ccc-8796-48dc67ee546c\" (UID: \"ba021c11-13f6-4ccc-8796-48dc67ee546c\") " Jul 10 14:58:45.366204 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:45.366028 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qgxv4\" (UniqueName: \"kubernetes.io/projected/ba021c11-13f6-4ccc-8796-48dc67ee546c-kube-api-access-qgxv4\") pod \"ba021c11-13f6-4ccc-8796-48dc67ee546c\" (UID: \"ba021c11-13f6-4ccc-8796-48dc67ee546c\") " Jul 10 14:58:45.366204 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:45.366051 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba021c11-13f6-4ccc-8796-48dc67ee546c-istio-token\") pod \"ba021c11-13f6-4ccc-8796-48dc67ee546c\" (UID: \"ba021c11-13f6-4ccc-8796-48dc67ee546c\") " Jul 10 14:58:45.366204 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:45.366073 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba021c11-13f6-4ccc-8796-48dc67ee546c-istio-podinfo\") pod \"ba021c11-13f6-4ccc-8796-48dc67ee546c\" (UID: \"ba021c11-13f6-4ccc-8796-48dc67ee546c\") " Jul 10 14:58:45.366204 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:45.366096 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba021c11-13f6-4ccc-8796-48dc67ee546c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ba021c11-13f6-4ccc-8796-48dc67ee546c" (UID: "ba021c11-13f6-4ccc-8796-48dc67ee546c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:58:45.366204 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:45.366113 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba021c11-13f6-4ccc-8796-48dc67ee546c-workload-certs\") pod \"ba021c11-13f6-4ccc-8796-48dc67ee546c\" (UID: \"ba021c11-13f6-4ccc-8796-48dc67ee546c\") " Jul 10 14:58:45.366534 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:45.366206 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba021c11-13f6-4ccc-8796-48dc67ee546c-istio-envoy\") pod \"ba021c11-13f6-4ccc-8796-48dc67ee546c\" (UID: \"ba021c11-13f6-4ccc-8796-48dc67ee546c\") " Jul 10 14:58:45.366534 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:45.366339 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba021c11-13f6-4ccc-8796-48dc67ee546c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ba021c11-13f6-4ccc-8796-48dc67ee546c" (UID: "ba021c11-13f6-4ccc-8796-48dc67ee546c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:58:45.366534 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:45.366417 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba021c11-13f6-4ccc-8796-48dc67ee546c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ba021c11-13f6-4ccc-8796-48dc67ee546c" (UID: "ba021c11-13f6-4ccc-8796-48dc67ee546c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:58:45.366534 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:45.366452 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba021c11-13f6-4ccc-8796-48dc67ee546c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ba021c11-13f6-4ccc-8796-48dc67ee546c" (UID: "ba021c11-13f6-4ccc-8796-48dc67ee546c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 14:58:45.366776 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:45.366580 2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba021c11-13f6-4ccc-8796-48dc67ee546c-workload-certs\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 14:58:45.366776 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:45.366598 2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba021c11-13f6-4ccc-8796-48dc67ee546c-credential-socket\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 14:58:45.366776 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:45.366613 2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba021c11-13f6-4ccc-8796-48dc67ee546c-istiod-ca-cert\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 14:58:45.366776 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:45.366627 2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba021c11-13f6-4ccc-8796-48dc67ee546c-workload-socket\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 14:58:45.367068 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:45.366974 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba021c11-13f6-4ccc-8796-48dc67ee546c-istio-data" (OuterVolumeSpecName: "istio-data") pod "ba021c11-13f6-4ccc-8796-48dc67ee546c" (UID: "ba021c11-13f6-4ccc-8796-48dc67ee546c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:58:45.368783 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:45.368731 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba021c11-13f6-4ccc-8796-48dc67ee546c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ba021c11-13f6-4ccc-8796-48dc67ee546c" (UID: "ba021c11-13f6-4ccc-8796-48dc67ee546c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 14:58:45.368903 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:45.368867 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba021c11-13f6-4ccc-8796-48dc67ee546c-kube-api-access-qgxv4" (OuterVolumeSpecName: "kube-api-access-qgxv4") pod "ba021c11-13f6-4ccc-8796-48dc67ee546c" (UID: "ba021c11-13f6-4ccc-8796-48dc67ee546c"). InnerVolumeSpecName "kube-api-access-qgxv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 14:58:45.368960 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:45.368891 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba021c11-13f6-4ccc-8796-48dc67ee546c-istio-token" (OuterVolumeSpecName: "istio-token") pod "ba021c11-13f6-4ccc-8796-48dc67ee546c" (UID: "ba021c11-13f6-4ccc-8796-48dc67ee546c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 14:58:45.368960 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:45.368905 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba021c11-13f6-4ccc-8796-48dc67ee546c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ba021c11-13f6-4ccc-8796-48dc67ee546c" (UID: "ba021c11-13f6-4ccc-8796-48dc67ee546c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 14:58:45.467982 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:45.467959 2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba021c11-13f6-4ccc-8796-48dc67ee546c-istio-envoy\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 14:58:45.467982 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:45.467979 2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba021c11-13f6-4ccc-8796-48dc67ee546c-istio-data\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 14:58:45.468129 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:45.467988 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgxv4\" (UniqueName: \"kubernetes.io/projected/ba021c11-13f6-4ccc-8796-48dc67ee546c-kube-api-access-qgxv4\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 14:58:45.468129 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:45.467997 2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba021c11-13f6-4ccc-8796-48dc67ee546c-istio-token\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 14:58:45.468129 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:45.468005 2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba021c11-13f6-4ccc-8796-48dc67ee546c-istio-podinfo\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 14:58:45.745539 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:45.745502 2569 generic.go:356] "Generic (PLEG): container finished" podID="ba021c11-13f6-4ccc-8796-48dc67ee546c" containerID="5a0cfba539ba9e6e83de347edee502e847008987dfae85cf88317a67bc8a7cb6" exitCode=0 Jul 10 14:58:45.745734 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:45.745550 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s" Jul 10 14:58:45.745734 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:45.745563 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s" event={"ID":"ba021c11-13f6-4ccc-8796-48dc67ee546c","Type":"ContainerDied","Data":"5a0cfba539ba9e6e83de347edee502e847008987dfae85cf88317a67bc8a7cb6"} Jul 10 14:58:45.745734 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:45.745605 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s" event={"ID":"ba021c11-13f6-4ccc-8796-48dc67ee546c","Type":"ContainerDied","Data":"510408ccef7fa228905a9c882b5cd56d5bac213f56f4c125243e7898a36c2494"} Jul 10 14:58:45.745734 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:45.745626 2569 scope.go:117] "RemoveContainer" containerID="5a0cfba539ba9e6e83de347edee502e847008987dfae85cf88317a67bc8a7cb6" Jul 10 14:58:45.754906 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:45.754888 2569 scope.go:117] "RemoveContainer" containerID="5a0cfba539ba9e6e83de347edee502e847008987dfae85cf88317a67bc8a7cb6" Jul 10 14:58:45.755174 ip-10-0-140-35 kubenswrapper[2569]: E0710 14:58:45.755154 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a0cfba539ba9e6e83de347edee502e847008987dfae85cf88317a67bc8a7cb6\": container with ID starting with 5a0cfba539ba9e6e83de347edee502e847008987dfae85cf88317a67bc8a7cb6 not found: ID does not exist" containerID="5a0cfba539ba9e6e83de347edee502e847008987dfae85cf88317a67bc8a7cb6" Jul 10 14:58:45.755235 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:45.755188 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a0cfba539ba9e6e83de347edee502e847008987dfae85cf88317a67bc8a7cb6"} err="failed to get container status \"5a0cfba539ba9e6e83de347edee502e847008987dfae85cf88317a67bc8a7cb6\": rpc error: code = NotFound desc = could not find container \"5a0cfba539ba9e6e83de347edee502e847008987dfae85cf88317a67bc8a7cb6\": container with ID starting with 5a0cfba539ba9e6e83de347edee502e847008987dfae85cf88317a67bc8a7cb6 not found: ID does not exist" Jul 10 14:58:45.766783 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:45.766741 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s"] Jul 10 14:58:45.770567 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:45.770548 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-5d3267f0-openshift-default-57b556796c-42n7s"] Jul 10 14:58:46.932702 ip-10-0-140-35 kubenswrapper[2569]: I0710 14:58:46.932667 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba021c11-13f6-4ccc-8796-48dc67ee546c" path="/var/lib/kubelet/pods/ba021c11-13f6-4ccc-8796-48dc67ee546c/volumes" Jul 10 15:00:54.793171 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:00:54.793138 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-n6pfx_296be890-64e9-4fb3-8c68-e4c2a7c8cc2c/console-operator/1.log" Jul 10 15:00:54.795631 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:00:54.795607 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-n6pfx_296be890-64e9-4fb3-8c68-e4c2a7c8cc2c/console-operator/1.log" Jul 10 15:00:54.799732 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:00:54.799710 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxvq2_016248a0-bbe2-44c9-8e51-5d4695ee6a52/ovn-acl-logging/0.log" Jul 10 15:00:54.802008 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:00:54.801990 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxvq2_016248a0-bbe2-44c9-8e51-5d4695ee6a52/ovn-acl-logging/0.log" Jul 10 15:01:23.189824 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:01:23.189780 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-f34a9b9d/e2e-lifecycle-model-f34a9b9d-kserve-57bb749cc9-qdm5w"] Jul 10 15:01:23.190375 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:01:23.190356 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba021c11-13f6-4ccc-8796-48dc67ee546c" containerName="istio-proxy" Jul 10 15:01:23.190418 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:01:23.190380 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba021c11-13f6-4ccc-8796-48dc67ee546c" containerName="istio-proxy" Jul 10 15:01:23.190488 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:01:23.190476 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba021c11-13f6-4ccc-8796-48dc67ee546c" containerName="istio-proxy" Jul 10 15:01:23.193937 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:01:23.193916 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f34a9b9d/e2e-lifecycle-model-f34a9b9d-kserve-57bb749cc9-qdm5w" Jul 10 15:01:23.197467 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:01:23.197448 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-f34a9b9d\"/\"e2e-lifecycle-model-f34a9b9d-kserve-self-signed-certs\"" Jul 10 15:01:23.197467 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:01:23.197462 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-f34a9b9d\"/\"default-dockercfg-h6vq6\"" Jul 10 15:01:23.197620 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:01:23.197491 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-f34a9b9d\"/\"openshift-service-ca.crt\"" Jul 10 15:01:23.197620 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:01:23.197463 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-f34a9b9d\"/\"kube-root-ca.crt\"" Jul 10 15:01:23.202090 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:01:23.202046 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-f34a9b9d/e2e-lifecycle-model-f34a9b9d-kserve-57bb749cc9-qdm5w"] Jul 10 15:01:23.304837 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:01:23.304802 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca072f79-80d2-4e12-a16e-80a5ecb257a6-dshm\") pod \"e2e-lifecycle-model-f34a9b9d-kserve-57bb749cc9-qdm5w\" (UID: \"ca072f79-80d2-4e12-a16e-80a5ecb257a6\") " pod="ai-tenant-e2e-mt-f34a9b9d/e2e-lifecycle-model-f34a9b9d-kserve-57bb749cc9-qdm5w" Jul 10 15:01:23.305008 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:01:23.304874 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca072f79-80d2-4e12-a16e-80a5ecb257a6-tmp-dir\") pod \"e2e-lifecycle-model-f34a9b9d-kserve-57bb749cc9-qdm5w\" (UID: \"ca072f79-80d2-4e12-a16e-80a5ecb257a6\") " pod="ai-tenant-e2e-mt-f34a9b9d/e2e-lifecycle-model-f34a9b9d-kserve-57bb749cc9-qdm5w" Jul 10 15:01:23.305008 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:01:23.304906 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca072f79-80d2-4e12-a16e-80a5ecb257a6-home\") pod \"e2e-lifecycle-model-f34a9b9d-kserve-57bb749cc9-qdm5w\" (UID: \"ca072f79-80d2-4e12-a16e-80a5ecb257a6\") " pod="ai-tenant-e2e-mt-f34a9b9d/e2e-lifecycle-model-f34a9b9d-kserve-57bb749cc9-qdm5w" Jul 10 15:01:23.305008 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:01:23.304938 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca072f79-80d2-4e12-a16e-80a5ecb257a6-tls-certs\") pod \"e2e-lifecycle-model-f34a9b9d-kserve-57bb749cc9-qdm5w\" (UID: \"ca072f79-80d2-4e12-a16e-80a5ecb257a6\") " pod="ai-tenant-e2e-mt-f34a9b9d/e2e-lifecycle-model-f34a9b9d-kserve-57bb749cc9-qdm5w" Jul 10 15:01:23.305008 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:01:23.304984 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8hlk\" (UniqueName: \"kubernetes.io/projected/ca072f79-80d2-4e12-a16e-80a5ecb257a6-kube-api-access-p8hlk\") pod \"e2e-lifecycle-model-f34a9b9d-kserve-57bb749cc9-qdm5w\" (UID: \"ca072f79-80d2-4e12-a16e-80a5ecb257a6\") " pod="ai-tenant-e2e-mt-f34a9b9d/e2e-lifecycle-model-f34a9b9d-kserve-57bb749cc9-qdm5w" Jul 10 15:01:23.305153 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:01:23.305035 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca072f79-80d2-4e12-a16e-80a5ecb257a6-model-cache\") pod \"e2e-lifecycle-model-f34a9b9d-kserve-57bb749cc9-qdm5w\" (UID: \"ca072f79-80d2-4e12-a16e-80a5ecb257a6\") " pod="ai-tenant-e2e-mt-f34a9b9d/e2e-lifecycle-model-f34a9b9d-kserve-57bb749cc9-qdm5w" Jul 10 15:01:23.406088 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:01:23.406048 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca072f79-80d2-4e12-a16e-80a5ecb257a6-dshm\") pod \"e2e-lifecycle-model-f34a9b9d-kserve-57bb749cc9-qdm5w\" (UID: \"ca072f79-80d2-4e12-a16e-80a5ecb257a6\") " pod="ai-tenant-e2e-mt-f34a9b9d/e2e-lifecycle-model-f34a9b9d-kserve-57bb749cc9-qdm5w" Jul 10 15:01:23.406276 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:01:23.406106 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca072f79-80d2-4e12-a16e-80a5ecb257a6-tmp-dir\") pod \"e2e-lifecycle-model-f34a9b9d-kserve-57bb749cc9-qdm5w\" (UID: \"ca072f79-80d2-4e12-a16e-80a5ecb257a6\") " pod="ai-tenant-e2e-mt-f34a9b9d/e2e-lifecycle-model-f34a9b9d-kserve-57bb749cc9-qdm5w" Jul 10 15:01:23.406276 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:01:23.406132 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca072f79-80d2-4e12-a16e-80a5ecb257a6-home\") pod \"e2e-lifecycle-model-f34a9b9d-kserve-57bb749cc9-qdm5w\" (UID: \"ca072f79-80d2-4e12-a16e-80a5ecb257a6\") " pod="ai-tenant-e2e-mt-f34a9b9d/e2e-lifecycle-model-f34a9b9d-kserve-57bb749cc9-qdm5w" Jul 10 15:01:23.406276 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:01:23.406156 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca072f79-80d2-4e12-a16e-80a5ecb257a6-tls-certs\") pod \"e2e-lifecycle-model-f34a9b9d-kserve-57bb749cc9-qdm5w\" (UID: \"ca072f79-80d2-4e12-a16e-80a5ecb257a6\") " pod="ai-tenant-e2e-mt-f34a9b9d/e2e-lifecycle-model-f34a9b9d-kserve-57bb749cc9-qdm5w" Jul 10 15:01:23.406276 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:01:23.406184 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8hlk\" (UniqueName: \"kubernetes.io/projected/ca072f79-80d2-4e12-a16e-80a5ecb257a6-kube-api-access-p8hlk\") pod \"e2e-lifecycle-model-f34a9b9d-kserve-57bb749cc9-qdm5w\" (UID: \"ca072f79-80d2-4e12-a16e-80a5ecb257a6\") " pod="ai-tenant-e2e-mt-f34a9b9d/e2e-lifecycle-model-f34a9b9d-kserve-57bb749cc9-qdm5w" Jul 10 15:01:23.406276 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:01:23.406218 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca072f79-80d2-4e12-a16e-80a5ecb257a6-model-cache\") pod \"e2e-lifecycle-model-f34a9b9d-kserve-57bb749cc9-qdm5w\" (UID: \"ca072f79-80d2-4e12-a16e-80a5ecb257a6\") " pod="ai-tenant-e2e-mt-f34a9b9d/e2e-lifecycle-model-f34a9b9d-kserve-57bb749cc9-qdm5w" Jul 10 15:01:23.406600 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:01:23.406547 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca072f79-80d2-4e12-a16e-80a5ecb257a6-home\") pod \"e2e-lifecycle-model-f34a9b9d-kserve-57bb749cc9-qdm5w\" (UID: \"ca072f79-80d2-4e12-a16e-80a5ecb257a6\") " pod="ai-tenant-e2e-mt-f34a9b9d/e2e-lifecycle-model-f34a9b9d-kserve-57bb749cc9-qdm5w" Jul 10 15:01:23.406600 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:01:23.406574 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca072f79-80d2-4e12-a16e-80a5ecb257a6-tmp-dir\") pod \"e2e-lifecycle-model-f34a9b9d-kserve-57bb749cc9-qdm5w\" (UID: \"ca072f79-80d2-4e12-a16e-80a5ecb257a6\") " pod="ai-tenant-e2e-mt-f34a9b9d/e2e-lifecycle-model-f34a9b9d-kserve-57bb749cc9-qdm5w" Jul 10 15:01:23.406710 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:01:23.406648 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca072f79-80d2-4e12-a16e-80a5ecb257a6-model-cache\") pod \"e2e-lifecycle-model-f34a9b9d-kserve-57bb749cc9-qdm5w\" (UID: \"ca072f79-80d2-4e12-a16e-80a5ecb257a6\") " pod="ai-tenant-e2e-mt-f34a9b9d/e2e-lifecycle-model-f34a9b9d-kserve-57bb749cc9-qdm5w" Jul 10 15:01:23.408262 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:01:23.408234 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca072f79-80d2-4e12-a16e-80a5ecb257a6-dshm\") pod \"e2e-lifecycle-model-f34a9b9d-kserve-57bb749cc9-qdm5w\" (UID: \"ca072f79-80d2-4e12-a16e-80a5ecb257a6\") " pod="ai-tenant-e2e-mt-f34a9b9d/e2e-lifecycle-model-f34a9b9d-kserve-57bb749cc9-qdm5w" Jul 10 15:01:23.408621 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:01:23.408601 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca072f79-80d2-4e12-a16e-80a5ecb257a6-tls-certs\") pod \"e2e-lifecycle-model-f34a9b9d-kserve-57bb749cc9-qdm5w\" (UID: \"ca072f79-80d2-4e12-a16e-80a5ecb257a6\") " pod="ai-tenant-e2e-mt-f34a9b9d/e2e-lifecycle-model-f34a9b9d-kserve-57bb749cc9-qdm5w" Jul 10 15:01:23.414992 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:01:23.414969 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8hlk\" (UniqueName: \"kubernetes.io/projected/ca072f79-80d2-4e12-a16e-80a5ecb257a6-kube-api-access-p8hlk\") pod \"e2e-lifecycle-model-f34a9b9d-kserve-57bb749cc9-qdm5w\" (UID: \"ca072f79-80d2-4e12-a16e-80a5ecb257a6\") " pod="ai-tenant-e2e-mt-f34a9b9d/e2e-lifecycle-model-f34a9b9d-kserve-57bb749cc9-qdm5w" Jul 10 15:01:23.503864 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:01:23.503798 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f34a9b9d/e2e-lifecycle-model-f34a9b9d-kserve-57bb749cc9-qdm5w" Jul 10 15:01:23.634734 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:01:23.634712 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-f34a9b9d/e2e-lifecycle-model-f34a9b9d-kserve-57bb749cc9-qdm5w"] Jul 10 15:01:23.636731 ip-10-0-140-35 kubenswrapper[2569]: W0710 15:01:23.636706 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca072f79_80d2_4e12_a16e_80a5ecb257a6.slice/crio-6e0cbb93d2fc99cd6a9ddb8ab5b962ebe8bb126715163cfcb77f51d2164ba3b9 WatchSource:0}: Error finding container 6e0cbb93d2fc99cd6a9ddb8ab5b962ebe8bb126715163cfcb77f51d2164ba3b9: Status 404 returned error can't find the container with id 6e0cbb93d2fc99cd6a9ddb8ab5b962ebe8bb126715163cfcb77f51d2164ba3b9 Jul 10 15:01:24.496053 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:01:24.496012 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f34a9b9d/e2e-lifecycle-model-f34a9b9d-kserve-57bb749cc9-qdm5w" event={"ID":"ca072f79-80d2-4e12-a16e-80a5ecb257a6","Type":"ContainerStarted","Data":"9d7dfc5e3aaddb37ec18f932e0b06502136010bfe2ab69243cdaf834971c6695"} Jul 10 15:01:24.496053 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:01:24.496062 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f34a9b9d/e2e-lifecycle-model-f34a9b9d-kserve-57bb749cc9-qdm5w" event={"ID":"ca072f79-80d2-4e12-a16e-80a5ecb257a6","Type":"ContainerStarted","Data":"6e0cbb93d2fc99cd6a9ddb8ab5b962ebe8bb126715163cfcb77f51d2164ba3b9"} Jul 10 15:01:24.512580 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:01:24.512529 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-f34a9b9d/e2e-lifecycle-model-f34a9b9d-kserve-57bb749cc9-qdm5w" podStartSLOduration=1.512512307 podStartE2EDuration="1.512512307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 15:01:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 15:01:24.512459973 +0000 UTC m=+1830.263831689" watchObservedRunningTime="2026-07-10 15:01:24.512512307 +0000 UTC m=+1830.263884004" Jul 10 15:01:33.504503 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:01:33.504466 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-f34a9b9d/e2e-lifecycle-model-f34a9b9d-kserve-57bb749cc9-qdm5w" Jul 10 15:01:33.504972 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:01:33.504515 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-f34a9b9d/e2e-lifecycle-model-f34a9b9d-kserve-57bb749cc9-qdm5w" Jul 10 15:01:33.517017 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:01:33.516996 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-f34a9b9d/e2e-lifecycle-model-f34a9b9d-kserve-57bb749cc9-qdm5w" Jul 10 15:01:33.553701 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:01:33.553669 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-f34a9b9d/e2e-lifecycle-model-f34a9b9d-kserve-57bb749cc9-qdm5w" Jul 10 15:01:52.390521 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:01:52.390486 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-f34a9b9d/e2e-lifecycle-model-f34a9b9d-kserve-57bb749cc9-qdm5w"] Jul 10 15:02:07.411082 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:02:07.411017 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-f34a9b9d/e2e-lifecycle-model-f34a9b9d-kserve-57bb749cc9-qdm5w" podUID="ca072f79-80d2-4e12-a16e-80a5ecb257a6" containerName="main" containerID="cri-o://9d7dfc5e3aaddb37ec18f932e0b06502136010bfe2ab69243cdaf834971c6695" gracePeriod=45 Jul 10 15:02:07.664479 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:02:07.664425 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f34a9b9d/e2e-lifecycle-model-f34a9b9d-kserve-57bb749cc9-qdm5w" Jul 10 15:02:07.706023 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:02:07.705992 2569 generic.go:356] "Generic (PLEG): container finished" podID="ca072f79-80d2-4e12-a16e-80a5ecb257a6" containerID="9d7dfc5e3aaddb37ec18f932e0b06502136010bfe2ab69243cdaf834971c6695" exitCode=0 Jul 10 15:02:07.706196 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:02:07.706038 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f34a9b9d/e2e-lifecycle-model-f34a9b9d-kserve-57bb749cc9-qdm5w" Jul 10 15:02:07.706196 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:02:07.706060 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f34a9b9d/e2e-lifecycle-model-f34a9b9d-kserve-57bb749cc9-qdm5w" event={"ID":"ca072f79-80d2-4e12-a16e-80a5ecb257a6","Type":"ContainerDied","Data":"9d7dfc5e3aaddb37ec18f932e0b06502136010bfe2ab69243cdaf834971c6695"} Jul 10 15:02:07.706196 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:02:07.706093 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f34a9b9d/e2e-lifecycle-model-f34a9b9d-kserve-57bb749cc9-qdm5w" event={"ID":"ca072f79-80d2-4e12-a16e-80a5ecb257a6","Type":"ContainerDied","Data":"6e0cbb93d2fc99cd6a9ddb8ab5b962ebe8bb126715163cfcb77f51d2164ba3b9"} Jul 10 15:02:07.706196 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:02:07.706113 2569 scope.go:117] "RemoveContainer" containerID="9d7dfc5e3aaddb37ec18f932e0b06502136010bfe2ab69243cdaf834971c6695" Jul 10 15:02:07.718549 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:02:07.718527 2569 scope.go:117] "RemoveContainer" containerID="9d7dfc5e3aaddb37ec18f932e0b06502136010bfe2ab69243cdaf834971c6695" Jul 10 15:02:07.718871 ip-10-0-140-35 kubenswrapper[2569]: E0710 15:02:07.718844 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d7dfc5e3aaddb37ec18f932e0b06502136010bfe2ab69243cdaf834971c6695\": container with ID starting with 9d7dfc5e3aaddb37ec18f932e0b06502136010bfe2ab69243cdaf834971c6695 not found: ID does not exist" containerID="9d7dfc5e3aaddb37ec18f932e0b06502136010bfe2ab69243cdaf834971c6695" Jul 10 15:02:07.718992 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:02:07.718884 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d7dfc5e3aaddb37ec18f932e0b06502136010bfe2ab69243cdaf834971c6695"} err="failed to get container status \"9d7dfc5e3aaddb37ec18f932e0b06502136010bfe2ab69243cdaf834971c6695\": rpc error: code = NotFound desc = could not find container \"9d7dfc5e3aaddb37ec18f932e0b06502136010bfe2ab69243cdaf834971c6695\": container with ID starting with 9d7dfc5e3aaddb37ec18f932e0b06502136010bfe2ab69243cdaf834971c6695 not found: ID does not exist" Jul 10 15:02:07.779624 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:02:07.779602 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca072f79-80d2-4e12-a16e-80a5ecb257a6-tls-certs\") pod \"ca072f79-80d2-4e12-a16e-80a5ecb257a6\" (UID: \"ca072f79-80d2-4e12-a16e-80a5ecb257a6\") " Jul 10 15:02:07.779724 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:02:07.779639 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca072f79-80d2-4e12-a16e-80a5ecb257a6-home\") pod \"ca072f79-80d2-4e12-a16e-80a5ecb257a6\" (UID: \"ca072f79-80d2-4e12-a16e-80a5ecb257a6\") " Jul 10 15:02:07.779724 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:02:07.779684 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca072f79-80d2-4e12-a16e-80a5ecb257a6-dshm\") pod \"ca072f79-80d2-4e12-a16e-80a5ecb257a6\" (UID: \"ca072f79-80d2-4e12-a16e-80a5ecb257a6\") " Jul 10 15:02:07.779865 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:02:07.779781 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca072f79-80d2-4e12-a16e-80a5ecb257a6-model-cache\") pod \"ca072f79-80d2-4e12-a16e-80a5ecb257a6\" (UID: \"ca072f79-80d2-4e12-a16e-80a5ecb257a6\") " Jul 10 15:02:07.779865 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:02:07.779850 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca072f79-80d2-4e12-a16e-80a5ecb257a6-tmp-dir\") pod \"ca072f79-80d2-4e12-a16e-80a5ecb257a6\" (UID: \"ca072f79-80d2-4e12-a16e-80a5ecb257a6\") " Jul 10 15:02:07.779967 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:02:07.779890 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p8hlk\" (UniqueName: \"kubernetes.io/projected/ca072f79-80d2-4e12-a16e-80a5ecb257a6-kube-api-access-p8hlk\") pod \"ca072f79-80d2-4e12-a16e-80a5ecb257a6\" (UID: \"ca072f79-80d2-4e12-a16e-80a5ecb257a6\") " Jul 10 15:02:07.780021 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:02:07.779964 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca072f79-80d2-4e12-a16e-80a5ecb257a6-home" (OuterVolumeSpecName: "home") pod "ca072f79-80d2-4e12-a16e-80a5ecb257a6" (UID: "ca072f79-80d2-4e12-a16e-80a5ecb257a6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 15:02:07.780085 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:02:07.780045 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca072f79-80d2-4e12-a16e-80a5ecb257a6-model-cache" (OuterVolumeSpecName: "model-cache") pod "ca072f79-80d2-4e12-a16e-80a5ecb257a6" (UID: "ca072f79-80d2-4e12-a16e-80a5ecb257a6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 15:02:07.780131 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:02:07.780109 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca072f79-80d2-4e12-a16e-80a5ecb257a6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ca072f79-80d2-4e12-a16e-80a5ecb257a6" (UID: "ca072f79-80d2-4e12-a16e-80a5ecb257a6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 15:02:07.780179 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:02:07.780120 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca072f79-80d2-4e12-a16e-80a5ecb257a6-home\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 15:02:07.781816 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:02:07.781794 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca072f79-80d2-4e12-a16e-80a5ecb257a6-dshm" (OuterVolumeSpecName: "dshm") pod "ca072f79-80d2-4e12-a16e-80a5ecb257a6" (UID: "ca072f79-80d2-4e12-a16e-80a5ecb257a6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 15:02:07.781910 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:02:07.781893 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca072f79-80d2-4e12-a16e-80a5ecb257a6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ca072f79-80d2-4e12-a16e-80a5ecb257a6" (UID: "ca072f79-80d2-4e12-a16e-80a5ecb257a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 15:02:07.781910 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:02:07.781894 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca072f79-80d2-4e12-a16e-80a5ecb257a6-kube-api-access-p8hlk" (OuterVolumeSpecName: "kube-api-access-p8hlk") pod "ca072f79-80d2-4e12-a16e-80a5ecb257a6" (UID: "ca072f79-80d2-4e12-a16e-80a5ecb257a6"). InnerVolumeSpecName "kube-api-access-p8hlk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 15:02:07.881030 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:02:07.881006 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca072f79-80d2-4e12-a16e-80a5ecb257a6-dshm\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 15:02:07.881030 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:02:07.881028 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca072f79-80d2-4e12-a16e-80a5ecb257a6-model-cache\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 15:02:07.881163 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:02:07.881039 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca072f79-80d2-4e12-a16e-80a5ecb257a6-tmp-dir\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 15:02:07.881163 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:02:07.881047 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8hlk\" (UniqueName: \"kubernetes.io/projected/ca072f79-80d2-4e12-a16e-80a5ecb257a6-kube-api-access-p8hlk\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 15:02:07.881163 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:02:07.881056 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca072f79-80d2-4e12-a16e-80a5ecb257a6-tls-certs\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 15:02:08.026300 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:02:08.026275 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-f34a9b9d/e2e-lifecycle-model-f34a9b9d-kserve-57bb749cc9-qdm5w"] Jul 10 15:02:08.031857 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:02:08.031835 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-f34a9b9d/e2e-lifecycle-model-f34a9b9d-kserve-57bb749cc9-qdm5w"] Jul 10 15:02:08.932016 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:02:08.931975 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca072f79-80d2-4e12-a16e-80a5ecb257a6" path="/var/lib/kubelet/pods/ca072f79-80d2-4e12-a16e-80a5ecb257a6/volumes" Jul 10 15:03:12.643904 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:12.643868 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp"] Jul 10 15:03:12.644387 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:12.644266 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca072f79-80d2-4e12-a16e-80a5ecb257a6" containerName="main" Jul 10 15:03:12.644387 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:12.644278 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca072f79-80d2-4e12-a16e-80a5ecb257a6" containerName="main" Jul 10 15:03:12.644387 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:12.644341 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca072f79-80d2-4e12-a16e-80a5ecb257a6" containerName="main" Jul 10 15:03:12.647438 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:12.647420 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp" Jul 10 15:03:12.650075 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:12.650058 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-72c145-openshift-default-dockercfg-sm6r9\"" Jul 10 15:03:12.657587 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:12.657564 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp"] Jul 10 15:03:12.695139 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:12.695103 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97bb8b27-20e2-4214-b9e6-e2c4f6769f0e-istiod-ca-cert\") pod \"e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp\" (UID: \"97bb8b27-20e2-4214-b9e6-e2c4f6769f0e\") " pod="openshift-ingress/e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp" Jul 10 15:03:12.695341 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:12.695152 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97bb8b27-20e2-4214-b9e6-e2c4f6769f0e-credential-socket\") pod \"e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp\" (UID: \"97bb8b27-20e2-4214-b9e6-e2c4f6769f0e\") " pod="openshift-ingress/e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp" Jul 10 15:03:12.695341 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:12.695174 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97bb8b27-20e2-4214-b9e6-e2c4f6769f0e-istio-envoy\") pod \"e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp\" (UID: \"97bb8b27-20e2-4214-b9e6-e2c4f6769f0e\") " pod="openshift-ingress/e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp" Jul 10 15:03:12.695341 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:12.695235 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97bb8b27-20e2-4214-b9e6-e2c4f6769f0e-istio-podinfo\") pod \"e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp\" (UID: \"97bb8b27-20e2-4214-b9e6-e2c4f6769f0e\") " pod="openshift-ingress/e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp" Jul 10 15:03:12.695341 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:12.695293 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97bb8b27-20e2-4214-b9e6-e2c4f6769f0e-istio-token\") pod \"e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp\" (UID: \"97bb8b27-20e2-4214-b9e6-e2c4f6769f0e\") " pod="openshift-ingress/e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp" Jul 10 15:03:12.695341 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:12.695318 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97bb8b27-20e2-4214-b9e6-e2c4f6769f0e-workload-socket\") pod \"e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp\" (UID: \"97bb8b27-20e2-4214-b9e6-e2c4f6769f0e\") " pod="openshift-ingress/e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp" Jul 10 15:03:12.695606 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:12.695383 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97bb8b27-20e2-4214-b9e6-e2c4f6769f0e-workload-certs\") pod \"e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp\" (UID: \"97bb8b27-20e2-4214-b9e6-e2c4f6769f0e\") " pod="openshift-ingress/e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp" Jul 10 15:03:12.695606 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:12.695407 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97bb8b27-20e2-4214-b9e6-e2c4f6769f0e-istio-data\") pod \"e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp\" (UID: \"97bb8b27-20e2-4214-b9e6-e2c4f6769f0e\") " pod="openshift-ingress/e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp" Jul 10 15:03:12.695606 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:12.695423 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsq2f\" (UniqueName: \"kubernetes.io/projected/97bb8b27-20e2-4214-b9e6-e2c4f6769f0e-kube-api-access-zsq2f\") pod \"e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp\" (UID: \"97bb8b27-20e2-4214-b9e6-e2c4f6769f0e\") " pod="openshift-ingress/e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp" Jul 10 15:03:12.796727 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:12.796695 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97bb8b27-20e2-4214-b9e6-e2c4f6769f0e-workload-certs\") pod \"e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp\" (UID: \"97bb8b27-20e2-4214-b9e6-e2c4f6769f0e\") " pod="openshift-ingress/e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp" Jul 10 15:03:12.796727 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:12.796733 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97bb8b27-20e2-4214-b9e6-e2c4f6769f0e-istio-data\") pod \"e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp\" (UID: \"97bb8b27-20e2-4214-b9e6-e2c4f6769f0e\") " pod="openshift-ingress/e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp" Jul 10 15:03:12.797017 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:12.796784 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsq2f\" (UniqueName: \"kubernetes.io/projected/97bb8b27-20e2-4214-b9e6-e2c4f6769f0e-kube-api-access-zsq2f\") pod \"e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp\" (UID: \"97bb8b27-20e2-4214-b9e6-e2c4f6769f0e\") " pod="openshift-ingress/e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp" Jul 10 15:03:12.797017 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:12.796825 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97bb8b27-20e2-4214-b9e6-e2c4f6769f0e-istiod-ca-cert\") pod \"e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp\" (UID: \"97bb8b27-20e2-4214-b9e6-e2c4f6769f0e\") " pod="openshift-ingress/e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp" Jul 10 15:03:12.797017 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:12.796863 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97bb8b27-20e2-4214-b9e6-e2c4f6769f0e-credential-socket\") pod \"e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp\" (UID: \"97bb8b27-20e2-4214-b9e6-e2c4f6769f0e\") " pod="openshift-ingress/e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp" Jul 10 15:03:12.797017 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:12.796879 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97bb8b27-20e2-4214-b9e6-e2c4f6769f0e-istio-envoy\") pod \"e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp\" (UID: \"97bb8b27-20e2-4214-b9e6-e2c4f6769f0e\") " pod="openshift-ingress/e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp" Jul 10 15:03:12.797017 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:12.796902 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97bb8b27-20e2-4214-b9e6-e2c4f6769f0e-istio-podinfo\") pod \"e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp\" (UID: \"97bb8b27-20e2-4214-b9e6-e2c4f6769f0e\") " pod="openshift-ingress/e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp" Jul 10 15:03:12.797017 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:12.796935 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97bb8b27-20e2-4214-b9e6-e2c4f6769f0e-istio-token\") pod \"e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp\" (UID: \"97bb8b27-20e2-4214-b9e6-e2c4f6769f0e\") " pod="openshift-ingress/e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp" Jul 10 15:03:12.797017 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:12.796962 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97bb8b27-20e2-4214-b9e6-e2c4f6769f0e-workload-socket\") pod \"e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp\" (UID: \"97bb8b27-20e2-4214-b9e6-e2c4f6769f0e\") " pod="openshift-ingress/e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp" Jul 10 15:03:12.797364 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:12.797202 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97bb8b27-20e2-4214-b9e6-e2c4f6769f0e-workload-certs\") pod \"e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp\" (UID: \"97bb8b27-20e2-4214-b9e6-e2c4f6769f0e\") " pod="openshift-ingress/e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp" Jul 10 15:03:12.797364 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:12.797291 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97bb8b27-20e2-4214-b9e6-e2c4f6769f0e-credential-socket\") pod \"e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp\" (UID: \"97bb8b27-20e2-4214-b9e6-e2c4f6769f0e\") " pod="openshift-ingress/e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp" Jul 10 15:03:12.797470 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:12.797418 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97bb8b27-20e2-4214-b9e6-e2c4f6769f0e-workload-socket\") pod \"e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp\" (UID: \"97bb8b27-20e2-4214-b9e6-e2c4f6769f0e\") " pod="openshift-ingress/e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp" Jul 10 15:03:12.797523 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:12.797470 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97bb8b27-20e2-4214-b9e6-e2c4f6769f0e-istio-data\") pod \"e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp\" (UID: \"97bb8b27-20e2-4214-b9e6-e2c4f6769f0e\") " pod="openshift-ingress/e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp" Jul 10 15:03:12.797632 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:12.797612 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97bb8b27-20e2-4214-b9e6-e2c4f6769f0e-istiod-ca-cert\") pod \"e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp\" (UID: \"97bb8b27-20e2-4214-b9e6-e2c4f6769f0e\") " pod="openshift-ingress/e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp" Jul 10 15:03:12.799485 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:12.799460 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97bb8b27-20e2-4214-b9e6-e2c4f6769f0e-istio-envoy\") pod \"e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp\" (UID: \"97bb8b27-20e2-4214-b9e6-e2c4f6769f0e\") " pod="openshift-ingress/e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp" Jul 10 15:03:12.799796 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:12.799742 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97bb8b27-20e2-4214-b9e6-e2c4f6769f0e-istio-podinfo\") pod \"e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp\" (UID: \"97bb8b27-20e2-4214-b9e6-e2c4f6769f0e\") " pod="openshift-ingress/e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp" Jul 10 15:03:12.808424 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:12.808387 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97bb8b27-20e2-4214-b9e6-e2c4f6769f0e-istio-token\") pod \"e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp\" (UID: \"97bb8b27-20e2-4214-b9e6-e2c4f6769f0e\") " pod="openshift-ingress/e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp" Jul 10 15:03:12.809010 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:12.808987 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsq2f\" (UniqueName: \"kubernetes.io/projected/97bb8b27-20e2-4214-b9e6-e2c4f6769f0e-kube-api-access-zsq2f\") pod \"e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp\" (UID: \"97bb8b27-20e2-4214-b9e6-e2c4f6769f0e\") " pod="openshift-ingress/e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp" Jul 10 15:03:12.958308 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:12.958231 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp" Jul 10 15:03:13.097043 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:13.097020 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp"] Jul 10 15:03:13.098878 ip-10-0-140-35 kubenswrapper[2569]: W0710 15:03:13.098844 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97bb8b27_20e2_4214_b9e6_e2c4f6769f0e.slice/crio-8f1708e710d1a7d1d234bf331b08901f62f3a247e67fa3beb8f55fc78ab1eceb WatchSource:0}: Error finding container 8f1708e710d1a7d1d234bf331b08901f62f3a247e67fa3beb8f55fc78ab1eceb: Status 404 returned error can't find the container with id 8f1708e710d1a7d1d234bf331b08901f62f3a247e67fa3beb8f55fc78ab1eceb Jul 10 15:03:13.100794 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:13.100734 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 15:03:13.100861 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:13.100838 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 15:03:13.100897 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:13.100873 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 15:03:14.031464 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:14.031429 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp" event={"ID":"97bb8b27-20e2-4214-b9e6-e2c4f6769f0e","Type":"ContainerStarted","Data":"6525481db3487ecfb4a3a2324ba77781de8019878c41c99801295d002d3145bc"} Jul 10 15:03:14.031891 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:14.031481 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp" event={"ID":"97bb8b27-20e2-4214-b9e6-e2c4f6769f0e","Type":"ContainerStarted","Data":"8f1708e710d1a7d1d234bf331b08901f62f3a247e67fa3beb8f55fc78ab1eceb"} Jul 10 15:03:14.053464 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:14.053422 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp" podStartSLOduration=2.053406786 podStartE2EDuration="2.053406786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 15:03:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 15:03:14.051127467 +0000 UTC m=+1939.802499163" watchObservedRunningTime="2026-07-10 15:03:14.053406786 +0000 UTC m=+1939.804778482" Jul 10 15:03:14.958771 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:14.958725 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp" Jul 10 15:03:14.963516 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:14.963477 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp" Jul 10 15:03:15.035967 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:15.035943 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp" Jul 10 15:03:15.036715 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:15.036696 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp" Jul 10 15:03:31.077932 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:31.077888 2569 patch_prober.go:28] interesting pod/e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 10 15:03:31.078406 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:31.077952 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp" podUID="97bb8b27-20e2-4214-b9e6-e2c4f6769f0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 10 15:03:32.116600 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:32.116564 2569 generic.go:356] "Generic (PLEG): container finished" podID="97bb8b27-20e2-4214-b9e6-e2c4f6769f0e" containerID="6525481db3487ecfb4a3a2324ba77781de8019878c41c99801295d002d3145bc" exitCode=0 Jul 10 15:03:32.116600 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:32.116602 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp" event={"ID":"97bb8b27-20e2-4214-b9e6-e2c4f6769f0e","Type":"ContainerDied","Data":"6525481db3487ecfb4a3a2324ba77781de8019878c41c99801295d002d3145bc"} Jul 10 15:03:32.117142 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:32.117023 2569 scope.go:117] "RemoveContainer" containerID="6525481db3487ecfb4a3a2324ba77781de8019878c41c99801295d002d3145bc" Jul 10 15:03:32.118093 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:32.118056 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 15:03:32.118169 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:32.118150 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 15:03:32.118225 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:32.118209 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 15:03:33.123438 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:33.123406 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp" event={"ID":"97bb8b27-20e2-4214-b9e6-e2c4f6769f0e","Type":"ContainerStarted","Data":"175ae8b0a89ecd470046e833f094149b37de638a51555f09387ad555041657a9"} Jul 10 15:03:33.959224 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:33.959181 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp" Jul 10 15:03:33.959444 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:33.959379 2569 patch_prober.go:28] interesting pod/e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Jul 10 15:03:33.959515 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:33.959440 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp" podUID="97bb8b27-20e2-4214-b9e6-e2c4f6769f0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Jul 10 15:03:34.958805 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:34.958764 2569 patch_prober.go:28] interesting pod/e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Jul 10 15:03:34.959221 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:34.958829 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp" podUID="97bb8b27-20e2-4214-b9e6-e2c4f6769f0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Jul 10 15:03:35.959578 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:35.959539 2569 patch_prober.go:28] interesting pod/e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Jul 10 15:03:35.959974 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:35.959599 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp" podUID="97bb8b27-20e2-4214-b9e6-e2c4f6769f0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Jul 10 15:03:36.959024 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:36.958988 2569 patch_prober.go:28] interesting pod/e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Jul 10 15:03:36.959186 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:36.959043 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp" podUID="97bb8b27-20e2-4214-b9e6-e2c4f6769f0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Jul 10 15:03:37.962525 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:37.962494 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp" Jul 10 15:03:37.962948 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:37.962762 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp" Jul 10 15:03:37.963407 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:37.963389 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp" Jul 10 15:03:38.999737 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:38.999694 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-72c145/test-model-72c145-kserve-58f76fd4d5-2bbp5"] Jul 10 15:03:39.003672 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:39.003652 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-72c145/test-model-72c145-kserve-58f76fd4d5-2bbp5" Jul 10 15:03:39.006357 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:39.006337 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-72c145\"/\"kube-root-ca.crt\"" Jul 10 15:03:39.006450 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:39.006436 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-72c145\"/\"openshift-service-ca.crt\"" Jul 10 15:03:39.007543 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:39.007526 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-72c145\"/\"test-model-72c145-kserve-self-signed-certs\"" Jul 10 15:03:39.007639 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:39.007549 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-72c145\"/\"default-dockercfg-sg2p9\"" Jul 10 15:03:39.013305 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:39.013285 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-72c145/test-model-72c145-kserve-58f76fd4d5-2bbp5"] Jul 10 15:03:39.133405 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:39.133371 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b36d9c70-19e6-47d1-818b-9bb6ffff3d30-home\") pod \"test-model-72c145-kserve-58f76fd4d5-2bbp5\" (UID: \"b36d9c70-19e6-47d1-818b-9bb6ffff3d30\") " pod="ai-tenant-e2e-inf-a-72c145/test-model-72c145-kserve-58f76fd4d5-2bbp5" Jul 10 15:03:39.133583 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:39.133409 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b36d9c70-19e6-47d1-818b-9bb6ffff3d30-tls-certs\") pod \"test-model-72c145-kserve-58f76fd4d5-2bbp5\" (UID: \"b36d9c70-19e6-47d1-818b-9bb6ffff3d30\") " pod="ai-tenant-e2e-inf-a-72c145/test-model-72c145-kserve-58f76fd4d5-2bbp5" Jul 10 15:03:39.133583 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:39.133461 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b36d9c70-19e6-47d1-818b-9bb6ffff3d30-model-cache\") pod \"test-model-72c145-kserve-58f76fd4d5-2bbp5\" (UID: \"b36d9c70-19e6-47d1-818b-9bb6ffff3d30\") " pod="ai-tenant-e2e-inf-a-72c145/test-model-72c145-kserve-58f76fd4d5-2bbp5" Jul 10 15:03:39.133583 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:39.133521 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b36d9c70-19e6-47d1-818b-9bb6ffff3d30-dshm\") pod \"test-model-72c145-kserve-58f76fd4d5-2bbp5\" (UID: \"b36d9c70-19e6-47d1-818b-9bb6ffff3d30\") " pod="ai-tenant-e2e-inf-a-72c145/test-model-72c145-kserve-58f76fd4d5-2bbp5" Jul 10 15:03:39.133583 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:39.133560 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trnkg\" (UniqueName: \"kubernetes.io/projected/b36d9c70-19e6-47d1-818b-9bb6ffff3d30-kube-api-access-trnkg\") pod \"test-model-72c145-kserve-58f76fd4d5-2bbp5\" (UID: \"b36d9c70-19e6-47d1-818b-9bb6ffff3d30\") " pod="ai-tenant-e2e-inf-a-72c145/test-model-72c145-kserve-58f76fd4d5-2bbp5" Jul 10 15:03:39.133716 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:39.133606 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b36d9c70-19e6-47d1-818b-9bb6ffff3d30-tmp-dir\") pod \"test-model-72c145-kserve-58f76fd4d5-2bbp5\" (UID: \"b36d9c70-19e6-47d1-818b-9bb6ffff3d30\") " pod="ai-tenant-e2e-inf-a-72c145/test-model-72c145-kserve-58f76fd4d5-2bbp5" Jul 10 15:03:39.234031 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:39.234000 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b36d9c70-19e6-47d1-818b-9bb6ffff3d30-tmp-dir\") pod \"test-model-72c145-kserve-58f76fd4d5-2bbp5\" (UID: \"b36d9c70-19e6-47d1-818b-9bb6ffff3d30\") " pod="ai-tenant-e2e-inf-a-72c145/test-model-72c145-kserve-58f76fd4d5-2bbp5" Jul 10 15:03:39.234031 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:39.234034 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b36d9c70-19e6-47d1-818b-9bb6ffff3d30-home\") pod \"test-model-72c145-kserve-58f76fd4d5-2bbp5\" (UID: \"b36d9c70-19e6-47d1-818b-9bb6ffff3d30\") " pod="ai-tenant-e2e-inf-a-72c145/test-model-72c145-kserve-58f76fd4d5-2bbp5" Jul 10 15:03:39.234275 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:39.234051 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b36d9c70-19e6-47d1-818b-9bb6ffff3d30-tls-certs\") pod \"test-model-72c145-kserve-58f76fd4d5-2bbp5\" (UID: \"b36d9c70-19e6-47d1-818b-9bb6ffff3d30\") " pod="ai-tenant-e2e-inf-a-72c145/test-model-72c145-kserve-58f76fd4d5-2bbp5" Jul 10 15:03:39.234275 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:39.234079 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b36d9c70-19e6-47d1-818b-9bb6ffff3d30-model-cache\") pod \"test-model-72c145-kserve-58f76fd4d5-2bbp5\" (UID: \"b36d9c70-19e6-47d1-818b-9bb6ffff3d30\") " pod="ai-tenant-e2e-inf-a-72c145/test-model-72c145-kserve-58f76fd4d5-2bbp5" Jul 10 15:03:39.234275 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:39.234127 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b36d9c70-19e6-47d1-818b-9bb6ffff3d30-dshm\") pod \"test-model-72c145-kserve-58f76fd4d5-2bbp5\" (UID: \"b36d9c70-19e6-47d1-818b-9bb6ffff3d30\") " pod="ai-tenant-e2e-inf-a-72c145/test-model-72c145-kserve-58f76fd4d5-2bbp5" Jul 10 15:03:39.234275 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:39.234146 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trnkg\" (UniqueName: \"kubernetes.io/projected/b36d9c70-19e6-47d1-818b-9bb6ffff3d30-kube-api-access-trnkg\") pod \"test-model-72c145-kserve-58f76fd4d5-2bbp5\" (UID: \"b36d9c70-19e6-47d1-818b-9bb6ffff3d30\") " pod="ai-tenant-e2e-inf-a-72c145/test-model-72c145-kserve-58f76fd4d5-2bbp5" Jul 10 15:03:39.234507 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:39.234482 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b36d9c70-19e6-47d1-818b-9bb6ffff3d30-home\") pod \"test-model-72c145-kserve-58f76fd4d5-2bbp5\" (UID: \"b36d9c70-19e6-47d1-818b-9bb6ffff3d30\") " pod="ai-tenant-e2e-inf-a-72c145/test-model-72c145-kserve-58f76fd4d5-2bbp5" Jul 10 15:03:39.234507 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:39.234499 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b36d9c70-19e6-47d1-818b-9bb6ffff3d30-tmp-dir\") pod \"test-model-72c145-kserve-58f76fd4d5-2bbp5\" (UID: \"b36d9c70-19e6-47d1-818b-9bb6ffff3d30\") " pod="ai-tenant-e2e-inf-a-72c145/test-model-72c145-kserve-58f76fd4d5-2bbp5" Jul 10 15:03:39.234613 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:39.234546 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b36d9c70-19e6-47d1-818b-9bb6ffff3d30-model-cache\") pod \"test-model-72c145-kserve-58f76fd4d5-2bbp5\" (UID: \"b36d9c70-19e6-47d1-818b-9bb6ffff3d30\") " pod="ai-tenant-e2e-inf-a-72c145/test-model-72c145-kserve-58f76fd4d5-2bbp5" Jul 10 15:03:39.236510 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:39.236484 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b36d9c70-19e6-47d1-818b-9bb6ffff3d30-dshm\") pod \"test-model-72c145-kserve-58f76fd4d5-2bbp5\" (UID: \"b36d9c70-19e6-47d1-818b-9bb6ffff3d30\") " pod="ai-tenant-e2e-inf-a-72c145/test-model-72c145-kserve-58f76fd4d5-2bbp5" Jul 10 15:03:39.236671 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:39.236655 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b36d9c70-19e6-47d1-818b-9bb6ffff3d30-tls-certs\") pod \"test-model-72c145-kserve-58f76fd4d5-2bbp5\" (UID: \"b36d9c70-19e6-47d1-818b-9bb6ffff3d30\") " pod="ai-tenant-e2e-inf-a-72c145/test-model-72c145-kserve-58f76fd4d5-2bbp5" Jul 10 15:03:39.242095 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:39.242076 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trnkg\" (UniqueName: \"kubernetes.io/projected/b36d9c70-19e6-47d1-818b-9bb6ffff3d30-kube-api-access-trnkg\") pod \"test-model-72c145-kserve-58f76fd4d5-2bbp5\" (UID: \"b36d9c70-19e6-47d1-818b-9bb6ffff3d30\") " pod="ai-tenant-e2e-inf-a-72c145/test-model-72c145-kserve-58f76fd4d5-2bbp5" Jul 10 15:03:39.313530 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:39.313466 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-72c145/test-model-72c145-kserve-58f76fd4d5-2bbp5" Jul 10 15:03:39.445969 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:39.445941 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-72c145/test-model-72c145-kserve-58f76fd4d5-2bbp5"] Jul 10 15:03:39.448145 ip-10-0-140-35 kubenswrapper[2569]: W0710 15:03:39.448115 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb36d9c70_19e6_47d1_818b_9bb6ffff3d30.slice/crio-e655ead30d9f48ae30509a0735ee285f942f3f91ef1e6226837f15256690c43c WatchSource:0}: Error finding container e655ead30d9f48ae30509a0735ee285f942f3f91ef1e6226837f15256690c43c: Status 404 returned error can't find the container with id e655ead30d9f48ae30509a0735ee285f942f3f91ef1e6226837f15256690c43c Jul 10 15:03:40.161926 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:40.161889 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-72c145/test-model-72c145-kserve-58f76fd4d5-2bbp5" event={"ID":"b36d9c70-19e6-47d1-818b-9bb6ffff3d30","Type":"ContainerStarted","Data":"7def236b70a12e390493fc366504b60ada8c5a9a2658280fecb91f63d81a0491"} Jul 10 15:03:40.161926 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:40.161928 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-72c145/test-model-72c145-kserve-58f76fd4d5-2bbp5" event={"ID":"b36d9c70-19e6-47d1-818b-9bb6ffff3d30","Type":"ContainerStarted","Data":"e655ead30d9f48ae30509a0735ee285f942f3f91ef1e6226837f15256690c43c"} Jul 10 15:03:40.178770 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:40.178709 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-72c145/test-model-72c145-kserve-58f76fd4d5-2bbp5" podStartSLOduration=2.178694382 podStartE2EDuration="2.178694382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 15:03:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 15:03:40.176628138 +0000 UTC m=+1965.927999833" watchObservedRunningTime="2026-07-10 15:03:40.178694382 +0000 UTC m=+1965.930066109" Jul 10 15:03:49.314298 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:49.314216 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-72c145/test-model-72c145-kserve-58f76fd4d5-2bbp5" Jul 10 15:03:49.314298 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:49.314276 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-72c145/test-model-72c145-kserve-58f76fd4d5-2bbp5" Jul 10 15:03:49.326945 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:49.326920 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-72c145/test-model-72c145-kserve-58f76fd4d5-2bbp5" Jul 10 15:03:50.222411 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:03:50.222384 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-72c145/test-model-72c145-kserve-58f76fd4d5-2bbp5" Jul 10 15:04:17.572689 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:04:17.572654 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-72c145/test-model-72c145-kserve-58f76fd4d5-2bbp5"] Jul 10 15:04:32.593120 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:04:32.593064 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-72c145/test-model-72c145-kserve-58f76fd4d5-2bbp5" podUID="b36d9c70-19e6-47d1-818b-9bb6ffff3d30" containerName="main" containerID="cri-o://7def236b70a12e390493fc366504b60ada8c5a9a2658280fecb91f63d81a0491" gracePeriod=45 Jul 10 15:04:32.839610 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:04:32.839586 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-72c145/test-model-72c145-kserve-58f76fd4d5-2bbp5" Jul 10 15:04:33.010300 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:04:33.010270 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-trnkg\" (UniqueName: \"kubernetes.io/projected/b36d9c70-19e6-47d1-818b-9bb6ffff3d30-kube-api-access-trnkg\") pod \"b36d9c70-19e6-47d1-818b-9bb6ffff3d30\" (UID: \"b36d9c70-19e6-47d1-818b-9bb6ffff3d30\") " Jul 10 15:04:33.010477 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:04:33.010309 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b36d9c70-19e6-47d1-818b-9bb6ffff3d30-model-cache\") pod \"b36d9c70-19e6-47d1-818b-9bb6ffff3d30\" (UID: \"b36d9c70-19e6-47d1-818b-9bb6ffff3d30\") " Jul 10 15:04:33.010477 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:04:33.010347 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b36d9c70-19e6-47d1-818b-9bb6ffff3d30-home\") pod \"b36d9c70-19e6-47d1-818b-9bb6ffff3d30\" (UID: \"b36d9c70-19e6-47d1-818b-9bb6ffff3d30\") " Jul 10 15:04:33.010477 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:04:33.010383 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b36d9c70-19e6-47d1-818b-9bb6ffff3d30-dshm\") pod \"b36d9c70-19e6-47d1-818b-9bb6ffff3d30\" (UID: \"b36d9c70-19e6-47d1-818b-9bb6ffff3d30\") " Jul 10 15:04:33.010477 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:04:33.010429 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b36d9c70-19e6-47d1-818b-9bb6ffff3d30-tls-certs\") pod \"b36d9c70-19e6-47d1-818b-9bb6ffff3d30\" (UID: \"b36d9c70-19e6-47d1-818b-9bb6ffff3d30\") " Jul 10 15:04:33.010477 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:04:33.010451 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b36d9c70-19e6-47d1-818b-9bb6ffff3d30-tmp-dir\") pod \"b36d9c70-19e6-47d1-818b-9bb6ffff3d30\" (UID: \"b36d9c70-19e6-47d1-818b-9bb6ffff3d30\") " Jul 10 15:04:33.010714 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:04:33.010623 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b36d9c70-19e6-47d1-818b-9bb6ffff3d30-home" (OuterVolumeSpecName: "home") pod "b36d9c70-19e6-47d1-818b-9bb6ffff3d30" (UID: "b36d9c70-19e6-47d1-818b-9bb6ffff3d30"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 15:04:33.010714 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:04:33.010633 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b36d9c70-19e6-47d1-818b-9bb6ffff3d30-model-cache" (OuterVolumeSpecName: "model-cache") pod "b36d9c70-19e6-47d1-818b-9bb6ffff3d30" (UID: "b36d9c70-19e6-47d1-818b-9bb6ffff3d30"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 15:04:33.010833 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:04:33.010793 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b36d9c70-19e6-47d1-818b-9bb6ffff3d30-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b36d9c70-19e6-47d1-818b-9bb6ffff3d30" (UID: "b36d9c70-19e6-47d1-818b-9bb6ffff3d30"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 15:04:33.010833 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:04:33.010808 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b36d9c70-19e6-47d1-818b-9bb6ffff3d30-model-cache\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 15:04:33.010833 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:04:33.010825 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b36d9c70-19e6-47d1-818b-9bb6ffff3d30-home\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 15:04:33.012618 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:04:33.012558 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b36d9c70-19e6-47d1-818b-9bb6ffff3d30-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b36d9c70-19e6-47d1-818b-9bb6ffff3d30" (UID: "b36d9c70-19e6-47d1-818b-9bb6ffff3d30"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 15:04:33.012618 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:04:33.012585 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b36d9c70-19e6-47d1-818b-9bb6ffff3d30-kube-api-access-trnkg" (OuterVolumeSpecName: "kube-api-access-trnkg") pod "b36d9c70-19e6-47d1-818b-9bb6ffff3d30" (UID: "b36d9c70-19e6-47d1-818b-9bb6ffff3d30"). InnerVolumeSpecName "kube-api-access-trnkg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 15:04:33.012618 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:04:33.012597 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b36d9c70-19e6-47d1-818b-9bb6ffff3d30-dshm" (OuterVolumeSpecName: "dshm") pod "b36d9c70-19e6-47d1-818b-9bb6ffff3d30" (UID: "b36d9c70-19e6-47d1-818b-9bb6ffff3d30"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 15:04:33.112090 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:04:33.112059 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trnkg\" (UniqueName: \"kubernetes.io/projected/b36d9c70-19e6-47d1-818b-9bb6ffff3d30-kube-api-access-trnkg\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 15:04:33.112090 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:04:33.112092 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b36d9c70-19e6-47d1-818b-9bb6ffff3d30-dshm\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 15:04:33.112229 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:04:33.112102 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b36d9c70-19e6-47d1-818b-9bb6ffff3d30-tls-certs\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 15:04:33.112229 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:04:33.112112 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b36d9c70-19e6-47d1-818b-9bb6ffff3d30-tmp-dir\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 15:04:33.431628 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:04:33.431599 2569 generic.go:356] "Generic (PLEG): container finished" podID="b36d9c70-19e6-47d1-818b-9bb6ffff3d30" containerID="7def236b70a12e390493fc366504b60ada8c5a9a2658280fecb91f63d81a0491" exitCode=0 Jul 10 15:04:33.431802 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:04:33.431646 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-72c145/test-model-72c145-kserve-58f76fd4d5-2bbp5" Jul 10 15:04:33.431802 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:04:33.431685 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-72c145/test-model-72c145-kserve-58f76fd4d5-2bbp5" event={"ID":"b36d9c70-19e6-47d1-818b-9bb6ffff3d30","Type":"ContainerDied","Data":"7def236b70a12e390493fc366504b60ada8c5a9a2658280fecb91f63d81a0491"} Jul 10 15:04:33.431802 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:04:33.431732 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-72c145/test-model-72c145-kserve-58f76fd4d5-2bbp5" event={"ID":"b36d9c70-19e6-47d1-818b-9bb6ffff3d30","Type":"ContainerDied","Data":"e655ead30d9f48ae30509a0735ee285f942f3f91ef1e6226837f15256690c43c"} Jul 10 15:04:33.431802 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:04:33.431768 2569 scope.go:117] "RemoveContainer" containerID="7def236b70a12e390493fc366504b60ada8c5a9a2658280fecb91f63d81a0491" Jul 10 15:04:33.440605 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:04:33.440588 2569 scope.go:117] "RemoveContainer" containerID="7def236b70a12e390493fc366504b60ada8c5a9a2658280fecb91f63d81a0491" Jul 10 15:04:33.440880 ip-10-0-140-35 kubenswrapper[2569]: E0710 15:04:33.440863 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7def236b70a12e390493fc366504b60ada8c5a9a2658280fecb91f63d81a0491\": container with ID starting with 7def236b70a12e390493fc366504b60ada8c5a9a2658280fecb91f63d81a0491 not found: ID does not exist" containerID="7def236b70a12e390493fc366504b60ada8c5a9a2658280fecb91f63d81a0491" Jul 10 15:04:33.440930 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:04:33.440889 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7def236b70a12e390493fc366504b60ada8c5a9a2658280fecb91f63d81a0491"} err="failed to get container status \"7def236b70a12e390493fc366504b60ada8c5a9a2658280fecb91f63d81a0491\": rpc error: code = NotFound desc = could not find container \"7def236b70a12e390493fc366504b60ada8c5a9a2658280fecb91f63d81a0491\": container with ID starting with 7def236b70a12e390493fc366504b60ada8c5a9a2658280fecb91f63d81a0491 not found: ID does not exist" Jul 10 15:04:33.459203 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:04:33.458078 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-72c145/test-model-72c145-kserve-58f76fd4d5-2bbp5"] Jul 10 15:04:33.463623 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:04:33.463594 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-72c145/test-model-72c145-kserve-58f76fd4d5-2bbp5"] Jul 10 15:04:34.931462 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:04:34.931430 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b36d9c70-19e6-47d1-818b-9bb6ffff3d30" path="/var/lib/kubelet/pods/b36d9c70-19e6-47d1-818b-9bb6ffff3d30/volumes" Jul 10 15:05:01.197701 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:01.197662 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp"] Jul 10 15:05:01.198110 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:01.197970 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp" podUID="97bb8b27-20e2-4214-b9e6-e2c4f6769f0e" containerName="istio-proxy" containerID="cri-o://175ae8b0a89ecd470046e833f094149b37de638a51555f09387ad555041657a9" gracePeriod=30 Jul 10 15:05:06.466341 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:06.466280 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp" Jul 10 15:05:06.590877 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:06.590849 2569 generic.go:356] "Generic (PLEG): container finished" podID="97bb8b27-20e2-4214-b9e6-e2c4f6769f0e" containerID="175ae8b0a89ecd470046e833f094149b37de638a51555f09387ad555041657a9" exitCode=0 Jul 10 15:05:06.591026 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:06.590893 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp" Jul 10 15:05:06.591026 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:06.590935 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp" event={"ID":"97bb8b27-20e2-4214-b9e6-e2c4f6769f0e","Type":"ContainerDied","Data":"175ae8b0a89ecd470046e833f094149b37de638a51555f09387ad555041657a9"} Jul 10 15:05:06.591026 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:06.590989 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp" event={"ID":"97bb8b27-20e2-4214-b9e6-e2c4f6769f0e","Type":"ContainerDied","Data":"8f1708e710d1a7d1d234bf331b08901f62f3a247e67fa3beb8f55fc78ab1eceb"} Jul 10 15:05:06.591026 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:06.591010 2569 scope.go:117] "RemoveContainer" containerID="175ae8b0a89ecd470046e833f094149b37de638a51555f09387ad555041657a9" Jul 10 15:05:06.597340 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:06.597320 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97bb8b27-20e2-4214-b9e6-e2c4f6769f0e-workload-certs\") pod \"97bb8b27-20e2-4214-b9e6-e2c4f6769f0e\" (UID: \"97bb8b27-20e2-4214-b9e6-e2c4f6769f0e\") " Jul 10 15:05:06.597433 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:06.597378 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97bb8b27-20e2-4214-b9e6-e2c4f6769f0e-istio-token\") pod \"97bb8b27-20e2-4214-b9e6-e2c4f6769f0e\" (UID: \"97bb8b27-20e2-4214-b9e6-e2c4f6769f0e\") " Jul 10 15:05:06.597433 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:06.597407 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97bb8b27-20e2-4214-b9e6-e2c4f6769f0e-istio-envoy\") pod \"97bb8b27-20e2-4214-b9e6-e2c4f6769f0e\" (UID: \"97bb8b27-20e2-4214-b9e6-e2c4f6769f0e\") " Jul 10 15:05:06.597433 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:06.597423 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97bb8b27-20e2-4214-b9e6-e2c4f6769f0e-istio-data\") pod \"97bb8b27-20e2-4214-b9e6-e2c4f6769f0e\" (UID: \"97bb8b27-20e2-4214-b9e6-e2c4f6769f0e\") " Jul 10 15:05:06.597589 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:06.597444 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97bb8b27-20e2-4214-b9e6-e2c4f6769f0e-istio-podinfo\") pod \"97bb8b27-20e2-4214-b9e6-e2c4f6769f0e\" (UID: \"97bb8b27-20e2-4214-b9e6-e2c4f6769f0e\") " Jul 10 15:05:06.597589 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:06.597466 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97bb8b27-20e2-4214-b9e6-e2c4f6769f0e-workload-socket\") pod \"97bb8b27-20e2-4214-b9e6-e2c4f6769f0e\" (UID: \"97bb8b27-20e2-4214-b9e6-e2c4f6769f0e\") " Jul 10 15:05:06.597701 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:06.597623 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97bb8b27-20e2-4214-b9e6-e2c4f6769f0e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "97bb8b27-20e2-4214-b9e6-e2c4f6769f0e" (UID: "97bb8b27-20e2-4214-b9e6-e2c4f6769f0e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 15:05:06.597783 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:06.597742 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97bb8b27-20e2-4214-b9e6-e2c4f6769f0e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "97bb8b27-20e2-4214-b9e6-e2c4f6769f0e" (UID: "97bb8b27-20e2-4214-b9e6-e2c4f6769f0e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 15:05:06.597891 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:06.597874 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zsq2f\" (UniqueName: \"kubernetes.io/projected/97bb8b27-20e2-4214-b9e6-e2c4f6769f0e-kube-api-access-zsq2f\") pod \"97bb8b27-20e2-4214-b9e6-e2c4f6769f0e\" (UID: \"97bb8b27-20e2-4214-b9e6-e2c4f6769f0e\") " Jul 10 15:05:06.597964 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:06.597940 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97bb8b27-20e2-4214-b9e6-e2c4f6769f0e-credential-socket\") pod \"97bb8b27-20e2-4214-b9e6-e2c4f6769f0e\" (UID: \"97bb8b27-20e2-4214-b9e6-e2c4f6769f0e\") " Jul 10 15:05:06.598017 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:06.597970 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97bb8b27-20e2-4214-b9e6-e2c4f6769f0e-istiod-ca-cert\") pod \"97bb8b27-20e2-4214-b9e6-e2c4f6769f0e\" (UID: \"97bb8b27-20e2-4214-b9e6-e2c4f6769f0e\") " Jul 10 15:05:06.598266 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:06.598242 2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97bb8b27-20e2-4214-b9e6-e2c4f6769f0e-workload-socket\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 15:05:06.598356 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:06.598271 2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97bb8b27-20e2-4214-b9e6-e2c4f6769f0e-workload-certs\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 15:05:06.598356 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:06.598330 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97bb8b27-20e2-4214-b9e6-e2c4f6769f0e-istio-data" (OuterVolumeSpecName: "istio-data") pod "97bb8b27-20e2-4214-b9e6-e2c4f6769f0e" (UID: "97bb8b27-20e2-4214-b9e6-e2c4f6769f0e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 15:05:06.598552 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:06.598531 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97bb8b27-20e2-4214-b9e6-e2c4f6769f0e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "97bb8b27-20e2-4214-b9e6-e2c4f6769f0e" (UID: "97bb8b27-20e2-4214-b9e6-e2c4f6769f0e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 15:05:06.598663 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:06.598628 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97bb8b27-20e2-4214-b9e6-e2c4f6769f0e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "97bb8b27-20e2-4214-b9e6-e2c4f6769f0e" (UID: "97bb8b27-20e2-4214-b9e6-e2c4f6769f0e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 15:05:06.599665 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:06.599645 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97bb8b27-20e2-4214-b9e6-e2c4f6769f0e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "97bb8b27-20e2-4214-b9e6-e2c4f6769f0e" (UID: "97bb8b27-20e2-4214-b9e6-e2c4f6769f0e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 15:05:06.599810 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:06.599790 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97bb8b27-20e2-4214-b9e6-e2c4f6769f0e-istio-token" (OuterVolumeSpecName: "istio-token") pod "97bb8b27-20e2-4214-b9e6-e2c4f6769f0e" (UID: "97bb8b27-20e2-4214-b9e6-e2c4f6769f0e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 15:05:06.599899 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:06.599878 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/97bb8b27-20e2-4214-b9e6-e2c4f6769f0e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "97bb8b27-20e2-4214-b9e6-e2c4f6769f0e" (UID: "97bb8b27-20e2-4214-b9e6-e2c4f6769f0e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 15:05:06.600637 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:06.600615 2569 scope.go:117] "RemoveContainer" containerID="6525481db3487ecfb4a3a2324ba77781de8019878c41c99801295d002d3145bc" Jul 10 15:05:06.600722 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:06.600675 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97bb8b27-20e2-4214-b9e6-e2c4f6769f0e-kube-api-access-zsq2f" (OuterVolumeSpecName: "kube-api-access-zsq2f") pod "97bb8b27-20e2-4214-b9e6-e2c4f6769f0e" (UID: "97bb8b27-20e2-4214-b9e6-e2c4f6769f0e"). InnerVolumeSpecName "kube-api-access-zsq2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 15:05:06.619947 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:06.619928 2569 scope.go:117] "RemoveContainer" containerID="175ae8b0a89ecd470046e833f094149b37de638a51555f09387ad555041657a9" Jul 10 15:05:06.620184 ip-10-0-140-35 kubenswrapper[2569]: E0710 15:05:06.620163 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"175ae8b0a89ecd470046e833f094149b37de638a51555f09387ad555041657a9\": container with ID starting with 175ae8b0a89ecd470046e833f094149b37de638a51555f09387ad555041657a9 not found: ID does not exist" containerID="175ae8b0a89ecd470046e833f094149b37de638a51555f09387ad555041657a9" Jul 10 15:05:06.620234 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:06.620191 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"175ae8b0a89ecd470046e833f094149b37de638a51555f09387ad555041657a9"} err="failed to get container status \"175ae8b0a89ecd470046e833f094149b37de638a51555f09387ad555041657a9\": rpc error: code = NotFound desc = could not find container \"175ae8b0a89ecd470046e833f094149b37de638a51555f09387ad555041657a9\": container with ID starting with 175ae8b0a89ecd470046e833f094149b37de638a51555f09387ad555041657a9 not found: ID does not exist" Jul 10 15:05:06.620234 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:06.620209 2569 scope.go:117] "RemoveContainer" containerID="6525481db3487ecfb4a3a2324ba77781de8019878c41c99801295d002d3145bc" Jul 10 15:05:06.620442 ip-10-0-140-35 kubenswrapper[2569]: E0710 15:05:06.620423 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6525481db3487ecfb4a3a2324ba77781de8019878c41c99801295d002d3145bc\": container with ID starting with 6525481db3487ecfb4a3a2324ba77781de8019878c41c99801295d002d3145bc not found: ID does not exist" containerID="6525481db3487ecfb4a3a2324ba77781de8019878c41c99801295d002d3145bc" Jul 10 15:05:06.620498 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:06.620454 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6525481db3487ecfb4a3a2324ba77781de8019878c41c99801295d002d3145bc"} err="failed to get container status \"6525481db3487ecfb4a3a2324ba77781de8019878c41c99801295d002d3145bc\": rpc error: code = NotFound desc = could not find container \"6525481db3487ecfb4a3a2324ba77781de8019878c41c99801295d002d3145bc\": container with ID starting with 6525481db3487ecfb4a3a2324ba77781de8019878c41c99801295d002d3145bc not found: ID does not exist" Jul 10 15:05:06.699403 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:06.699373 2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97bb8b27-20e2-4214-b9e6-e2c4f6769f0e-istio-podinfo\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 15:05:06.699403 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:06.699398 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsq2f\" (UniqueName: \"kubernetes.io/projected/97bb8b27-20e2-4214-b9e6-e2c4f6769f0e-kube-api-access-zsq2f\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 15:05:06.699535 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:06.699412 2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97bb8b27-20e2-4214-b9e6-e2c4f6769f0e-credential-socket\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 15:05:06.699535 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:06.699424 2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97bb8b27-20e2-4214-b9e6-e2c4f6769f0e-istiod-ca-cert\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 15:05:06.699535 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:06.699436 2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97bb8b27-20e2-4214-b9e6-e2c4f6769f0e-istio-token\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 15:05:06.699535 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:06.699447 2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97bb8b27-20e2-4214-b9e6-e2c4f6769f0e-istio-envoy\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 15:05:06.699535 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:06.699458 2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97bb8b27-20e2-4214-b9e6-e2c4f6769f0e-istio-data\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 15:05:06.912859 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:06.912832 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp"] Jul 10 15:05:06.918080 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:06.918059 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-72c145-openshift-default-5c5c5f45bf-mtkxp"] Jul 10 15:05:06.932045 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:06.932021 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97bb8b27-20e2-4214-b9e6-e2c4f6769f0e" path="/var/lib/kubelet/pods/97bb8b27-20e2-4214-b9e6-e2c4f6769f0e/volumes" Jul 10 15:05:09.274404 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:09.274366 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl"] Jul 10 15:05:09.274942 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:09.274920 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97bb8b27-20e2-4214-b9e6-e2c4f6769f0e" containerName="istio-proxy" Jul 10 15:05:09.274942 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:09.274942 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="97bb8b27-20e2-4214-b9e6-e2c4f6769f0e" containerName="istio-proxy" Jul 10 15:05:09.275176 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:09.274952 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b36d9c70-19e6-47d1-818b-9bb6ffff3d30" containerName="main" Jul 10 15:05:09.275176 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:09.274961 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b36d9c70-19e6-47d1-818b-9bb6ffff3d30" containerName="main" Jul 10 15:05:09.275176 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:09.274976 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97bb8b27-20e2-4214-b9e6-e2c4f6769f0e" containerName="istio-proxy" Jul 10 15:05:09.275176 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:09.274983 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="97bb8b27-20e2-4214-b9e6-e2c4f6769f0e" containerName="istio-proxy" Jul 10 15:05:09.275176 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:09.275082 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b36d9c70-19e6-47d1-818b-9bb6ffff3d30" containerName="main" Jul 10 15:05:09.275176 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:09.275097 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="97bb8b27-20e2-4214-b9e6-e2c4f6769f0e" containerName="istio-proxy" Jul 10 15:05:09.279914 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:09.279894 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl" Jul 10 15:05:09.282514 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:09.282493 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-6d41a0-openshift-default-dockercfg-z6zxm\"" Jul 10 15:05:09.289372 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:09.289347 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl"] Jul 10 15:05:09.423183 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:09.423150 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a7290a-07da-40cb-acc2-2a6bf6f5d2c6-workload-socket\") pod \"e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl\" (UID: \"84a7290a-07da-40cb-acc2-2a6bf6f5d2c6\") " pod="openshift-ingress/e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl" Jul 10 15:05:09.423327 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:09.423195 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84a7290a-07da-40cb-acc2-2a6bf6f5d2c6-istiod-ca-cert\") pod \"e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl\" (UID: \"84a7290a-07da-40cb-acc2-2a6bf6f5d2c6\") " pod="openshift-ingress/e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl" Jul 10 15:05:09.423327 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:09.423231 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84a7290a-07da-40cb-acc2-2a6bf6f5d2c6-istio-token\") pod \"e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl\" (UID: \"84a7290a-07da-40cb-acc2-2a6bf6f5d2c6\") " pod="openshift-ingress/e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl" Jul 10 15:05:09.423327 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:09.423248 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84a7290a-07da-40cb-acc2-2a6bf6f5d2c6-workload-certs\") pod \"e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl\" (UID: \"84a7290a-07da-40cb-acc2-2a6bf6f5d2c6\") " pod="openshift-ingress/e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl" Jul 10 15:05:09.423327 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:09.423265 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a7290a-07da-40cb-acc2-2a6bf6f5d2c6-credential-socket\") pod \"e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl\" (UID: \"84a7290a-07da-40cb-acc2-2a6bf6f5d2c6\") " pod="openshift-ingress/e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl" Jul 10 15:05:09.423476 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:09.423345 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84a7290a-07da-40cb-acc2-2a6bf6f5d2c6-istio-envoy\") pod \"e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl\" (UID: \"84a7290a-07da-40cb-acc2-2a6bf6f5d2c6\") " pod="openshift-ingress/e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl" Jul 10 15:05:09.423476 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:09.423369 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84a7290a-07da-40cb-acc2-2a6bf6f5d2c6-istio-podinfo\") pod \"e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl\" (UID: \"84a7290a-07da-40cb-acc2-2a6bf6f5d2c6\") " pod="openshift-ingress/e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl" Jul 10 15:05:09.423476 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:09.423396 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhc7c\" (UniqueName: \"kubernetes.io/projected/84a7290a-07da-40cb-acc2-2a6bf6f5d2c6-kube-api-access-qhc7c\") pod \"e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl\" (UID: \"84a7290a-07da-40cb-acc2-2a6bf6f5d2c6\") " pod="openshift-ingress/e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl" Jul 10 15:05:09.423476 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:09.423413 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84a7290a-07da-40cb-acc2-2a6bf6f5d2c6-istio-data\") pod \"e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl\" (UID: \"84a7290a-07da-40cb-acc2-2a6bf6f5d2c6\") " pod="openshift-ingress/e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl" Jul 10 15:05:09.524002 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:09.523969 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84a7290a-07da-40cb-acc2-2a6bf6f5d2c6-istio-envoy\") pod \"e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl\" (UID: \"84a7290a-07da-40cb-acc2-2a6bf6f5d2c6\") " pod="openshift-ingress/e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl" Jul 10 15:05:09.524002 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:09.524003 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84a7290a-07da-40cb-acc2-2a6bf6f5d2c6-istio-podinfo\") pod \"e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl\" (UID: \"84a7290a-07da-40cb-acc2-2a6bf6f5d2c6\") " pod="openshift-ingress/e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl" Jul 10 15:05:09.524259 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:09.524021 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhc7c\" (UniqueName: \"kubernetes.io/projected/84a7290a-07da-40cb-acc2-2a6bf6f5d2c6-kube-api-access-qhc7c\") pod \"e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl\" (UID: \"84a7290a-07da-40cb-acc2-2a6bf6f5d2c6\") " pod="openshift-ingress/e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl" Jul 10 15:05:09.524259 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:09.524037 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84a7290a-07da-40cb-acc2-2a6bf6f5d2c6-istio-data\") pod \"e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl\" (UID: \"84a7290a-07da-40cb-acc2-2a6bf6f5d2c6\") " pod="openshift-ingress/e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl" Jul 10 15:05:09.524259 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:09.524068 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a7290a-07da-40cb-acc2-2a6bf6f5d2c6-workload-socket\") pod \"e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl\" (UID: \"84a7290a-07da-40cb-acc2-2a6bf6f5d2c6\") " pod="openshift-ingress/e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl" Jul 10 15:05:09.524259 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:09.524107 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84a7290a-07da-40cb-acc2-2a6bf6f5d2c6-istiod-ca-cert\") pod \"e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl\" (UID: \"84a7290a-07da-40cb-acc2-2a6bf6f5d2c6\") " pod="openshift-ingress/e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl" Jul 10 15:05:09.524259 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:09.524137 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84a7290a-07da-40cb-acc2-2a6bf6f5d2c6-istio-token\") pod \"e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl\" (UID: \"84a7290a-07da-40cb-acc2-2a6bf6f5d2c6\") " pod="openshift-ingress/e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl" Jul 10 15:05:09.524259 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:09.524161 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84a7290a-07da-40cb-acc2-2a6bf6f5d2c6-workload-certs\") pod \"e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl\" (UID: \"84a7290a-07da-40cb-acc2-2a6bf6f5d2c6\") " pod="openshift-ingress/e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl" Jul 10 15:05:09.524584 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:09.524357 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a7290a-07da-40cb-acc2-2a6bf6f5d2c6-credential-socket\") pod \"e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl\" (UID: \"84a7290a-07da-40cb-acc2-2a6bf6f5d2c6\") " pod="openshift-ingress/e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl" Jul 10 15:05:09.524584 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:09.524490 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84a7290a-07da-40cb-acc2-2a6bf6f5d2c6-istio-data\") pod \"e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl\" (UID: \"84a7290a-07da-40cb-acc2-2a6bf6f5d2c6\") " pod="openshift-ingress/e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl" Jul 10 15:05:09.524584 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:09.524531 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a7290a-07da-40cb-acc2-2a6bf6f5d2c6-workload-socket\") pod \"e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl\" (UID: \"84a7290a-07da-40cb-acc2-2a6bf6f5d2c6\") " pod="openshift-ingress/e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl" Jul 10 15:05:09.524791 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:09.524700 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a7290a-07da-40cb-acc2-2a6bf6f5d2c6-credential-socket\") pod \"e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl\" (UID: \"84a7290a-07da-40cb-acc2-2a6bf6f5d2c6\") " pod="openshift-ingress/e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl" Jul 10 15:05:09.524791 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:09.524718 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84a7290a-07da-40cb-acc2-2a6bf6f5d2c6-workload-certs\") pod \"e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl\" (UID: \"84a7290a-07da-40cb-acc2-2a6bf6f5d2c6\") " pod="openshift-ingress/e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl" Jul 10 15:05:09.525136 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:09.525113 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84a7290a-07da-40cb-acc2-2a6bf6f5d2c6-istiod-ca-cert\") pod \"e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl\" (UID: \"84a7290a-07da-40cb-acc2-2a6bf6f5d2c6\") " pod="openshift-ingress/e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl" Jul 10 15:05:09.526324 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:09.526307 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84a7290a-07da-40cb-acc2-2a6bf6f5d2c6-istio-envoy\") pod \"e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl\" (UID: \"84a7290a-07da-40cb-acc2-2a6bf6f5d2c6\") " pod="openshift-ingress/e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl" Jul 10 15:05:09.526714 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:09.526696 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84a7290a-07da-40cb-acc2-2a6bf6f5d2c6-istio-podinfo\") pod \"e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl\" (UID: \"84a7290a-07da-40cb-acc2-2a6bf6f5d2c6\") " pod="openshift-ingress/e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl" Jul 10 15:05:09.532073 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:09.532049 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84a7290a-07da-40cb-acc2-2a6bf6f5d2c6-istio-token\") pod \"e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl\" (UID: \"84a7290a-07da-40cb-acc2-2a6bf6f5d2c6\") " pod="openshift-ingress/e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl" Jul 10 15:05:09.532586 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:09.532567 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhc7c\" (UniqueName: \"kubernetes.io/projected/84a7290a-07da-40cb-acc2-2a6bf6f5d2c6-kube-api-access-qhc7c\") pod \"e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl\" (UID: \"84a7290a-07da-40cb-acc2-2a6bf6f5d2c6\") " pod="openshift-ingress/e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl" Jul 10 15:05:09.590924 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:09.590902 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl" Jul 10 15:05:09.727079 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:09.727053 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl"] Jul 10 15:05:09.728985 ip-10-0-140-35 kubenswrapper[2569]: W0710 15:05:09.728955 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84a7290a_07da_40cb_acc2_2a6bf6f5d2c6.slice/crio-1acdef366753d4c3dab2628c128a87a281be09da016daeee31c49830fa26f043 WatchSource:0}: Error finding container 1acdef366753d4c3dab2628c128a87a281be09da016daeee31c49830fa26f043: Status 404 returned error can't find the container with id 1acdef366753d4c3dab2628c128a87a281be09da016daeee31c49830fa26f043 Jul 10 15:05:09.731041 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:09.731008 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 15:05:09.731146 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:09.731072 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 15:05:09.731146 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:09.731103 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 15:05:10.614297 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:10.614259 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl" event={"ID":"84a7290a-07da-40cb-acc2-2a6bf6f5d2c6","Type":"ContainerStarted","Data":"7b18e333362c8cb27570d73905701fa2bdc9962c0e21b49f28e86113ef32295f"} Jul 10 15:05:10.614660 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:10.614320 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl" event={"ID":"84a7290a-07da-40cb-acc2-2a6bf6f5d2c6","Type":"ContainerStarted","Data":"1acdef366753d4c3dab2628c128a87a281be09da016daeee31c49830fa26f043"} Jul 10 15:05:10.636208 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:10.636161 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl" podStartSLOduration=1.636146342 podStartE2EDuration="1.636146342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 15:05:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 15:05:10.633335446 +0000 UTC m=+2056.384707141" watchObservedRunningTime="2026-07-10 15:05:10.636146342 +0000 UTC m=+2056.387518039" Jul 10 15:05:11.591429 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:11.591387 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl" Jul 10 15:05:11.597481 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:11.597449 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl" Jul 10 15:05:11.619577 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:11.619541 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl" Jul 10 15:05:11.620551 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:11.620531 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl" Jul 10 15:05:26.620954 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:26.620917 2569 patch_prober.go:28] interesting pod/e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Jul 10 15:05:26.621362 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:26.620975 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl" podUID="84a7290a-07da-40cb-acc2-2a6bf6f5d2c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Jul 10 15:05:26.701814 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:26.701782 2569 generic.go:356] "Generic (PLEG): container finished" podID="84a7290a-07da-40cb-acc2-2a6bf6f5d2c6" containerID="7b18e333362c8cb27570d73905701fa2bdc9962c0e21b49f28e86113ef32295f" exitCode=0 Jul 10 15:05:26.701962 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:26.701866 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl" event={"ID":"84a7290a-07da-40cb-acc2-2a6bf6f5d2c6","Type":"ContainerDied","Data":"7b18e333362c8cb27570d73905701fa2bdc9962c0e21b49f28e86113ef32295f"} Jul 10 15:05:26.702412 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:26.702396 2569 scope.go:117] "RemoveContainer" containerID="7b18e333362c8cb27570d73905701fa2bdc9962c0e21b49f28e86113ef32295f" Jul 10 15:05:26.703533 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:26.703503 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 15:05:26.703642 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:26.703573 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 15:05:26.703642 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:26.703627 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 15:05:27.708388 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:27.708355 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl" event={"ID":"84a7290a-07da-40cb-acc2-2a6bf6f5d2c6","Type":"ContainerStarted","Data":"b99adf93c60716ef5acaa8df0ae059d27b5e542214d416f99cd993b9056b9d94"} Jul 10 15:05:28.591916 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:28.591875 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl" Jul 10 15:05:28.592174 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:28.592146 2569 patch_prober.go:28] interesting pod/e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Jul 10 15:05:28.592266 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:28.592207 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl" podUID="84a7290a-07da-40cb-acc2-2a6bf6f5d2c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Jul 10 15:05:29.591820 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:29.591781 2569 patch_prober.go:28] interesting pod/e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Jul 10 15:05:29.592222 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:29.591840 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl" podUID="84a7290a-07da-40cb-acc2-2a6bf6f5d2c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Jul 10 15:05:30.591497 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:30.591459 2569 patch_prober.go:28] interesting pod/e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Jul 10 15:05:30.591674 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:30.591514 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl" podUID="84a7290a-07da-40cb-acc2-2a6bf6f5d2c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Jul 10 15:05:31.591916 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:31.591875 2569 patch_prober.go:28] interesting pod/e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Jul 10 15:05:31.592304 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:31.591943 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl" podUID="84a7290a-07da-40cb-acc2-2a6bf6f5d2c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Jul 10 15:05:32.595321 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:32.595280 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl" Jul 10 15:05:32.595680 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:32.595601 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl" Jul 10 15:05:32.596246 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:32.596229 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl" Jul 10 15:05:40.544118 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:40.544082 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl"] Jul 10 15:05:40.544923 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:40.544304 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl" podUID="84a7290a-07da-40cb-acc2-2a6bf6f5d2c6" containerName="istio-proxy" containerID="cri-o://b99adf93c60716ef5acaa8df0ae059d27b5e542214d416f99cd993b9056b9d94" gracePeriod=30 Jul 10 15:05:45.798153 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:45.798128 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl" Jul 10 15:05:45.799575 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:45.799514 2569 generic.go:356] "Generic (PLEG): container finished" podID="84a7290a-07da-40cb-acc2-2a6bf6f5d2c6" containerID="b99adf93c60716ef5acaa8df0ae059d27b5e542214d416f99cd993b9056b9d94" exitCode=0 Jul 10 15:05:45.799675 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:45.799585 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl" event={"ID":"84a7290a-07da-40cb-acc2-2a6bf6f5d2c6","Type":"ContainerDied","Data":"b99adf93c60716ef5acaa8df0ae059d27b5e542214d416f99cd993b9056b9d94"} Jul 10 15:05:45.799675 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:45.799662 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl" event={"ID":"84a7290a-07da-40cb-acc2-2a6bf6f5d2c6","Type":"ContainerDied","Data":"1acdef366753d4c3dab2628c128a87a281be09da016daeee31c49830fa26f043"} Jul 10 15:05:45.799745 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:45.799685 2569 scope.go:117] "RemoveContainer" containerID="b99adf93c60716ef5acaa8df0ae059d27b5e542214d416f99cd993b9056b9d94" Jul 10 15:05:45.808059 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:45.808030 2569 scope.go:117] "RemoveContainer" containerID="7b18e333362c8cb27570d73905701fa2bdc9962c0e21b49f28e86113ef32295f" Jul 10 15:05:45.815723 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:45.815706 2569 scope.go:117] "RemoveContainer" containerID="b99adf93c60716ef5acaa8df0ae059d27b5e542214d416f99cd993b9056b9d94" Jul 10 15:05:45.816082 ip-10-0-140-35 kubenswrapper[2569]: E0710 15:05:45.816060 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b99adf93c60716ef5acaa8df0ae059d27b5e542214d416f99cd993b9056b9d94\": container with ID starting with b99adf93c60716ef5acaa8df0ae059d27b5e542214d416f99cd993b9056b9d94 not found: ID does not exist" containerID="b99adf93c60716ef5acaa8df0ae059d27b5e542214d416f99cd993b9056b9d94" Jul 10 15:05:45.816180 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:45.816093 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b99adf93c60716ef5acaa8df0ae059d27b5e542214d416f99cd993b9056b9d94"} err="failed to get container status \"b99adf93c60716ef5acaa8df0ae059d27b5e542214d416f99cd993b9056b9d94\": rpc error: code = NotFound desc = could not find container \"b99adf93c60716ef5acaa8df0ae059d27b5e542214d416f99cd993b9056b9d94\": container with ID starting with b99adf93c60716ef5acaa8df0ae059d27b5e542214d416f99cd993b9056b9d94 not found: ID does not exist" Jul 10 15:05:45.816180 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:45.816116 2569 scope.go:117] "RemoveContainer" containerID="7b18e333362c8cb27570d73905701fa2bdc9962c0e21b49f28e86113ef32295f" Jul 10 15:05:45.816439 ip-10-0-140-35 kubenswrapper[2569]: E0710 15:05:45.816422 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b18e333362c8cb27570d73905701fa2bdc9962c0e21b49f28e86113ef32295f\": container with ID starting with 7b18e333362c8cb27570d73905701fa2bdc9962c0e21b49f28e86113ef32295f not found: ID does not exist" containerID="7b18e333362c8cb27570d73905701fa2bdc9962c0e21b49f28e86113ef32295f" Jul 10 15:05:45.816492 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:45.816444 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b18e333362c8cb27570d73905701fa2bdc9962c0e21b49f28e86113ef32295f"} err="failed to get container status \"7b18e333362c8cb27570d73905701fa2bdc9962c0e21b49f28e86113ef32295f\": rpc error: code = NotFound desc = could not find container \"7b18e333362c8cb27570d73905701fa2bdc9962c0e21b49f28e86113ef32295f\": container with ID starting with 7b18e333362c8cb27570d73905701fa2bdc9962c0e21b49f28e86113ef32295f not found: ID does not exist" Jul 10 15:05:45.879030 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:45.878988 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84a7290a-07da-40cb-acc2-2a6bf6f5d2c6-istio-token\") pod \"84a7290a-07da-40cb-acc2-2a6bf6f5d2c6\" (UID: \"84a7290a-07da-40cb-acc2-2a6bf6f5d2c6\") " Jul 10 15:05:45.879200 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:45.879049 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84a7290a-07da-40cb-acc2-2a6bf6f5d2c6-istio-podinfo\") pod \"84a7290a-07da-40cb-acc2-2a6bf6f5d2c6\" (UID: \"84a7290a-07da-40cb-acc2-2a6bf6f5d2c6\") " Jul 10 15:05:45.879200 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:45.879115 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a7290a-07da-40cb-acc2-2a6bf6f5d2c6-workload-socket\") pod \"84a7290a-07da-40cb-acc2-2a6bf6f5d2c6\" (UID: \"84a7290a-07da-40cb-acc2-2a6bf6f5d2c6\") " Jul 10 15:05:45.879200 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:45.879137 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a7290a-07da-40cb-acc2-2a6bf6f5d2c6-credential-socket\") pod \"84a7290a-07da-40cb-acc2-2a6bf6f5d2c6\" (UID: \"84a7290a-07da-40cb-acc2-2a6bf6f5d2c6\") " Jul 10 15:05:45.879200 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:45.879183 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84a7290a-07da-40cb-acc2-2a6bf6f5d2c6-istiod-ca-cert\") pod \"84a7290a-07da-40cb-acc2-2a6bf6f5d2c6\" (UID: \"84a7290a-07da-40cb-acc2-2a6bf6f5d2c6\") " Jul 10 15:05:45.879411 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:45.879218 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qhc7c\" (UniqueName: \"kubernetes.io/projected/84a7290a-07da-40cb-acc2-2a6bf6f5d2c6-kube-api-access-qhc7c\") pod \"84a7290a-07da-40cb-acc2-2a6bf6f5d2c6\" (UID: \"84a7290a-07da-40cb-acc2-2a6bf6f5d2c6\") " Jul 10 15:05:45.879411 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:45.879250 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84a7290a-07da-40cb-acc2-2a6bf6f5d2c6-workload-certs\") pod \"84a7290a-07da-40cb-acc2-2a6bf6f5d2c6\" (UID: \"84a7290a-07da-40cb-acc2-2a6bf6f5d2c6\") " Jul 10 15:05:45.879411 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:45.879273 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84a7290a-07da-40cb-acc2-2a6bf6f5d2c6-istio-envoy\") pod \"84a7290a-07da-40cb-acc2-2a6bf6f5d2c6\" (UID: \"84a7290a-07da-40cb-acc2-2a6bf6f5d2c6\") " Jul 10 15:05:45.879411 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:45.879308 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84a7290a-07da-40cb-acc2-2a6bf6f5d2c6-istio-data\") pod \"84a7290a-07da-40cb-acc2-2a6bf6f5d2c6\" (UID: \"84a7290a-07da-40cb-acc2-2a6bf6f5d2c6\") " Jul 10 15:05:45.879619 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:45.879479 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84a7290a-07da-40cb-acc2-2a6bf6f5d2c6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "84a7290a-07da-40cb-acc2-2a6bf6f5d2c6" (UID: "84a7290a-07da-40cb-acc2-2a6bf6f5d2c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 15:05:45.879619 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:45.879498 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84a7290a-07da-40cb-acc2-2a6bf6f5d2c6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "84a7290a-07da-40cb-acc2-2a6bf6f5d2c6" (UID: "84a7290a-07da-40cb-acc2-2a6bf6f5d2c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 15:05:45.879781 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:45.879635 2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a7290a-07da-40cb-acc2-2a6bf6f5d2c6-workload-socket\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 15:05:45.879781 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:45.879652 2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a7290a-07da-40cb-acc2-2a6bf6f5d2c6-credential-socket\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 15:05:45.879891 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:45.879671 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84a7290a-07da-40cb-acc2-2a6bf6f5d2c6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "84a7290a-07da-40cb-acc2-2a6bf6f5d2c6" (UID: "84a7290a-07da-40cb-acc2-2a6bf6f5d2c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 15:05:45.879984 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:45.879960 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84a7290a-07da-40cb-acc2-2a6bf6f5d2c6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "84a7290a-07da-40cb-acc2-2a6bf6f5d2c6" (UID: "84a7290a-07da-40cb-acc2-2a6bf6f5d2c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 15:05:45.880362 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:45.880341 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84a7290a-07da-40cb-acc2-2a6bf6f5d2c6-istio-data" (OuterVolumeSpecName: "istio-data") pod "84a7290a-07da-40cb-acc2-2a6bf6f5d2c6" (UID: "84a7290a-07da-40cb-acc2-2a6bf6f5d2c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 15:05:45.881364 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:45.881335 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/84a7290a-07da-40cb-acc2-2a6bf6f5d2c6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "84a7290a-07da-40cb-acc2-2a6bf6f5d2c6" (UID: "84a7290a-07da-40cb-acc2-2a6bf6f5d2c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 15:05:45.881461 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:45.881415 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84a7290a-07da-40cb-acc2-2a6bf6f5d2c6-kube-api-access-qhc7c" (OuterVolumeSpecName: "kube-api-access-qhc7c") pod "84a7290a-07da-40cb-acc2-2a6bf6f5d2c6" (UID: "84a7290a-07da-40cb-acc2-2a6bf6f5d2c6"). InnerVolumeSpecName "kube-api-access-qhc7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 15:05:45.881648 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:45.881621 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84a7290a-07da-40cb-acc2-2a6bf6f5d2c6-istio-token" (OuterVolumeSpecName: "istio-token") pod "84a7290a-07da-40cb-acc2-2a6bf6f5d2c6" (UID: "84a7290a-07da-40cb-acc2-2a6bf6f5d2c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 15:05:45.881739 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:45.881713 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84a7290a-07da-40cb-acc2-2a6bf6f5d2c6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "84a7290a-07da-40cb-acc2-2a6bf6f5d2c6" (UID: "84a7290a-07da-40cb-acc2-2a6bf6f5d2c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 15:05:45.980665 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:45.980637 2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84a7290a-07da-40cb-acc2-2a6bf6f5d2c6-istiod-ca-cert\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 15:05:45.980665 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:45.980666 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhc7c\" (UniqueName: \"kubernetes.io/projected/84a7290a-07da-40cb-acc2-2a6bf6f5d2c6-kube-api-access-qhc7c\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 15:05:45.980923 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:45.980680 2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84a7290a-07da-40cb-acc2-2a6bf6f5d2c6-workload-certs\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 15:05:45.980923 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:45.980696 2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84a7290a-07da-40cb-acc2-2a6bf6f5d2c6-istio-envoy\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 15:05:45.980923 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:45.980708 2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84a7290a-07da-40cb-acc2-2a6bf6f5d2c6-istio-data\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 15:05:45.980923 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:45.980721 2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84a7290a-07da-40cb-acc2-2a6bf6f5d2c6-istio-token\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 15:05:45.980923 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:45.980735 2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84a7290a-07da-40cb-acc2-2a6bf6f5d2c6-istio-podinfo\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 15:05:46.805499 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:46.805464 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl" Jul 10 15:05:46.827540 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:46.827511 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl"] Jul 10 15:05:46.831460 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:46.831434 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-6d41a0-openshift-default-544dc66c79-br8rl"] Jul 10 15:05:46.932884 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:46.932853 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84a7290a-07da-40cb-acc2-2a6bf6f5d2c6" path="/var/lib/kubelet/pods/84a7290a-07da-40cb-acc2-2a6bf6f5d2c6/volumes" Jul 10 15:05:48.455072 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:48.455037 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr"] Jul 10 15:05:48.455599 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:48.455579 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84a7290a-07da-40cb-acc2-2a6bf6f5d2c6" containerName="istio-proxy" Jul 10 15:05:48.455599 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:48.455597 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="84a7290a-07da-40cb-acc2-2a6bf6f5d2c6" containerName="istio-proxy" Jul 10 15:05:48.455701 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:48.455691 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="84a7290a-07da-40cb-acc2-2a6bf6f5d2c6" containerName="istio-proxy" Jul 10 15:05:48.455737 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:48.455702 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="84a7290a-07da-40cb-acc2-2a6bf6f5d2c6" containerName="istio-proxy" Jul 10 15:05:48.455737 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:48.455712 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="97bb8b27-20e2-4214-b9e6-e2c4f6769f0e" containerName="istio-proxy" Jul 10 15:05:48.460685 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:48.460666 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr" Jul 10 15:05:48.463277 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:48.463257 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-1f1c9a-openshift-default-dockercfg-fqf95\"" Jul 10 15:05:48.472896 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:48.472870 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr"] Jul 10 15:05:48.603597 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:48.603563 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83c864ba-4d1f-4161-8587-c38bb2a17cc1-istio-podinfo\") pod \"e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr\" (UID: \"83c864ba-4d1f-4161-8587-c38bb2a17cc1\") " pod="openshift-ingress/e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr" Jul 10 15:05:48.603597 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:48.603598 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83c864ba-4d1f-4161-8587-c38bb2a17cc1-istio-token\") pod \"e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr\" (UID: \"83c864ba-4d1f-4161-8587-c38bb2a17cc1\") " pod="openshift-ingress/e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr" Jul 10 15:05:48.603824 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:48.603625 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c864ba-4d1f-4161-8587-c38bb2a17cc1-credential-socket\") pod \"e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr\" (UID: \"83c864ba-4d1f-4161-8587-c38bb2a17cc1\") " pod="openshift-ingress/e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr" Jul 10 15:05:48.603824 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:48.603651 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83c864ba-4d1f-4161-8587-c38bb2a17cc1-istiod-ca-cert\") pod \"e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr\" (UID: \"83c864ba-4d1f-4161-8587-c38bb2a17cc1\") " pod="openshift-ingress/e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr" Jul 10 15:05:48.603824 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:48.603698 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83c864ba-4d1f-4161-8587-c38bb2a17cc1-workload-certs\") pod \"e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr\" (UID: \"83c864ba-4d1f-4161-8587-c38bb2a17cc1\") " pod="openshift-ingress/e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr" Jul 10 15:05:48.603824 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:48.603715 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83c864ba-4d1f-4161-8587-c38bb2a17cc1-istio-data\") pod \"e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr\" (UID: \"83c864ba-4d1f-4161-8587-c38bb2a17cc1\") " pod="openshift-ingress/e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr" Jul 10 15:05:48.603824 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:48.603776 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c864ba-4d1f-4161-8587-c38bb2a17cc1-workload-socket\") pod \"e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr\" (UID: \"83c864ba-4d1f-4161-8587-c38bb2a17cc1\") " pod="openshift-ingress/e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr" Jul 10 15:05:48.603824 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:48.603800 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6ws8\" (UniqueName: \"kubernetes.io/projected/83c864ba-4d1f-4161-8587-c38bb2a17cc1-kube-api-access-t6ws8\") pod \"e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr\" (UID: \"83c864ba-4d1f-4161-8587-c38bb2a17cc1\") " pod="openshift-ingress/e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr" Jul 10 15:05:48.603824 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:48.603822 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83c864ba-4d1f-4161-8587-c38bb2a17cc1-istio-envoy\") pod \"e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr\" (UID: \"83c864ba-4d1f-4161-8587-c38bb2a17cc1\") " pod="openshift-ingress/e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr" Jul 10 15:05:48.705382 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:48.705283 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83c864ba-4d1f-4161-8587-c38bb2a17cc1-istio-podinfo\") pod \"e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr\" (UID: \"83c864ba-4d1f-4161-8587-c38bb2a17cc1\") " pod="openshift-ingress/e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr" Jul 10 15:05:48.705382 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:48.705322 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83c864ba-4d1f-4161-8587-c38bb2a17cc1-istio-token\") pod \"e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr\" (UID: \"83c864ba-4d1f-4161-8587-c38bb2a17cc1\") " pod="openshift-ingress/e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr" Jul 10 15:05:48.705617 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:48.705455 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c864ba-4d1f-4161-8587-c38bb2a17cc1-credential-socket\") pod \"e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr\" (UID: \"83c864ba-4d1f-4161-8587-c38bb2a17cc1\") " pod="openshift-ingress/e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr" Jul 10 15:05:48.705617 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:48.705496 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83c864ba-4d1f-4161-8587-c38bb2a17cc1-istiod-ca-cert\") pod \"e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr\" (UID: \"83c864ba-4d1f-4161-8587-c38bb2a17cc1\") " pod="openshift-ingress/e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr" Jul 10 15:05:48.705617 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:48.705570 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83c864ba-4d1f-4161-8587-c38bb2a17cc1-workload-certs\") pod \"e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr\" (UID: \"83c864ba-4d1f-4161-8587-c38bb2a17cc1\") " pod="openshift-ingress/e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr" Jul 10 15:05:48.705617 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:48.705594 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83c864ba-4d1f-4161-8587-c38bb2a17cc1-istio-data\") pod \"e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr\" (UID: \"83c864ba-4d1f-4161-8587-c38bb2a17cc1\") " pod="openshift-ingress/e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr" Jul 10 15:05:48.705894 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:48.705634 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c864ba-4d1f-4161-8587-c38bb2a17cc1-workload-socket\") pod \"e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr\" (UID: \"83c864ba-4d1f-4161-8587-c38bb2a17cc1\") " pod="openshift-ingress/e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr" Jul 10 15:05:48.705894 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:48.705680 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6ws8\" (UniqueName: \"kubernetes.io/projected/83c864ba-4d1f-4161-8587-c38bb2a17cc1-kube-api-access-t6ws8\") pod \"e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr\" (UID: \"83c864ba-4d1f-4161-8587-c38bb2a17cc1\") " pod="openshift-ingress/e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr" Jul 10 15:05:48.705894 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:48.705728 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83c864ba-4d1f-4161-8587-c38bb2a17cc1-istio-envoy\") pod \"e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr\" (UID: \"83c864ba-4d1f-4161-8587-c38bb2a17cc1\") " pod="openshift-ingress/e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr" Jul 10 15:05:48.706056 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:48.705981 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c864ba-4d1f-4161-8587-c38bb2a17cc1-credential-socket\") pod \"e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr\" (UID: \"83c864ba-4d1f-4161-8587-c38bb2a17cc1\") " pod="openshift-ingress/e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr" Jul 10 15:05:48.706056 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:48.706033 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83c864ba-4d1f-4161-8587-c38bb2a17cc1-workload-certs\") pod \"e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr\" (UID: \"83c864ba-4d1f-4161-8587-c38bb2a17cc1\") " pod="openshift-ingress/e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr" Jul 10 15:05:48.706172 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:48.706072 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83c864ba-4d1f-4161-8587-c38bb2a17cc1-istio-data\") pod \"e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr\" (UID: \"83c864ba-4d1f-4161-8587-c38bb2a17cc1\") " pod="openshift-ingress/e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr" Jul 10 15:05:48.706223 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:48.706206 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c864ba-4d1f-4161-8587-c38bb2a17cc1-workload-socket\") pod \"e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr\" (UID: \"83c864ba-4d1f-4161-8587-c38bb2a17cc1\") " pod="openshift-ingress/e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr" Jul 10 15:05:48.706401 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:48.706382 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83c864ba-4d1f-4161-8587-c38bb2a17cc1-istiod-ca-cert\") pod \"e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr\" (UID: \"83c864ba-4d1f-4161-8587-c38bb2a17cc1\") " pod="openshift-ingress/e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr" Jul 10 15:05:48.707839 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:48.707819 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83c864ba-4d1f-4161-8587-c38bb2a17cc1-istio-envoy\") pod \"e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr\" (UID: \"83c864ba-4d1f-4161-8587-c38bb2a17cc1\") " pod="openshift-ingress/e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr" Jul 10 15:05:48.707968 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:48.707952 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83c864ba-4d1f-4161-8587-c38bb2a17cc1-istio-podinfo\") pod \"e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr\" (UID: \"83c864ba-4d1f-4161-8587-c38bb2a17cc1\") " pod="openshift-ingress/e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr" Jul 10 15:05:48.712945 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:48.712924 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83c864ba-4d1f-4161-8587-c38bb2a17cc1-istio-token\") pod \"e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr\" (UID: \"83c864ba-4d1f-4161-8587-c38bb2a17cc1\") " pod="openshift-ingress/e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr" Jul 10 15:05:48.714044 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:48.714020 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6ws8\" (UniqueName: \"kubernetes.io/projected/83c864ba-4d1f-4161-8587-c38bb2a17cc1-kube-api-access-t6ws8\") pod \"e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr\" (UID: \"83c864ba-4d1f-4161-8587-c38bb2a17cc1\") " pod="openshift-ingress/e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr" Jul 10 15:05:48.771801 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:48.771771 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr" Jul 10 15:05:48.912207 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:48.912180 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr"] Jul 10 15:05:48.913532 ip-10-0-140-35 kubenswrapper[2569]: W0710 15:05:48.913504 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83c864ba_4d1f_4161_8587_c38bb2a17cc1.slice/crio-dfe99fc908ed46ecdc0d0fdd75b07485f52eff134e39d1cbb342c7781abe8004 WatchSource:0}: Error finding container dfe99fc908ed46ecdc0d0fdd75b07485f52eff134e39d1cbb342c7781abe8004: Status 404 returned error can't find the container with id dfe99fc908ed46ecdc0d0fdd75b07485f52eff134e39d1cbb342c7781abe8004 Jul 10 15:05:48.915729 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:48.915697 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 15:05:48.915815 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:48.915771 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 15:05:48.915815 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:48.915803 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 15:05:49.823398 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:49.823355 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr" event={"ID":"83c864ba-4d1f-4161-8587-c38bb2a17cc1","Type":"ContainerStarted","Data":"b46eae7e3b3eaaf06cc62b992d93460f0c653140105c2f2ef763d0a4725ad8ba"} Jul 10 15:05:49.823940 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:49.823412 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr" event={"ID":"83c864ba-4d1f-4161-8587-c38bb2a17cc1","Type":"ContainerStarted","Data":"dfe99fc908ed46ecdc0d0fdd75b07485f52eff134e39d1cbb342c7781abe8004"} Jul 10 15:05:49.843479 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:49.843420 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr" podStartSLOduration=1.8434014890000001 podStartE2EDuration="1.843401489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 15:05:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 15:05:49.842925599 +0000 UTC m=+2095.594297295" watchObservedRunningTime="2026-07-10 15:05:49.843401489 +0000 UTC m=+2095.594773185" Jul 10 15:05:50.772863 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:50.772832 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr" Jul 10 15:05:50.777833 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:50.777808 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr" Jul 10 15:05:50.828810 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:50.828775 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr" Jul 10 15:05:50.829738 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:50.829718 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr" Jul 10 15:05:54.822266 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:54.822235 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-n6pfx_296be890-64e9-4fb3-8c68-e4c2a7c8cc2c/console-operator/1.log" Jul 10 15:05:54.825213 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:54.825190 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-n6pfx_296be890-64e9-4fb3-8c68-e4c2a7c8cc2c/console-operator/1.log" Jul 10 15:05:54.828943 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:54.828917 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxvq2_016248a0-bbe2-44c9-8e51-5d4695ee6a52/ovn-acl-logging/0.log" Jul 10 15:05:54.831520 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:05:54.831502 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxvq2_016248a0-bbe2-44c9-8e51-5d4695ee6a52/ovn-acl-logging/0.log" Jul 10 15:06:06.870907 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:06:06.870865 2569 patch_prober.go:28] interesting pod/e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 10 15:06:06.871284 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:06:06.870943 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr" podUID="83c864ba-4d1f-4161-8587-c38bb2a17cc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 10 15:06:07.913360 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:06:07.913327 2569 generic.go:356] "Generic (PLEG): container finished" podID="83c864ba-4d1f-4161-8587-c38bb2a17cc1" containerID="b46eae7e3b3eaaf06cc62b992d93460f0c653140105c2f2ef763d0a4725ad8ba" exitCode=0 Jul 10 15:06:07.913843 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:06:07.913401 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr" event={"ID":"83c864ba-4d1f-4161-8587-c38bb2a17cc1","Type":"ContainerDied","Data":"b46eae7e3b3eaaf06cc62b992d93460f0c653140105c2f2ef763d0a4725ad8ba"} Jul 10 15:06:07.913843 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:06:07.913788 2569 scope.go:117] "RemoveContainer" containerID="b46eae7e3b3eaaf06cc62b992d93460f0c653140105c2f2ef763d0a4725ad8ba" Jul 10 15:06:07.914800 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:06:07.914770 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 15:06:07.914870 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:06:07.914844 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 15:06:07.914909 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:06:07.914883 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 15:06:08.920028 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:06:08.919995 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr" event={"ID":"83c864ba-4d1f-4161-8587-c38bb2a17cc1","Type":"ContainerStarted","Data":"cfbe211d43f503e9befc516748603ce1eab3463451d6f47f96c081c826aad107"} Jul 10 15:06:09.772308 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:06:09.772276 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr" Jul 10 15:06:09.772504 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:06:09.772449 2569 patch_prober.go:28] interesting pod/e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Jul 10 15:06:09.772577 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:06:09.772502 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr" podUID="83c864ba-4d1f-4161-8587-c38bb2a17cc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Jul 10 15:06:10.772932 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:06:10.772890 2569 patch_prober.go:28] interesting pod/e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Jul 10 15:06:10.773332 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:06:10.772961 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr" podUID="83c864ba-4d1f-4161-8587-c38bb2a17cc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Jul 10 15:06:11.772231 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:06:11.772199 2569 patch_prober.go:28] interesting pod/e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Jul 10 15:06:11.772445 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:06:11.772257 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr" podUID="83c864ba-4d1f-4161-8587-c38bb2a17cc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Jul 10 15:06:12.772190 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:06:12.772154 2569 patch_prober.go:28] interesting pod/e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Jul 10 15:06:12.772591 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:06:12.772208 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr" podUID="83c864ba-4d1f-4161-8587-c38bb2a17cc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Jul 10 15:06:13.775392 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:06:13.775356 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr" Jul 10 15:06:13.775980 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:06:13.775764 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr" Jul 10 15:06:13.776434 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:06:13.776413 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr" Jul 10 15:06:16.516448 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:06:16.516414 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-1f1c9a/auth-test-model-1f1c9a-kserve-797cd549f-x6cvm"] Jul 10 15:06:16.516830 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:06:16.516805 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84a7290a-07da-40cb-acc2-2a6bf6f5d2c6" containerName="istio-proxy" Jul 10 15:06:16.516830 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:06:16.516818 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="84a7290a-07da-40cb-acc2-2a6bf6f5d2c6" containerName="istio-proxy" Jul 10 15:06:16.519073 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:06:16.519056 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1f1c9a/auth-test-model-1f1c9a-kserve-797cd549f-x6cvm" Jul 10 15:06:16.523020 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:06:16.522989 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-1f1c9a\"/\"default-dockercfg-lvz4r\"" Jul 10 15:06:16.523165 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:06:16.523037 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-1f1c9a\"/\"auth-test-model-1f1c9a-kserve-self-signed-certs\"" Jul 10 15:06:16.523165 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:06:16.523091 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-1f1c9a\"/\"openshift-service-ca.crt\"" Jul 10 15:06:16.523165 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:06:16.523101 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-1f1c9a\"/\"kube-root-ca.crt\"" Jul 10 15:06:16.530451 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:06:16.530428 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1f1c9a/auth-test-model-1f1c9a-kserve-797cd549f-x6cvm"] Jul 10 15:06:16.565971 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:06:16.565943 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/babaf7a1-de49-40e3-b708-0d2386b387af-dshm\") pod \"auth-test-model-1f1c9a-kserve-797cd549f-x6cvm\" (UID: \"babaf7a1-de49-40e3-b708-0d2386b387af\") " pod="ai-tenant-e2e-shared-a-1f1c9a/auth-test-model-1f1c9a-kserve-797cd549f-x6cvm" Jul 10 15:06:16.566117 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:06:16.565983 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/babaf7a1-de49-40e3-b708-0d2386b387af-tmp-dir\") pod \"auth-test-model-1f1c9a-kserve-797cd549f-x6cvm\" (UID: \"babaf7a1-de49-40e3-b708-0d2386b387af\") " pod="ai-tenant-e2e-shared-a-1f1c9a/auth-test-model-1f1c9a-kserve-797cd549f-x6cvm" Jul 10 15:06:16.566117 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:06:16.566002 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/babaf7a1-de49-40e3-b708-0d2386b387af-home\") pod \"auth-test-model-1f1c9a-kserve-797cd549f-x6cvm\" (UID: \"babaf7a1-de49-40e3-b708-0d2386b387af\") " pod="ai-tenant-e2e-shared-a-1f1c9a/auth-test-model-1f1c9a-kserve-797cd549f-x6cvm" Jul 10 15:06:16.566117 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:06:16.566043 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/babaf7a1-de49-40e3-b708-0d2386b387af-model-cache\") pod \"auth-test-model-1f1c9a-kserve-797cd549f-x6cvm\" (UID: \"babaf7a1-de49-40e3-b708-0d2386b387af\") " pod="ai-tenant-e2e-shared-a-1f1c9a/auth-test-model-1f1c9a-kserve-797cd549f-x6cvm" Jul 10 15:06:16.566117 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:06:16.566092 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/babaf7a1-de49-40e3-b708-0d2386b387af-tls-certs\") pod \"auth-test-model-1f1c9a-kserve-797cd549f-x6cvm\" (UID: \"babaf7a1-de49-40e3-b708-0d2386b387af\") " pod="ai-tenant-e2e-shared-a-1f1c9a/auth-test-model-1f1c9a-kserve-797cd549f-x6cvm" Jul 10 15:06:16.566255 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:06:16.566139 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdwhp\" (UniqueName: \"kubernetes.io/projected/babaf7a1-de49-40e3-b708-0d2386b387af-kube-api-access-kdwhp\") pod \"auth-test-model-1f1c9a-kserve-797cd549f-x6cvm\" (UID: \"babaf7a1-de49-40e3-b708-0d2386b387af\") " pod="ai-tenant-e2e-shared-a-1f1c9a/auth-test-model-1f1c9a-kserve-797cd549f-x6cvm" Jul 10 15:06:16.667361 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:06:16.667329 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/babaf7a1-de49-40e3-b708-0d2386b387af-dshm\") pod \"auth-test-model-1f1c9a-kserve-797cd549f-x6cvm\" (UID: \"babaf7a1-de49-40e3-b708-0d2386b387af\") " pod="ai-tenant-e2e-shared-a-1f1c9a/auth-test-model-1f1c9a-kserve-797cd549f-x6cvm" Jul 10 15:06:16.667558 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:06:16.667376 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/babaf7a1-de49-40e3-b708-0d2386b387af-tmp-dir\") pod \"auth-test-model-1f1c9a-kserve-797cd549f-x6cvm\" (UID: \"babaf7a1-de49-40e3-b708-0d2386b387af\") " pod="ai-tenant-e2e-shared-a-1f1c9a/auth-test-model-1f1c9a-kserve-797cd549f-x6cvm" Jul 10 15:06:16.667558 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:06:16.667396 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/babaf7a1-de49-40e3-b708-0d2386b387af-home\") pod \"auth-test-model-1f1c9a-kserve-797cd549f-x6cvm\" (UID: \"babaf7a1-de49-40e3-b708-0d2386b387af\") " pod="ai-tenant-e2e-shared-a-1f1c9a/auth-test-model-1f1c9a-kserve-797cd549f-x6cvm" Jul 10 15:06:16.667558 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:06:16.667414 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/babaf7a1-de49-40e3-b708-0d2386b387af-model-cache\") pod \"auth-test-model-1f1c9a-kserve-797cd549f-x6cvm\" (UID: \"babaf7a1-de49-40e3-b708-0d2386b387af\") " pod="ai-tenant-e2e-shared-a-1f1c9a/auth-test-model-1f1c9a-kserve-797cd549f-x6cvm" Jul 10 15:06:16.667558 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:06:16.667451 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/babaf7a1-de49-40e3-b708-0d2386b387af-tls-certs\") pod \"auth-test-model-1f1c9a-kserve-797cd549f-x6cvm\" (UID: \"babaf7a1-de49-40e3-b708-0d2386b387af\") " pod="ai-tenant-e2e-shared-a-1f1c9a/auth-test-model-1f1c9a-kserve-797cd549f-x6cvm" Jul 10 15:06:16.667558 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:06:16.667489 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdwhp\" (UniqueName: \"kubernetes.io/projected/babaf7a1-de49-40e3-b708-0d2386b387af-kube-api-access-kdwhp\") pod \"auth-test-model-1f1c9a-kserve-797cd549f-x6cvm\" (UID: \"babaf7a1-de49-40e3-b708-0d2386b387af\") " pod="ai-tenant-e2e-shared-a-1f1c9a/auth-test-model-1f1c9a-kserve-797cd549f-x6cvm" Jul 10 15:06:16.667883 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:06:16.667862 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/babaf7a1-de49-40e3-b708-0d2386b387af-home\") pod \"auth-test-model-1f1c9a-kserve-797cd549f-x6cvm\" (UID: \"babaf7a1-de49-40e3-b708-0d2386b387af\") " pod="ai-tenant-e2e-shared-a-1f1c9a/auth-test-model-1f1c9a-kserve-797cd549f-x6cvm" Jul 10 15:06:16.667977 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:06:16.667950 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/babaf7a1-de49-40e3-b708-0d2386b387af-tmp-dir\") pod \"auth-test-model-1f1c9a-kserve-797cd549f-x6cvm\" (UID: \"babaf7a1-de49-40e3-b708-0d2386b387af\") " pod="ai-tenant-e2e-shared-a-1f1c9a/auth-test-model-1f1c9a-kserve-797cd549f-x6cvm" Jul 10 15:06:16.668088 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:06:16.667988 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/babaf7a1-de49-40e3-b708-0d2386b387af-model-cache\") pod \"auth-test-model-1f1c9a-kserve-797cd549f-x6cvm\" (UID: \"babaf7a1-de49-40e3-b708-0d2386b387af\") " pod="ai-tenant-e2e-shared-a-1f1c9a/auth-test-model-1f1c9a-kserve-797cd549f-x6cvm" Jul 10 15:06:16.669615 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:06:16.669596 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/babaf7a1-de49-40e3-b708-0d2386b387af-dshm\") pod \"auth-test-model-1f1c9a-kserve-797cd549f-x6cvm\" (UID: \"babaf7a1-de49-40e3-b708-0d2386b387af\") " pod="ai-tenant-e2e-shared-a-1f1c9a/auth-test-model-1f1c9a-kserve-797cd549f-x6cvm" Jul 10 15:06:16.669902 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:06:16.669885 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/babaf7a1-de49-40e3-b708-0d2386b387af-tls-certs\") pod \"auth-test-model-1f1c9a-kserve-797cd549f-x6cvm\" (UID: \"babaf7a1-de49-40e3-b708-0d2386b387af\") " pod="ai-tenant-e2e-shared-a-1f1c9a/auth-test-model-1f1c9a-kserve-797cd549f-x6cvm" Jul 10 15:06:16.676397 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:06:16.676373 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdwhp\" (UniqueName: \"kubernetes.io/projected/babaf7a1-de49-40e3-b708-0d2386b387af-kube-api-access-kdwhp\") pod \"auth-test-model-1f1c9a-kserve-797cd549f-x6cvm\" (UID: \"babaf7a1-de49-40e3-b708-0d2386b387af\") " pod="ai-tenant-e2e-shared-a-1f1c9a/auth-test-model-1f1c9a-kserve-797cd549f-x6cvm" Jul 10 15:06:16.829186 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:06:16.829114 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1f1c9a/auth-test-model-1f1c9a-kserve-797cd549f-x6cvm" Jul 10 15:06:16.961309 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:06:16.961285 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1f1c9a/auth-test-model-1f1c9a-kserve-797cd549f-x6cvm"] Jul 10 15:06:16.963160 ip-10-0-140-35 kubenswrapper[2569]: W0710 15:06:16.963137 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbabaf7a1_de49_40e3_b708_0d2386b387af.slice/crio-1079248ae999a86522e3415b1f234095ddae04d39ec23ccd906e5d72d0963085 WatchSource:0}: Error finding container 1079248ae999a86522e3415b1f234095ddae04d39ec23ccd906e5d72d0963085: Status 404 returned error can't find the container with id 1079248ae999a86522e3415b1f234095ddae04d39ec23ccd906e5d72d0963085 Jul 10 15:06:17.964080 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:06:17.964046 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1f1c9a/auth-test-model-1f1c9a-kserve-797cd549f-x6cvm" event={"ID":"babaf7a1-de49-40e3-b708-0d2386b387af","Type":"ContainerStarted","Data":"1cb772d4381e4b5e36a1fe39fc55a0a9e8a5d4185f9cc6e32fa7cd768707abeb"} Jul 10 15:06:17.964458 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:06:17.964094 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1f1c9a/auth-test-model-1f1c9a-kserve-797cd549f-x6cvm" event={"ID":"babaf7a1-de49-40e3-b708-0d2386b387af","Type":"ContainerStarted","Data":"1079248ae999a86522e3415b1f234095ddae04d39ec23ccd906e5d72d0963085"} Jul 10 15:06:17.980164 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:06:17.980116 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-1f1c9a/auth-test-model-1f1c9a-kserve-797cd549f-x6cvm" podStartSLOduration=1.980103374 podStartE2EDuration="1.980103374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 15:06:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 15:06:17.979621403 +0000 UTC m=+2123.730993100" watchObservedRunningTime="2026-07-10 15:06:17.980103374 +0000 UTC m=+2123.731475069" Jul 10 15:06:26.829550 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:06:26.829512 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-1f1c9a/auth-test-model-1f1c9a-kserve-797cd549f-x6cvm" Jul 10 15:06:26.830062 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:06:26.829574 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-1f1c9a/auth-test-model-1f1c9a-kserve-797cd549f-x6cvm" Jul 10 15:06:26.841884 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:06:26.841859 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-1f1c9a/auth-test-model-1f1c9a-kserve-797cd549f-x6cvm" Jul 10 15:06:27.020039 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:06:27.020002 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-1f1c9a/auth-test-model-1f1c9a-kserve-797cd549f-x6cvm" Jul 10 15:08:45.183152 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:08:45.183069 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1f1c9a/auth-test-model-1f1c9a-kserve-797cd549f-x6cvm"] Jul 10 15:09:00.203051 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:09:00.202987 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-1f1c9a/auth-test-model-1f1c9a-kserve-797cd549f-x6cvm" podUID="babaf7a1-de49-40e3-b708-0d2386b387af" containerName="main" containerID="cri-o://1cb772d4381e4b5e36a1fe39fc55a0a9e8a5d4185f9cc6e32fa7cd768707abeb" gracePeriod=45 Jul 10 15:09:00.448933 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:09:00.448910 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1f1c9a/auth-test-model-1f1c9a-kserve-797cd549f-x6cvm" Jul 10 15:09:00.603089 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:09:00.603016 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/babaf7a1-de49-40e3-b708-0d2386b387af-dshm\") pod \"babaf7a1-de49-40e3-b708-0d2386b387af\" (UID: \"babaf7a1-de49-40e3-b708-0d2386b387af\") " Jul 10 15:09:00.603222 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:09:00.603118 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/babaf7a1-de49-40e3-b708-0d2386b387af-tmp-dir\") pod \"babaf7a1-de49-40e3-b708-0d2386b387af\" (UID: \"babaf7a1-de49-40e3-b708-0d2386b387af\") " Jul 10 15:09:00.603222 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:09:00.603168 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/babaf7a1-de49-40e3-b708-0d2386b387af-home\") pod \"babaf7a1-de49-40e3-b708-0d2386b387af\" (UID: \"babaf7a1-de49-40e3-b708-0d2386b387af\") " Jul 10 15:09:00.603222 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:09:00.603203 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/babaf7a1-de49-40e3-b708-0d2386b387af-tls-certs\") pod \"babaf7a1-de49-40e3-b708-0d2386b387af\" (UID: \"babaf7a1-de49-40e3-b708-0d2386b387af\") " Jul 10 15:09:00.603420 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:09:00.603234 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kdwhp\" (UniqueName: \"kubernetes.io/projected/babaf7a1-de49-40e3-b708-0d2386b387af-kube-api-access-kdwhp\") pod \"babaf7a1-de49-40e3-b708-0d2386b387af\" (UID: \"babaf7a1-de49-40e3-b708-0d2386b387af\") " Jul 10 15:09:00.603420 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:09:00.603285 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/babaf7a1-de49-40e3-b708-0d2386b387af-model-cache\") pod \"babaf7a1-de49-40e3-b708-0d2386b387af\" (UID: \"babaf7a1-de49-40e3-b708-0d2386b387af\") " Jul 10 15:09:00.603420 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:09:00.603400 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/babaf7a1-de49-40e3-b708-0d2386b387af-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "babaf7a1-de49-40e3-b708-0d2386b387af" (UID: "babaf7a1-de49-40e3-b708-0d2386b387af"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 15:09:00.603586 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:09:00.603460 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/babaf7a1-de49-40e3-b708-0d2386b387af-home" (OuterVolumeSpecName: "home") pod "babaf7a1-de49-40e3-b708-0d2386b387af" (UID: "babaf7a1-de49-40e3-b708-0d2386b387af"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 15:09:00.603650 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:09:00.603634 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/babaf7a1-de49-40e3-b708-0d2386b387af-model-cache" (OuterVolumeSpecName: "model-cache") pod "babaf7a1-de49-40e3-b708-0d2386b387af" (UID: "babaf7a1-de49-40e3-b708-0d2386b387af"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 15:09:00.603697 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:09:00.603662 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/babaf7a1-de49-40e3-b708-0d2386b387af-tmp-dir\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 15:09:00.603697 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:09:00.603679 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/babaf7a1-de49-40e3-b708-0d2386b387af-home\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 15:09:00.605355 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:09:00.605330 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/babaf7a1-de49-40e3-b708-0d2386b387af-dshm" (OuterVolumeSpecName: "dshm") pod "babaf7a1-de49-40e3-b708-0d2386b387af" (UID: "babaf7a1-de49-40e3-b708-0d2386b387af"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 15:09:00.605469 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:09:00.605376 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/babaf7a1-de49-40e3-b708-0d2386b387af-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "babaf7a1-de49-40e3-b708-0d2386b387af" (UID: "babaf7a1-de49-40e3-b708-0d2386b387af"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 15:09:00.605469 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:09:00.605380 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/babaf7a1-de49-40e3-b708-0d2386b387af-kube-api-access-kdwhp" (OuterVolumeSpecName: "kube-api-access-kdwhp") pod "babaf7a1-de49-40e3-b708-0d2386b387af" (UID: "babaf7a1-de49-40e3-b708-0d2386b387af"). InnerVolumeSpecName "kube-api-access-kdwhp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 15:09:00.704698 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:09:00.704676 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdwhp\" (UniqueName: \"kubernetes.io/projected/babaf7a1-de49-40e3-b708-0d2386b387af-kube-api-access-kdwhp\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 15:09:00.704698 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:09:00.704699 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/babaf7a1-de49-40e3-b708-0d2386b387af-model-cache\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 15:09:00.704889 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:09:00.704709 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/babaf7a1-de49-40e3-b708-0d2386b387af-dshm\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 15:09:00.704889 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:09:00.704717 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/babaf7a1-de49-40e3-b708-0d2386b387af-tls-certs\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 15:09:00.747276 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:09:00.747246 2569 generic.go:356] "Generic (PLEG): container finished" podID="babaf7a1-de49-40e3-b708-0d2386b387af" containerID="1cb772d4381e4b5e36a1fe39fc55a0a9e8a5d4185f9cc6e32fa7cd768707abeb" exitCode=0 Jul 10 15:09:00.747427 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:09:00.747293 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1f1c9a/auth-test-model-1f1c9a-kserve-797cd549f-x6cvm" Jul 10 15:09:00.747427 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:09:00.747324 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1f1c9a/auth-test-model-1f1c9a-kserve-797cd549f-x6cvm" event={"ID":"babaf7a1-de49-40e3-b708-0d2386b387af","Type":"ContainerDied","Data":"1cb772d4381e4b5e36a1fe39fc55a0a9e8a5d4185f9cc6e32fa7cd768707abeb"} Jul 10 15:09:00.747427 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:09:00.747368 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1f1c9a/auth-test-model-1f1c9a-kserve-797cd549f-x6cvm" event={"ID":"babaf7a1-de49-40e3-b708-0d2386b387af","Type":"ContainerDied","Data":"1079248ae999a86522e3415b1f234095ddae04d39ec23ccd906e5d72d0963085"} Jul 10 15:09:00.747427 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:09:00.747388 2569 scope.go:117] "RemoveContainer" containerID="1cb772d4381e4b5e36a1fe39fc55a0a9e8a5d4185f9cc6e32fa7cd768707abeb" Jul 10 15:09:00.756084 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:09:00.756069 2569 scope.go:117] "RemoveContainer" containerID="1cb772d4381e4b5e36a1fe39fc55a0a9e8a5d4185f9cc6e32fa7cd768707abeb" Jul 10 15:09:00.756327 ip-10-0-140-35 kubenswrapper[2569]: E0710 15:09:00.756311 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cb772d4381e4b5e36a1fe39fc55a0a9e8a5d4185f9cc6e32fa7cd768707abeb\": container with ID starting with 1cb772d4381e4b5e36a1fe39fc55a0a9e8a5d4185f9cc6e32fa7cd768707abeb not found: ID does not exist" containerID="1cb772d4381e4b5e36a1fe39fc55a0a9e8a5d4185f9cc6e32fa7cd768707abeb" Jul 10 15:09:00.756364 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:09:00.756335 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cb772d4381e4b5e36a1fe39fc55a0a9e8a5d4185f9cc6e32fa7cd768707abeb"} err="failed to get container status \"1cb772d4381e4b5e36a1fe39fc55a0a9e8a5d4185f9cc6e32fa7cd768707abeb\": rpc error: code = NotFound desc = could not find container \"1cb772d4381e4b5e36a1fe39fc55a0a9e8a5d4185f9cc6e32fa7cd768707abeb\": container with ID starting with 1cb772d4381e4b5e36a1fe39fc55a0a9e8a5d4185f9cc6e32fa7cd768707abeb not found: ID does not exist" Jul 10 15:09:00.767999 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:09:00.767973 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1f1c9a/auth-test-model-1f1c9a-kserve-797cd549f-x6cvm"] Jul 10 15:09:00.771839 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:09:00.771818 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-1f1c9a/auth-test-model-1f1c9a-kserve-797cd549f-x6cvm"] Jul 10 15:09:00.937002 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:09:00.936963 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="babaf7a1-de49-40e3-b708-0d2386b387af" path="/var/lib/kubelet/pods/babaf7a1-de49-40e3-b708-0d2386b387af/volumes" Jul 10 15:09:06.996675 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:09:06.996641 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-1f1c9a/sub-test-model-1f1c9a-kserve-778878d5-ztn9c"] Jul 10 15:09:06.997070 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:09:06.997053 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="babaf7a1-de49-40e3-b708-0d2386b387af" containerName="main" Jul 10 15:09:06.997070 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:09:06.997066 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="babaf7a1-de49-40e3-b708-0d2386b387af" containerName="main" Jul 10 15:09:06.997142 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:09:06.997128 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="babaf7a1-de49-40e3-b708-0d2386b387af" containerName="main" Jul 10 15:09:07.001274 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:09:07.001254 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1f1c9a/sub-test-model-1f1c9a-kserve-778878d5-ztn9c" Jul 10 15:09:07.007972 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:09:07.007945 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-1f1c9a\"/\"kube-root-ca.crt\"" Jul 10 15:09:07.008095 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:09:07.008009 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-1f1c9a\"/\"openshift-service-ca.crt\"" Jul 10 15:09:07.008168 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:09:07.007945 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-1f1c9a\"/\"sub-test-model-1f1c9a-kserve-self-signed-certs\"" Jul 10 15:09:07.008496 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:09:07.008197 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-1f1c9a\"/\"default-dockercfg-lvz4r\"" Jul 10 15:09:07.013087 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:09:07.013065 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1f1c9a/sub-test-model-1f1c9a-kserve-778878d5-ztn9c"] Jul 10 15:09:07.162179 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:09:07.162143 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdz47\" (UniqueName: \"kubernetes.io/projected/629a7581-b804-409c-96b0-3640c50aecfa-kube-api-access-zdz47\") pod \"sub-test-model-1f1c9a-kserve-778878d5-ztn9c\" (UID: \"629a7581-b804-409c-96b0-3640c50aecfa\") " pod="ai-tenant-e2e-shared-a-1f1c9a/sub-test-model-1f1c9a-kserve-778878d5-ztn9c" Jul 10 15:09:07.162373 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:09:07.162192 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/629a7581-b804-409c-96b0-3640c50aecfa-tmp-dir\") pod \"sub-test-model-1f1c9a-kserve-778878d5-ztn9c\" (UID: \"629a7581-b804-409c-96b0-3640c50aecfa\") " pod="ai-tenant-e2e-shared-a-1f1c9a/sub-test-model-1f1c9a-kserve-778878d5-ztn9c" Jul 10 15:09:07.162373 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:09:07.162243 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/629a7581-b804-409c-96b0-3640c50aecfa-model-cache\") pod \"sub-test-model-1f1c9a-kserve-778878d5-ztn9c\" (UID: \"629a7581-b804-409c-96b0-3640c50aecfa\") " pod="ai-tenant-e2e-shared-a-1f1c9a/sub-test-model-1f1c9a-kserve-778878d5-ztn9c" Jul 10 15:09:07.162373 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:09:07.162279 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/629a7581-b804-409c-96b0-3640c50aecfa-dshm\") pod \"sub-test-model-1f1c9a-kserve-778878d5-ztn9c\" (UID: \"629a7581-b804-409c-96b0-3640c50aecfa\") " pod="ai-tenant-e2e-shared-a-1f1c9a/sub-test-model-1f1c9a-kserve-778878d5-ztn9c" Jul 10 15:09:07.162507 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:09:07.162368 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/629a7581-b804-409c-96b0-3640c50aecfa-home\") pod \"sub-test-model-1f1c9a-kserve-778878d5-ztn9c\" (UID: \"629a7581-b804-409c-96b0-3640c50aecfa\") " pod="ai-tenant-e2e-shared-a-1f1c9a/sub-test-model-1f1c9a-kserve-778878d5-ztn9c" Jul 10 15:09:07.162507 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:09:07.162402 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/629a7581-b804-409c-96b0-3640c50aecfa-tls-certs\") pod \"sub-test-model-1f1c9a-kserve-778878d5-ztn9c\" (UID: \"629a7581-b804-409c-96b0-3640c50aecfa\") " pod="ai-tenant-e2e-shared-a-1f1c9a/sub-test-model-1f1c9a-kserve-778878d5-ztn9c" Jul 10 15:09:07.263194 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:09:07.263101 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/629a7581-b804-409c-96b0-3640c50aecfa-home\") pod \"sub-test-model-1f1c9a-kserve-778878d5-ztn9c\" (UID: \"629a7581-b804-409c-96b0-3640c50aecfa\") " pod="ai-tenant-e2e-shared-a-1f1c9a/sub-test-model-1f1c9a-kserve-778878d5-ztn9c" Jul 10 15:09:07.263194 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:09:07.263143 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/629a7581-b804-409c-96b0-3640c50aecfa-tls-certs\") pod \"sub-test-model-1f1c9a-kserve-778878d5-ztn9c\" (UID: \"629a7581-b804-409c-96b0-3640c50aecfa\") " pod="ai-tenant-e2e-shared-a-1f1c9a/sub-test-model-1f1c9a-kserve-778878d5-ztn9c" Jul 10 15:09:07.263194 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:09:07.263165 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdz47\" (UniqueName: \"kubernetes.io/projected/629a7581-b804-409c-96b0-3640c50aecfa-kube-api-access-zdz47\") pod \"sub-test-model-1f1c9a-kserve-778878d5-ztn9c\" (UID: \"629a7581-b804-409c-96b0-3640c50aecfa\") " pod="ai-tenant-e2e-shared-a-1f1c9a/sub-test-model-1f1c9a-kserve-778878d5-ztn9c" Jul 10 15:09:07.263427 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:09:07.263198 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/629a7581-b804-409c-96b0-3640c50aecfa-tmp-dir\") pod \"sub-test-model-1f1c9a-kserve-778878d5-ztn9c\" (UID: \"629a7581-b804-409c-96b0-3640c50aecfa\") " pod="ai-tenant-e2e-shared-a-1f1c9a/sub-test-model-1f1c9a-kserve-778878d5-ztn9c" Jul 10 15:09:07.263427 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:09:07.263234 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/629a7581-b804-409c-96b0-3640c50aecfa-model-cache\") pod \"sub-test-model-1f1c9a-kserve-778878d5-ztn9c\" (UID: \"629a7581-b804-409c-96b0-3640c50aecfa\") " pod="ai-tenant-e2e-shared-a-1f1c9a/sub-test-model-1f1c9a-kserve-778878d5-ztn9c" Jul 10 15:09:07.263427 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:09:07.263287 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/629a7581-b804-409c-96b0-3640c50aecfa-dshm\") pod \"sub-test-model-1f1c9a-kserve-778878d5-ztn9c\" (UID: \"629a7581-b804-409c-96b0-3640c50aecfa\") " pod="ai-tenant-e2e-shared-a-1f1c9a/sub-test-model-1f1c9a-kserve-778878d5-ztn9c" Jul 10 15:09:07.263562 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:09:07.263539 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/629a7581-b804-409c-96b0-3640c50aecfa-home\") pod \"sub-test-model-1f1c9a-kserve-778878d5-ztn9c\" (UID: \"629a7581-b804-409c-96b0-3640c50aecfa\") " pod="ai-tenant-e2e-shared-a-1f1c9a/sub-test-model-1f1c9a-kserve-778878d5-ztn9c" Jul 10 15:09:07.263614 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:09:07.263595 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/629a7581-b804-409c-96b0-3640c50aecfa-model-cache\") pod \"sub-test-model-1f1c9a-kserve-778878d5-ztn9c\" (UID: \"629a7581-b804-409c-96b0-3640c50aecfa\") " pod="ai-tenant-e2e-shared-a-1f1c9a/sub-test-model-1f1c9a-kserve-778878d5-ztn9c" Jul 10 15:09:07.263676 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:09:07.263617 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/629a7581-b804-409c-96b0-3640c50aecfa-tmp-dir\") pod \"sub-test-model-1f1c9a-kserve-778878d5-ztn9c\" (UID: \"629a7581-b804-409c-96b0-3640c50aecfa\") " pod="ai-tenant-e2e-shared-a-1f1c9a/sub-test-model-1f1c9a-kserve-778878d5-ztn9c" Jul 10 15:09:07.265558 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:09:07.265536 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/629a7581-b804-409c-96b0-3640c50aecfa-dshm\") pod \"sub-test-model-1f1c9a-kserve-778878d5-ztn9c\" (UID: \"629a7581-b804-409c-96b0-3640c50aecfa\") " pod="ai-tenant-e2e-shared-a-1f1c9a/sub-test-model-1f1c9a-kserve-778878d5-ztn9c" Jul 10 15:09:07.265877 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:09:07.265855 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/629a7581-b804-409c-96b0-3640c50aecfa-tls-certs\") pod \"sub-test-model-1f1c9a-kserve-778878d5-ztn9c\" (UID: \"629a7581-b804-409c-96b0-3640c50aecfa\") " pod="ai-tenant-e2e-shared-a-1f1c9a/sub-test-model-1f1c9a-kserve-778878d5-ztn9c" Jul 10 15:09:07.272297 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:09:07.272274 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdz47\" (UniqueName: \"kubernetes.io/projected/629a7581-b804-409c-96b0-3640c50aecfa-kube-api-access-zdz47\") pod \"sub-test-model-1f1c9a-kserve-778878d5-ztn9c\" (UID: \"629a7581-b804-409c-96b0-3640c50aecfa\") " pod="ai-tenant-e2e-shared-a-1f1c9a/sub-test-model-1f1c9a-kserve-778878d5-ztn9c" Jul 10 15:09:07.312156 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:09:07.312126 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1f1c9a/sub-test-model-1f1c9a-kserve-778878d5-ztn9c" Jul 10 15:09:07.442471 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:09:07.442443 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1f1c9a/sub-test-model-1f1c9a-kserve-778878d5-ztn9c"] Jul 10 15:09:07.446513 ip-10-0-140-35 kubenswrapper[2569]: W0710 15:09:07.444708 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod629a7581_b804_409c_96b0_3640c50aecfa.slice/crio-2337e4252b745628860e7a49badf11d308edbd3fd5c508ac2e8fef5a1cb156be WatchSource:0}: Error finding container 2337e4252b745628860e7a49badf11d308edbd3fd5c508ac2e8fef5a1cb156be: Status 404 returned error can't find the container with id 2337e4252b745628860e7a49badf11d308edbd3fd5c508ac2e8fef5a1cb156be Jul 10 15:09:07.782494 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:09:07.782455 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1f1c9a/sub-test-model-1f1c9a-kserve-778878d5-ztn9c" event={"ID":"629a7581-b804-409c-96b0-3640c50aecfa","Type":"ContainerStarted","Data":"427f19e6d369fe284aba20b2035700e7f677c87b602818f556896825d4e22cc5"} Jul 10 15:09:07.782699 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:09:07.782503 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1f1c9a/sub-test-model-1f1c9a-kserve-778878d5-ztn9c" event={"ID":"629a7581-b804-409c-96b0-3640c50aecfa","Type":"ContainerStarted","Data":"2337e4252b745628860e7a49badf11d308edbd3fd5c508ac2e8fef5a1cb156be"} Jul 10 15:09:07.800832 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:09:07.800784 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-1f1c9a/sub-test-model-1f1c9a-kserve-778878d5-ztn9c" podStartSLOduration=1.800744621 podStartE2EDuration="1.800744621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 15:09:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 15:09:07.799209486 +0000 UTC m=+2293.550581201" watchObservedRunningTime="2026-07-10 15:09:07.800744621 +0000 UTC m=+2293.552116317" Jul 10 15:09:17.312378 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:09:17.312348 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-1f1c9a/sub-test-model-1f1c9a-kserve-778878d5-ztn9c" Jul 10 15:09:17.312378 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:09:17.312386 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-1f1c9a/sub-test-model-1f1c9a-kserve-778878d5-ztn9c" Jul 10 15:09:17.324906 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:09:17.324881 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-1f1c9a/sub-test-model-1f1c9a-kserve-778878d5-ztn9c" Jul 10 15:09:17.842051 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:09:17.842020 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-1f1c9a/sub-test-model-1f1c9a-kserve-778878d5-ztn9c" Jul 10 15:10:21.476229 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:21.476196 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1f1c9a/sub-test-model-1f1c9a-kserve-778878d5-ztn9c"] Jul 10 15:10:36.496395 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:36.496345 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-1f1c9a/sub-test-model-1f1c9a-kserve-778878d5-ztn9c" podUID="629a7581-b804-409c-96b0-3640c50aecfa" containerName="main" containerID="cri-o://427f19e6d369fe284aba20b2035700e7f677c87b602818f556896825d4e22cc5" gracePeriod=45 Jul 10 15:10:36.754530 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:36.754471 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1f1c9a/sub-test-model-1f1c9a-kserve-778878d5-ztn9c" Jul 10 15:10:36.845629 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:36.845600 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zdz47\" (UniqueName: \"kubernetes.io/projected/629a7581-b804-409c-96b0-3640c50aecfa-kube-api-access-zdz47\") pod \"629a7581-b804-409c-96b0-3640c50aecfa\" (UID: \"629a7581-b804-409c-96b0-3640c50aecfa\") " Jul 10 15:10:36.845629 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:36.845635 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/629a7581-b804-409c-96b0-3640c50aecfa-model-cache\") pod \"629a7581-b804-409c-96b0-3640c50aecfa\" (UID: \"629a7581-b804-409c-96b0-3640c50aecfa\") " Jul 10 15:10:36.845902 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:36.845671 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/629a7581-b804-409c-96b0-3640c50aecfa-dshm\") pod \"629a7581-b804-409c-96b0-3640c50aecfa\" (UID: \"629a7581-b804-409c-96b0-3640c50aecfa\") " Jul 10 15:10:36.845902 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:36.845800 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/629a7581-b804-409c-96b0-3640c50aecfa-tmp-dir\") pod \"629a7581-b804-409c-96b0-3640c50aecfa\" (UID: \"629a7581-b804-409c-96b0-3640c50aecfa\") " Jul 10 15:10:36.845902 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:36.845838 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/629a7581-b804-409c-96b0-3640c50aecfa-tls-certs\") pod \"629a7581-b804-409c-96b0-3640c50aecfa\" (UID: \"629a7581-b804-409c-96b0-3640c50aecfa\") " Jul 10 15:10:36.845902 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:36.845877 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/629a7581-b804-409c-96b0-3640c50aecfa-home\") pod \"629a7581-b804-409c-96b0-3640c50aecfa\" (UID: \"629a7581-b804-409c-96b0-3640c50aecfa\") " Jul 10 15:10:36.846102 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:36.845899 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/629a7581-b804-409c-96b0-3640c50aecfa-model-cache" (OuterVolumeSpecName: "model-cache") pod "629a7581-b804-409c-96b0-3640c50aecfa" (UID: "629a7581-b804-409c-96b0-3640c50aecfa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 15:10:36.846102 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:36.846009 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/629a7581-b804-409c-96b0-3640c50aecfa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "629a7581-b804-409c-96b0-3640c50aecfa" (UID: "629a7581-b804-409c-96b0-3640c50aecfa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 15:10:36.846252 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:36.846146 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/629a7581-b804-409c-96b0-3640c50aecfa-home" (OuterVolumeSpecName: "home") pod "629a7581-b804-409c-96b0-3640c50aecfa" (UID: "629a7581-b804-409c-96b0-3640c50aecfa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 15:10:36.846252 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:36.846248 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/629a7581-b804-409c-96b0-3640c50aecfa-model-cache\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 15:10:36.846361 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:36.846261 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/629a7581-b804-409c-96b0-3640c50aecfa-tmp-dir\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 15:10:36.846361 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:36.846269 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/629a7581-b804-409c-96b0-3640c50aecfa-home\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 15:10:36.847743 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:36.847722 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/629a7581-b804-409c-96b0-3640c50aecfa-dshm" (OuterVolumeSpecName: "dshm") pod "629a7581-b804-409c-96b0-3640c50aecfa" (UID: "629a7581-b804-409c-96b0-3640c50aecfa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 15:10:36.847893 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:36.847880 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/629a7581-b804-409c-96b0-3640c50aecfa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "629a7581-b804-409c-96b0-3640c50aecfa" (UID: "629a7581-b804-409c-96b0-3640c50aecfa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 15:10:36.847939 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:36.847911 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/629a7581-b804-409c-96b0-3640c50aecfa-kube-api-access-zdz47" (OuterVolumeSpecName: "kube-api-access-zdz47") pod "629a7581-b804-409c-96b0-3640c50aecfa" (UID: "629a7581-b804-409c-96b0-3640c50aecfa"). InnerVolumeSpecName "kube-api-access-zdz47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 15:10:36.947100 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:36.947076 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdz47\" (UniqueName: \"kubernetes.io/projected/629a7581-b804-409c-96b0-3640c50aecfa-kube-api-access-zdz47\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 15:10:36.947100 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:36.947100 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/629a7581-b804-409c-96b0-3640c50aecfa-dshm\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 15:10:36.947261 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:36.947108 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/629a7581-b804-409c-96b0-3640c50aecfa-tls-certs\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 15:10:37.212608 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:37.212578 2569 generic.go:356] "Generic (PLEG): container finished" podID="629a7581-b804-409c-96b0-3640c50aecfa" containerID="427f19e6d369fe284aba20b2035700e7f677c87b602818f556896825d4e22cc5" exitCode=0 Jul 10 15:10:37.212820 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:37.212623 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1f1c9a/sub-test-model-1f1c9a-kserve-778878d5-ztn9c" Jul 10 15:10:37.212820 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:37.212665 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1f1c9a/sub-test-model-1f1c9a-kserve-778878d5-ztn9c" event={"ID":"629a7581-b804-409c-96b0-3640c50aecfa","Type":"ContainerDied","Data":"427f19e6d369fe284aba20b2035700e7f677c87b602818f556896825d4e22cc5"} Jul 10 15:10:37.212820 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:37.212718 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1f1c9a/sub-test-model-1f1c9a-kserve-778878d5-ztn9c" event={"ID":"629a7581-b804-409c-96b0-3640c50aecfa","Type":"ContainerDied","Data":"2337e4252b745628860e7a49badf11d308edbd3fd5c508ac2e8fef5a1cb156be"} Jul 10 15:10:37.212820 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:37.212741 2569 scope.go:117] "RemoveContainer" containerID="427f19e6d369fe284aba20b2035700e7f677c87b602818f556896825d4e22cc5" Jul 10 15:10:37.221932 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:37.221908 2569 scope.go:117] "RemoveContainer" containerID="427f19e6d369fe284aba20b2035700e7f677c87b602818f556896825d4e22cc5" Jul 10 15:10:37.222185 ip-10-0-140-35 kubenswrapper[2569]: E0710 15:10:37.222168 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"427f19e6d369fe284aba20b2035700e7f677c87b602818f556896825d4e22cc5\": container with ID starting with 427f19e6d369fe284aba20b2035700e7f677c87b602818f556896825d4e22cc5 not found: ID does not exist" containerID="427f19e6d369fe284aba20b2035700e7f677c87b602818f556896825d4e22cc5" Jul 10 15:10:37.222238 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:37.222193 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"427f19e6d369fe284aba20b2035700e7f677c87b602818f556896825d4e22cc5"} err="failed to get container status \"427f19e6d369fe284aba20b2035700e7f677c87b602818f556896825d4e22cc5\": rpc error: code = NotFound desc = could not find container \"427f19e6d369fe284aba20b2035700e7f677c87b602818f556896825d4e22cc5\": container with ID starting with 427f19e6d369fe284aba20b2035700e7f677c87b602818f556896825d4e22cc5 not found: ID does not exist" Jul 10 15:10:37.228462 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:37.228438 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1f1c9a/sub-test-model-1f1c9a-kserve-778878d5-ztn9c"] Jul 10 15:10:37.234719 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:37.234694 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-1f1c9a/sub-test-model-1f1c9a-kserve-778878d5-ztn9c"] Jul 10 15:10:38.931072 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:38.931036 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="629a7581-b804-409c-96b0-3640c50aecfa" path="/var/lib/kubelet/pods/629a7581-b804-409c-96b0-3640c50aecfa/volumes" Jul 10 15:10:54.852469 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:54.852440 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-n6pfx_296be890-64e9-4fb3-8c68-e4c2a7c8cc2c/console-operator/1.log" Jul 10 15:10:54.854934 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:54.854909 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-n6pfx_296be890-64e9-4fb3-8c68-e4c2a7c8cc2c/console-operator/1.log" Jul 10 15:10:54.858509 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:54.858490 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxvq2_016248a0-bbe2-44c9-8e51-5d4695ee6a52/ovn-acl-logging/0.log" Jul 10 15:10:54.860842 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:54.860822 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxvq2_016248a0-bbe2-44c9-8e51-5d4695ee6a52/ovn-acl-logging/0.log" Jul 10 15:10:57.101218 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:57.101179 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-f923ab/rate-test-model-f923ab-kserve-6b578647dc-r895t"] Jul 10 15:10:57.101650 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:57.101588 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="629a7581-b804-409c-96b0-3640c50aecfa" containerName="main" Jul 10 15:10:57.101650 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:57.101607 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="629a7581-b804-409c-96b0-3640c50aecfa" containerName="main" Jul 10 15:10:57.101736 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:57.101686 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="629a7581-b804-409c-96b0-3640c50aecfa" containerName="main" Jul 10 15:10:57.104891 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:57.104869 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f923ab/rate-test-model-f923ab-kserve-6b578647dc-r895t" Jul 10 15:10:57.107819 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:57.107800 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-f923ab\"/\"kube-root-ca.crt\"" Jul 10 15:10:57.108985 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:57.108960 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-f923ab\"/\"openshift-service-ca.crt\"" Jul 10 15:10:57.108985 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:57.108975 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-f923ab\"/\"default-dockercfg-kdtmk\"" Jul 10 15:10:57.109159 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:57.109013 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-f923ab\"/\"rate-test-model-f923ab-kserve-self-signed-certs\"" Jul 10 15:10:57.118418 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:57.118394 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-f923ab/rate-test-model-f923ab-kserve-6b578647dc-r895t"] Jul 10 15:10:57.221687 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:57.221660 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da9b03d0-4137-4cb6-b5f9-0ab64bf8b240-tls-certs\") pod \"rate-test-model-f923ab-kserve-6b578647dc-r895t\" (UID: \"da9b03d0-4137-4cb6-b5f9-0ab64bf8b240\") " pod="ai-tenant-e2e-shared-b-f923ab/rate-test-model-f923ab-kserve-6b578647dc-r895t" Jul 10 15:10:57.221859 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:57.221698 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da9b03d0-4137-4cb6-b5f9-0ab64bf8b240-tmp-dir\") pod \"rate-test-model-f923ab-kserve-6b578647dc-r895t\" (UID: \"da9b03d0-4137-4cb6-b5f9-0ab64bf8b240\") " pod="ai-tenant-e2e-shared-b-f923ab/rate-test-model-f923ab-kserve-6b578647dc-r895t" Jul 10 15:10:57.221859 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:57.221732 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da9b03d0-4137-4cb6-b5f9-0ab64bf8b240-model-cache\") pod \"rate-test-model-f923ab-kserve-6b578647dc-r895t\" (UID: \"da9b03d0-4137-4cb6-b5f9-0ab64bf8b240\") " pod="ai-tenant-e2e-shared-b-f923ab/rate-test-model-f923ab-kserve-6b578647dc-r895t" Jul 10 15:10:57.221859 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:57.221813 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da9b03d0-4137-4cb6-b5f9-0ab64bf8b240-dshm\") pod \"rate-test-model-f923ab-kserve-6b578647dc-r895t\" (UID: \"da9b03d0-4137-4cb6-b5f9-0ab64bf8b240\") " pod="ai-tenant-e2e-shared-b-f923ab/rate-test-model-f923ab-kserve-6b578647dc-r895t" Jul 10 15:10:57.221859 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:57.221856 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da9b03d0-4137-4cb6-b5f9-0ab64bf8b240-home\") pod \"rate-test-model-f923ab-kserve-6b578647dc-r895t\" (UID: \"da9b03d0-4137-4cb6-b5f9-0ab64bf8b240\") " pod="ai-tenant-e2e-shared-b-f923ab/rate-test-model-f923ab-kserve-6b578647dc-r895t" Jul 10 15:10:57.222034 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:57.221875 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5bmr\" (UniqueName: \"kubernetes.io/projected/da9b03d0-4137-4cb6-b5f9-0ab64bf8b240-kube-api-access-q5bmr\") pod \"rate-test-model-f923ab-kserve-6b578647dc-r895t\" (UID: \"da9b03d0-4137-4cb6-b5f9-0ab64bf8b240\") " pod="ai-tenant-e2e-shared-b-f923ab/rate-test-model-f923ab-kserve-6b578647dc-r895t" Jul 10 15:10:57.323316 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:57.323289 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da9b03d0-4137-4cb6-b5f9-0ab64bf8b240-model-cache\") pod \"rate-test-model-f923ab-kserve-6b578647dc-r895t\" (UID: \"da9b03d0-4137-4cb6-b5f9-0ab64bf8b240\") " pod="ai-tenant-e2e-shared-b-f923ab/rate-test-model-f923ab-kserve-6b578647dc-r895t" Jul 10 15:10:57.323316 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:57.323319 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da9b03d0-4137-4cb6-b5f9-0ab64bf8b240-dshm\") pod \"rate-test-model-f923ab-kserve-6b578647dc-r895t\" (UID: \"da9b03d0-4137-4cb6-b5f9-0ab64bf8b240\") " pod="ai-tenant-e2e-shared-b-f923ab/rate-test-model-f923ab-kserve-6b578647dc-r895t" Jul 10 15:10:57.323523 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:57.323340 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da9b03d0-4137-4cb6-b5f9-0ab64bf8b240-home\") pod \"rate-test-model-f923ab-kserve-6b578647dc-r895t\" (UID: \"da9b03d0-4137-4cb6-b5f9-0ab64bf8b240\") " pod="ai-tenant-e2e-shared-b-f923ab/rate-test-model-f923ab-kserve-6b578647dc-r895t" Jul 10 15:10:57.323523 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:57.323357 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5bmr\" (UniqueName: \"kubernetes.io/projected/da9b03d0-4137-4cb6-b5f9-0ab64bf8b240-kube-api-access-q5bmr\") pod \"rate-test-model-f923ab-kserve-6b578647dc-r895t\" (UID: \"da9b03d0-4137-4cb6-b5f9-0ab64bf8b240\") " pod="ai-tenant-e2e-shared-b-f923ab/rate-test-model-f923ab-kserve-6b578647dc-r895t" Jul 10 15:10:57.323523 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:57.323401 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da9b03d0-4137-4cb6-b5f9-0ab64bf8b240-tls-certs\") pod \"rate-test-model-f923ab-kserve-6b578647dc-r895t\" (UID: \"da9b03d0-4137-4cb6-b5f9-0ab64bf8b240\") " pod="ai-tenant-e2e-shared-b-f923ab/rate-test-model-f923ab-kserve-6b578647dc-r895t" Jul 10 15:10:57.323523 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:57.323425 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da9b03d0-4137-4cb6-b5f9-0ab64bf8b240-tmp-dir\") pod \"rate-test-model-f923ab-kserve-6b578647dc-r895t\" (UID: \"da9b03d0-4137-4cb6-b5f9-0ab64bf8b240\") " pod="ai-tenant-e2e-shared-b-f923ab/rate-test-model-f923ab-kserve-6b578647dc-r895t" Jul 10 15:10:57.323728 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:57.323681 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da9b03d0-4137-4cb6-b5f9-0ab64bf8b240-model-cache\") pod \"rate-test-model-f923ab-kserve-6b578647dc-r895t\" (UID: \"da9b03d0-4137-4cb6-b5f9-0ab64bf8b240\") " pod="ai-tenant-e2e-shared-b-f923ab/rate-test-model-f923ab-kserve-6b578647dc-r895t" Jul 10 15:10:57.323817 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:57.323737 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da9b03d0-4137-4cb6-b5f9-0ab64bf8b240-home\") pod \"rate-test-model-f923ab-kserve-6b578647dc-r895t\" (UID: \"da9b03d0-4137-4cb6-b5f9-0ab64bf8b240\") " pod="ai-tenant-e2e-shared-b-f923ab/rate-test-model-f923ab-kserve-6b578647dc-r895t" Jul 10 15:10:57.323877 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:57.323857 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da9b03d0-4137-4cb6-b5f9-0ab64bf8b240-tmp-dir\") pod \"rate-test-model-f923ab-kserve-6b578647dc-r895t\" (UID: \"da9b03d0-4137-4cb6-b5f9-0ab64bf8b240\") " pod="ai-tenant-e2e-shared-b-f923ab/rate-test-model-f923ab-kserve-6b578647dc-r895t" Jul 10 15:10:57.325674 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:57.325653 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da9b03d0-4137-4cb6-b5f9-0ab64bf8b240-dshm\") pod \"rate-test-model-f923ab-kserve-6b578647dc-r895t\" (UID: \"da9b03d0-4137-4cb6-b5f9-0ab64bf8b240\") " pod="ai-tenant-e2e-shared-b-f923ab/rate-test-model-f923ab-kserve-6b578647dc-r895t" Jul 10 15:10:57.325782 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:57.325768 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da9b03d0-4137-4cb6-b5f9-0ab64bf8b240-tls-certs\") pod \"rate-test-model-f923ab-kserve-6b578647dc-r895t\" (UID: \"da9b03d0-4137-4cb6-b5f9-0ab64bf8b240\") " pod="ai-tenant-e2e-shared-b-f923ab/rate-test-model-f923ab-kserve-6b578647dc-r895t" Jul 10 15:10:57.331737 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:57.331713 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5bmr\" (UniqueName: \"kubernetes.io/projected/da9b03d0-4137-4cb6-b5f9-0ab64bf8b240-kube-api-access-q5bmr\") pod \"rate-test-model-f923ab-kserve-6b578647dc-r895t\" (UID: \"da9b03d0-4137-4cb6-b5f9-0ab64bf8b240\") " pod="ai-tenant-e2e-shared-b-f923ab/rate-test-model-f923ab-kserve-6b578647dc-r895t" Jul 10 15:10:57.415214 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:57.415190 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f923ab/rate-test-model-f923ab-kserve-6b578647dc-r895t" Jul 10 15:10:57.545736 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:57.545706 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-f923ab/rate-test-model-f923ab-kserve-6b578647dc-r895t"] Jul 10 15:10:57.547737 ip-10-0-140-35 kubenswrapper[2569]: W0710 15:10:57.547714 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda9b03d0_4137_4cb6_b5f9_0ab64bf8b240.slice/crio-598d58c1455c5ed2f55e59c8a70b20f7336fc20ffe1c4bc5bf7b0c2925a8f718 WatchSource:0}: Error finding container 598d58c1455c5ed2f55e59c8a70b20f7336fc20ffe1c4bc5bf7b0c2925a8f718: Status 404 returned error can't find the container with id 598d58c1455c5ed2f55e59c8a70b20f7336fc20ffe1c4bc5bf7b0c2925a8f718 Jul 10 15:10:58.313634 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:58.313599 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f923ab/rate-test-model-f923ab-kserve-6b578647dc-r895t" event={"ID":"da9b03d0-4137-4cb6-b5f9-0ab64bf8b240","Type":"ContainerStarted","Data":"a5c24236e422c4b6901d9170adc1d492cecf52b22f89765b6c7300c10e753222"} Jul 10 15:10:58.313634 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:58.313645 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f923ab/rate-test-model-f923ab-kserve-6b578647dc-r895t" event={"ID":"da9b03d0-4137-4cb6-b5f9-0ab64bf8b240","Type":"ContainerStarted","Data":"598d58c1455c5ed2f55e59c8a70b20f7336fc20ffe1c4bc5bf7b0c2925a8f718"} Jul 10 15:10:58.331510 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:10:58.331467 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-f923ab/rate-test-model-f923ab-kserve-6b578647dc-r895t" podStartSLOduration=1.331452333 podStartE2EDuration="1.331452333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 15:10:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 15:10:58.32984927 +0000 UTC m=+2404.081220991" watchObservedRunningTime="2026-07-10 15:10:58.331452333 +0000 UTC m=+2404.082824029" Jul 10 15:11:07.415708 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:11:07.415676 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-f923ab/rate-test-model-f923ab-kserve-6b578647dc-r895t" Jul 10 15:11:07.416269 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:11:07.415955 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-f923ab/rate-test-model-f923ab-kserve-6b578647dc-r895t" Jul 10 15:11:07.428256 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:11:07.428231 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-f923ab/rate-test-model-f923ab-kserve-6b578647dc-r895t" Jul 10 15:11:08.379331 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:11:08.379299 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-f923ab/rate-test-model-f923ab-kserve-6b578647dc-r895t" Jul 10 15:11:50.183675 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:11:50.183599 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-f923ab/rate-test-model-f923ab-kserve-6b578647dc-r895t"] Jul 10 15:12:05.203795 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:05.203740 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-f923ab/rate-test-model-f923ab-kserve-6b578647dc-r895t" podUID="da9b03d0-4137-4cb6-b5f9-0ab64bf8b240" containerName="main" containerID="cri-o://a5c24236e422c4b6901d9170adc1d492cecf52b22f89765b6c7300c10e753222" gracePeriod=45 Jul 10 15:12:05.458893 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:05.458827 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f923ab/rate-test-model-f923ab-kserve-6b578647dc-r895t" Jul 10 15:12:05.530344 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:05.530320 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da9b03d0-4137-4cb6-b5f9-0ab64bf8b240-tls-certs\") pod \"da9b03d0-4137-4cb6-b5f9-0ab64bf8b240\" (UID: \"da9b03d0-4137-4cb6-b5f9-0ab64bf8b240\") " Jul 10 15:12:05.530470 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:05.530362 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da9b03d0-4137-4cb6-b5f9-0ab64bf8b240-tmp-dir\") pod \"da9b03d0-4137-4cb6-b5f9-0ab64bf8b240\" (UID: \"da9b03d0-4137-4cb6-b5f9-0ab64bf8b240\") " Jul 10 15:12:05.530470 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:05.530384 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da9b03d0-4137-4cb6-b5f9-0ab64bf8b240-dshm\") pod \"da9b03d0-4137-4cb6-b5f9-0ab64bf8b240\" (UID: \"da9b03d0-4137-4cb6-b5f9-0ab64bf8b240\") " Jul 10 15:12:05.530470 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:05.530420 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da9b03d0-4137-4cb6-b5f9-0ab64bf8b240-model-cache\") pod \"da9b03d0-4137-4cb6-b5f9-0ab64bf8b240\" (UID: \"da9b03d0-4137-4cb6-b5f9-0ab64bf8b240\") " Jul 10 15:12:05.530576 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:05.530486 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da9b03d0-4137-4cb6-b5f9-0ab64bf8b240-home\") pod \"da9b03d0-4137-4cb6-b5f9-0ab64bf8b240\" (UID: \"da9b03d0-4137-4cb6-b5f9-0ab64bf8b240\") " Jul 10 15:12:05.530576 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:05.530527 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q5bmr\" (UniqueName: \"kubernetes.io/projected/da9b03d0-4137-4cb6-b5f9-0ab64bf8b240-kube-api-access-q5bmr\") pod \"da9b03d0-4137-4cb6-b5f9-0ab64bf8b240\" (UID: \"da9b03d0-4137-4cb6-b5f9-0ab64bf8b240\") " Jul 10 15:12:05.530660 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:05.530616 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da9b03d0-4137-4cb6-b5f9-0ab64bf8b240-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "da9b03d0-4137-4cb6-b5f9-0ab64bf8b240" (UID: "da9b03d0-4137-4cb6-b5f9-0ab64bf8b240"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 15:12:05.530792 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:05.530766 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da9b03d0-4137-4cb6-b5f9-0ab64bf8b240-home" (OuterVolumeSpecName: "home") pod "da9b03d0-4137-4cb6-b5f9-0ab64bf8b240" (UID: "da9b03d0-4137-4cb6-b5f9-0ab64bf8b240"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 15:12:05.530792 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:05.530781 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da9b03d0-4137-4cb6-b5f9-0ab64bf8b240-model-cache" (OuterVolumeSpecName: "model-cache") pod "da9b03d0-4137-4cb6-b5f9-0ab64bf8b240" (UID: "da9b03d0-4137-4cb6-b5f9-0ab64bf8b240"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 15:12:05.530904 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:05.530886 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da9b03d0-4137-4cb6-b5f9-0ab64bf8b240-tmp-dir\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 15:12:05.530944 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:05.530903 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da9b03d0-4137-4cb6-b5f9-0ab64bf8b240-model-cache\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 15:12:05.530944 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:05.530917 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da9b03d0-4137-4cb6-b5f9-0ab64bf8b240-home\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 15:12:05.532644 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:05.532612 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da9b03d0-4137-4cb6-b5f9-0ab64bf8b240-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "da9b03d0-4137-4cb6-b5f9-0ab64bf8b240" (UID: "da9b03d0-4137-4cb6-b5f9-0ab64bf8b240"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 15:12:05.532644 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:05.532633 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da9b03d0-4137-4cb6-b5f9-0ab64bf8b240-dshm" (OuterVolumeSpecName: "dshm") pod "da9b03d0-4137-4cb6-b5f9-0ab64bf8b240" (UID: "da9b03d0-4137-4cb6-b5f9-0ab64bf8b240"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 15:12:05.532809 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:05.532640 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da9b03d0-4137-4cb6-b5f9-0ab64bf8b240-kube-api-access-q5bmr" (OuterVolumeSpecName: "kube-api-access-q5bmr") pod "da9b03d0-4137-4cb6-b5f9-0ab64bf8b240" (UID: "da9b03d0-4137-4cb6-b5f9-0ab64bf8b240"). InnerVolumeSpecName "kube-api-access-q5bmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 15:12:05.631393 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:05.631368 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5bmr\" (UniqueName: \"kubernetes.io/projected/da9b03d0-4137-4cb6-b5f9-0ab64bf8b240-kube-api-access-q5bmr\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 15:12:05.631393 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:05.631391 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da9b03d0-4137-4cb6-b5f9-0ab64bf8b240-tls-certs\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 15:12:05.631537 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:05.631401 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da9b03d0-4137-4cb6-b5f9-0ab64bf8b240-dshm\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 15:12:05.650538 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:05.650512 2569 generic.go:356] "Generic (PLEG): container finished" podID="da9b03d0-4137-4cb6-b5f9-0ab64bf8b240" containerID="a5c24236e422c4b6901d9170adc1d492cecf52b22f89765b6c7300c10e753222" exitCode=0 Jul 10 15:12:05.650636 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:05.650556 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f923ab/rate-test-model-f923ab-kserve-6b578647dc-r895t" Jul 10 15:12:05.650636 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:05.650596 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f923ab/rate-test-model-f923ab-kserve-6b578647dc-r895t" event={"ID":"da9b03d0-4137-4cb6-b5f9-0ab64bf8b240","Type":"ContainerDied","Data":"a5c24236e422c4b6901d9170adc1d492cecf52b22f89765b6c7300c10e753222"} Jul 10 15:12:05.650709 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:05.650637 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f923ab/rate-test-model-f923ab-kserve-6b578647dc-r895t" event={"ID":"da9b03d0-4137-4cb6-b5f9-0ab64bf8b240","Type":"ContainerDied","Data":"598d58c1455c5ed2f55e59c8a70b20f7336fc20ffe1c4bc5bf7b0c2925a8f718"} Jul 10 15:12:05.650709 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:05.650654 2569 scope.go:117] "RemoveContainer" containerID="a5c24236e422c4b6901d9170adc1d492cecf52b22f89765b6c7300c10e753222" Jul 10 15:12:05.659986 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:05.659969 2569 scope.go:117] "RemoveContainer" containerID="a5c24236e422c4b6901d9170adc1d492cecf52b22f89765b6c7300c10e753222" Jul 10 15:12:05.660204 ip-10-0-140-35 kubenswrapper[2569]: E0710 15:12:05.660186 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5c24236e422c4b6901d9170adc1d492cecf52b22f89765b6c7300c10e753222\": container with ID starting with a5c24236e422c4b6901d9170adc1d492cecf52b22f89765b6c7300c10e753222 not found: ID does not exist" containerID="a5c24236e422c4b6901d9170adc1d492cecf52b22f89765b6c7300c10e753222" Jul 10 15:12:05.660264 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:05.660215 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5c24236e422c4b6901d9170adc1d492cecf52b22f89765b6c7300c10e753222"} err="failed to get container status \"a5c24236e422c4b6901d9170adc1d492cecf52b22f89765b6c7300c10e753222\": rpc error: code = NotFound desc = could not find container \"a5c24236e422c4b6901d9170adc1d492cecf52b22f89765b6c7300c10e753222\": container with ID starting with a5c24236e422c4b6901d9170adc1d492cecf52b22f89765b6c7300c10e753222 not found: ID does not exist" Jul 10 15:12:05.672648 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:05.672627 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-f923ab/rate-test-model-f923ab-kserve-6b578647dc-r895t"] Jul 10 15:12:05.674313 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:05.674294 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-f923ab/rate-test-model-f923ab-kserve-6b578647dc-r895t"] Jul 10 15:12:06.932050 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:06.932021 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da9b03d0-4137-4cb6-b5f9-0ab64bf8b240" path="/var/lib/kubelet/pods/da9b03d0-4137-4cb6-b5f9-0ab64bf8b240/volumes" Jul 10 15:12:15.773349 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:15.773314 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr"] Jul 10 15:12:15.773829 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:15.773559 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr" podUID="83c864ba-4d1f-4161-8587-c38bb2a17cc1" containerName="istio-proxy" containerID="cri-o://cfbe211d43f503e9befc516748603ce1eab3463451d6f47f96c081c826aad107" gracePeriod=30 Jul 10 15:12:21.029336 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:21.029311 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr" Jul 10 15:12:21.171679 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:21.171650 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83c864ba-4d1f-4161-8587-c38bb2a17cc1-istio-data\") pod \"83c864ba-4d1f-4161-8587-c38bb2a17cc1\" (UID: \"83c864ba-4d1f-4161-8587-c38bb2a17cc1\") " Jul 10 15:12:21.171891 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:21.171713 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c864ba-4d1f-4161-8587-c38bb2a17cc1-credential-socket\") pod \"83c864ba-4d1f-4161-8587-c38bb2a17cc1\" (UID: \"83c864ba-4d1f-4161-8587-c38bb2a17cc1\") " Jul 10 15:12:21.171891 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:21.171730 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83c864ba-4d1f-4161-8587-c38bb2a17cc1-istio-token\") pod \"83c864ba-4d1f-4161-8587-c38bb2a17cc1\" (UID: \"83c864ba-4d1f-4161-8587-c38bb2a17cc1\") " Jul 10 15:12:21.171891 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:21.171771 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t6ws8\" (UniqueName: \"kubernetes.io/projected/83c864ba-4d1f-4161-8587-c38bb2a17cc1-kube-api-access-t6ws8\") pod \"83c864ba-4d1f-4161-8587-c38bb2a17cc1\" (UID: \"83c864ba-4d1f-4161-8587-c38bb2a17cc1\") " Jul 10 15:12:21.171891 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:21.171790 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83c864ba-4d1f-4161-8587-c38bb2a17cc1-istiod-ca-cert\") pod \"83c864ba-4d1f-4161-8587-c38bb2a17cc1\" (UID: \"83c864ba-4d1f-4161-8587-c38bb2a17cc1\") " Jul 10 15:12:21.171891 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:21.171840 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83c864ba-4d1f-4161-8587-c38bb2a17cc1-workload-certs\") pod \"83c864ba-4d1f-4161-8587-c38bb2a17cc1\" (UID: \"83c864ba-4d1f-4161-8587-c38bb2a17cc1\") " Jul 10 15:12:21.171891 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:21.171884 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83c864ba-4d1f-4161-8587-c38bb2a17cc1-istio-envoy\") pod \"83c864ba-4d1f-4161-8587-c38bb2a17cc1\" (UID: \"83c864ba-4d1f-4161-8587-c38bb2a17cc1\") " Jul 10 15:12:21.172233 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:21.171913 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83c864ba-4d1f-4161-8587-c38bb2a17cc1-istio-podinfo\") pod \"83c864ba-4d1f-4161-8587-c38bb2a17cc1\" (UID: \"83c864ba-4d1f-4161-8587-c38bb2a17cc1\") " Jul 10 15:12:21.172233 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:21.171941 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c864ba-4d1f-4161-8587-c38bb2a17cc1-workload-socket\") pod \"83c864ba-4d1f-4161-8587-c38bb2a17cc1\" (UID: \"83c864ba-4d1f-4161-8587-c38bb2a17cc1\") " Jul 10 15:12:21.172233 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:21.172001 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83c864ba-4d1f-4161-8587-c38bb2a17cc1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "83c864ba-4d1f-4161-8587-c38bb2a17cc1" (UID: "83c864ba-4d1f-4161-8587-c38bb2a17cc1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 15:12:21.172233 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:21.172154 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83c864ba-4d1f-4161-8587-c38bb2a17cc1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "83c864ba-4d1f-4161-8587-c38bb2a17cc1" (UID: "83c864ba-4d1f-4161-8587-c38bb2a17cc1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 15:12:21.172233 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:21.172213 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83c864ba-4d1f-4161-8587-c38bb2a17cc1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "83c864ba-4d1f-4161-8587-c38bb2a17cc1" (UID: "83c864ba-4d1f-4161-8587-c38bb2a17cc1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 15:12:21.172485 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:21.172315 2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c864ba-4d1f-4161-8587-c38bb2a17cc1-credential-socket\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 15:12:21.172485 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:21.172333 2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83c864ba-4d1f-4161-8587-c38bb2a17cc1-istiod-ca-cert\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 15:12:21.172485 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:21.172346 2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83c864ba-4d1f-4161-8587-c38bb2a17cc1-workload-certs\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 15:12:21.172485 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:21.172361 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83c864ba-4d1f-4161-8587-c38bb2a17cc1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "83c864ba-4d1f-4161-8587-c38bb2a17cc1" (UID: "83c864ba-4d1f-4161-8587-c38bb2a17cc1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 15:12:21.172889 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:21.172868 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83c864ba-4d1f-4161-8587-c38bb2a17cc1-istio-data" (OuterVolumeSpecName: "istio-data") pod "83c864ba-4d1f-4161-8587-c38bb2a17cc1" (UID: "83c864ba-4d1f-4161-8587-c38bb2a17cc1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 15:12:21.174043 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:21.174024 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83c864ba-4d1f-4161-8587-c38bb2a17cc1-istio-token" (OuterVolumeSpecName: "istio-token") pod "83c864ba-4d1f-4161-8587-c38bb2a17cc1" (UID: "83c864ba-4d1f-4161-8587-c38bb2a17cc1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 15:12:21.174157 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:21.174134 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83c864ba-4d1f-4161-8587-c38bb2a17cc1-kube-api-access-t6ws8" (OuterVolumeSpecName: "kube-api-access-t6ws8") pod "83c864ba-4d1f-4161-8587-c38bb2a17cc1" (UID: "83c864ba-4d1f-4161-8587-c38bb2a17cc1"). InnerVolumeSpecName "kube-api-access-t6ws8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 15:12:21.174458 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:21.174435 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83c864ba-4d1f-4161-8587-c38bb2a17cc1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "83c864ba-4d1f-4161-8587-c38bb2a17cc1" (UID: "83c864ba-4d1f-4161-8587-c38bb2a17cc1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 15:12:21.174458 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:21.174435 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83c864ba-4d1f-4161-8587-c38bb2a17cc1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "83c864ba-4d1f-4161-8587-c38bb2a17cc1" (UID: "83c864ba-4d1f-4161-8587-c38bb2a17cc1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 15:12:21.272784 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:21.272729 2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83c864ba-4d1f-4161-8587-c38bb2a17cc1-istio-token\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 15:12:21.272784 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:21.272772 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6ws8\" (UniqueName: \"kubernetes.io/projected/83c864ba-4d1f-4161-8587-c38bb2a17cc1-kube-api-access-t6ws8\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 15:12:21.272784 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:21.272788 2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83c864ba-4d1f-4161-8587-c38bb2a17cc1-istio-envoy\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 15:12:21.273059 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:21.272803 2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83c864ba-4d1f-4161-8587-c38bb2a17cc1-istio-podinfo\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 15:12:21.273059 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:21.272815 2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c864ba-4d1f-4161-8587-c38bb2a17cc1-workload-socket\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 15:12:21.273059 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:21.272827 2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83c864ba-4d1f-4161-8587-c38bb2a17cc1-istio-data\") on node \"ip-10-0-140-35.ec2.internal\" DevicePath \"\"" Jul 10 15:12:21.729909 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:21.729879 2569 generic.go:356] "Generic (PLEG): container finished" podID="83c864ba-4d1f-4161-8587-c38bb2a17cc1" containerID="cfbe211d43f503e9befc516748603ce1eab3463451d6f47f96c081c826aad107" exitCode=0 Jul 10 15:12:21.730089 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:21.729930 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr" Jul 10 15:12:21.730089 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:21.729968 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr" event={"ID":"83c864ba-4d1f-4161-8587-c38bb2a17cc1","Type":"ContainerDied","Data":"cfbe211d43f503e9befc516748603ce1eab3463451d6f47f96c081c826aad107"} Jul 10 15:12:21.730089 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:21.730023 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr" event={"ID":"83c864ba-4d1f-4161-8587-c38bb2a17cc1","Type":"ContainerDied","Data":"dfe99fc908ed46ecdc0d0fdd75b07485f52eff134e39d1cbb342c7781abe8004"} Jul 10 15:12:21.730089 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:21.730045 2569 scope.go:117] "RemoveContainer" containerID="cfbe211d43f503e9befc516748603ce1eab3463451d6f47f96c081c826aad107" Jul 10 15:12:21.740132 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:21.740104 2569 scope.go:117] "RemoveContainer" containerID="b46eae7e3b3eaaf06cc62b992d93460f0c653140105c2f2ef763d0a4725ad8ba" Jul 10 15:12:21.747958 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:21.747931 2569 scope.go:117] "RemoveContainer" containerID="cfbe211d43f503e9befc516748603ce1eab3463451d6f47f96c081c826aad107" Jul 10 15:12:21.748179 ip-10-0-140-35 kubenswrapper[2569]: E0710 15:12:21.748161 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfbe211d43f503e9befc516748603ce1eab3463451d6f47f96c081c826aad107\": container with ID starting with cfbe211d43f503e9befc516748603ce1eab3463451d6f47f96c081c826aad107 not found: ID does not exist" containerID="cfbe211d43f503e9befc516748603ce1eab3463451d6f47f96c081c826aad107" Jul 10 15:12:21.748231 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:21.748188 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfbe211d43f503e9befc516748603ce1eab3463451d6f47f96c081c826aad107"} err="failed to get container status \"cfbe211d43f503e9befc516748603ce1eab3463451d6f47f96c081c826aad107\": rpc error: code = NotFound desc = could not find container \"cfbe211d43f503e9befc516748603ce1eab3463451d6f47f96c081c826aad107\": container with ID starting with cfbe211d43f503e9befc516748603ce1eab3463451d6f47f96c081c826aad107 not found: ID does not exist" Jul 10 15:12:21.748231 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:21.748205 2569 scope.go:117] "RemoveContainer" containerID="b46eae7e3b3eaaf06cc62b992d93460f0c653140105c2f2ef763d0a4725ad8ba" Jul 10 15:12:21.748470 ip-10-0-140-35 kubenswrapper[2569]: E0710 15:12:21.748451 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b46eae7e3b3eaaf06cc62b992d93460f0c653140105c2f2ef763d0a4725ad8ba\": container with ID starting with b46eae7e3b3eaaf06cc62b992d93460f0c653140105c2f2ef763d0a4725ad8ba not found: ID does not exist" containerID="b46eae7e3b3eaaf06cc62b992d93460f0c653140105c2f2ef763d0a4725ad8ba" Jul 10 15:12:21.748522 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:21.748479 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b46eae7e3b3eaaf06cc62b992d93460f0c653140105c2f2ef763d0a4725ad8ba"} err="failed to get container status \"b46eae7e3b3eaaf06cc62b992d93460f0c653140105c2f2ef763d0a4725ad8ba\": rpc error: code = NotFound desc = could not find container \"b46eae7e3b3eaaf06cc62b992d93460f0c653140105c2f2ef763d0a4725ad8ba\": container with ID starting with b46eae7e3b3eaaf06cc62b992d93460f0c653140105c2f2ef763d0a4725ad8ba not found: ID does not exist" Jul 10 15:12:21.751370 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:21.751351 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr"] Jul 10 15:12:21.757559 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:21.757538 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-1f1c9a-openshift-default-645bd655cf-88vbr"] Jul 10 15:12:22.932058 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:22.932027 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83c864ba-4d1f-4161-8587-c38bb2a17cc1" path="/var/lib/kubelet/pods/83c864ba-4d1f-4161-8587-c38bb2a17cc1/volumes" Jul 10 15:12:29.613151 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:29.613124 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-547957cd78-s4l9r_1b90b50f-1257-447e-97cc-0b89ace148c5/copy-manifests/0.log" Jul 10 15:12:29.625679 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:29.625642 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-547957cd78-s4l9r_1b90b50f-1257-447e-97cc-0b89ace148c5/manager/0.log" Jul 10 15:12:30.266338 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:30.266304 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-6kvml_9e667200-d548-4673-9cb8-d923f2eba1fc/postgres/0.log" Jul 10 15:12:31.937715 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:31.937686 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-n872j_1bb64e24-930b-41c3-8bfd-b1d54cfef5c3/manager/0.log" Jul 10 15:12:32.617567 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:32.617533 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-97b9r_400f05b5-cf78-464e-9e82-898cee7e117b/manager/0.log" Jul 10 15:12:33.021041 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:33.020964 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf427n9s_b9c6d657-9f43-4734-bb7f-a28dcdfa2704/istio-proxy/0.log" Jul 10 15:12:33.674508 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:33.674472 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-7f974_992df02a-bd7d-4cbb-92f8-35c7d92035a7/payload-pre-processing/0.log" Jul 10 15:12:33.930687 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:33.930610 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-795d96bbd4-2nql9_13a97abe-27b2-4b7b-ae18-8ae1ff61f043/router/0.log" Jul 10 15:12:34.406211 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:34.406181 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-hl72k_ea6fe6ba-3aaf-4967-8653-26de92c01e33/main/0.log" Jul 10 15:12:34.648813 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:34.648782 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbfqgxb_4568e4d3-668b-4eed-86d7-e1b49c357b36/main/0.log" Jul 10 15:12:34.766820 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:34.766695 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-xfvgk_ce819691-846a-49b3-a923-5b053cfe39d0/main/0.log" Jul 10 15:12:41.797328 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:41.797286 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-rxmgq_4bba05ad-b3f8-4bc4-acef-c5e573ea8e4f/global-pull-secret-syncer/0.log" Jul 10 15:12:41.946744 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:41.946713 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-j2g7m_4ee5894d-0007-499f-a401-2f11724910ac/konnectivity-agent/0.log" Jul 10 15:12:41.987262 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:41.987232 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-35.ec2.internal_486f4ab7baea4188ee524f2052e08a91/haproxy/0.log" Jul 10 15:12:47.439731 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:47.439656 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-n872j_1bb64e24-930b-41c3-8bfd-b1d54cfef5c3/manager/0.log" Jul 10 15:12:47.752435 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:47.752339 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-97b9r_400f05b5-cf78-464e-9e82-898cee7e117b/manager/0.log" Jul 10 15:12:49.627234 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:49.627184 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-c8ztx_cb41c5b6-dab8-4be4-b2be-56040d1659e7/cluster-monitoring-operator/0.log" Jul 10 15:12:49.773461 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:49.773429 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7r8x9_c84e238b-1e80-4718-af8c-e6fdf5c92b03/node-exporter/0.log" Jul 10 15:12:49.791314 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:49.791292 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7r8x9_c84e238b-1e80-4718-af8c-e6fdf5c92b03/kube-rbac-proxy/0.log" Jul 10 15:12:49.810026 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:49.810006 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7r8x9_c84e238b-1e80-4718-af8c-e6fdf5c92b03/init-textfile/0.log" Jul 10 15:12:50.225111 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:50.225077 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-77d98bc786-tbg8l_7558a80d-ed5d-4625-9297-995bfd84b9de/prometheus-operator-admission-webhook/0.log" Jul 10 15:12:51.506641 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:51.506611 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cf856758c-95jwg_bc08d681-36cd-4339-ad25-349ee4e4f916/networking-console-plugin/0.log" Jul 10 15:12:51.779420 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:51.779335 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-n89zh/perf-node-gather-daemonset-sbk9j"] Jul 10 15:12:51.779816 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:51.779798 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83c864ba-4d1f-4161-8587-c38bb2a17cc1" containerName="istio-proxy" Jul 10 15:12:51.779920 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:51.779817 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="83c864ba-4d1f-4161-8587-c38bb2a17cc1" containerName="istio-proxy" Jul 10 15:12:51.779920 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:51.779834 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83c864ba-4d1f-4161-8587-c38bb2a17cc1" containerName="istio-proxy" Jul 10 15:12:51.779920 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:51.779841 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="83c864ba-4d1f-4161-8587-c38bb2a17cc1" containerName="istio-proxy" Jul 10 15:12:51.779920 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:51.779854 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da9b03d0-4137-4cb6-b5f9-0ab64bf8b240" containerName="main" Jul 10 15:12:51.779920 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:51.779862 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="da9b03d0-4137-4cb6-b5f9-0ab64bf8b240" containerName="main" Jul 10 15:12:51.780156 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:51.779952 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="83c864ba-4d1f-4161-8587-c38bb2a17cc1" containerName="istio-proxy" Jul 10 15:12:51.780156 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:51.779963 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="da9b03d0-4137-4cb6-b5f9-0ab64bf8b240" containerName="main" Jul 10 15:12:51.783034 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:51.783013 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n89zh/perf-node-gather-daemonset-sbk9j" Jul 10 15:12:51.785552 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:51.785532 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-n89zh\"/\"kube-root-ca.crt\"" Jul 10 15:12:51.785637 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:51.785586 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-n89zh\"/\"openshift-service-ca.crt\"" Jul 10 15:12:51.786444 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:51.786429 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-n89zh\"/\"default-dockercfg-h7l2f\"" Jul 10 15:12:51.791595 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:51.791575 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n89zh/perf-node-gather-daemonset-sbk9j"] Jul 10 15:12:51.839477 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:51.839450 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4415b1f6-e967-4f91-94ca-ba052d6f07a6-lib-modules\") pod \"perf-node-gather-daemonset-sbk9j\" (UID: \"4415b1f6-e967-4f91-94ca-ba052d6f07a6\") " pod="openshift-must-gather-n89zh/perf-node-gather-daemonset-sbk9j" Jul 10 15:12:51.839588 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:51.839484 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzm2l\" (UniqueName: \"kubernetes.io/projected/4415b1f6-e967-4f91-94ca-ba052d6f07a6-kube-api-access-zzm2l\") pod \"perf-node-gather-daemonset-sbk9j\" (UID: \"4415b1f6-e967-4f91-94ca-ba052d6f07a6\") " pod="openshift-must-gather-n89zh/perf-node-gather-daemonset-sbk9j" Jul 10 15:12:51.839588 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:51.839516 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4415b1f6-e967-4f91-94ca-ba052d6f07a6-sys\") pod \"perf-node-gather-daemonset-sbk9j\" (UID: \"4415b1f6-e967-4f91-94ca-ba052d6f07a6\") " pod="openshift-must-gather-n89zh/perf-node-gather-daemonset-sbk9j" Jul 10 15:12:51.839588 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:51.839534 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4415b1f6-e967-4f91-94ca-ba052d6f07a6-podres\") pod \"perf-node-gather-daemonset-sbk9j\" (UID: \"4415b1f6-e967-4f91-94ca-ba052d6f07a6\") " pod="openshift-must-gather-n89zh/perf-node-gather-daemonset-sbk9j" Jul 10 15:12:51.839588 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:51.839577 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4415b1f6-e967-4f91-94ca-ba052d6f07a6-proc\") pod \"perf-node-gather-daemonset-sbk9j\" (UID: \"4415b1f6-e967-4f91-94ca-ba052d6f07a6\") " pod="openshift-must-gather-n89zh/perf-node-gather-daemonset-sbk9j" Jul 10 15:12:51.940280 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:51.940250 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4415b1f6-e967-4f91-94ca-ba052d6f07a6-proc\") pod \"perf-node-gather-daemonset-sbk9j\" (UID: \"4415b1f6-e967-4f91-94ca-ba052d6f07a6\") " pod="openshift-must-gather-n89zh/perf-node-gather-daemonset-sbk9j" Jul 10 15:12:51.940487 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:51.940290 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4415b1f6-e967-4f91-94ca-ba052d6f07a6-lib-modules\") pod \"perf-node-gather-daemonset-sbk9j\" (UID: \"4415b1f6-e967-4f91-94ca-ba052d6f07a6\") " pod="openshift-must-gather-n89zh/perf-node-gather-daemonset-sbk9j" Jul 10 15:12:51.940487 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:51.940313 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzm2l\" (UniqueName: \"kubernetes.io/projected/4415b1f6-e967-4f91-94ca-ba052d6f07a6-kube-api-access-zzm2l\") pod \"perf-node-gather-daemonset-sbk9j\" (UID: \"4415b1f6-e967-4f91-94ca-ba052d6f07a6\") " pod="openshift-must-gather-n89zh/perf-node-gather-daemonset-sbk9j" Jul 10 15:12:51.940487 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:51.940343 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4415b1f6-e967-4f91-94ca-ba052d6f07a6-sys\") pod \"perf-node-gather-daemonset-sbk9j\" (UID: \"4415b1f6-e967-4f91-94ca-ba052d6f07a6\") " pod="openshift-must-gather-n89zh/perf-node-gather-daemonset-sbk9j" Jul 10 15:12:51.940487 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:51.940358 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4415b1f6-e967-4f91-94ca-ba052d6f07a6-podres\") pod \"perf-node-gather-daemonset-sbk9j\" (UID: \"4415b1f6-e967-4f91-94ca-ba052d6f07a6\") " pod="openshift-must-gather-n89zh/perf-node-gather-daemonset-sbk9j" Jul 10 15:12:51.940487 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:51.940376 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4415b1f6-e967-4f91-94ca-ba052d6f07a6-proc\") pod \"perf-node-gather-daemonset-sbk9j\" (UID: \"4415b1f6-e967-4f91-94ca-ba052d6f07a6\") " pod="openshift-must-gather-n89zh/perf-node-gather-daemonset-sbk9j" Jul 10 15:12:51.940487 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:51.940427 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4415b1f6-e967-4f91-94ca-ba052d6f07a6-lib-modules\") pod \"perf-node-gather-daemonset-sbk9j\" (UID: \"4415b1f6-e967-4f91-94ca-ba052d6f07a6\") " pod="openshift-must-gather-n89zh/perf-node-gather-daemonset-sbk9j" Jul 10 15:12:51.940487 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:51.940437 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4415b1f6-e967-4f91-94ca-ba052d6f07a6-sys\") pod \"perf-node-gather-daemonset-sbk9j\" (UID: \"4415b1f6-e967-4f91-94ca-ba052d6f07a6\") " pod="openshift-must-gather-n89zh/perf-node-gather-daemonset-sbk9j" Jul 10 15:12:51.940487 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:51.940453 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4415b1f6-e967-4f91-94ca-ba052d6f07a6-podres\") pod \"perf-node-gather-daemonset-sbk9j\" (UID: \"4415b1f6-e967-4f91-94ca-ba052d6f07a6\") " pod="openshift-must-gather-n89zh/perf-node-gather-daemonset-sbk9j" Jul 10 15:12:51.947632 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:51.947611 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzm2l\" (UniqueName: \"kubernetes.io/projected/4415b1f6-e967-4f91-94ca-ba052d6f07a6-kube-api-access-zzm2l\") pod \"perf-node-gather-daemonset-sbk9j\" (UID: \"4415b1f6-e967-4f91-94ca-ba052d6f07a6\") " pod="openshift-must-gather-n89zh/perf-node-gather-daemonset-sbk9j" Jul 10 15:12:52.094139 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:52.094067 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n89zh/perf-node-gather-daemonset-sbk9j" Jul 10 15:12:52.131925 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:52.131900 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-n6pfx_296be890-64e9-4fb3-8c68-e4c2a7c8cc2c/console-operator/1.log" Jul 10 15:12:52.136710 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:52.136670 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-n6pfx_296be890-64e9-4fb3-8c68-e4c2a7c8cc2c/console-operator/2.log" Jul 10 15:12:52.230533 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:52.230493 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n89zh/perf-node-gather-daemonset-sbk9j"] Jul 10 15:12:52.233292 ip-10-0-140-35 kubenswrapper[2569]: W0710 15:12:52.233268 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4415b1f6_e967_4f91_94ca_ba052d6f07a6.slice/crio-bc7da2f82f3a1e5081887e43b792e805229088eabd9b0389bd221f03e7b5b8e1 WatchSource:0}: Error finding container bc7da2f82f3a1e5081887e43b792e805229088eabd9b0389bd221f03e7b5b8e1: Status 404 returned error can't find the container with id bc7da2f82f3a1e5081887e43b792e805229088eabd9b0389bd221f03e7b5b8e1 Jul 10 15:12:52.711989 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:52.711963 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-68c9cbb497-7jcj6_ae441373-d195-4c90-b77a-d189ae1951af/download-server/0.log" Jul 10 15:12:52.884365 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:52.884331 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n89zh/perf-node-gather-daemonset-sbk9j" event={"ID":"4415b1f6-e967-4f91-94ca-ba052d6f07a6","Type":"ContainerStarted","Data":"fbe3a886394cac2056175f7ab36f962f70648593763e0483c9ef0bf7c8ae1270"} Jul 10 15:12:52.884534 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:52.884388 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n89zh/perf-node-gather-daemonset-sbk9j" event={"ID":"4415b1f6-e967-4f91-94ca-ba052d6f07a6","Type":"ContainerStarted","Data":"bc7da2f82f3a1e5081887e43b792e805229088eabd9b0389bd221f03e7b5b8e1"} Jul 10 15:12:52.884534 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:52.884495 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-n89zh/perf-node-gather-daemonset-sbk9j" Jul 10 15:12:52.902250 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:52.902206 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-n89zh/perf-node-gather-daemonset-sbk9j" podStartSLOduration=1.902190512 podStartE2EDuration="1.902190512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 15:12:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 15:12:52.899747631 +0000 UTC m=+2518.651119345" watchObservedRunningTime="2026-07-10 15:12:52.902190512 +0000 UTC m=+2518.653562208" Jul 10 15:12:53.251419 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:53.251391 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78495597f6-bttbd_4d2d05db-87c6-4dff-9d05-599de4cfa9e4/volume-data-source-validator/0.log" Jul 10 15:12:54.180792 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:54.180744 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-f698x_68e91fda-6611-4951-806d-2514fe8bd1cb/dns/0.log" Jul 10 15:12:54.198763 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:54.198736 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-f698x_68e91fda-6611-4951-806d-2514fe8bd1cb/kube-rbac-proxy/0.log" Jul 10 15:12:54.294434 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:54.294402 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-842qb_270982ae-963c-42a2-a10f-06cd88f30f81/dns-node-resolver/0.log" Jul 10 15:12:54.885607 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:54.885573 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-w9ww7_21295856-adfb-486d-a3dc-b9ff8aa3ff50/node-ca/0.log" Jul 10 15:12:55.763097 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:55.763068 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf427n9s_b9c6d657-9f43-4734-bb7f-a28dcdfa2704/istio-proxy/0.log" Jul 10 15:12:56.101415 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:56.101339 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-7f974_992df02a-bd7d-4cbb-92f8-35c7d92035a7/payload-pre-processing/0.log" Jul 10 15:12:56.162699 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:56.162673 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-795d96bbd4-2nql9_13a97abe-27b2-4b7b-ae18-8ae1ff61f043/router/0.log" Jul 10 15:12:56.840987 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:56.840957 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wbbhv_80537ce8-cbd4-4a4c-8604-7377993cd5b1/serve-healthcheck-canary/0.log" Jul 10 15:12:57.264504 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:57.264475 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-qtr8d_dc5a5ba3-52b4-4b58-923e-df7e716d9438/insights-operator/0.log" Jul 10 15:12:57.264966 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:57.264953 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-qtr8d_dc5a5ba3-52b4-4b58-923e-df7e716d9438/insights-operator/1.log" Jul 10 15:12:57.395525 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:57.395497 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-q9tkw_1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0/kube-rbac-proxy/0.log" Jul 10 15:12:57.411840 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:57.411815 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-q9tkw_1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0/exporter/0.log" Jul 10 15:12:57.429191 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:57.429164 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-q9tkw_1dd7a68f-7b1e-44c0-afbe-ab1bb7df83f0/extractor/0.log" Jul 10 15:12:58.900016 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:12:58.899987 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-n89zh/perf-node-gather-daemonset-sbk9j" Jul 10 15:13:00.118439 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:13:00.118410 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-547957cd78-s4l9r_1b90b50f-1257-447e-97cc-0b89ace148c5/manager/0.log" Jul 10 15:13:00.133269 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:13:00.133249 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-547957cd78-s4l9r_1b90b50f-1257-447e-97cc-0b89ace148c5/copy-manifests/0.log" Jul 10 15:13:01.398554 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:13:01.398513 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-d76757ddd-kt87g_5b4e00a7-48d5-4566-8e92-83cbd3bfea97/manager/0.log" Jul 10 15:13:02.277114 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:13:02.277085 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf427n9s_b9c6d657-9f43-4734-bb7f-a28dcdfa2704/istio-proxy/0.log" Jul 10 15:13:02.596654 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:13:02.596630 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-7f974_992df02a-bd7d-4cbb-92f8-35c7d92035a7/payload-pre-processing/0.log" Jul 10 15:13:02.642870 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:13:02.642836 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-795d96bbd4-2nql9_13a97abe-27b2-4b7b-ae18-8ae1ff61f043/router/0.log" Jul 10 15:13:03.823926 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:13:03.823897 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf427n9s_b9c6d657-9f43-4734-bb7f-a28dcdfa2704/istio-proxy/0.log" Jul 10 15:13:04.137467 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:13:04.137437 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-7f974_992df02a-bd7d-4cbb-92f8-35c7d92035a7/payload-pre-processing/0.log" Jul 10 15:13:04.187872 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:13:04.187842 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-795d96bbd4-2nql9_13a97abe-27b2-4b7b-ae18-8ae1ff61f043/router/0.log" Jul 10 15:13:06.462115 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:13:06.462087 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-9z7n5_064387da-b900-4cda-9839-2b5c166bc246/migrator/0.log" Jul 10 15:13:06.480021 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:13:06.479994 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-9z7n5_064387da-b900-4cda-9839-2b5c166bc246/graceful-termination/0.log" Jul 10 15:13:06.806860 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:13:06.806787 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-d7kg8_70c38fec-83c3-4e81-90c3-ab3f7204fe3c/kube-storage-version-migrator-operator/1.log" Jul 10 15:13:06.807548 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:13:06.807531 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-d7kg8_70c38fec-83c3-4e81-90c3-ab3f7204fe3c/kube-storage-version-migrator-operator/0.log" Jul 10 15:13:07.736879 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:13:07.736852 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2sjq2_ba16bc50-891a-4477-8bd5-de3ef058d04f/kube-multus-additional-cni-plugins/0.log" Jul 10 15:13:07.755899 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:13:07.755876 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2sjq2_ba16bc50-891a-4477-8bd5-de3ef058d04f/egress-router-binary-copy/0.log" Jul 10 15:13:07.772176 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:13:07.772156 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2sjq2_ba16bc50-891a-4477-8bd5-de3ef058d04f/cni-plugins/0.log" Jul 10 15:13:07.788185 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:13:07.788164 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2sjq2_ba16bc50-891a-4477-8bd5-de3ef058d04f/bond-cni-plugin/0.log" Jul 10 15:13:07.804901 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:13:07.804882 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2sjq2_ba16bc50-891a-4477-8bd5-de3ef058d04f/routeoverride-cni/0.log" Jul 10 15:13:07.821971 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:13:07.821952 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2sjq2_ba16bc50-891a-4477-8bd5-de3ef058d04f/whereabouts-cni-bincopy/0.log" Jul 10 15:13:07.838521 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:13:07.838494 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2sjq2_ba16bc50-891a-4477-8bd5-de3ef058d04f/whereabouts-cni/0.log" Jul 10 15:13:08.191924 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:13:08.191890 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-r6rkz_fc922b38-8f9b-4ec4-95c0-c132253a3158/kube-multus/0.log" Jul 10 15:13:08.211790 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:13:08.211725 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-fdhw7_ff3c9cb2-1924-4ea9-9a93-282213b083db/network-metrics-daemon/0.log" Jul 10 15:13:08.226939 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:13:08.226917 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-fdhw7_ff3c9cb2-1924-4ea9-9a93-282213b083db/kube-rbac-proxy/0.log" Jul 10 15:13:09.493619 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:13:09.493590 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxvq2_016248a0-bbe2-44c9-8e51-5d4695ee6a52/ovn-controller/0.log" Jul 10 15:13:09.507293 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:13:09.507273 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxvq2_016248a0-bbe2-44c9-8e51-5d4695ee6a52/ovn-acl-logging/0.log" Jul 10 15:13:09.507463 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:13:09.507306 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxvq2_016248a0-bbe2-44c9-8e51-5d4695ee6a52/ovn-acl-logging/1.log" Jul 10 15:13:09.521321 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:13:09.521298 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxvq2_016248a0-bbe2-44c9-8e51-5d4695ee6a52/kube-rbac-proxy-node/0.log" Jul 10 15:13:09.539949 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:13:09.539929 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxvq2_016248a0-bbe2-44c9-8e51-5d4695ee6a52/kube-rbac-proxy-ovn-metrics/0.log" Jul 10 15:13:09.556621 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:13:09.556602 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxvq2_016248a0-bbe2-44c9-8e51-5d4695ee6a52/northd/0.log" Jul 10 15:13:09.576318 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:13:09.576295 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxvq2_016248a0-bbe2-44c9-8e51-5d4695ee6a52/nbdb/0.log" Jul 10 15:13:09.593680 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:13:09.593655 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxvq2_016248a0-bbe2-44c9-8e51-5d4695ee6a52/sbdb/0.log" Jul 10 15:13:09.638905 ip-10-0-140-35 kubenswrapper[2569]: I0710 15:13:09.638877 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kxvq2_016248a0-bbe2-44c9-8e51-5d4695ee6a52/ovnkube-controller/0.log"