Jul 06 22:09:08.568348 ip-10-0-128-71 systemd[1]: Starting Kubernetes Kubelet... Jul 06 22:09:09.115631 ip-10-0-128-71 kubenswrapper[2555]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 22:09:09.115631 ip-10-0-128-71 kubenswrapper[2555]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 06 22:09:09.115631 ip-10-0-128-71 kubenswrapper[2555]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 22:09:09.115631 ip-10-0-128-71 kubenswrapper[2555]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 06 22:09:09.115631 ip-10-0-128-71 kubenswrapper[2555]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 22:09:09.115631 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.027271 2555 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 06 22:09:09.115631 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.032858 2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 22:09:09.115631 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.032873 2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 22:09:09.115631 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.032877 2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 22:09:09.115631 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.032880 2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 22:09:09.115631 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.032883 2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 22:09:09.115631 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.032886 2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 22:09:09.115631 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.032888 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 22:09:09.114779 ip-10-0-128-71 systemd[1]: Started Kubernetes Kubelet. Jul 06 22:09:09.118858 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.032891 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 22:09:09.118858 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.032894 2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 22:09:09.118858 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.032896 2555 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 22:09:09.118858 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.032899 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 22:09:09.118858 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.032901 2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 22:09:09.118858 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.032904 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 22:09:09.118858 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.032907 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 22:09:09.118858 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.032910 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 22:09:09.118858 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.032913 2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 22:09:09.118858 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.032917 2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 22:09:09.118858 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.032920 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 22:09:09.118858 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.032922 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 22:09:09.118858 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.032927 2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 22:09:09.118858 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.032931 2555 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 22:09:09.118858 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.032935 2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 22:09:09.118858 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.032938 2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 22:09:09.118858 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.032941 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 22:09:09.118858 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.032945 2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 22:09:09.118858 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.032949 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 22:09:09.121483 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.032952 2555 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 22:09:09.121483 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.032955 2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 22:09:09.121483 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.032958 2555 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 22:09:09.121483 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.032961 2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 22:09:09.121483 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.032964 2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 22:09:09.121483 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.032966 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 22:09:09.121483 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.032969 2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 22:09:09.121483 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.032972 2555 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 22:09:09.121483 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.032974 2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 22:09:09.121483 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.032977 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 22:09:09.121483 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.032979 2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 22:09:09.121483 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.032982 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 22:09:09.121483 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.032984 2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 22:09:09.121483 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.032988 2555 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 22:09:09.121483 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.032991 2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 22:09:09.121483 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.032993 2555 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 22:09:09.121483 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.032996 2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 22:09:09.121483 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.032998 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 22:09:09.121483 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033013 2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 22:09:09.121483 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033020 2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 22:09:09.127298 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033024 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 22:09:09.127298 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033027 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 22:09:09.127298 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033031 2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 22:09:09.127298 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033034 2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 22:09:09.127298 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033037 2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 22:09:09.127298 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033040 2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 22:09:09.127298 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033043 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 22:09:09.127298 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033045 2555 feature_gate.go:328] unrecognized feature gate: Example Jul 06 22:09:09.127298 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033048 2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 22:09:09.127298 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033050 2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 22:09:09.127298 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033053 2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 22:09:09.127298 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033055 2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 22:09:09.127298 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033058 2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 22:09:09.127298 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033061 2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 22:09:09.127298 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033063 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 22:09:09.127298 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033066 2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 22:09:09.127298 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033068 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 22:09:09.127298 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033071 2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 22:09:09.127298 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033073 2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 22:09:09.127298 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033076 2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 22:09:09.128729 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033078 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 22:09:09.128729 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033081 2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 22:09:09.128729 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033083 2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 22:09:09.128729 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033086 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 22:09:09.128729 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033089 2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 22:09:09.128729 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033092 2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 22:09:09.128729 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033094 2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 22:09:09.128729 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033097 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 22:09:09.128729 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033100 2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 22:09:09.128729 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033102 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 22:09:09.128729 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033105 2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 22:09:09.128729 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033108 2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 22:09:09.128729 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033111 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 22:09:09.128729 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033113 2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 22:09:09.128729 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033116 2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 22:09:09.128729 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033118 2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 22:09:09.128729 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033120 2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 22:09:09.128729 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033123 2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 22:09:09.128729 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033125 2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 22:09:09.128729 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033128 2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 22:09:09.143297 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033131 2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 22:09:09.143297 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033135 2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 22:09:09.143297 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033137 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 22:09:09.143297 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033140 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 22:09:09.143297 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033143 2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 22:09:09.143297 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033145 2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 22:09:09.143297 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033148 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 22:09:09.143297 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033152 2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 22:09:09.143297 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033156 2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 22:09:09.143297 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033159 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 22:09:09.143297 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033161 2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 22:09:09.143297 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033164 2555 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 22:09:09.143297 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033166 2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 22:09:09.143297 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033880 2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 22:09:09.143297 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033887 2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 22:09:09.143297 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033891 2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 22:09:09.143297 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033894 2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 22:09:09.143297 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033897 2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 22:09:09.143297 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033900 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 22:09:09.143297 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033903 2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 22:09:09.156125 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033906 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 22:09:09.156125 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033908 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 22:09:09.156125 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033911 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 22:09:09.156125 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033914 2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 22:09:09.156125 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033917 2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 22:09:09.156125 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033919 2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 22:09:09.156125 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033922 2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 22:09:09.156125 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033925 2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 22:09:09.156125 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033927 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 22:09:09.156125 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033930 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 22:09:09.156125 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033933 2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 22:09:09.156125 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033935 2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 22:09:09.156125 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033938 2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 22:09:09.156125 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033941 2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 22:09:09.156125 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033943 2555 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 22:09:09.156125 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033946 2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 22:09:09.156125 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033949 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 22:09:09.156125 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033951 2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 22:09:09.156125 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033955 2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 22:09:09.156125 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033957 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 22:09:09.159212 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033962 2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 22:09:09.159212 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033966 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 22:09:09.159212 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033969 2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 22:09:09.159212 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033972 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 22:09:09.159212 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033975 2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 22:09:09.159212 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033978 2555 feature_gate.go:328] unrecognized feature gate: Example Jul 06 22:09:09.159212 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033981 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 22:09:09.159212 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033984 2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 22:09:09.159212 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033987 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 22:09:09.159212 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033989 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 22:09:09.159212 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033992 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 22:09:09.159212 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033994 2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 22:09:09.159212 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.033997 2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 22:09:09.159212 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034000 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 22:09:09.159212 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034021 2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 22:09:09.159212 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034027 2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 22:09:09.159212 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034030 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 22:09:09.159212 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034034 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 22:09:09.159212 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034037 2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 22:09:09.309981 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034040 2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 22:09:09.309981 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034043 2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 22:09:09.309981 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034045 2555 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 22:09:09.309981 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034048 2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 22:09:09.309981 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034053 2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 22:09:09.309981 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034057 2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 22:09:09.309981 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034060 2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 22:09:09.309981 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034064 2555 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 22:09:09.309981 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034067 2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 22:09:09.309981 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034070 2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 22:09:09.309981 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034073 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 22:09:09.309981 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034075 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 22:09:09.309981 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034078 2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 22:09:09.309981 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034081 2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 22:09:09.309981 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034083 2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 22:09:09.309981 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034089 2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 22:09:09.309981 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034093 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 22:09:09.309981 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034096 2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 22:09:09.309981 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034106 2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 22:09:09.309981 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034109 2555 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 22:09:09.310995 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034114 2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 22:09:09.310995 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034118 2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 22:09:09.310995 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034121 2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 22:09:09.310995 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034123 2555 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 22:09:09.310995 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034126 2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 22:09:09.310995 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034129 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 22:09:09.310995 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034132 2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 22:09:09.310995 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034134 2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 22:09:09.310995 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034136 2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 22:09:09.310995 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034139 2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 22:09:09.310995 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034142 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 22:09:09.310995 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034145 2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 22:09:09.310995 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034148 2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 22:09:09.310995 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034150 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 22:09:09.310995 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034153 2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 22:09:09.310995 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034156 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 22:09:09.310995 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034158 2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 22:09:09.310995 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034161 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 22:09:09.310995 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034164 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 22:09:09.311755 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034166 2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 22:09:09.311755 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034169 2555 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 22:09:09.311755 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034171 2555 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 22:09:09.311755 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034173 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 22:09:09.311755 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034176 2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 22:09:09.311755 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034178 2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 22:09:09.311755 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034181 2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 22:09:09.311755 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034183 2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 22:09:09.311755 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034189 2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 22:09:09.311755 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034191 2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 22:09:09.311755 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034194 2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 22:09:09.311755 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034196 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 22:09:09.311755 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034199 2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 22:09:09.311755 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034201 2555 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 22:09:09.311755 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034594 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 22:09:09.311755 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034599 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 22:09:09.311755 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034602 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 22:09:09.311755 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034604 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 22:09:09.311755 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034607 2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 22:09:09.311755 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034610 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 22:09:09.312615 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034613 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 22:09:09.312615 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034615 2555 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 22:09:09.312615 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034618 2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 22:09:09.312615 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034621 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 22:09:09.312615 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034624 2555 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 22:09:09.312615 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034626 2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 22:09:09.312615 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034629 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 22:09:09.312615 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034631 2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 22:09:09.312615 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034634 2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 22:09:09.312615 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034639 2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 22:09:09.312615 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034641 2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 22:09:09.312615 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034644 2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 22:09:09.312615 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034646 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 22:09:09.312615 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034649 2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 22:09:09.312615 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034652 2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 22:09:09.312615 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034654 2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 22:09:09.312615 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034657 2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 22:09:09.312615 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034659 2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 22:09:09.312615 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034661 2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 22:09:09.312615 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034664 2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 22:09:09.313395 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034668 2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 22:09:09.313395 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034671 2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 22:09:09.313395 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034674 2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 22:09:09.313395 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034677 2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 22:09:09.313395 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034680 2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 22:09:09.313395 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034682 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 22:09:09.313395 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034685 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 22:09:09.313395 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034687 2555 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 22:09:09.313395 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034692 2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 22:09:09.313395 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034696 2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 22:09:09.313395 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034699 2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 22:09:09.313395 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034702 2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 22:09:09.313395 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034704 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 22:09:09.313395 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034707 2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 22:09:09.313395 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034710 2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 22:09:09.313395 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034713 2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 22:09:09.313395 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034715 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 22:09:09.313395 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034719 2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 22:09:09.313395 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034721 2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 22:09:09.314237 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034724 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 22:09:09.314237 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034726 2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 22:09:09.314237 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034730 2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 22:09:09.314237 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034733 2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 22:09:09.314237 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034735 2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 22:09:09.314237 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034738 2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 22:09:09.314237 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034740 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 22:09:09.314237 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034743 2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 22:09:09.314237 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034745 2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 22:09:09.314237 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034748 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 22:09:09.314237 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034750 2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 22:09:09.314237 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034753 2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 22:09:09.314237 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034756 2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 22:09:09.314237 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034760 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 22:09:09.314237 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034762 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 22:09:09.314237 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034765 2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 22:09:09.314237 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034768 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 22:09:09.314237 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034771 2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 22:09:09.314237 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034773 2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 22:09:09.315117 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034776 2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 22:09:09.315117 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034778 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 22:09:09.315117 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034781 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 22:09:09.315117 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034785 2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 22:09:09.315117 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034789 2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 22:09:09.315117 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034792 2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 22:09:09.315117 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034796 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 22:09:09.315117 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034799 2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 22:09:09.315117 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034802 2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 22:09:09.315117 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034805 2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 22:09:09.315117 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034808 2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 22:09:09.315117 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034811 2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 22:09:09.315117 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034814 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 22:09:09.315117 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034817 2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 22:09:09.315117 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034820 2555 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 22:09:09.315117 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034824 2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 22:09:09.315117 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034826 2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 22:09:09.315117 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034829 2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 22:09:09.315117 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034832 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 22:09:09.315929 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034834 2555 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 22:09:09.315929 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034837 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 22:09:09.315929 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034839 2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 22:09:09.315929 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034842 2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 22:09:09.315929 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034845 2555 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 22:09:09.315929 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034847 2555 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 22:09:09.315929 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034850 2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 22:09:09.315929 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034853 2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 22:09:09.315929 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034856 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 22:09:09.315929 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034859 2555 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 22:09:09.315929 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034861 2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 22:09:09.315929 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034864 2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 22:09:09.315929 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034867 2555 feature_gate.go:328] unrecognized feature gate: Example Jul 06 22:09:09.315929 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034869 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 22:09:09.315929 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034872 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 22:09:09.315929 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.034874 2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 22:09:09.315929 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.034962 2555 flags.go:64] FLAG: --address="0.0.0.0" Jul 06 22:09:09.315929 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.034974 2555 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 06 22:09:09.315929 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.034980 2555 flags.go:64] FLAG: --anonymous-auth="true" Jul 06 22:09:09.315929 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.034984 2555 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 06 22:09:09.315929 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.034989 2555 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 06 22:09:09.316740 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.034992 2555 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 06 22:09:09.316740 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.034997 2555 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 06 22:09:09.316740 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035002 2555 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 06 22:09:09.316740 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035022 2555 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 06 22:09:09.316740 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035025 2555 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 06 22:09:09.316740 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035030 2555 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 06 22:09:09.316740 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035033 2555 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 06 22:09:09.316740 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035038 2555 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 06 22:09:09.316740 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035042 2555 flags.go:64] FLAG: --cgroup-root="" Jul 06 22:09:09.316740 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035045 2555 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 06 22:09:09.316740 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035048 2555 flags.go:64] FLAG: --client-ca-file="" Jul 06 22:09:09.316740 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035051 2555 flags.go:64] FLAG: --cloud-provider="external" Jul 06 22:09:09.316740 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035054 2555 flags.go:64] FLAG: --cluster-dns="[]" Jul 06 22:09:09.316740 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035059 2555 flags.go:64] FLAG: --cluster-domain="" Jul 06 22:09:09.316740 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035062 2555 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 06 22:09:09.316740 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035065 2555 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 06 22:09:09.316740 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035068 2555 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 06 22:09:09.316740 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035072 2555 flags.go:64] FLAG: --container-log-max-files="5" Jul 06 22:09:09.316740 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035077 2555 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 06 22:09:09.316740 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035080 2555 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 06 22:09:09.316740 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035083 2555 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 06 22:09:09.316740 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035087 2555 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 06 22:09:09.316740 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035090 2555 flags.go:64] FLAG: --contention-profiling="false" Jul 06 22:09:09.316740 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035093 2555 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 06 22:09:09.317563 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035096 2555 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 06 22:09:09.317563 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035100 2555 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 06 22:09:09.317563 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035102 2555 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 06 22:09:09.317563 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035107 2555 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 06 22:09:09.317563 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035110 2555 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 06 22:09:09.317563 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035112 2555 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 06 22:09:09.317563 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035115 2555 flags.go:64] FLAG: --enable-load-reader="false" Jul 06 22:09:09.317563 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035119 2555 flags.go:64] FLAG: --enable-server="true" Jul 06 22:09:09.317563 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035122 2555 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 06 22:09:09.317563 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035126 2555 flags.go:64] FLAG: --event-burst="100" Jul 06 22:09:09.317563 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035129 2555 flags.go:64] FLAG: --event-qps="50" Jul 06 22:09:09.317563 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035132 2555 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 06 22:09:09.317563 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035136 2555 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 06 22:09:09.317563 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035139 2555 flags.go:64] FLAG: --eviction-hard="" Jul 06 22:09:09.317563 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035143 2555 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 06 22:09:09.317563 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035147 2555 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 06 22:09:09.317563 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035150 2555 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 06 22:09:09.317563 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035153 2555 flags.go:64] FLAG: --eviction-soft="" Jul 06 22:09:09.317563 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035156 2555 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 06 22:09:09.317563 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035159 2555 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 06 22:09:09.317563 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035162 2555 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 06 22:09:09.317563 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035165 2555 flags.go:64] FLAG: --experimental-mounter-path="" Jul 06 22:09:09.317563 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035168 2555 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 06 22:09:09.317563 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035170 2555 flags.go:64] FLAG: --fail-swap-on="true" Jul 06 22:09:09.317563 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035173 2555 flags.go:64] FLAG: --feature-gates="" Jul 06 22:09:09.318373 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035177 2555 flags.go:64] FLAG: --file-check-frequency="20s" Jul 06 22:09:09.318373 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035181 2555 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 06 22:09:09.318373 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035184 2555 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 06 22:09:09.318373 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035188 2555 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 06 22:09:09.318373 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035191 2555 flags.go:64] FLAG: --healthz-port="10248" Jul 06 22:09:09.318373 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035193 2555 flags.go:64] FLAG: --help="false" Jul 06 22:09:09.318373 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035196 2555 flags.go:64] FLAG: --hostname-override="ip-10-0-128-71.ec2.internal" Jul 06 22:09:09.318373 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035200 2555 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 06 22:09:09.318373 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035203 2555 flags.go:64] FLAG: --http-check-frequency="20s" Jul 06 22:09:09.318373 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035206 2555 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 06 22:09:09.318373 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035209 2555 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 06 22:09:09.318373 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035212 2555 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 06 22:09:09.318373 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035216 2555 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 06 22:09:09.318373 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035219 2555 flags.go:64] FLAG: --image-service-endpoint="" Jul 06 22:09:09.318373 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035222 2555 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 06 22:09:09.318373 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035225 2555 flags.go:64] FLAG: --kube-api-burst="100" Jul 06 22:09:09.318373 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035228 2555 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 06 22:09:09.318373 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035231 2555 flags.go:64] FLAG: --kube-api-qps="50" Jul 06 22:09:09.318373 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035234 2555 flags.go:64] FLAG: --kube-reserved="" Jul 06 22:09:09.318373 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035237 2555 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 06 22:09:09.318373 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035239 2555 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 06 22:09:09.318373 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035242 2555 flags.go:64] FLAG: --kubelet-cgroups="" Jul 06 22:09:09.318373 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035247 2555 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 06 22:09:09.318373 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035250 2555 flags.go:64] FLAG: --lock-file="" Jul 06 22:09:09.324074 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035252 2555 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 06 22:09:09.324074 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035255 2555 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 06 22:09:09.324074 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035258 2555 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 06 22:09:09.324074 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035263 2555 flags.go:64] FLAG: --log-json-split-stream="false" Jul 06 22:09:09.324074 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035266 2555 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 06 22:09:09.324074 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035269 2555 flags.go:64] FLAG: --log-text-split-stream="false" Jul 06 22:09:09.324074 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035272 2555 flags.go:64] FLAG: --logging-format="text" Jul 06 22:09:09.324074 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035275 2555 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 06 22:09:09.324074 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035278 2555 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 06 22:09:09.324074 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035283 2555 flags.go:64] FLAG: --manifest-url="" Jul 06 22:09:09.324074 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035286 2555 flags.go:64] FLAG: --manifest-url-header="" Jul 06 22:09:09.324074 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035290 2555 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 06 22:09:09.324074 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035293 2555 flags.go:64] FLAG: --max-open-files="1000000" Jul 06 22:09:09.324074 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035297 2555 flags.go:64] FLAG: --max-pods="110" Jul 06 22:09:09.324074 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035300 2555 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 06 22:09:09.324074 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035303 2555 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 06 22:09:09.324074 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035306 2555 flags.go:64] FLAG: --memory-manager-policy="None" Jul 06 22:09:09.324074 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035309 2555 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 06 22:09:09.324074 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035312 2555 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 06 22:09:09.324074 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035315 2555 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 06 22:09:09.324074 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035318 2555 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 06 22:09:09.324074 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035326 2555 flags.go:64] FLAG: --node-status-max-images="50" Jul 06 22:09:09.324074 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035330 2555 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 06 22:09:09.324074 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035333 2555 flags.go:64] FLAG: --oom-score-adj="-999" Jul 06 22:09:09.324074 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035336 2555 flags.go:64] FLAG: --pod-cidr="" Jul 06 22:09:09.324897 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035338 2555 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 06 22:09:09.324897 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035344 2555 flags.go:64] FLAG: --pod-manifest-path="" Jul 06 22:09:09.324897 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035347 2555 flags.go:64] FLAG: --pod-max-pids="-1" Jul 06 22:09:09.324897 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035350 2555 flags.go:64] FLAG: --pods-per-core="0" Jul 06 22:09:09.324897 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035353 2555 flags.go:64] FLAG: --port="10250" Jul 06 22:09:09.324897 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035358 2555 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 06 22:09:09.324897 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035362 2555 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b929ac0cbc6db716" Jul 06 22:09:09.324897 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035365 2555 flags.go:64] FLAG: --qos-reserved="" Jul 06 22:09:09.324897 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035368 2555 flags.go:64] FLAG: --read-only-port="10255" Jul 06 22:09:09.324897 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035371 2555 flags.go:64] FLAG: --register-node="true" Jul 06 22:09:09.324897 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035374 2555 flags.go:64] FLAG: --register-with-taints="" Jul 06 22:09:09.324897 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035378 2555 flags.go:64] FLAG: --registry-burst="10" Jul 06 22:09:09.324897 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035380 2555 flags.go:64] FLAG: --registry-qps="5" Jul 06 22:09:09.324897 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035383 2555 flags.go:64] FLAG: --reserved-cpus="" Jul 06 22:09:09.324897 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035386 2555 flags.go:64] FLAG: --reserved-memory="" Jul 06 22:09:09.324897 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035390 2555 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 06 22:09:09.324897 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035393 2555 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 06 22:09:09.324897 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035398 2555 flags.go:64] FLAG: --rotate-certificates="false" Jul 06 22:09:09.324897 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035401 2555 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 06 22:09:09.324897 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035404 2555 flags.go:64] FLAG: --runonce="false" Jul 06 22:09:09.324897 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035407 2555 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 06 22:09:09.324897 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035410 2555 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 06 22:09:09.324897 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035413 2555 flags.go:64] FLAG: --seccomp-default="false" Jul 06 22:09:09.324897 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035416 2555 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 06 22:09:09.324897 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035419 2555 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 06 22:09:09.337803 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035422 2555 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 06 22:09:09.337803 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035425 2555 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 06 22:09:09.337803 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035428 2555 flags.go:64] FLAG: --storage-driver-password="root" Jul 06 22:09:09.337803 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035431 2555 flags.go:64] FLAG: --storage-driver-secure="false" Jul 06 22:09:09.337803 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035434 2555 flags.go:64] FLAG: --storage-driver-table="stats" Jul 06 22:09:09.337803 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035437 2555 flags.go:64] FLAG: --storage-driver-user="root" Jul 06 22:09:09.337803 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035440 2555 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 06 22:09:09.337803 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035443 2555 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 06 22:09:09.337803 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035446 2555 flags.go:64] FLAG: --system-cgroups="" Jul 06 22:09:09.337803 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035449 2555 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 06 22:09:09.337803 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035454 2555 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 06 22:09:09.337803 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035457 2555 flags.go:64] FLAG: --tls-cert-file="" Jul 06 22:09:09.337803 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035461 2555 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 06 22:09:09.337803 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035466 2555 flags.go:64] FLAG: --tls-min-version="" Jul 06 22:09:09.337803 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035470 2555 flags.go:64] FLAG: --tls-private-key-file="" Jul 06 22:09:09.337803 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035473 2555 flags.go:64] FLAG: --topology-manager-policy="none" Jul 06 22:09:09.337803 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035477 2555 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 06 22:09:09.337803 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035480 2555 flags.go:64] FLAG: --topology-manager-scope="container" Jul 06 22:09:09.337803 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035483 2555 flags.go:64] FLAG: --v="2" Jul 06 22:09:09.337803 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035487 2555 flags.go:64] FLAG: --version="false" Jul 06 22:09:09.337803 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035491 2555 flags.go:64] FLAG: --vmodule="" Jul 06 22:09:09.337803 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035494 2555 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 06 22:09:09.337803 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.035498 2555 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 06 22:09:09.337803 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035619 2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 22:09:09.337803 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035625 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 22:09:09.338598 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035629 2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 22:09:09.338598 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035632 2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 22:09:09.338598 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035635 2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 22:09:09.338598 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035638 2555 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 22:09:09.338598 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035641 2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 22:09:09.338598 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035644 2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 22:09:09.338598 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035646 2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 22:09:09.338598 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035649 2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 22:09:09.338598 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035652 2555 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 22:09:09.338598 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035654 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 22:09:09.338598 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035657 2555 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 22:09:09.338598 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035659 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 22:09:09.338598 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035662 2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 22:09:09.338598 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035665 2555 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 22:09:09.338598 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035667 2555 feature_gate.go:328] unrecognized feature gate: Example Jul 06 22:09:09.338598 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035670 2555 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 22:09:09.338598 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035672 2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 22:09:09.338598 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035675 2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 22:09:09.338598 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035678 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 22:09:09.338598 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035682 2555 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 22:09:09.338598 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035686 2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 22:09:09.339398 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035688 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 22:09:09.339398 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035691 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 22:09:09.339398 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035694 2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 22:09:09.339398 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035697 2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 22:09:09.339398 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035699 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 22:09:09.339398 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035701 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 22:09:09.339398 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035704 2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 22:09:09.339398 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035707 2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 22:09:09.339398 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035709 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 22:09:09.339398 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035712 2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 22:09:09.339398 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035716 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 22:09:09.339398 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035719 2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 22:09:09.339398 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035721 2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 22:09:09.339398 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035724 2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 22:09:09.339398 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035726 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 22:09:09.339398 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035729 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 22:09:09.339398 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035732 2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 22:09:09.339398 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035734 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 22:09:09.339398 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035737 2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 22:09:09.437185 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035740 2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 22:09:09.437185 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035742 2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 22:09:09.437185 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035745 2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 22:09:09.437185 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035748 2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 22:09:09.437185 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035752 2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 22:09:09.437185 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035755 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 22:09:09.437185 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035758 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 22:09:09.437185 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035761 2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 22:09:09.437185 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035763 2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 22:09:09.437185 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035766 2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 22:09:09.437185 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035769 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 22:09:09.437185 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035774 2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 22:09:09.437185 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035777 2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 22:09:09.437185 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035781 2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 22:09:09.437185 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035784 2555 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 22:09:09.437185 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035788 2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 22:09:09.437185 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035790 2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 22:09:09.437185 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035793 2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 22:09:09.437185 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035796 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 22:09:09.437881 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035799 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 22:09:09.437881 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035802 2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 22:09:09.437881 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035804 2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 22:09:09.437881 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035807 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 22:09:09.437881 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035810 2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 22:09:09.437881 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035813 2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 22:09:09.437881 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035816 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 22:09:09.437881 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035818 2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 22:09:09.437881 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035821 2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 22:09:09.437881 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035824 2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 22:09:09.437881 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035826 2555 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 22:09:09.437881 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035829 2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 22:09:09.437881 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035831 2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 22:09:09.437881 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035834 2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 22:09:09.437881 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035836 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 22:09:09.437881 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035839 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 22:09:09.437881 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035842 2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 22:09:09.437881 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035844 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 22:09:09.437881 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035847 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 22:09:09.437881 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035849 2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 22:09:09.438695 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035852 2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 22:09:09.438695 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035855 2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 22:09:09.438695 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035859 2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 22:09:09.438695 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035861 2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 22:09:09.438695 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035865 2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 22:09:09.438695 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035867 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 22:09:09.438695 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035870 2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 22:09:09.438695 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035872 2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 22:09:09.438695 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035875 2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 22:09:09.438695 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035878 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 22:09:09.438695 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035880 2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 22:09:09.438695 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035883 2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 22:09:09.438695 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035885 2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 22:09:09.438695 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035888 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 22:09:09.438695 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035890 2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 22:09:09.438695 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035893 2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 22:09:09.438695 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035896 2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 22:09:09.438695 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.035899 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 22:09:09.439521 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.036681 2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 22:09:09.439521 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.045671 2555 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 06 22:09:09.439521 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.045691 2555 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 06 22:09:09.439521 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.045711 2555 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 06 22:09:09.439521 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.045717 2555 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 06 22:09:09.439521 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045781 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 22:09:09.439521 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045787 2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 22:09:09.439521 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045790 2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 22:09:09.439521 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045793 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 22:09:09.439521 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045796 2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 22:09:09.439521 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045799 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 22:09:09.439521 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045802 2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 22:09:09.439521 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045805 2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 22:09:09.439521 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045807 2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 22:09:09.439521 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045810 2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 22:09:09.440322 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045812 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 22:09:09.440322 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045815 2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 22:09:09.440322 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045818 2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 22:09:09.440322 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045821 2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 22:09:09.440322 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045823 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 22:09:09.440322 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045826 2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 22:09:09.440322 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045829 2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 22:09:09.440322 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045831 2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 22:09:09.440322 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045834 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 22:09:09.440322 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045837 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 22:09:09.440322 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045839 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 22:09:09.440322 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045842 2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 22:09:09.440322 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045844 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 22:09:09.440322 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045847 2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 22:09:09.440322 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045850 2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 22:09:09.440322 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045852 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 22:09:09.440322 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045857 2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 22:09:09.440322 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045861 2555 feature_gate.go:328] unrecognized feature gate: Example Jul 06 22:09:09.440322 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045864 2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 22:09:09.441112 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045870 2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 22:09:09.441112 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045873 2555 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 22:09:09.441112 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045876 2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 22:09:09.441112 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045879 2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 22:09:09.441112 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045881 2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 22:09:09.441112 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045884 2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 22:09:09.441112 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045887 2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 22:09:09.441112 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045889 2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 22:09:09.441112 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045892 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 22:09:09.441112 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045895 2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 22:09:09.441112 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045897 2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 22:09:09.441112 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045900 2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 22:09:09.441112 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045903 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 22:09:09.441112 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045905 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 22:09:09.441112 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045908 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 22:09:09.441112 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045913 2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 22:09:09.441112 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045917 2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 22:09:09.441112 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045920 2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 22:09:09.441112 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045924 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 22:09:09.441897 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045927 2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 22:09:09.441897 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045930 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 22:09:09.441897 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045933 2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 22:09:09.441897 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045936 2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 22:09:09.441897 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045939 2555 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 22:09:09.441897 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045942 2555 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 22:09:09.441897 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045945 2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 22:09:09.441897 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045947 2555 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 22:09:09.441897 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045950 2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 22:09:09.441897 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045953 2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 22:09:09.441897 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045956 2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 22:09:09.441897 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045959 2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 22:09:09.441897 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045962 2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 22:09:09.441897 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045967 2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 22:09:09.441897 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045970 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 22:09:09.441897 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045973 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 22:09:09.441897 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045976 2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 22:09:09.441897 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045979 2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 22:09:09.441897 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045981 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 22:09:09.441897 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045984 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 22:09:09.442699 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045987 2555 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 22:09:09.442699 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045989 2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 22:09:09.442699 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045992 2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 22:09:09.442699 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045994 2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 22:09:09.442699 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.045997 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 22:09:09.442699 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046000 2555 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 22:09:09.442699 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046021 2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 22:09:09.442699 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046025 2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 22:09:09.442699 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046030 2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 22:09:09.442699 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046032 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 22:09:09.442699 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046035 2555 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 22:09:09.442699 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046037 2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 22:09:09.442699 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046040 2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 22:09:09.442699 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046043 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 22:09:09.442699 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046045 2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 22:09:09.442699 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046048 2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 22:09:09.442699 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046050 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 22:09:09.442699 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046053 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 22:09:09.442699 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046055 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 22:09:09.442699 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046058 2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 22:09:09.443512 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046062 2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 22:09:09.443512 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046065 2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 22:09:09.443512 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046068 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 22:09:09.443512 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046071 2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 22:09:09.443512 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046074 2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 22:09:09.443512 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046078 2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 22:09:09.443512 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046080 2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 22:09:09.443512 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046084 2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 22:09:09.443512 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046086 2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 22:09:09.443512 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046089 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 22:09:09.443512 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046091 2555 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 22:09:09.443512 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.046104 2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 22:09:09.443512 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046272 2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 22:09:09.443512 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046278 2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 22:09:09.443512 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046282 2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 22:09:09.443921 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046285 2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 22:09:09.443921 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046288 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 22:09:09.443921 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046291 2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 22:09:09.443921 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046293 2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 22:09:09.443921 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046296 2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 22:09:09.443921 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046301 2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 22:09:09.443921 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046304 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 22:09:09.443921 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046306 2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 22:09:09.443921 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046309 2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 22:09:09.443921 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046312 2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 22:09:09.443921 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046314 2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 22:09:09.443921 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046317 2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 22:09:09.443921 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046320 2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 22:09:09.443921 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046322 2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 22:09:09.443921 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046325 2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 22:09:09.443921 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046327 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 22:09:09.443921 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046330 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 22:09:09.443921 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046332 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 22:09:09.443921 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046335 2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 22:09:09.444729 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046337 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 22:09:09.444729 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046340 2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 22:09:09.444729 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046343 2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 22:09:09.444729 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046345 2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 22:09:09.444729 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046349 2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 22:09:09.444729 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046352 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 22:09:09.444729 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046364 2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 22:09:09.444729 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046367 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 22:09:09.444729 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046370 2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 22:09:09.444729 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046372 2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 22:09:09.444729 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046375 2555 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 22:09:09.444729 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046378 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 22:09:09.444729 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046380 2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 22:09:09.444729 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046382 2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 22:09:09.444729 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046385 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 22:09:09.444729 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046388 2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 22:09:09.444729 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046390 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 22:09:09.444729 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046393 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 22:09:09.444729 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046397 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 22:09:09.445527 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046400 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 22:09:09.445527 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046403 2555 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 22:09:09.445527 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046406 2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 22:09:09.445527 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046408 2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 22:09:09.445527 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046411 2555 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 22:09:09.445527 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046414 2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 22:09:09.445527 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046416 2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 22:09:09.445527 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046419 2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 22:09:09.445527 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046421 2555 feature_gate.go:328] unrecognized feature gate: Example Jul 06 22:09:09.445527 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046425 2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 22:09:09.445527 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046428 2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 22:09:09.445527 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046430 2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 22:09:09.445527 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046432 2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 22:09:09.445527 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046435 2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 22:09:09.445527 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046437 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 22:09:09.445527 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046441 2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 22:09:09.445527 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046444 2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 22:09:09.445527 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046450 2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 22:09:09.445527 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046453 2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 22:09:09.445527 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046456 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 22:09:09.446340 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046464 2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 22:09:09.446340 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046467 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 22:09:09.446340 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046469 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 22:09:09.446340 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046472 2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 22:09:09.446340 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046474 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 22:09:09.446340 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046477 2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 22:09:09.446340 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046480 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 22:09:09.446340 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046483 2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 22:09:09.446340 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046485 2555 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 22:09:09.446340 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046488 2555 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 22:09:09.446340 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046491 2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 22:09:09.446340 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046495 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 22:09:09.446340 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046497 2555 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 22:09:09.446340 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046500 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 22:09:09.446340 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046503 2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 22:09:09.446340 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046507 2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 22:09:09.446340 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046510 2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 22:09:09.446340 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046512 2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 22:09:09.446340 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046515 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 22:09:09.446340 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046518 2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 22:09:09.447151 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046521 2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 22:09:09.447151 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046523 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 22:09:09.447151 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046526 2555 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 22:09:09.447151 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046529 2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 22:09:09.447151 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046531 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 22:09:09.447151 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046534 2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 22:09:09.447151 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046536 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 22:09:09.447151 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046539 2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 22:09:09.447151 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046541 2555 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 22:09:09.447151 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046545 2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 22:09:09.447151 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046548 2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 22:09:09.447151 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046550 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 22:09:09.447151 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046553 2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 22:09:09.447151 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046561 2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 22:09:09.447151 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046564 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 22:09:09.447151 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046566 2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 22:09:09.447151 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046569 2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 22:09:09.447151 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:09.046571 2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 22:09:09.447954 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.046576 2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 22:09:09.447954 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.046723 2555 server.go:962] "Client rotation is on, will bootstrap in background" Jul 06 22:09:09.447954 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.048960 2555 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 06 22:09:09.447954 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.050964 2555 server.go:1019] "Starting client certificate rotation" Jul 06 22:09:09.447954 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.051086 2555 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 22:09:09.447954 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.051129 2555 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 22:09:09.447954 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.054485 2555 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 22:09:09.447954 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.063845 2555 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 22:09:09.447954 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.075054 2555 log.go:25] "Validated CRI v1 runtime API" Jul 06 22:09:09.447954 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.078552 2555 log.go:25] "Validated CRI v1 image API" Jul 06 22:09:09.447954 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.080308 2555 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 06 22:09:09.447954 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.082693 2555 fs.go:135] Filesystem UUIDs: map[258c2603-0b07-403e-b347-9ac0fc949d3f:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 ea443d61-bb4c-405d-8b1d-e4882f1ff0d3:/dev/nvme0n1p4] Jul 06 22:09:09.448357 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.082709 2555 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 06 22:09:09.448357 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.084889 2555 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 22:09:09.454835 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.087120 2555 manager.go:216] Machine: {Timestamp:2026-07-06 22:09:09.086625075 +0000 UTC m=+0.386336916 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098225 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec273da3ac2344132448883be2ecca44 SystemUUID:ec273da3-ac23-4413-2448-883be2ecca44 BootID:dd6c6624-f2f6-4b49-b492-cdcf5ef7e6ac Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:61:5c:0d:6a:cb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:61:5c:0d:6a:cb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ca:d0:c1:eb:4b:97 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 06 22:09:09.454835 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.087222 2555 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 06 22:09:09.454835 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.087308 2555 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 06 22:09:09.454835 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.090466 2555 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 06 22:09:09.454835 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.090526 2555 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-71.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 06 22:09:09.454835 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.091583 2555 topology_manager.go:138] "Creating topology manager with none policy" Jul 06 22:09:09.454835 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.091598 2555 container_manager_linux.go:309] "Creating device plugin manager" Jul 06 22:09:09.454835 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.091614 2555 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 22:09:09.454835 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.092423 2555 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 22:09:09.454835 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.093690 2555 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 06 22:09:09.454835 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.097117 2555 state_mem.go:36] "Initialized new in-memory state store" Jul 06 22:09:09.454835 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.097284 2555 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 06 22:09:09.454835 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.097354 2555 kubelet.go:486] "Attempting to sync node with API server" Jul 06 22:09:09.454835 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.097370 2555 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 06 22:09:09.454835 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.097388 2555 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 06 22:09:09.454835 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.097399 2555 kubelet.go:398] "Adding apiserver pod source" Jul 06 22:09:09.454835 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.097413 2555 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 06 22:09:09.454835 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.097921 2555 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 22:09:09.455654 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.097946 2555 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 22:09:09.455654 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.100498 2555 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 06 22:09:09.455654 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.103874 2555 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 06 22:09:09.455654 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.103909 2555 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 06 22:09:09.455654 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.105474 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 06 22:09:09.455654 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.105504 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 06 22:09:09.455654 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.105515 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 06 22:09:09.455654 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.105527 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 06 22:09:09.455654 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.105545 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 06 22:09:09.455654 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.105554 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 06 22:09:09.455654 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.105566 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 06 22:09:09.455654 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.105575 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 06 22:09:09.455654 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.105590 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 06 22:09:09.455654 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.105602 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 06 22:09:09.455654 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.105603 2555 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mmbv6" Jul 06 22:09:09.455654 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.105617 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 06 22:09:09.455654 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.105648 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 06 22:09:09.455654 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.106465 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 06 22:09:09.455654 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.106479 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 06 22:09:09.455654 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:09.107441 2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 06 22:09:09.455654 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:09.107703 2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-71.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 06 22:09:09.455654 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.110662 2555 server.go:1268] "Started kubelet" Jul 06 22:09:09.455654 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.110751 2555 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 06 22:09:09.455654 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.110762 2555 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 06 22:09:09.455654 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.113621 2555 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 06 22:09:09.455654 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.116346 2555 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 06 22:09:09.455654 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.116528 2555 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-71.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 06 22:09:09.455654 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.116682 2555 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mmbv6" Jul 06 22:09:09.455654 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.117367 2555 server.go:310] "Adding debug handlers to kubelet server" Jul 06 22:09:09.455654 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:09.118914 2555 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 06 22:09:09.455654 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:09.121767 2555 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-71.ec2.internal.18bfd2cb8ac8f355 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-71.ec2.internal,UID:ip-10-0-128-71.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-71.ec2.internal,},FirstTimestamp:2026-07-06 22:09:09.110625109 +0000 UTC m=+0.410336949,LastTimestamp:2026-07-06 22:09:09.110625109 +0000 UTC m=+0.410336949,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-71.ec2.internal,}" Jul 06 22:09:09.455654 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.128238 2555 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 06 22:09:09.455654 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.128254 2555 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 06 22:09:09.455654 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:09.128712 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-71.ec2.internal\" not found" Jul 06 22:09:09.455654 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.129051 2555 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 06 22:09:09.455654 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.129132 2555 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 06 22:09:09.455654 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.129150 2555 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 06 22:09:09.455654 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.129307 2555 reconstruct.go:97] "Volume reconstruction finished" Jul 06 22:09:09.455654 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.129317 2555 reconciler.go:29] "Reconciler: start to sync state" Jul 06 22:09:09.455654 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.129943 2555 factory.go:55] Registering systemd factory Jul 06 22:09:09.564315 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.129963 2555 factory.go:223] Registration of the systemd container factory successfully Jul 06 22:09:09.564315 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.130309 2555 factory.go:153] Registering CRI-O factory Jul 06 22:09:09.564315 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.130326 2555 factory.go:223] Registration of the crio container factory successfully Jul 06 22:09:09.564315 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.130385 2555 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 06 22:09:09.564315 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.130415 2555 factory.go:103] Registering Raw factory Jul 06 22:09:09.564315 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.130431 2555 manager.go:1196] Started watching for new ooms in manager Jul 06 22:09:09.564315 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.131051 2555 manager.go:349] Starting recovery of all containers Jul 06 22:09:09.564315 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.141160 2555 manager.go:354] Recovery completed Jul 06 22:09:09.564315 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.141432 2555 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 22:09:09.564315 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.146788 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 22:09:09.564315 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.149161 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-71.ec2.internal" event="NodeHasSufficientMemory" Jul 06 22:09:09.564315 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.149200 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-71.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 22:09:09.564315 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.149212 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-71.ec2.internal" event="NodeHasSufficientPID" Jul 06 22:09:09.564315 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.149743 2555 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 06 22:09:09.564315 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.149755 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 22:09:09.564315 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.149756 2555 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 06 22:09:09.564315 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.149782 2555 state_mem.go:36] "Initialized new in-memory state store" Jul 06 22:09:09.564315 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.150548 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-71.ec2.internal" event="NodeHasSufficientMemory" Jul 06 22:09:09.564315 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.150578 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-71.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 22:09:09.564315 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.150590 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-71.ec2.internal" event="NodeHasSufficientPID" Jul 06 22:09:09.564315 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.151302 2555 policy_none.go:49] "None policy: Start" Jul 06 22:09:09.564315 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.151317 2555 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 06 22:09:09.564315 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.151328 2555 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 06 22:09:09.564315 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.152570 2555 policy_none.go:47] "Start" Jul 06 22:09:09.564315 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:09.156333 2555 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-128-71.ec2.internal\" not found" node="ip-10-0-128-71.ec2.internal" Jul 06 22:09:09.564315 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.185833 2555 manager.go:337] "Starting Device Plugin manager" Jul 06 22:09:09.564315 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:09.185879 2555 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 06 22:09:09.564315 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.185892 2555 server.go:87] "Starting device plugin registration server" Jul 06 22:09:09.564315 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.186255 2555 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 06 22:09:09.564315 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.186269 2555 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 06 22:09:09.564315 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.186399 2555 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 06 22:09:09.564315 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.186485 2555 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 06 22:09:09.564315 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.186494 2555 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 06 22:09:09.564315 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:09.187046 2555 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 06 22:09:09.564315 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:09.187106 2555 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-71.ec2.internal\" not found" Jul 06 22:09:09.564315 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.286907 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 22:09:09.564315 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.288586 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-71.ec2.internal" event="NodeHasSufficientMemory" Jul 06 22:09:09.564315 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.288616 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-71.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 22:09:09.564315 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.288628 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-71.ec2.internal" event="NodeHasSufficientPID" Jul 06 22:09:09.564315 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.288659 2555 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-71.ec2.internal" Jul 06 22:09:09.564315 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:09.296442 2555 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-71.ec2.internal" Jul 06 22:09:09.564315 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:09.296472 2555 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-71.ec2.internal\": node \"ip-10-0-128-71.ec2.internal\" not found" Jul 06 22:09:09.564315 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:09.314637 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-71.ec2.internal\" not found" Jul 06 22:09:09.564315 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:09.415123 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-71.ec2.internal\" not found" Jul 06 22:09:09.564315 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:09.515557 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-71.ec2.internal\" not found" Jul 06 22:09:09.696978 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:09.616262 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-71.ec2.internal\" not found" Jul 06 22:09:09.827306 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:09.721249 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-71.ec2.internal\" not found" Jul 06 22:09:09.827306 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:09.826393 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-71.ec2.internal\" not found" Jul 06 22:09:09.961807 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:09.926659 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-71.ec2.internal\" not found" Jul 06 22:09:10.092669 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:10.027205 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-71.ec2.internal\" not found" Jul 06 22:09:10.092669 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:10.050394 2555 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 06 22:09:10.092669 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:10.050560 2555 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 22:09:10.231308 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:10.125123 2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-05 22:04:09 +0000 UTC" deadline="2028-01-03 16:13:34.673832863 +0000 UTC" Jul 06 22:09:10.231308 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:10.125157 2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13098h4m24.548678931s" Jul 06 22:09:10.231308 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:10.128380 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-71.ec2.internal\" not found" Jul 06 22:09:10.231308 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:10.189007 2555 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 22:09:10.231308 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:10.228762 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-71.ec2.internal\" not found" Jul 06 22:09:10.356740 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:10.332563 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-71.ec2.internal\" not found" Jul 06 22:09:10.488469 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:10.433379 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-71.ec2.internal\" not found" Jul 06 22:09:10.488469 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:10.447621 2555 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 22:09:11.135012 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.099780 2555 apiserver.go:52] "Watching apiserver" Jul 06 22:09:11.135012 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.111096 2555 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 06 22:09:11.135012 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.128904 2555 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 06 22:09:11.142131 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.140728 2555 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 22:09:11.168114 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.168071 2555 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9qtcw" Jul 06 22:09:11.176857 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.176839 2555 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9qtcw" Jul 06 22:09:11.262621 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.262430 2555 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 22:09:11.331832 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.331739 2555 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 06 22:09:11.333065 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.333049 2555 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 06 22:09:11.333158 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.333102 2555 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 06 22:09:11.333202 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.333168 2555 kubelet.go:2448] "Starting kubelet main sync loop" Jul 06 22:09:11.333250 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:11.333215 2555 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 06 22:09:11.336618 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.336594 2555 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 22:09:11.433631 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.433590 2555 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-71.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-71.ec2.internal"] Jul 06 22:09:11.435486 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.435467 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-71.ec2.internal" Jul 06 22:09:11.436574 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.436533 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-zt94q","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhckn","openshift-image-registry/node-ca-9tlpj","openshift-network-operator/iptables-alerter-h7bb2","kube-system/konnectivity-agent-grmjd","openshift-cluster-node-tuning-operator/tuned-hqgv6","openshift-dns/node-resolver-dfd8v","openshift-multus/multus-additional-cni-plugins-t5rgx","openshift-multus/multus-kwwj2","openshift-multus/network-metrics-daemon-shdh8","openshift-network-diagnostics/network-check-target-vdbr6","openshift-ovn-kubernetes/ovnkube-node-p7mq8"] Jul 06 22:09:11.436710 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.436672 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-71.ec2.internal" Jul 06 22:09:11.438009 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.437989 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zt94q" Jul 06 22:09:11.438117 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:11.438085 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zt94q" podUID="d681bb3e-9bb1-4705-98cb-fd2cd5dc7063" Jul 06 22:09:11.439494 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.439478 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhckn" Jul 06 22:09:11.441189 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.441167 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9tlpj" Jul 06 22:09:11.442843 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.442195 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 06 22:09:11.442843 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.442543 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-h7bb2" Jul 06 22:09:11.442843 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.442604 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 06 22:09:11.443019 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.442855 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 06 22:09:11.443019 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.442862 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-t8zkp\"" Jul 06 22:09:11.445023 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.443704 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 06 22:09:11.445023 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.444042 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 06 22:09:11.445023 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.444176 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lzpnm\"" Jul 06 22:09:11.445023 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.444384 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 06 22:09:11.445023 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.444519 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-grmjd" Jul 06 22:09:11.445565 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.445547 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 06 22:09:11.445632 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.445589 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-d2xcg\"" Jul 06 22:09:11.445692 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.445674 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 06 22:09:11.446443 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.445865 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 06 22:09:11.446443 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.446113 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hqgv6" Jul 06 22:09:11.447476 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.447457 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 06 22:09:11.447566 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.447495 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 06 22:09:11.447620 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.447569 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-b9xc7\"" Jul 06 22:09:11.447935 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.447919 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dfd8v" Jul 06 22:09:11.448615 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.448600 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 06 22:09:11.448682 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.448618 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 06 22:09:11.448682 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.448620 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-wd69f\"" Jul 06 22:09:11.449507 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.449494 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-t5rgx" Jul 06 22:09:11.450481 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.450467 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ncx2v\"" Jul 06 22:09:11.450706 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.450693 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 06 22:09:11.450749 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.450734 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 06 22:09:11.451068 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.451053 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kwwj2" Jul 06 22:09:11.451974 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.451955 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 06 22:09:11.452037 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.452022 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 06 22:09:11.452076 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.452053 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 06 22:09:11.452076 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.452066 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 06 22:09:11.452187 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.452167 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-d76rs\"" Jul 06 22:09:11.452224 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.452191 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 06 22:09:11.452707 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.452691 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-shdh8" Jul 06 22:09:11.452799 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:11.452776 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-shdh8" podUID="838f4166-d0dc-478f-aa98-4e1fd7ad300a" Jul 06 22:09:11.453521 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.453505 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 06 22:09:11.453592 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.453579 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-qgn5c\"" Jul 06 22:09:11.454351 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.454333 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdbr6" Jul 06 22:09:11.454427 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:11.454401 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdbr6" podUID="852d3c7b-6b13-4603-9a80-7b07a035d22f" Jul 06 22:09:11.455829 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.455810 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:09:11.458629 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.458603 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 06 22:09:11.458629 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.458614 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 06 22:09:11.458859 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.458723 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 06 22:09:11.458859 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.458731 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 06 22:09:11.458859 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.458757 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 06 22:09:11.458985 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.458959 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xpkj6\"" Jul 06 22:09:11.459344 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.459329 2555 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-71.ec2.internal" Jul 06 22:09:11.459940 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.459926 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 06 22:09:11.472073 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.472051 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-71.ec2.internal"] Jul 06 22:09:11.472182 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.472060 2555 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 22:09:11.474887 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.474869 2555 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-71.ec2.internal" Jul 06 22:09:11.487933 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.487910 2555 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 22:09:11.488060 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.487973 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-128-71.ec2.internal"] Jul 06 22:09:11.532778 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.530631 2555 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 06 22:09:11.539501 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.539477 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/327650c8-6a1d-4167-9a90-f9b24f67ffe4-hostroot\") pod \"multus-kwwj2\" (UID: \"327650c8-6a1d-4167-9a90-f9b24f67ffe4\") " pod="openshift-multus/multus-kwwj2" Jul 06 22:09:11.539624 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.539506 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e50d3b5f-832e-481f-8807-7774ff8d5521-host-cni-netd\") pod \"ovnkube-node-p7mq8\" (UID: \"e50d3b5f-832e-481f-8807-7774ff8d5521\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:09:11.539624 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.539524 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/245656d5-d53d-4319-b3d9-364a463bfeda-sys-fs\") pod \"aws-ebs-csi-driver-node-vhckn\" (UID: \"245656d5-d53d-4319-b3d9-364a463bfeda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhckn" Jul 06 22:09:11.539624 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.539550 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f7d6b6f8-91a4-4341-b018-4e768abdd9a0-agent-certs\") pod \"konnectivity-agent-grmjd\" (UID: \"f7d6b6f8-91a4-4341-b018-4e768abdd9a0\") " pod="kube-system/konnectivity-agent-grmjd" Jul 06 22:09:11.539624 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.539590 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8frlx\" (UniqueName: \"kubernetes.io/projected/327650c8-6a1d-4167-9a90-f9b24f67ffe4-kube-api-access-8frlx\") pod \"multus-kwwj2\" (UID: \"327650c8-6a1d-4167-9a90-f9b24f67ffe4\") " pod="openshift-multus/multus-kwwj2" Jul 06 22:09:11.539811 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.539635 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e50d3b5f-832e-481f-8807-7774ff8d5521-host-kubelet\") pod \"ovnkube-node-p7mq8\" (UID: \"e50d3b5f-832e-481f-8807-7774ff8d5521\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:09:11.539811 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.539655 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dbf68764-0a98-4778-9b27-8e03df2a1c1b-etc-tuned\") pod \"tuned-hqgv6\" (UID: \"dbf68764-0a98-4778-9b27-8e03df2a1c1b\") " pod="openshift-cluster-node-tuning-operator/tuned-hqgv6" Jul 06 22:09:11.539811 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.539672 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/327650c8-6a1d-4167-9a90-f9b24f67ffe4-multus-cni-dir\") pod \"multus-kwwj2\" (UID: \"327650c8-6a1d-4167-9a90-f9b24f67ffe4\") " pod="openshift-multus/multus-kwwj2" Jul 06 22:09:11.539811 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.539686 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e50d3b5f-832e-481f-8807-7774ff8d5521-host-run-ovn-kubernetes\") pod \"ovnkube-node-p7mq8\" (UID: \"e50d3b5f-832e-481f-8807-7774ff8d5521\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:09:11.539811 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.539700 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dbf68764-0a98-4778-9b27-8e03df2a1c1b-tmp\") pod \"tuned-hqgv6\" (UID: \"dbf68764-0a98-4778-9b27-8e03df2a1c1b\") " pod="openshift-cluster-node-tuning-operator/tuned-hqgv6" Jul 06 22:09:11.539811 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.539740 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d681bb3e-9bb1-4705-98cb-fd2cd5dc7063-original-pull-secret\") pod \"global-pull-secret-syncer-zt94q\" (UID: \"d681bb3e-9bb1-4705-98cb-fd2cd5dc7063\") " pod="kube-system/global-pull-secret-syncer-zt94q" Jul 06 22:09:11.539811 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.539765 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7c3b8e45-4537-4010-97c9-6081a2e3edcf-serviceca\") pod \"node-ca-9tlpj\" (UID: \"7c3b8e45-4537-4010-97c9-6081a2e3edcf\") " pod="openshift-image-registry/node-ca-9tlpj" Jul 06 22:09:11.539811 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.539781 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dbf68764-0a98-4778-9b27-8e03df2a1c1b-run\") pod \"tuned-hqgv6\" (UID: \"dbf68764-0a98-4778-9b27-8e03df2a1c1b\") " pod="openshift-cluster-node-tuning-operator/tuned-hqgv6" Jul 06 22:09:11.539811 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.539798 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f7d6b6f8-91a4-4341-b018-4e768abdd9a0-konnectivity-ca\") pod \"konnectivity-agent-grmjd\" (UID: \"f7d6b6f8-91a4-4341-b018-4e768abdd9a0\") " pod="kube-system/konnectivity-agent-grmjd" Jul 06 22:09:11.539811 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.539814 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/327650c8-6a1d-4167-9a90-f9b24f67ffe4-system-cni-dir\") pod \"multus-kwwj2\" (UID: \"327650c8-6a1d-4167-9a90-f9b24f67ffe4\") " pod="openshift-multus/multus-kwwj2" Jul 06 22:09:11.540178 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.539830 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/327650c8-6a1d-4167-9a90-f9b24f67ffe4-host-var-lib-cni-multus\") pod \"multus-kwwj2\" (UID: \"327650c8-6a1d-4167-9a90-f9b24f67ffe4\") " pod="openshift-multus/multus-kwwj2" Jul 06 22:09:11.540178 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.539845 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/327650c8-6a1d-4167-9a90-f9b24f67ffe4-etc-kubernetes\") pod \"multus-kwwj2\" (UID: \"327650c8-6a1d-4167-9a90-f9b24f67ffe4\") " pod="openshift-multus/multus-kwwj2" Jul 06 22:09:11.540178 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.539859 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b05f823fcae217aa6cef90a2100bffba-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-71.ec2.internal\" (UID: \"b05f823fcae217aa6cef90a2100bffba\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-71.ec2.internal" Jul 06 22:09:11.540178 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.539880 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7c3b8e45-4537-4010-97c9-6081a2e3edcf-host\") pod \"node-ca-9tlpj\" (UID: \"7c3b8e45-4537-4010-97c9-6081a2e3edcf\") " pod="openshift-image-registry/node-ca-9tlpj" Jul 06 22:09:11.540178 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.539920 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dbf68764-0a98-4778-9b27-8e03df2a1c1b-etc-sysconfig\") pod \"tuned-hqgv6\" (UID: \"dbf68764-0a98-4778-9b27-8e03df2a1c1b\") " pod="openshift-cluster-node-tuning-operator/tuned-hqgv6" Jul 06 22:09:11.540178 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.539953 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dbf68764-0a98-4778-9b27-8e03df2a1c1b-host\") pod \"tuned-hqgv6\" (UID: \"dbf68764-0a98-4778-9b27-8e03df2a1c1b\") " pod="openshift-cluster-node-tuning-operator/tuned-hqgv6" Jul 06 22:09:11.540178 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.539985 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ff073db8-8f49-4832-8b5e-928a864bd41d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t5rgx\" (UID: \"ff073db8-8f49-4832-8b5e-928a864bd41d\") " pod="openshift-multus/multus-additional-cni-plugins-t5rgx" Jul 06 22:09:11.540178 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.540016 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/838f4166-d0dc-478f-aa98-4e1fd7ad300a-metrics-certs\") pod \"network-metrics-daemon-shdh8\" (UID: \"838f4166-d0dc-478f-aa98-4e1fd7ad300a\") " pod="openshift-multus/network-metrics-daemon-shdh8" Jul 06 22:09:11.540178 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.540048 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e50d3b5f-832e-481f-8807-7774ff8d5521-env-overrides\") pod \"ovnkube-node-p7mq8\" (UID: \"e50d3b5f-832e-481f-8807-7774ff8d5521\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:09:11.540178 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.540064 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ff073db8-8f49-4832-8b5e-928a864bd41d-cni-binary-copy\") pod \"multus-additional-cni-plugins-t5rgx\" (UID: \"ff073db8-8f49-4832-8b5e-928a864bd41d\") " pod="openshift-multus/multus-additional-cni-plugins-t5rgx" Jul 06 22:09:11.540178 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.540079 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhq7d\" (UniqueName: \"kubernetes.io/projected/ff073db8-8f49-4832-8b5e-928a864bd41d-kube-api-access-bhq7d\") pod \"multus-additional-cni-plugins-t5rgx\" (UID: \"ff073db8-8f49-4832-8b5e-928a864bd41d\") " pod="openshift-multus/multus-additional-cni-plugins-t5rgx" Jul 06 22:09:11.540178 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.540101 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/327650c8-6a1d-4167-9a90-f9b24f67ffe4-cnibin\") pod \"multus-kwwj2\" (UID: \"327650c8-6a1d-4167-9a90-f9b24f67ffe4\") " pod="openshift-multus/multus-kwwj2" Jul 06 22:09:11.540178 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.540117 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e50d3b5f-832e-481f-8807-7774ff8d5521-ovnkube-script-lib\") pod \"ovnkube-node-p7mq8\" (UID: \"e50d3b5f-832e-481f-8807-7774ff8d5521\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:09:11.540178 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.540133 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d681bb3e-9bb1-4705-98cb-fd2cd5dc7063-kubelet-config\") pod \"global-pull-secret-syncer-zt94q\" (UID: \"d681bb3e-9bb1-4705-98cb-fd2cd5dc7063\") " pod="kube-system/global-pull-secret-syncer-zt94q" Jul 06 22:09:11.540178 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.540152 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e50d3b5f-832e-481f-8807-7774ff8d5521-host-slash\") pod \"ovnkube-node-p7mq8\" (UID: \"e50d3b5f-832e-481f-8807-7774ff8d5521\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:09:11.540178 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.540170 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dbf68764-0a98-4778-9b27-8e03df2a1c1b-etc-sysctl-conf\") pod \"tuned-hqgv6\" (UID: \"dbf68764-0a98-4778-9b27-8e03df2a1c1b\") " pod="openshift-cluster-node-tuning-operator/tuned-hqgv6" Jul 06 22:09:11.540822 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.540185 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ff073db8-8f49-4832-8b5e-928a864bd41d-system-cni-dir\") pod \"multus-additional-cni-plugins-t5rgx\" (UID: \"ff073db8-8f49-4832-8b5e-928a864bd41d\") " pod="openshift-multus/multus-additional-cni-plugins-t5rgx" Jul 06 22:09:11.540822 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.540199 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ff073db8-8f49-4832-8b5e-928a864bd41d-cnibin\") pod \"multus-additional-cni-plugins-t5rgx\" (UID: \"ff073db8-8f49-4832-8b5e-928a864bd41d\") " pod="openshift-multus/multus-additional-cni-plugins-t5rgx" Jul 06 22:09:11.540822 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.540217 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ff073db8-8f49-4832-8b5e-928a864bd41d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t5rgx\" (UID: \"ff073db8-8f49-4832-8b5e-928a864bd41d\") " pod="openshift-multus/multus-additional-cni-plugins-t5rgx" Jul 06 22:09:11.540822 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.540242 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/245656d5-d53d-4319-b3d9-364a463bfeda-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vhckn\" (UID: \"245656d5-d53d-4319-b3d9-364a463bfeda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhckn" Jul 06 22:09:11.540822 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.540259 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/245656d5-d53d-4319-b3d9-364a463bfeda-device-dir\") pod \"aws-ebs-csi-driver-node-vhckn\" (UID: \"245656d5-d53d-4319-b3d9-364a463bfeda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhckn" Jul 06 22:09:11.540822 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.540281 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/327650c8-6a1d-4167-9a90-f9b24f67ffe4-os-release\") pod \"multus-kwwj2\" (UID: \"327650c8-6a1d-4167-9a90-f9b24f67ffe4\") " pod="openshift-multus/multus-kwwj2" Jul 06 22:09:11.540822 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.540296 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/327650c8-6a1d-4167-9a90-f9b24f67ffe4-cni-binary-copy\") pod \"multus-kwwj2\" (UID: \"327650c8-6a1d-4167-9a90-f9b24f67ffe4\") " pod="openshift-multus/multus-kwwj2" Jul 06 22:09:11.540822 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.540319 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5147af52-3533-40a1-9797-566a9f3b2c86-iptables-alerter-script\") pod \"iptables-alerter-h7bb2\" (UID: \"5147af52-3533-40a1-9797-566a9f3b2c86\") " pod="openshift-network-operator/iptables-alerter-h7bb2" Jul 06 22:09:11.540822 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.540369 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dbf68764-0a98-4778-9b27-8e03df2a1c1b-var-lib-kubelet\") pod \"tuned-hqgv6\" (UID: \"dbf68764-0a98-4778-9b27-8e03df2a1c1b\") " pod="openshift-cluster-node-tuning-operator/tuned-hqgv6" Jul 06 22:09:11.540822 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.540408 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/327650c8-6a1d-4167-9a90-f9b24f67ffe4-multus-daemon-config\") pod \"multus-kwwj2\" (UID: \"327650c8-6a1d-4167-9a90-f9b24f67ffe4\") " pod="openshift-multus/multus-kwwj2" Jul 06 22:09:11.540822 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.540435 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/327650c8-6a1d-4167-9a90-f9b24f67ffe4-host-run-multus-certs\") pod \"multus-kwwj2\" (UID: \"327650c8-6a1d-4167-9a90-f9b24f67ffe4\") " pod="openshift-multus/multus-kwwj2" Jul 06 22:09:11.540822 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.540469 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e50d3b5f-832e-481f-8807-7774ff8d5521-host-run-netns\") pod \"ovnkube-node-p7mq8\" (UID: \"e50d3b5f-832e-481f-8807-7774ff8d5521\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:09:11.540822 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.540494 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e50d3b5f-832e-481f-8807-7774ff8d5521-run-ovn\") pod \"ovnkube-node-p7mq8\" (UID: \"e50d3b5f-832e-481f-8807-7774ff8d5521\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:09:11.540822 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.540520 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e50d3b5f-832e-481f-8807-7774ff8d5521-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-p7mq8\" (UID: \"e50d3b5f-832e-481f-8807-7774ff8d5521\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:09:11.540822 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.540545 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj994\" (UniqueName: \"kubernetes.io/projected/7c3b8e45-4537-4010-97c9-6081a2e3edcf-kube-api-access-wj994\") pod \"node-ca-9tlpj\" (UID: \"7c3b8e45-4537-4010-97c9-6081a2e3edcf\") " pod="openshift-image-registry/node-ca-9tlpj" Jul 06 22:09:11.540822 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.540569 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dbf68764-0a98-4778-9b27-8e03df2a1c1b-sys\") pod \"tuned-hqgv6\" (UID: \"dbf68764-0a98-4778-9b27-8e03df2a1c1b\") " pod="openshift-cluster-node-tuning-operator/tuned-hqgv6" Jul 06 22:09:11.541393 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.540588 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hccq5\" (UniqueName: \"kubernetes.io/projected/7139257f-7bcb-4486-8818-83a0c465eea4-kube-api-access-hccq5\") pod \"node-resolver-dfd8v\" (UID: \"7139257f-7bcb-4486-8818-83a0c465eea4\") " pod="openshift-dns/node-resolver-dfd8v" Jul 06 22:09:11.541393 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.540605 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/327650c8-6a1d-4167-9a90-f9b24f67ffe4-host-run-netns\") pod \"multus-kwwj2\" (UID: \"327650c8-6a1d-4167-9a90-f9b24f67ffe4\") " pod="openshift-multus/multus-kwwj2" Jul 06 22:09:11.541393 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.540623 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e50d3b5f-832e-481f-8807-7774ff8d5521-systemd-units\") pod \"ovnkube-node-p7mq8\" (UID: \"e50d3b5f-832e-481f-8807-7774ff8d5521\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:09:11.541393 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.540638 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e50d3b5f-832e-481f-8807-7774ff8d5521-host-cni-bin\") pod \"ovnkube-node-p7mq8\" (UID: \"e50d3b5f-832e-481f-8807-7774ff8d5521\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:09:11.541393 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.540652 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dbf68764-0a98-4778-9b27-8e03df2a1c1b-etc-kubernetes\") pod \"tuned-hqgv6\" (UID: \"dbf68764-0a98-4778-9b27-8e03df2a1c1b\") " pod="openshift-cluster-node-tuning-operator/tuned-hqgv6" Jul 06 22:09:11.541393 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.540666 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5147af52-3533-40a1-9797-566a9f3b2c86-host-slash\") pod \"iptables-alerter-h7bb2\" (UID: \"5147af52-3533-40a1-9797-566a9f3b2c86\") " pod="openshift-network-operator/iptables-alerter-h7bb2" Jul 06 22:09:11.541393 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.540681 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dbf68764-0a98-4778-9b27-8e03df2a1c1b-etc-modprobe-d\") pod \"tuned-hqgv6\" (UID: \"dbf68764-0a98-4778-9b27-8e03df2a1c1b\") " pod="openshift-cluster-node-tuning-operator/tuned-hqgv6" Jul 06 22:09:11.541393 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.540697 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dbf68764-0a98-4778-9b27-8e03df2a1c1b-lib-modules\") pod \"tuned-hqgv6\" (UID: \"dbf68764-0a98-4778-9b27-8e03df2a1c1b\") " pod="openshift-cluster-node-tuning-operator/tuned-hqgv6" Jul 06 22:09:11.541393 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.540711 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ff073db8-8f49-4832-8b5e-928a864bd41d-os-release\") pod \"multus-additional-cni-plugins-t5rgx\" (UID: \"ff073db8-8f49-4832-8b5e-928a864bd41d\") " pod="openshift-multus/multus-additional-cni-plugins-t5rgx" Jul 06 22:09:11.541393 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.540726 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ff073db8-8f49-4832-8b5e-928a864bd41d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t5rgx\" (UID: \"ff073db8-8f49-4832-8b5e-928a864bd41d\") " pod="openshift-multus/multus-additional-cni-plugins-t5rgx" Jul 06 22:09:11.541393 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.540743 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtrhv\" (UniqueName: \"kubernetes.io/projected/245656d5-d53d-4319-b3d9-364a463bfeda-kube-api-access-jtrhv\") pod \"aws-ebs-csi-driver-node-vhckn\" (UID: \"245656d5-d53d-4319-b3d9-364a463bfeda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhckn" Jul 06 22:09:11.541393 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.540758 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/327650c8-6a1d-4167-9a90-f9b24f67ffe4-host-var-lib-cni-bin\") pod \"multus-kwwj2\" (UID: \"327650c8-6a1d-4167-9a90-f9b24f67ffe4\") " pod="openshift-multus/multus-kwwj2" Jul 06 22:09:11.541393 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.540773 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dbf68764-0a98-4778-9b27-8e03df2a1c1b-etc-sysctl-d\") pod \"tuned-hqgv6\" (UID: \"dbf68764-0a98-4778-9b27-8e03df2a1c1b\") " pod="openshift-cluster-node-tuning-operator/tuned-hqgv6" Jul 06 22:09:11.541393 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.540797 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7q86\" (UniqueName: \"kubernetes.io/projected/dbf68764-0a98-4778-9b27-8e03df2a1c1b-kube-api-access-p7q86\") pod \"tuned-hqgv6\" (UID: \"dbf68764-0a98-4778-9b27-8e03df2a1c1b\") " pod="openshift-cluster-node-tuning-operator/tuned-hqgv6" Jul 06 22:09:11.541393 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.540823 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/245656d5-d53d-4319-b3d9-364a463bfeda-registration-dir\") pod \"aws-ebs-csi-driver-node-vhckn\" (UID: \"245656d5-d53d-4319-b3d9-364a463bfeda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhckn" Jul 06 22:09:11.541393 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.540838 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/327650c8-6a1d-4167-9a90-f9b24f67ffe4-host-var-lib-kubelet\") pod \"multus-kwwj2\" (UID: \"327650c8-6a1d-4167-9a90-f9b24f67ffe4\") " pod="openshift-multus/multus-kwwj2" Jul 06 22:09:11.541904 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.540866 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/327650c8-6a1d-4167-9a90-f9b24f67ffe4-multus-conf-dir\") pod \"multus-kwwj2\" (UID: \"327650c8-6a1d-4167-9a90-f9b24f67ffe4\") " pod="openshift-multus/multus-kwwj2" Jul 06 22:09:11.541904 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.540890 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4z7z\" (UniqueName: \"kubernetes.io/projected/852d3c7b-6b13-4603-9a80-7b07a035d22f-kube-api-access-z4z7z\") pod \"network-check-target-vdbr6\" (UID: \"852d3c7b-6b13-4603-9a80-7b07a035d22f\") " pod="openshift-network-diagnostics/network-check-target-vdbr6" Jul 06 22:09:11.541904 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.540917 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e50d3b5f-832e-481f-8807-7774ff8d5521-run-systemd\") pod \"ovnkube-node-p7mq8\" (UID: \"e50d3b5f-832e-481f-8807-7774ff8d5521\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:09:11.541904 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.540933 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e50d3b5f-832e-481f-8807-7774ff8d5521-run-openvswitch\") pod \"ovnkube-node-p7mq8\" (UID: \"e50d3b5f-832e-481f-8807-7774ff8d5521\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:09:11.541904 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.540957 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b05f823fcae217aa6cef90a2100bffba-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-71.ec2.internal\" (UID: \"b05f823fcae217aa6cef90a2100bffba\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-71.ec2.internal" Jul 06 22:09:11.541904 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.540979 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rndz\" (UniqueName: \"kubernetes.io/projected/5147af52-3533-40a1-9797-566a9f3b2c86-kube-api-access-7rndz\") pod \"iptables-alerter-h7bb2\" (UID: \"5147af52-3533-40a1-9797-566a9f3b2c86\") " pod="openshift-network-operator/iptables-alerter-h7bb2" Jul 06 22:09:11.541904 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.540994 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/33df906486200f7aaa67d6b5660b85fa-config\") pod \"kube-apiserver-proxy-ip-10-0-128-71.ec2.internal\" (UID: \"33df906486200f7aaa67d6b5660b85fa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-71.ec2.internal" Jul 06 22:09:11.541904 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.541028 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/327650c8-6a1d-4167-9a90-f9b24f67ffe4-host-run-k8s-cni-cncf-io\") pod \"multus-kwwj2\" (UID: \"327650c8-6a1d-4167-9a90-f9b24f67ffe4\") " pod="openshift-multus/multus-kwwj2" Jul 06 22:09:11.541904 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.541045 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e50d3b5f-832e-481f-8807-7774ff8d5521-node-log\") pod \"ovnkube-node-p7mq8\" (UID: \"e50d3b5f-832e-481f-8807-7774ff8d5521\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:09:11.541904 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.541059 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e50d3b5f-832e-481f-8807-7774ff8d5521-ovn-node-metrics-cert\") pod \"ovnkube-node-p7mq8\" (UID: \"e50d3b5f-832e-481f-8807-7774ff8d5521\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:09:11.541904 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.541074 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7139257f-7bcb-4486-8818-83a0c465eea4-hosts-file\") pod \"node-resolver-dfd8v\" (UID: \"7139257f-7bcb-4486-8818-83a0c465eea4\") " pod="openshift-dns/node-resolver-dfd8v" Jul 06 22:09:11.541904 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.541088 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7139257f-7bcb-4486-8818-83a0c465eea4-tmp-dir\") pod \"node-resolver-dfd8v\" (UID: \"7139257f-7bcb-4486-8818-83a0c465eea4\") " pod="openshift-dns/node-resolver-dfd8v" Jul 06 22:09:11.541904 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.541103 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjrjn\" (UniqueName: \"kubernetes.io/projected/838f4166-d0dc-478f-aa98-4e1fd7ad300a-kube-api-access-tjrjn\") pod \"network-metrics-daemon-shdh8\" (UID: \"838f4166-d0dc-478f-aa98-4e1fd7ad300a\") " pod="openshift-multus/network-metrics-daemon-shdh8" Jul 06 22:09:11.541904 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.541139 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e50d3b5f-832e-481f-8807-7774ff8d5521-var-lib-openvswitch\") pod \"ovnkube-node-p7mq8\" (UID: \"e50d3b5f-832e-481f-8807-7774ff8d5521\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:09:11.541904 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.541156 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e50d3b5f-832e-481f-8807-7774ff8d5521-etc-openvswitch\") pod \"ovnkube-node-p7mq8\" (UID: \"e50d3b5f-832e-481f-8807-7774ff8d5521\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:09:11.541904 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.541171 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e50d3b5f-832e-481f-8807-7774ff8d5521-log-socket\") pod \"ovnkube-node-p7mq8\" (UID: \"e50d3b5f-832e-481f-8807-7774ff8d5521\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:09:11.542391 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.541186 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e50d3b5f-832e-481f-8807-7774ff8d5521-ovnkube-config\") pod \"ovnkube-node-p7mq8\" (UID: \"e50d3b5f-832e-481f-8807-7774ff8d5521\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:09:11.542391 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.541201 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpwfq\" (UniqueName: \"kubernetes.io/projected/e50d3b5f-832e-481f-8807-7774ff8d5521-kube-api-access-wpwfq\") pod \"ovnkube-node-p7mq8\" (UID: \"e50d3b5f-832e-481f-8807-7774ff8d5521\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:09:11.542391 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.541224 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dbf68764-0a98-4778-9b27-8e03df2a1c1b-etc-systemd\") pod \"tuned-hqgv6\" (UID: \"dbf68764-0a98-4778-9b27-8e03df2a1c1b\") " pod="openshift-cluster-node-tuning-operator/tuned-hqgv6" Jul 06 22:09:11.542391 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.541242 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/327650c8-6a1d-4167-9a90-f9b24f67ffe4-multus-socket-dir-parent\") pod \"multus-kwwj2\" (UID: \"327650c8-6a1d-4167-9a90-f9b24f67ffe4\") " pod="openshift-multus/multus-kwwj2" Jul 06 22:09:11.542391 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.541259 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d681bb3e-9bb1-4705-98cb-fd2cd5dc7063-dbus\") pod \"global-pull-secret-syncer-zt94q\" (UID: \"d681bb3e-9bb1-4705-98cb-fd2cd5dc7063\") " pod="kube-system/global-pull-secret-syncer-zt94q" Jul 06 22:09:11.542391 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.541274 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/245656d5-d53d-4319-b3d9-364a463bfeda-socket-dir\") pod \"aws-ebs-csi-driver-node-vhckn\" (UID: \"245656d5-d53d-4319-b3d9-364a463bfeda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhckn" Jul 06 22:09:11.542391 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.541292 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/245656d5-d53d-4319-b3d9-364a463bfeda-etc-selinux\") pod \"aws-ebs-csi-driver-node-vhckn\" (UID: \"245656d5-d53d-4319-b3d9-364a463bfeda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhckn" Jul 06 22:09:11.641772 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.641730 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/33df906486200f7aaa67d6b5660b85fa-config\") pod \"kube-apiserver-proxy-ip-10-0-128-71.ec2.internal\" (UID: \"33df906486200f7aaa67d6b5660b85fa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-71.ec2.internal" Jul 06 22:09:11.641922 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.641777 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/327650c8-6a1d-4167-9a90-f9b24f67ffe4-host-run-k8s-cni-cncf-io\") pod \"multus-kwwj2\" (UID: \"327650c8-6a1d-4167-9a90-f9b24f67ffe4\") " pod="openshift-multus/multus-kwwj2" Jul 06 22:09:11.641922 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.641809 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e50d3b5f-832e-481f-8807-7774ff8d5521-node-log\") pod \"ovnkube-node-p7mq8\" (UID: \"e50d3b5f-832e-481f-8807-7774ff8d5521\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:09:11.641922 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.641820 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/33df906486200f7aaa67d6b5660b85fa-config\") pod \"kube-apiserver-proxy-ip-10-0-128-71.ec2.internal\" (UID: \"33df906486200f7aaa67d6b5660b85fa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-71.ec2.internal" Jul 06 22:09:11.641922 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.641826 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e50d3b5f-832e-481f-8807-7774ff8d5521-ovn-node-metrics-cert\") pod \"ovnkube-node-p7mq8\" (UID: \"e50d3b5f-832e-481f-8807-7774ff8d5521\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:09:11.641922 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.641884 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7139257f-7bcb-4486-8818-83a0c465eea4-hosts-file\") pod \"node-resolver-dfd8v\" (UID: \"7139257f-7bcb-4486-8818-83a0c465eea4\") " pod="openshift-dns/node-resolver-dfd8v" Jul 06 22:09:11.641922 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.641909 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7139257f-7bcb-4486-8818-83a0c465eea4-tmp-dir\") pod \"node-resolver-dfd8v\" (UID: \"7139257f-7bcb-4486-8818-83a0c465eea4\") " pod="openshift-dns/node-resolver-dfd8v" Jul 06 22:09:11.641922 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.641912 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/327650c8-6a1d-4167-9a90-f9b24f67ffe4-host-run-k8s-cni-cncf-io\") pod \"multus-kwwj2\" (UID: \"327650c8-6a1d-4167-9a90-f9b24f67ffe4\") " pod="openshift-multus/multus-kwwj2" Jul 06 22:09:11.642305 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.641947 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e50d3b5f-832e-481f-8807-7774ff8d5521-node-log\") pod \"ovnkube-node-p7mq8\" (UID: \"e50d3b5f-832e-481f-8807-7774ff8d5521\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:09:11.642305 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.641972 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7139257f-7bcb-4486-8818-83a0c465eea4-hosts-file\") pod \"node-resolver-dfd8v\" (UID: \"7139257f-7bcb-4486-8818-83a0c465eea4\") " pod="openshift-dns/node-resolver-dfd8v" Jul 06 22:09:11.642305 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.641982 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjrjn\" (UniqueName: \"kubernetes.io/projected/838f4166-d0dc-478f-aa98-4e1fd7ad300a-kube-api-access-tjrjn\") pod \"network-metrics-daemon-shdh8\" (UID: \"838f4166-d0dc-478f-aa98-4e1fd7ad300a\") " pod="openshift-multus/network-metrics-daemon-shdh8" Jul 06 22:09:11.642305 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.642050 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e50d3b5f-832e-481f-8807-7774ff8d5521-var-lib-openvswitch\") pod \"ovnkube-node-p7mq8\" (UID: \"e50d3b5f-832e-481f-8807-7774ff8d5521\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:09:11.642305 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.642096 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e50d3b5f-832e-481f-8807-7774ff8d5521-etc-openvswitch\") pod \"ovnkube-node-p7mq8\" (UID: \"e50d3b5f-832e-481f-8807-7774ff8d5521\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:09:11.642305 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.642136 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e50d3b5f-832e-481f-8807-7774ff8d5521-log-socket\") pod \"ovnkube-node-p7mq8\" (UID: \"e50d3b5f-832e-481f-8807-7774ff8d5521\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:09:11.642305 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.642139 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e50d3b5f-832e-481f-8807-7774ff8d5521-etc-openvswitch\") pod \"ovnkube-node-p7mq8\" (UID: \"e50d3b5f-832e-481f-8807-7774ff8d5521\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:09:11.642305 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.642162 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e50d3b5f-832e-481f-8807-7774ff8d5521-ovnkube-config\") pod \"ovnkube-node-p7mq8\" (UID: \"e50d3b5f-832e-481f-8807-7774ff8d5521\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:09:11.642305 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.642180 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e50d3b5f-832e-481f-8807-7774ff8d5521-log-socket\") pod \"ovnkube-node-p7mq8\" (UID: \"e50d3b5f-832e-481f-8807-7774ff8d5521\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:09:11.642305 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.642187 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpwfq\" (UniqueName: \"kubernetes.io/projected/e50d3b5f-832e-481f-8807-7774ff8d5521-kube-api-access-wpwfq\") pod \"ovnkube-node-p7mq8\" (UID: \"e50d3b5f-832e-481f-8807-7774ff8d5521\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:09:11.642305 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.642158 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e50d3b5f-832e-481f-8807-7774ff8d5521-var-lib-openvswitch\") pod \"ovnkube-node-p7mq8\" (UID: \"e50d3b5f-832e-481f-8807-7774ff8d5521\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:09:11.642305 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.642209 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dbf68764-0a98-4778-9b27-8e03df2a1c1b-etc-systemd\") pod \"tuned-hqgv6\" (UID: \"dbf68764-0a98-4778-9b27-8e03df2a1c1b\") " pod="openshift-cluster-node-tuning-operator/tuned-hqgv6" Jul 06 22:09:11.642305 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.642280 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/327650c8-6a1d-4167-9a90-f9b24f67ffe4-multus-socket-dir-parent\") pod \"multus-kwwj2\" (UID: \"327650c8-6a1d-4167-9a90-f9b24f67ffe4\") " pod="openshift-multus/multus-kwwj2" Jul 06 22:09:11.642305 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.642299 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dbf68764-0a98-4778-9b27-8e03df2a1c1b-etc-systemd\") pod \"tuned-hqgv6\" (UID: \"dbf68764-0a98-4778-9b27-8e03df2a1c1b\") " pod="openshift-cluster-node-tuning-operator/tuned-hqgv6" Jul 06 22:09:11.642305 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.642311 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d681bb3e-9bb1-4705-98cb-fd2cd5dc7063-dbus\") pod \"global-pull-secret-syncer-zt94q\" (UID: \"d681bb3e-9bb1-4705-98cb-fd2cd5dc7063\") " pod="kube-system/global-pull-secret-syncer-zt94q" Jul 06 22:09:11.642305 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.642297 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7139257f-7bcb-4486-8818-83a0c465eea4-tmp-dir\") pod \"node-resolver-dfd8v\" (UID: \"7139257f-7bcb-4486-8818-83a0c465eea4\") " pod="openshift-dns/node-resolver-dfd8v" Jul 06 22:09:11.643116 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.642342 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/245656d5-d53d-4319-b3d9-364a463bfeda-socket-dir\") pod \"aws-ebs-csi-driver-node-vhckn\" (UID: \"245656d5-d53d-4319-b3d9-364a463bfeda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhckn" Jul 06 22:09:11.643116 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.642372 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/245656d5-d53d-4319-b3d9-364a463bfeda-etc-selinux\") pod \"aws-ebs-csi-driver-node-vhckn\" (UID: \"245656d5-d53d-4319-b3d9-364a463bfeda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhckn" Jul 06 22:09:11.643116 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.642430 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/327650c8-6a1d-4167-9a90-f9b24f67ffe4-multus-socket-dir-parent\") pod \"multus-kwwj2\" (UID: \"327650c8-6a1d-4167-9a90-f9b24f67ffe4\") " pod="openshift-multus/multus-kwwj2" Jul 06 22:09:11.643116 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.642450 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d681bb3e-9bb1-4705-98cb-fd2cd5dc7063-dbus\") pod \"global-pull-secret-syncer-zt94q\" (UID: \"d681bb3e-9bb1-4705-98cb-fd2cd5dc7063\") " pod="kube-system/global-pull-secret-syncer-zt94q" Jul 06 22:09:11.643116 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.642461 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/245656d5-d53d-4319-b3d9-364a463bfeda-etc-selinux\") pod \"aws-ebs-csi-driver-node-vhckn\" (UID: \"245656d5-d53d-4319-b3d9-364a463bfeda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhckn" Jul 06 22:09:11.643116 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.642432 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/327650c8-6a1d-4167-9a90-f9b24f67ffe4-hostroot\") pod \"multus-kwwj2\" (UID: \"327650c8-6a1d-4167-9a90-f9b24f67ffe4\") " pod="openshift-multus/multus-kwwj2" Jul 06 22:09:11.643116 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.642492 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/245656d5-d53d-4319-b3d9-364a463bfeda-socket-dir\") pod \"aws-ebs-csi-driver-node-vhckn\" (UID: \"245656d5-d53d-4319-b3d9-364a463bfeda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhckn" Jul 06 22:09:11.643116 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.642473 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/327650c8-6a1d-4167-9a90-f9b24f67ffe4-hostroot\") pod \"multus-kwwj2\" (UID: \"327650c8-6a1d-4167-9a90-f9b24f67ffe4\") " pod="openshift-multus/multus-kwwj2" Jul 06 22:09:11.643116 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.642510 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e50d3b5f-832e-481f-8807-7774ff8d5521-host-cni-netd\") pod \"ovnkube-node-p7mq8\" (UID: \"e50d3b5f-832e-481f-8807-7774ff8d5521\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:09:11.643116 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.642534 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e50d3b5f-832e-481f-8807-7774ff8d5521-host-cni-netd\") pod \"ovnkube-node-p7mq8\" (UID: \"e50d3b5f-832e-481f-8807-7774ff8d5521\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:09:11.643116 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.642557 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/245656d5-d53d-4319-b3d9-364a463bfeda-sys-fs\") pod \"aws-ebs-csi-driver-node-vhckn\" (UID: \"245656d5-d53d-4319-b3d9-364a463bfeda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhckn" Jul 06 22:09:11.643116 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.642493 2555 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 06 22:09:11.643116 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.642587 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f7d6b6f8-91a4-4341-b018-4e768abdd9a0-agent-certs\") pod \"konnectivity-agent-grmjd\" (UID: \"f7d6b6f8-91a4-4341-b018-4e768abdd9a0\") " pod="kube-system/konnectivity-agent-grmjd" Jul 06 22:09:11.643116 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.642648 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/245656d5-d53d-4319-b3d9-364a463bfeda-sys-fs\") pod \"aws-ebs-csi-driver-node-vhckn\" (UID: \"245656d5-d53d-4319-b3d9-364a463bfeda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhckn" Jul 06 22:09:11.643116 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.642648 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8frlx\" (UniqueName: \"kubernetes.io/projected/327650c8-6a1d-4167-9a90-f9b24f67ffe4-kube-api-access-8frlx\") pod \"multus-kwwj2\" (UID: \"327650c8-6a1d-4167-9a90-f9b24f67ffe4\") " pod="openshift-multus/multus-kwwj2" Jul 06 22:09:11.643116 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.642695 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e50d3b5f-832e-481f-8807-7774ff8d5521-host-kubelet\") pod \"ovnkube-node-p7mq8\" (UID: \"e50d3b5f-832e-481f-8807-7774ff8d5521\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:09:11.643116 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.642722 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dbf68764-0a98-4778-9b27-8e03df2a1c1b-etc-tuned\") pod \"tuned-hqgv6\" (UID: \"dbf68764-0a98-4778-9b27-8e03df2a1c1b\") " pod="openshift-cluster-node-tuning-operator/tuned-hqgv6" Jul 06 22:09:11.643116 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.642748 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/327650c8-6a1d-4167-9a90-f9b24f67ffe4-multus-cni-dir\") pod \"multus-kwwj2\" (UID: \"327650c8-6a1d-4167-9a90-f9b24f67ffe4\") " pod="openshift-multus/multus-kwwj2" Jul 06 22:09:11.643998 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.642755 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e50d3b5f-832e-481f-8807-7774ff8d5521-ovnkube-config\") pod \"ovnkube-node-p7mq8\" (UID: \"e50d3b5f-832e-481f-8807-7774ff8d5521\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:09:11.643998 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.642770 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e50d3b5f-832e-481f-8807-7774ff8d5521-host-kubelet\") pod \"ovnkube-node-p7mq8\" (UID: \"e50d3b5f-832e-481f-8807-7774ff8d5521\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:09:11.643998 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.642771 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e50d3b5f-832e-481f-8807-7774ff8d5521-host-run-ovn-kubernetes\") pod \"ovnkube-node-p7mq8\" (UID: \"e50d3b5f-832e-481f-8807-7774ff8d5521\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:09:11.643998 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.642821 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dbf68764-0a98-4778-9b27-8e03df2a1c1b-tmp\") pod \"tuned-hqgv6\" (UID: \"dbf68764-0a98-4778-9b27-8e03df2a1c1b\") " pod="openshift-cluster-node-tuning-operator/tuned-hqgv6" Jul 06 22:09:11.643998 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.642829 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e50d3b5f-832e-481f-8807-7774ff8d5521-host-run-ovn-kubernetes\") pod \"ovnkube-node-p7mq8\" (UID: \"e50d3b5f-832e-481f-8807-7774ff8d5521\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:09:11.643998 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.642836 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/327650c8-6a1d-4167-9a90-f9b24f67ffe4-multus-cni-dir\") pod \"multus-kwwj2\" (UID: \"327650c8-6a1d-4167-9a90-f9b24f67ffe4\") " pod="openshift-multus/multus-kwwj2" Jul 06 22:09:11.643998 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.642846 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d681bb3e-9bb1-4705-98cb-fd2cd5dc7063-original-pull-secret\") pod \"global-pull-secret-syncer-zt94q\" (UID: \"d681bb3e-9bb1-4705-98cb-fd2cd5dc7063\") " pod="kube-system/global-pull-secret-syncer-zt94q" Jul 06 22:09:11.643998 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.642868 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7c3b8e45-4537-4010-97c9-6081a2e3edcf-serviceca\") pod \"node-ca-9tlpj\" (UID: \"7c3b8e45-4537-4010-97c9-6081a2e3edcf\") " pod="openshift-image-registry/node-ca-9tlpj" Jul 06 22:09:11.643998 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.642911 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dbf68764-0a98-4778-9b27-8e03df2a1c1b-run\") pod \"tuned-hqgv6\" (UID: \"dbf68764-0a98-4778-9b27-8e03df2a1c1b\") " pod="openshift-cluster-node-tuning-operator/tuned-hqgv6" Jul 06 22:09:11.643998 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.642937 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f7d6b6f8-91a4-4341-b018-4e768abdd9a0-konnectivity-ca\") pod \"konnectivity-agent-grmjd\" (UID: \"f7d6b6f8-91a4-4341-b018-4e768abdd9a0\") " pod="kube-system/konnectivity-agent-grmjd" Jul 06 22:09:11.643998 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:11.642965 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 22:09:11.643998 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:11.643055 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d681bb3e-9bb1-4705-98cb-fd2cd5dc7063-original-pull-secret podName:d681bb3e-9bb1-4705-98cb-fd2cd5dc7063 nodeName:}" failed. No retries permitted until 2026-07-06 22:09:12.143034514 +0000 UTC m=+3.442746365 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d681bb3e-9bb1-4705-98cb-fd2cd5dc7063-original-pull-secret") pod "global-pull-secret-syncer-zt94q" (UID: "d681bb3e-9bb1-4705-98cb-fd2cd5dc7063") : object "kube-system"/"original-pull-secret" not registered Jul 06 22:09:11.643998 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.643077 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/327650c8-6a1d-4167-9a90-f9b24f67ffe4-system-cni-dir\") pod \"multus-kwwj2\" (UID: \"327650c8-6a1d-4167-9a90-f9b24f67ffe4\") " pod="openshift-multus/multus-kwwj2" Jul 06 22:09:11.643998 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.643131 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dbf68764-0a98-4778-9b27-8e03df2a1c1b-run\") pod \"tuned-hqgv6\" (UID: \"dbf68764-0a98-4778-9b27-8e03df2a1c1b\") " pod="openshift-cluster-node-tuning-operator/tuned-hqgv6" Jul 06 22:09:11.643998 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.642965 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/327650c8-6a1d-4167-9a90-f9b24f67ffe4-system-cni-dir\") pod \"multus-kwwj2\" (UID: \"327650c8-6a1d-4167-9a90-f9b24f67ffe4\") " pod="openshift-multus/multus-kwwj2" Jul 06 22:09:11.643998 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.643277 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/327650c8-6a1d-4167-9a90-f9b24f67ffe4-host-var-lib-cni-multus\") pod \"multus-kwwj2\" (UID: \"327650c8-6a1d-4167-9a90-f9b24f67ffe4\") " pod="openshift-multus/multus-kwwj2" Jul 06 22:09:11.643998 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.643314 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/327650c8-6a1d-4167-9a90-f9b24f67ffe4-etc-kubernetes\") pod \"multus-kwwj2\" (UID: \"327650c8-6a1d-4167-9a90-f9b24f67ffe4\") " pod="openshift-multus/multus-kwwj2" Jul 06 22:09:11.644865 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.643342 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b05f823fcae217aa6cef90a2100bffba-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-71.ec2.internal\" (UID: \"b05f823fcae217aa6cef90a2100bffba\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-71.ec2.internal" Jul 06 22:09:11.644865 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.643367 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7c3b8e45-4537-4010-97c9-6081a2e3edcf-host\") pod \"node-ca-9tlpj\" (UID: \"7c3b8e45-4537-4010-97c9-6081a2e3edcf\") " pod="openshift-image-registry/node-ca-9tlpj" Jul 06 22:09:11.644865 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.643391 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dbf68764-0a98-4778-9b27-8e03df2a1c1b-etc-sysconfig\") pod \"tuned-hqgv6\" (UID: \"dbf68764-0a98-4778-9b27-8e03df2a1c1b\") " pod="openshift-cluster-node-tuning-operator/tuned-hqgv6" Jul 06 22:09:11.644865 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.643418 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dbf68764-0a98-4778-9b27-8e03df2a1c1b-host\") pod \"tuned-hqgv6\" (UID: \"dbf68764-0a98-4778-9b27-8e03df2a1c1b\") " pod="openshift-cluster-node-tuning-operator/tuned-hqgv6" Jul 06 22:09:11.644865 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.643437 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/327650c8-6a1d-4167-9a90-f9b24f67ffe4-etc-kubernetes\") pod \"multus-kwwj2\" (UID: \"327650c8-6a1d-4167-9a90-f9b24f67ffe4\") " pod="openshift-multus/multus-kwwj2" Jul 06 22:09:11.644865 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.643444 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ff073db8-8f49-4832-8b5e-928a864bd41d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t5rgx\" (UID: \"ff073db8-8f49-4832-8b5e-928a864bd41d\") " pod="openshift-multus/multus-additional-cni-plugins-t5rgx" Jul 06 22:09:11.644865 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.643484 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/838f4166-d0dc-478f-aa98-4e1fd7ad300a-metrics-certs\") pod \"network-metrics-daemon-shdh8\" (UID: \"838f4166-d0dc-478f-aa98-4e1fd7ad300a\") " pod="openshift-multus/network-metrics-daemon-shdh8" Jul 06 22:09:11.644865 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.643517 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e50d3b5f-832e-481f-8807-7774ff8d5521-env-overrides\") pod \"ovnkube-node-p7mq8\" (UID: \"e50d3b5f-832e-481f-8807-7774ff8d5521\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:09:11.644865 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.643543 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ff073db8-8f49-4832-8b5e-928a864bd41d-cni-binary-copy\") pod \"multus-additional-cni-plugins-t5rgx\" (UID: \"ff073db8-8f49-4832-8b5e-928a864bd41d\") " pod="openshift-multus/multus-additional-cni-plugins-t5rgx" Jul 06 22:09:11.644865 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.643576 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ff073db8-8f49-4832-8b5e-928a864bd41d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t5rgx\" (UID: \"ff073db8-8f49-4832-8b5e-928a864bd41d\") " pod="openshift-multus/multus-additional-cni-plugins-t5rgx" Jul 06 22:09:11.644865 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.643597 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhq7d\" (UniqueName: \"kubernetes.io/projected/ff073db8-8f49-4832-8b5e-928a864bd41d-kube-api-access-bhq7d\") pod \"multus-additional-cni-plugins-t5rgx\" (UID: \"ff073db8-8f49-4832-8b5e-928a864bd41d\") " pod="openshift-multus/multus-additional-cni-plugins-t5rgx" Jul 06 22:09:11.644865 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.643624 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/327650c8-6a1d-4167-9a90-f9b24f67ffe4-cnibin\") pod \"multus-kwwj2\" (UID: \"327650c8-6a1d-4167-9a90-f9b24f67ffe4\") " pod="openshift-multus/multus-kwwj2" Jul 06 22:09:11.644865 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.643653 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e50d3b5f-832e-481f-8807-7774ff8d5521-ovnkube-script-lib\") pod \"ovnkube-node-p7mq8\" (UID: \"e50d3b5f-832e-481f-8807-7774ff8d5521\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:09:11.644865 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.643656 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f7d6b6f8-91a4-4341-b018-4e768abdd9a0-konnectivity-ca\") pod \"konnectivity-agent-grmjd\" (UID: \"f7d6b6f8-91a4-4341-b018-4e768abdd9a0\") " pod="kube-system/konnectivity-agent-grmjd" Jul 06 22:09:11.644865 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.643659 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7c3b8e45-4537-4010-97c9-6081a2e3edcf-host\") pod \"node-ca-9tlpj\" (UID: \"7c3b8e45-4537-4010-97c9-6081a2e3edcf\") " pod="openshift-image-registry/node-ca-9tlpj" Jul 06 22:09:11.644865 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.643681 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d681bb3e-9bb1-4705-98cb-fd2cd5dc7063-kubelet-config\") pod \"global-pull-secret-syncer-zt94q\" (UID: \"d681bb3e-9bb1-4705-98cb-fd2cd5dc7063\") " pod="kube-system/global-pull-secret-syncer-zt94q" Jul 06 22:09:11.644865 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.643696 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dbf68764-0a98-4778-9b27-8e03df2a1c1b-etc-sysconfig\") pod \"tuned-hqgv6\" (UID: \"dbf68764-0a98-4778-9b27-8e03df2a1c1b\") " pod="openshift-cluster-node-tuning-operator/tuned-hqgv6" Jul 06 22:09:11.647344 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.643711 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e50d3b5f-832e-481f-8807-7774ff8d5521-host-slash\") pod \"ovnkube-node-p7mq8\" (UID: \"e50d3b5f-832e-481f-8807-7774ff8d5521\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:09:11.647344 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.643738 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/327650c8-6a1d-4167-9a90-f9b24f67ffe4-host-var-lib-cni-multus\") pod \"multus-kwwj2\" (UID: \"327650c8-6a1d-4167-9a90-f9b24f67ffe4\") " pod="openshift-multus/multus-kwwj2" Jul 06 22:09:11.647344 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.643739 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dbf68764-0a98-4778-9b27-8e03df2a1c1b-etc-sysctl-conf\") pod \"tuned-hqgv6\" (UID: \"dbf68764-0a98-4778-9b27-8e03df2a1c1b\") " pod="openshift-cluster-node-tuning-operator/tuned-hqgv6" Jul 06 22:09:11.647344 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.643808 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ff073db8-8f49-4832-8b5e-928a864bd41d-system-cni-dir\") pod \"multus-additional-cni-plugins-t5rgx\" (UID: \"ff073db8-8f49-4832-8b5e-928a864bd41d\") " pod="openshift-multus/multus-additional-cni-plugins-t5rgx" Jul 06 22:09:11.647344 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.643840 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ff073db8-8f49-4832-8b5e-928a864bd41d-cnibin\") pod \"multus-additional-cni-plugins-t5rgx\" (UID: \"ff073db8-8f49-4832-8b5e-928a864bd41d\") " pod="openshift-multus/multus-additional-cni-plugins-t5rgx" Jul 06 22:09:11.647344 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.643860 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dbf68764-0a98-4778-9b27-8e03df2a1c1b-etc-sysctl-conf\") pod \"tuned-hqgv6\" (UID: \"dbf68764-0a98-4778-9b27-8e03df2a1c1b\") " pod="openshift-cluster-node-tuning-operator/tuned-hqgv6" Jul 06 22:09:11.647344 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.643870 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ff073db8-8f49-4832-8b5e-928a864bd41d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t5rgx\" (UID: \"ff073db8-8f49-4832-8b5e-928a864bd41d\") " pod="openshift-multus/multus-additional-cni-plugins-t5rgx" Jul 06 22:09:11.647344 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.644060 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e50d3b5f-832e-481f-8807-7774ff8d5521-env-overrides\") pod \"ovnkube-node-p7mq8\" (UID: \"e50d3b5f-832e-481f-8807-7774ff8d5521\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:09:11.647344 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.644145 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d681bb3e-9bb1-4705-98cb-fd2cd5dc7063-kubelet-config\") pod \"global-pull-secret-syncer-zt94q\" (UID: \"d681bb3e-9bb1-4705-98cb-fd2cd5dc7063\") " pod="kube-system/global-pull-secret-syncer-zt94q" Jul 06 22:09:11.647344 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.644146 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e50d3b5f-832e-481f-8807-7774ff8d5521-host-slash\") pod \"ovnkube-node-p7mq8\" (UID: \"e50d3b5f-832e-481f-8807-7774ff8d5521\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:09:11.647344 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.643626 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b05f823fcae217aa6cef90a2100bffba-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-71.ec2.internal\" (UID: \"b05f823fcae217aa6cef90a2100bffba\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-71.ec2.internal" Jul 06 22:09:11.647344 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.644430 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ff073db8-8f49-4832-8b5e-928a864bd41d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t5rgx\" (UID: \"ff073db8-8f49-4832-8b5e-928a864bd41d\") " pod="openshift-multus/multus-additional-cni-plugins-t5rgx" Jul 06 22:09:11.647344 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.643389 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7c3b8e45-4537-4010-97c9-6081a2e3edcf-serviceca\") pod \"node-ca-9tlpj\" (UID: \"7c3b8e45-4537-4010-97c9-6081a2e3edcf\") " pod="openshift-image-registry/node-ca-9tlpj" Jul 06 22:09:11.647344 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.644507 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dbf68764-0a98-4778-9b27-8e03df2a1c1b-host\") pod \"tuned-hqgv6\" (UID: \"dbf68764-0a98-4778-9b27-8e03df2a1c1b\") " pod="openshift-cluster-node-tuning-operator/tuned-hqgv6" Jul 06 22:09:11.647344 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:11.644577 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:09:11.647344 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:11.644627 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/838f4166-d0dc-478f-aa98-4e1fd7ad300a-metrics-certs podName:838f4166-d0dc-478f-aa98-4e1fd7ad300a nodeName:}" failed. No retries permitted until 2026-07-06 22:09:12.144608757 +0000 UTC m=+3.444320598 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/838f4166-d0dc-478f-aa98-4e1fd7ad300a-metrics-certs") pod "network-metrics-daemon-shdh8" (UID: "838f4166-d0dc-478f-aa98-4e1fd7ad300a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:09:11.647344 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.644632 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ff073db8-8f49-4832-8b5e-928a864bd41d-cni-binary-copy\") pod \"multus-additional-cni-plugins-t5rgx\" (UID: \"ff073db8-8f49-4832-8b5e-928a864bd41d\") " pod="openshift-multus/multus-additional-cni-plugins-t5rgx" Jul 06 22:09:11.647859 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.644658 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e50d3b5f-832e-481f-8807-7774ff8d5521-ovnkube-script-lib\") pod \"ovnkube-node-p7mq8\" (UID: \"e50d3b5f-832e-481f-8807-7774ff8d5521\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:09:11.647859 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.644669 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ff073db8-8f49-4832-8b5e-928a864bd41d-system-cni-dir\") pod \"multus-additional-cni-plugins-t5rgx\" (UID: \"ff073db8-8f49-4832-8b5e-928a864bd41d\") " pod="openshift-multus/multus-additional-cni-plugins-t5rgx" Jul 06 22:09:11.647859 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.644676 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/245656d5-d53d-4319-b3d9-364a463bfeda-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vhckn\" (UID: \"245656d5-d53d-4319-b3d9-364a463bfeda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhckn" Jul 06 22:09:11.647859 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.644710 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/245656d5-d53d-4319-b3d9-364a463bfeda-device-dir\") pod \"aws-ebs-csi-driver-node-vhckn\" (UID: \"245656d5-d53d-4319-b3d9-364a463bfeda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhckn" Jul 06 22:09:11.647859 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.644716 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/327650c8-6a1d-4167-9a90-f9b24f67ffe4-cnibin\") pod \"multus-kwwj2\" (UID: \"327650c8-6a1d-4167-9a90-f9b24f67ffe4\") " pod="openshift-multus/multus-kwwj2" Jul 06 22:09:11.647859 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.644714 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ff073db8-8f49-4832-8b5e-928a864bd41d-cnibin\") pod \"multus-additional-cni-plugins-t5rgx\" (UID: \"ff073db8-8f49-4832-8b5e-928a864bd41d\") " pod="openshift-multus/multus-additional-cni-plugins-t5rgx" Jul 06 22:09:11.647859 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.644751 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/327650c8-6a1d-4167-9a90-f9b24f67ffe4-os-release\") pod \"multus-kwwj2\" (UID: \"327650c8-6a1d-4167-9a90-f9b24f67ffe4\") " pod="openshift-multus/multus-kwwj2" Jul 06 22:09:11.647859 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.644753 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/245656d5-d53d-4319-b3d9-364a463bfeda-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vhckn\" (UID: \"245656d5-d53d-4319-b3d9-364a463bfeda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhckn" Jul 06 22:09:11.647859 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.644782 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/327650c8-6a1d-4167-9a90-f9b24f67ffe4-cni-binary-copy\") pod \"multus-kwwj2\" (UID: \"327650c8-6a1d-4167-9a90-f9b24f67ffe4\") " pod="openshift-multus/multus-kwwj2" Jul 06 22:09:11.647859 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.644790 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/245656d5-d53d-4319-b3d9-364a463bfeda-device-dir\") pod \"aws-ebs-csi-driver-node-vhckn\" (UID: \"245656d5-d53d-4319-b3d9-364a463bfeda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhckn" Jul 06 22:09:11.647859 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.644810 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5147af52-3533-40a1-9797-566a9f3b2c86-iptables-alerter-script\") pod \"iptables-alerter-h7bb2\" (UID: \"5147af52-3533-40a1-9797-566a9f3b2c86\") " pod="openshift-network-operator/iptables-alerter-h7bb2" Jul 06 22:09:11.647859 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.644829 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/327650c8-6a1d-4167-9a90-f9b24f67ffe4-os-release\") pod \"multus-kwwj2\" (UID: \"327650c8-6a1d-4167-9a90-f9b24f67ffe4\") " pod="openshift-multus/multus-kwwj2" Jul 06 22:09:11.647859 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.644837 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dbf68764-0a98-4778-9b27-8e03df2a1c1b-var-lib-kubelet\") pod \"tuned-hqgv6\" (UID: \"dbf68764-0a98-4778-9b27-8e03df2a1c1b\") " pod="openshift-cluster-node-tuning-operator/tuned-hqgv6" Jul 06 22:09:11.647859 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.644863 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/327650c8-6a1d-4167-9a90-f9b24f67ffe4-multus-daemon-config\") pod \"multus-kwwj2\" (UID: \"327650c8-6a1d-4167-9a90-f9b24f67ffe4\") " pod="openshift-multus/multus-kwwj2" Jul 06 22:09:11.647859 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.644891 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/327650c8-6a1d-4167-9a90-f9b24f67ffe4-host-run-multus-certs\") pod \"multus-kwwj2\" (UID: \"327650c8-6a1d-4167-9a90-f9b24f67ffe4\") " pod="openshift-multus/multus-kwwj2" Jul 06 22:09:11.647859 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.644916 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e50d3b5f-832e-481f-8807-7774ff8d5521-host-run-netns\") pod \"ovnkube-node-p7mq8\" (UID: \"e50d3b5f-832e-481f-8807-7774ff8d5521\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:09:11.647859 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.644942 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e50d3b5f-832e-481f-8807-7774ff8d5521-run-ovn\") pod \"ovnkube-node-p7mq8\" (UID: \"e50d3b5f-832e-481f-8807-7774ff8d5521\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:09:11.648392 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.644970 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e50d3b5f-832e-481f-8807-7774ff8d5521-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-p7mq8\" (UID: \"e50d3b5f-832e-481f-8807-7774ff8d5521\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:09:11.648392 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.645000 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj994\" (UniqueName: \"kubernetes.io/projected/7c3b8e45-4537-4010-97c9-6081a2e3edcf-kube-api-access-wj994\") pod \"node-ca-9tlpj\" (UID: \"7c3b8e45-4537-4010-97c9-6081a2e3edcf\") " pod="openshift-image-registry/node-ca-9tlpj" Jul 06 22:09:11.648392 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.645001 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dbf68764-0a98-4778-9b27-8e03df2a1c1b-var-lib-kubelet\") pod \"tuned-hqgv6\" (UID: \"dbf68764-0a98-4778-9b27-8e03df2a1c1b\") " pod="openshift-cluster-node-tuning-operator/tuned-hqgv6" Jul 06 22:09:11.648392 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.645040 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e50d3b5f-832e-481f-8807-7774ff8d5521-run-ovn\") pod \"ovnkube-node-p7mq8\" (UID: \"e50d3b5f-832e-481f-8807-7774ff8d5521\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:09:11.648392 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.645076 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dbf68764-0a98-4778-9b27-8e03df2a1c1b-sys\") pod \"tuned-hqgv6\" (UID: \"dbf68764-0a98-4778-9b27-8e03df2a1c1b\") " pod="openshift-cluster-node-tuning-operator/tuned-hqgv6" Jul 06 22:09:11.648392 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.645128 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hccq5\" (UniqueName: \"kubernetes.io/projected/7139257f-7bcb-4486-8818-83a0c465eea4-kube-api-access-hccq5\") pod \"node-resolver-dfd8v\" (UID: \"7139257f-7bcb-4486-8818-83a0c465eea4\") " pod="openshift-dns/node-resolver-dfd8v" Jul 06 22:09:11.648392 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.645178 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/327650c8-6a1d-4167-9a90-f9b24f67ffe4-host-run-multus-certs\") pod \"multus-kwwj2\" (UID: \"327650c8-6a1d-4167-9a90-f9b24f67ffe4\") " pod="openshift-multus/multus-kwwj2" Jul 06 22:09:11.648392 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.645226 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e50d3b5f-832e-481f-8807-7774ff8d5521-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-p7mq8\" (UID: \"e50d3b5f-832e-481f-8807-7774ff8d5521\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:09:11.648392 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.645361 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dbf68764-0a98-4778-9b27-8e03df2a1c1b-sys\") pod \"tuned-hqgv6\" (UID: \"dbf68764-0a98-4778-9b27-8e03df2a1c1b\") " pod="openshift-cluster-node-tuning-operator/tuned-hqgv6" Jul 06 22:09:11.648392 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.645369 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e50d3b5f-832e-481f-8807-7774ff8d5521-host-run-netns\") pod \"ovnkube-node-p7mq8\" (UID: \"e50d3b5f-832e-481f-8807-7774ff8d5521\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:09:11.648392 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.645379 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5147af52-3533-40a1-9797-566a9f3b2c86-iptables-alerter-script\") pod \"iptables-alerter-h7bb2\" (UID: \"5147af52-3533-40a1-9797-566a9f3b2c86\") " pod="openshift-network-operator/iptables-alerter-h7bb2" Jul 06 22:09:11.648392 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.645402 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/327650c8-6a1d-4167-9a90-f9b24f67ffe4-host-run-netns\") pod \"multus-kwwj2\" (UID: \"327650c8-6a1d-4167-9a90-f9b24f67ffe4\") " pod="openshift-multus/multus-kwwj2" Jul 06 22:09:11.648392 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.645432 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e50d3b5f-832e-481f-8807-7774ff8d5521-systemd-units\") pod \"ovnkube-node-p7mq8\" (UID: \"e50d3b5f-832e-481f-8807-7774ff8d5521\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:09:11.648392 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.645457 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e50d3b5f-832e-481f-8807-7774ff8d5521-host-cni-bin\") pod \"ovnkube-node-p7mq8\" (UID: \"e50d3b5f-832e-481f-8807-7774ff8d5521\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:09:11.648392 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.645482 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dbf68764-0a98-4778-9b27-8e03df2a1c1b-etc-kubernetes\") pod \"tuned-hqgv6\" (UID: \"dbf68764-0a98-4778-9b27-8e03df2a1c1b\") " pod="openshift-cluster-node-tuning-operator/tuned-hqgv6" Jul 06 22:09:11.648392 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.645501 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/327650c8-6a1d-4167-9a90-f9b24f67ffe4-host-run-netns\") pod \"multus-kwwj2\" (UID: \"327650c8-6a1d-4167-9a90-f9b24f67ffe4\") " pod="openshift-multus/multus-kwwj2" Jul 06 22:09:11.648392 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.645516 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e50d3b5f-832e-481f-8807-7774ff8d5521-systemd-units\") pod \"ovnkube-node-p7mq8\" (UID: \"e50d3b5f-832e-481f-8807-7774ff8d5521\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:09:11.649279 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.645525 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e50d3b5f-832e-481f-8807-7774ff8d5521-host-cni-bin\") pod \"ovnkube-node-p7mq8\" (UID: \"e50d3b5f-832e-481f-8807-7774ff8d5521\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:09:11.649279 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.645539 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dbf68764-0a98-4778-9b27-8e03df2a1c1b-etc-kubernetes\") pod \"tuned-hqgv6\" (UID: \"dbf68764-0a98-4778-9b27-8e03df2a1c1b\") " pod="openshift-cluster-node-tuning-operator/tuned-hqgv6" Jul 06 22:09:11.649279 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.645604 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5147af52-3533-40a1-9797-566a9f3b2c86-host-slash\") pod \"iptables-alerter-h7bb2\" (UID: \"5147af52-3533-40a1-9797-566a9f3b2c86\") " pod="openshift-network-operator/iptables-alerter-h7bb2" Jul 06 22:09:11.649279 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.645633 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dbf68764-0a98-4778-9b27-8e03df2a1c1b-etc-modprobe-d\") pod \"tuned-hqgv6\" (UID: \"dbf68764-0a98-4778-9b27-8e03df2a1c1b\") " pod="openshift-cluster-node-tuning-operator/tuned-hqgv6" Jul 06 22:09:11.649279 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.645640 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5147af52-3533-40a1-9797-566a9f3b2c86-host-slash\") pod \"iptables-alerter-h7bb2\" (UID: \"5147af52-3533-40a1-9797-566a9f3b2c86\") " pod="openshift-network-operator/iptables-alerter-h7bb2" Jul 06 22:09:11.649279 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.645661 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dbf68764-0a98-4778-9b27-8e03df2a1c1b-lib-modules\") pod \"tuned-hqgv6\" (UID: \"dbf68764-0a98-4778-9b27-8e03df2a1c1b\") " pod="openshift-cluster-node-tuning-operator/tuned-hqgv6" Jul 06 22:09:11.649279 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.645686 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ff073db8-8f49-4832-8b5e-928a864bd41d-os-release\") pod \"multus-additional-cni-plugins-t5rgx\" (UID: \"ff073db8-8f49-4832-8b5e-928a864bd41d\") " pod="openshift-multus/multus-additional-cni-plugins-t5rgx" Jul 06 22:09:11.649279 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.645713 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ff073db8-8f49-4832-8b5e-928a864bd41d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t5rgx\" (UID: \"ff073db8-8f49-4832-8b5e-928a864bd41d\") " pod="openshift-multus/multus-additional-cni-plugins-t5rgx" Jul 06 22:09:11.649279 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.645743 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtrhv\" (UniqueName: \"kubernetes.io/projected/245656d5-d53d-4319-b3d9-364a463bfeda-kube-api-access-jtrhv\") pod \"aws-ebs-csi-driver-node-vhckn\" (UID: \"245656d5-d53d-4319-b3d9-364a463bfeda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhckn" Jul 06 22:09:11.649279 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.645735 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dbf68764-0a98-4778-9b27-8e03df2a1c1b-etc-modprobe-d\") pod \"tuned-hqgv6\" (UID: \"dbf68764-0a98-4778-9b27-8e03df2a1c1b\") " pod="openshift-cluster-node-tuning-operator/tuned-hqgv6" Jul 06 22:09:11.649279 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.645771 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/327650c8-6a1d-4167-9a90-f9b24f67ffe4-host-var-lib-cni-bin\") pod \"multus-kwwj2\" (UID: \"327650c8-6a1d-4167-9a90-f9b24f67ffe4\") " pod="openshift-multus/multus-kwwj2" Jul 06 22:09:11.649279 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.645800 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dbf68764-0a98-4778-9b27-8e03df2a1c1b-etc-sysctl-d\") pod \"tuned-hqgv6\" (UID: \"dbf68764-0a98-4778-9b27-8e03df2a1c1b\") " pod="openshift-cluster-node-tuning-operator/tuned-hqgv6" Jul 06 22:09:11.649279 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.645820 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ff073db8-8f49-4832-8b5e-928a864bd41d-os-release\") pod \"multus-additional-cni-plugins-t5rgx\" (UID: \"ff073db8-8f49-4832-8b5e-928a864bd41d\") " pod="openshift-multus/multus-additional-cni-plugins-t5rgx" Jul 06 22:09:11.649279 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.645826 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7q86\" (UniqueName: \"kubernetes.io/projected/dbf68764-0a98-4778-9b27-8e03df2a1c1b-kube-api-access-p7q86\") pod \"tuned-hqgv6\" (UID: \"dbf68764-0a98-4778-9b27-8e03df2a1c1b\") " pod="openshift-cluster-node-tuning-operator/tuned-hqgv6" Jul 06 22:09:11.649279 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.645903 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dbf68764-0a98-4778-9b27-8e03df2a1c1b-lib-modules\") pod \"tuned-hqgv6\" (UID: \"dbf68764-0a98-4778-9b27-8e03df2a1c1b\") " pod="openshift-cluster-node-tuning-operator/tuned-hqgv6" Jul 06 22:09:11.649279 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.645934 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/245656d5-d53d-4319-b3d9-364a463bfeda-registration-dir\") pod \"aws-ebs-csi-driver-node-vhckn\" (UID: \"245656d5-d53d-4319-b3d9-364a463bfeda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhckn" Jul 06 22:09:11.649279 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.645964 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/327650c8-6a1d-4167-9a90-f9b24f67ffe4-host-var-lib-kubelet\") pod \"multus-kwwj2\" (UID: \"327650c8-6a1d-4167-9a90-f9b24f67ffe4\") " pod="openshift-multus/multus-kwwj2" Jul 06 22:09:11.650058 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.645990 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/327650c8-6a1d-4167-9a90-f9b24f67ffe4-multus-conf-dir\") pod \"multus-kwwj2\" (UID: \"327650c8-6a1d-4167-9a90-f9b24f67ffe4\") " pod="openshift-multus/multus-kwwj2" Jul 06 22:09:11.650058 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.646017 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4z7z\" (UniqueName: \"kubernetes.io/projected/852d3c7b-6b13-4603-9a80-7b07a035d22f-kube-api-access-z4z7z\") pod \"network-check-target-vdbr6\" (UID: \"852d3c7b-6b13-4603-9a80-7b07a035d22f\") " pod="openshift-network-diagnostics/network-check-target-vdbr6" Jul 06 22:09:11.650058 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.646067 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/327650c8-6a1d-4167-9a90-f9b24f67ffe4-cni-binary-copy\") pod \"multus-kwwj2\" (UID: \"327650c8-6a1d-4167-9a90-f9b24f67ffe4\") " pod="openshift-multus/multus-kwwj2" Jul 06 22:09:11.650058 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.646114 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e50d3b5f-832e-481f-8807-7774ff8d5521-run-systemd\") pod \"ovnkube-node-p7mq8\" (UID: \"e50d3b5f-832e-481f-8807-7774ff8d5521\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:09:11.650058 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.646141 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e50d3b5f-832e-481f-8807-7774ff8d5521-run-openvswitch\") pod \"ovnkube-node-p7mq8\" (UID: \"e50d3b5f-832e-481f-8807-7774ff8d5521\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:09:11.650058 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.646169 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b05f823fcae217aa6cef90a2100bffba-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-71.ec2.internal\" (UID: \"b05f823fcae217aa6cef90a2100bffba\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-71.ec2.internal" Jul 06 22:09:11.650058 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.646253 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/327650c8-6a1d-4167-9a90-f9b24f67ffe4-multus-daemon-config\") pod \"multus-kwwj2\" (UID: \"327650c8-6a1d-4167-9a90-f9b24f67ffe4\") " pod="openshift-multus/multus-kwwj2" Jul 06 22:09:11.650058 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.646337 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e50d3b5f-832e-481f-8807-7774ff8d5521-run-openvswitch\") pod \"ovnkube-node-p7mq8\" (UID: \"e50d3b5f-832e-481f-8807-7774ff8d5521\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:09:11.650058 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.646375 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/245656d5-d53d-4319-b3d9-364a463bfeda-registration-dir\") pod \"aws-ebs-csi-driver-node-vhckn\" (UID: \"245656d5-d53d-4319-b3d9-364a463bfeda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhckn" Jul 06 22:09:11.650058 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.646399 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e50d3b5f-832e-481f-8807-7774ff8d5521-run-systemd\") pod \"ovnkube-node-p7mq8\" (UID: \"e50d3b5f-832e-481f-8807-7774ff8d5521\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:09:11.650058 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.646139 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dbf68764-0a98-4778-9b27-8e03df2a1c1b-etc-sysctl-d\") pod \"tuned-hqgv6\" (UID: \"dbf68764-0a98-4778-9b27-8e03df2a1c1b\") " pod="openshift-cluster-node-tuning-operator/tuned-hqgv6" Jul 06 22:09:11.650058 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.646474 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/327650c8-6a1d-4167-9a90-f9b24f67ffe4-host-var-lib-kubelet\") pod \"multus-kwwj2\" (UID: \"327650c8-6a1d-4167-9a90-f9b24f67ffe4\") " pod="openshift-multus/multus-kwwj2" Jul 06 22:09:11.650058 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.646517 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/327650c8-6a1d-4167-9a90-f9b24f67ffe4-multus-conf-dir\") pod \"multus-kwwj2\" (UID: \"327650c8-6a1d-4167-9a90-f9b24f67ffe4\") " pod="openshift-multus/multus-kwwj2" Jul 06 22:09:11.650058 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.646551 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/327650c8-6a1d-4167-9a90-f9b24f67ffe4-host-var-lib-cni-bin\") pod \"multus-kwwj2\" (UID: \"327650c8-6a1d-4167-9a90-f9b24f67ffe4\") " pod="openshift-multus/multus-kwwj2" Jul 06 22:09:11.650058 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.646582 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ff073db8-8f49-4832-8b5e-928a864bd41d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t5rgx\" (UID: \"ff073db8-8f49-4832-8b5e-928a864bd41d\") " pod="openshift-multus/multus-additional-cni-plugins-t5rgx" Jul 06 22:09:11.650058 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.646198 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rndz\" (UniqueName: \"kubernetes.io/projected/5147af52-3533-40a1-9797-566a9f3b2c86-kube-api-access-7rndz\") pod \"iptables-alerter-h7bb2\" (UID: \"5147af52-3533-40a1-9797-566a9f3b2c86\") " pod="openshift-network-operator/iptables-alerter-h7bb2" Jul 06 22:09:11.650058 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.646759 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b05f823fcae217aa6cef90a2100bffba-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-71.ec2.internal\" (UID: \"b05f823fcae217aa6cef90a2100bffba\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-71.ec2.internal" Jul 06 22:09:11.650901 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.646887 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dbf68764-0a98-4778-9b27-8e03df2a1c1b-etc-tuned\") pod \"tuned-hqgv6\" (UID: \"dbf68764-0a98-4778-9b27-8e03df2a1c1b\") " pod="openshift-cluster-node-tuning-operator/tuned-hqgv6" Jul 06 22:09:11.650901 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.646959 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e50d3b5f-832e-481f-8807-7774ff8d5521-ovn-node-metrics-cert\") pod \"ovnkube-node-p7mq8\" (UID: \"e50d3b5f-832e-481f-8807-7774ff8d5521\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:09:11.650901 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.646983 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dbf68764-0a98-4778-9b27-8e03df2a1c1b-tmp\") pod \"tuned-hqgv6\" (UID: \"dbf68764-0a98-4778-9b27-8e03df2a1c1b\") " pod="openshift-cluster-node-tuning-operator/tuned-hqgv6" Jul 06 22:09:11.650901 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.647075 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f7d6b6f8-91a4-4341-b018-4e768abdd9a0-agent-certs\") pod \"konnectivity-agent-grmjd\" (UID: \"f7d6b6f8-91a4-4341-b018-4e768abdd9a0\") " pod="kube-system/konnectivity-agent-grmjd" Jul 06 22:09:11.654601 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.654578 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjrjn\" (UniqueName: \"kubernetes.io/projected/838f4166-d0dc-478f-aa98-4e1fd7ad300a-kube-api-access-tjrjn\") pod \"network-metrics-daemon-shdh8\" (UID: \"838f4166-d0dc-478f-aa98-4e1fd7ad300a\") " pod="openshift-multus/network-metrics-daemon-shdh8" Jul 06 22:09:11.654964 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.654940 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpwfq\" (UniqueName: \"kubernetes.io/projected/e50d3b5f-832e-481f-8807-7774ff8d5521-kube-api-access-wpwfq\") pod \"ovnkube-node-p7mq8\" (UID: \"e50d3b5f-832e-481f-8807-7774ff8d5521\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:09:11.655017 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.654947 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8frlx\" (UniqueName: \"kubernetes.io/projected/327650c8-6a1d-4167-9a90-f9b24f67ffe4-kube-api-access-8frlx\") pod \"multus-kwwj2\" (UID: \"327650c8-6a1d-4167-9a90-f9b24f67ffe4\") " pod="openshift-multus/multus-kwwj2" Jul 06 22:09:11.655906 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.655884 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhq7d\" (UniqueName: \"kubernetes.io/projected/ff073db8-8f49-4832-8b5e-928a864bd41d-kube-api-access-bhq7d\") pod \"multus-additional-cni-plugins-t5rgx\" (UID: \"ff073db8-8f49-4832-8b5e-928a864bd41d\") " pod="openshift-multus/multus-additional-cni-plugins-t5rgx" Jul 06 22:09:11.658766 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:11.658743 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 22:09:11.658899 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:11.658769 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 22:09:11.658899 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:11.658787 2555 projected.go:196] Error preparing data for projected volume kube-api-access-z4z7z for pod openshift-network-diagnostics/network-check-target-vdbr6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:09:11.658899 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:11.658880 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/852d3c7b-6b13-4603-9a80-7b07a035d22f-kube-api-access-z4z7z podName:852d3c7b-6b13-4603-9a80-7b07a035d22f nodeName:}" failed. No retries permitted until 2026-07-06 22:09:12.158860244 +0000 UTC m=+3.458572100 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-z4z7z" (UniqueName: "kubernetes.io/projected/852d3c7b-6b13-4603-9a80-7b07a035d22f-kube-api-access-z4z7z") pod "network-check-target-vdbr6" (UID: "852d3c7b-6b13-4603-9a80-7b07a035d22f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:09:11.659461 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.659441 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rndz\" (UniqueName: \"kubernetes.io/projected/5147af52-3533-40a1-9797-566a9f3b2c86-kube-api-access-7rndz\") pod \"iptables-alerter-h7bb2\" (UID: \"5147af52-3533-40a1-9797-566a9f3b2c86\") " pod="openshift-network-operator/iptables-alerter-h7bb2" Jul 06 22:09:11.660125 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.660099 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj994\" (UniqueName: \"kubernetes.io/projected/7c3b8e45-4537-4010-97c9-6081a2e3edcf-kube-api-access-wj994\") pod \"node-ca-9tlpj\" (UID: \"7c3b8e45-4537-4010-97c9-6081a2e3edcf\") " pod="openshift-image-registry/node-ca-9tlpj" Jul 06 22:09:11.660241 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.660105 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtrhv\" (UniqueName: \"kubernetes.io/projected/245656d5-d53d-4319-b3d9-364a463bfeda-kube-api-access-jtrhv\") pod \"aws-ebs-csi-driver-node-vhckn\" (UID: \"245656d5-d53d-4319-b3d9-364a463bfeda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhckn" Jul 06 22:09:11.660311 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.660293 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7q86\" (UniqueName: \"kubernetes.io/projected/dbf68764-0a98-4778-9b27-8e03df2a1c1b-kube-api-access-p7q86\") pod \"tuned-hqgv6\" (UID: \"dbf68764-0a98-4778-9b27-8e03df2a1c1b\") " pod="openshift-cluster-node-tuning-operator/tuned-hqgv6" Jul 06 22:09:11.660358 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.660337 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hccq5\" (UniqueName: \"kubernetes.io/projected/7139257f-7bcb-4486-8818-83a0c465eea4-kube-api-access-hccq5\") pod \"node-resolver-dfd8v\" (UID: \"7139257f-7bcb-4486-8818-83a0c465eea4\") " pod="openshift-dns/node-resolver-dfd8v" Jul 06 22:09:11.777423 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.777390 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-71.ec2.internal" Jul 06 22:09:11.792767 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.792746 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-71.ec2.internal" Jul 06 22:09:11.802254 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.802228 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhckn" Jul 06 22:09:11.819803 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.819760 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9tlpj" Jul 06 22:09:11.840167 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.840133 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-h7bb2" Jul 06 22:09:11.859618 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.859589 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-grmjd" Jul 06 22:09:11.876615 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.876576 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hqgv6" Jul 06 22:09:11.889889 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.889857 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dfd8v" Jul 06 22:09:11.897096 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.897051 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-t5rgx" Jul 06 22:09:11.899987 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.899962 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kwwj2" Jul 06 22:09:11.920226 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:11.920190 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:09:12.149602 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:12.149514 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d681bb3e-9bb1-4705-98cb-fd2cd5dc7063-original-pull-secret\") pod \"global-pull-secret-syncer-zt94q\" (UID: \"d681bb3e-9bb1-4705-98cb-fd2cd5dc7063\") " pod="kube-system/global-pull-secret-syncer-zt94q" Jul 06 22:09:12.149602 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:12.149564 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/838f4166-d0dc-478f-aa98-4e1fd7ad300a-metrics-certs\") pod \"network-metrics-daemon-shdh8\" (UID: \"838f4166-d0dc-478f-aa98-4e1fd7ad300a\") " pod="openshift-multus/network-metrics-daemon-shdh8" Jul 06 22:09:12.149781 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:12.149671 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 22:09:12.149781 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:12.149692 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:09:12.149781 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:12.149738 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d681bb3e-9bb1-4705-98cb-fd2cd5dc7063-original-pull-secret podName:d681bb3e-9bb1-4705-98cb-fd2cd5dc7063 nodeName:}" failed. No retries permitted until 2026-07-06 22:09:13.149721326 +0000 UTC m=+4.449433156 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d681bb3e-9bb1-4705-98cb-fd2cd5dc7063-original-pull-secret") pod "global-pull-secret-syncer-zt94q" (UID: "d681bb3e-9bb1-4705-98cb-fd2cd5dc7063") : object "kube-system"/"original-pull-secret" not registered Jul 06 22:09:12.149781 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:12.149753 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/838f4166-d0dc-478f-aa98-4e1fd7ad300a-metrics-certs podName:838f4166-d0dc-478f-aa98-4e1fd7ad300a nodeName:}" failed. No retries permitted until 2026-07-06 22:09:13.149745894 +0000 UTC m=+4.449457722 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/838f4166-d0dc-478f-aa98-4e1fd7ad300a-metrics-certs") pod "network-metrics-daemon-shdh8" (UID: "838f4166-d0dc-478f-aa98-4e1fd7ad300a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:09:12.178902 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:12.178839 2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 22:04:11 +0000 UTC" deadline="2028-03-25 20:38:07.115407094 +0000 UTC" Jul 06 22:09:12.178902 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:12.178896 2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15070h28m54.93651395s" Jul 06 22:09:12.250196 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:12.250159 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4z7z\" (UniqueName: \"kubernetes.io/projected/852d3c7b-6b13-4603-9a80-7b07a035d22f-kube-api-access-z4z7z\") pod \"network-check-target-vdbr6\" (UID: \"852d3c7b-6b13-4603-9a80-7b07a035d22f\") " pod="openshift-network-diagnostics/network-check-target-vdbr6" Jul 06 22:09:12.250498 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:12.250316 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 22:09:12.250498 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:12.250345 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 22:09:12.250498 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:12.250358 2555 projected.go:196] Error preparing data for projected volume kube-api-access-z4z7z for pod openshift-network-diagnostics/network-check-target-vdbr6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:09:12.250498 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:12.250423 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/852d3c7b-6b13-4603-9a80-7b07a035d22f-kube-api-access-z4z7z podName:852d3c7b-6b13-4603-9a80-7b07a035d22f nodeName:}" failed. No retries permitted until 2026-07-06 22:09:13.250403565 +0000 UTC m=+4.550115400 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-z4z7z" (UniqueName: "kubernetes.io/projected/852d3c7b-6b13-4603-9a80-7b07a035d22f-kube-api-access-z4z7z") pod "network-check-target-vdbr6" (UID: "852d3c7b-6b13-4603-9a80-7b07a035d22f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:09:12.362640 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:12.362597 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod245656d5_d53d_4319_b3d9_364a463bfeda.slice/crio-4566e922b24c9ebdb9a71116f1045520e2f0fb323864fae8a3d8e8b61a2612e9 WatchSource:0}: Error finding container 4566e922b24c9ebdb9a71116f1045520e2f0fb323864fae8a3d8e8b61a2612e9: Status 404 returned error can't find the container with id 4566e922b24c9ebdb9a71116f1045520e2f0fb323864fae8a3d8e8b61a2612e9 Jul 06 22:09:12.363298 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:12.362828 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff073db8_8f49_4832_8b5e_928a864bd41d.slice/crio-1ee96b5b0632bb27a8ca711d1379bb9a0e6aa3c2726be36b11262db0b5607e91 WatchSource:0}: Error finding container 1ee96b5b0632bb27a8ca711d1379bb9a0e6aa3c2726be36b11262db0b5607e91: Status 404 returned error can't find the container with id 1ee96b5b0632bb27a8ca711d1379bb9a0e6aa3c2726be36b11262db0b5607e91 Jul 06 22:09:12.364198 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:12.364160 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5147af52_3533_40a1_9797_566a9f3b2c86.slice/crio-90a17b13ffce6f60e1e8e2c3883c7aa611f497e2ab41bc3f9fc922e739e054de WatchSource:0}: Error finding container 90a17b13ffce6f60e1e8e2c3883c7aa611f497e2ab41bc3f9fc922e739e054de: Status 404 returned error can't find the container with id 90a17b13ffce6f60e1e8e2c3883c7aa611f497e2ab41bc3f9fc922e739e054de Jul 06 22:09:12.364995 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:12.364969 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb05f823fcae217aa6cef90a2100bffba.slice/crio-0412d221168a7b42a317c1c40b5022a7c131c0a7f462dd6b4ca232d1225e0500 WatchSource:0}: Error finding container 0412d221168a7b42a317c1c40b5022a7c131c0a7f462dd6b4ca232d1225e0500: Status 404 returned error can't find the container with id 0412d221168a7b42a317c1c40b5022a7c131c0a7f462dd6b4ca232d1225e0500 Jul 06 22:09:12.370725 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:12.370652 2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 22:09:12.378427 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:12.378405 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod327650c8_6a1d_4167_9a90_f9b24f67ffe4.slice/crio-2f72abdc742012e81eacde70e0e2f8612b23e740cc5a5b3de0fe20370c04a0b4 WatchSource:0}: Error finding container 2f72abdc742012e81eacde70e0e2f8612b23e740cc5a5b3de0fe20370c04a0b4: Status 404 returned error can't find the container with id 2f72abdc742012e81eacde70e0e2f8612b23e740cc5a5b3de0fe20370c04a0b4 Jul 06 22:09:12.394242 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:12.394202 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33df906486200f7aaa67d6b5660b85fa.slice/crio-78eea6a51129dce496a1e926b7817c20c052f71ceeca936b99da0206cb534cf9 WatchSource:0}: Error finding container 78eea6a51129dce496a1e926b7817c20c052f71ceeca936b99da0206cb534cf9: Status 404 returned error can't find the container with id 78eea6a51129dce496a1e926b7817c20c052f71ceeca936b99da0206cb534cf9 Jul 06 22:09:12.394561 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:12.394509 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7d6b6f8_91a4_4341_b018_4e768abdd9a0.slice/crio-44570911c02e56775ec5ff03eb730537e7ab16c1c0c22675aefc6ed6abdcef36 WatchSource:0}: Error finding container 44570911c02e56775ec5ff03eb730537e7ab16c1c0c22675aefc6ed6abdcef36: Status 404 returned error can't find the container with id 44570911c02e56775ec5ff03eb730537e7ab16c1c0c22675aefc6ed6abdcef36 Jul 06 22:09:12.400615 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:12.400581 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7139257f_7bcb_4486_8818_83a0c465eea4.slice/crio-5d6e70326e60dc8f0c74c54bdf5819bf79c623bbe0d1debb445bae5e51f13184 WatchSource:0}: Error finding container 5d6e70326e60dc8f0c74c54bdf5819bf79c623bbe0d1debb445bae5e51f13184: Status 404 returned error can't find the container with id 5d6e70326e60dc8f0c74c54bdf5819bf79c623bbe0d1debb445bae5e51f13184 Jul 06 22:09:12.444718 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:12.444690 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c3b8e45_4537_4010_97c9_6081a2e3edcf.slice/crio-768d58fa35c6388dd5d1ae7d4e7ae3ba50490e964568fa91b243dba0bbfa0a03 WatchSource:0}: Error finding container 768d58fa35c6388dd5d1ae7d4e7ae3ba50490e964568fa91b243dba0bbfa0a03: Status 404 returned error can't find the container with id 768d58fa35c6388dd5d1ae7d4e7ae3ba50490e964568fa91b243dba0bbfa0a03 Jul 06 22:09:12.493857 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:12.493825 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode50d3b5f_832e_481f_8807_7774ff8d5521.slice/crio-d8c3d8104302cc16325758fb8518ee5111d39b7b145373007e4685fb4678269b WatchSource:0}: Error finding container d8c3d8104302cc16325758fb8518ee5111d39b7b145373007e4685fb4678269b: Status 404 returned error can't find the container with id d8c3d8104302cc16325758fb8518ee5111d39b7b145373007e4685fb4678269b Jul 06 22:09:12.544182 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:09:12.544155 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbf68764_0a98_4778_9b27_8e03df2a1c1b.slice/crio-baa64a27e3100ed0ff4c839d499897ec6a3d93cdcf98ec1817916f0d2b106b8f WatchSource:0}: Error finding container baa64a27e3100ed0ff4c839d499897ec6a3d93cdcf98ec1817916f0d2b106b8f: Status 404 returned error can't find the container with id baa64a27e3100ed0ff4c839d499897ec6a3d93cdcf98ec1817916f0d2b106b8f Jul 06 22:09:12.545991 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:12.545971 2555 kuberuntime_manager.go:1459] "Unhandled Error" err="container tuned start failed in pod tuned-hqgv6_openshift-cluster-node-tuning-operator(dbf68764-0a98-4778-9b27-8e03df2a1c1b): ErrImagePull: pull QPS exceeded" logger="UnhandledError" Jul 06 22:09:12.546084 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:12.546005 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"tuned\" with ErrImagePull: \"pull QPS exceeded\"" pod="openshift-cluster-node-tuning-operator/tuned-hqgv6" podUID="dbf68764-0a98-4778-9b27-8e03df2a1c1b" Jul 06 22:09:13.157806 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:13.157463 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d681bb3e-9bb1-4705-98cb-fd2cd5dc7063-original-pull-secret\") pod \"global-pull-secret-syncer-zt94q\" (UID: \"d681bb3e-9bb1-4705-98cb-fd2cd5dc7063\") " pod="kube-system/global-pull-secret-syncer-zt94q" Jul 06 22:09:13.157806 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:13.157538 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/838f4166-d0dc-478f-aa98-4e1fd7ad300a-metrics-certs\") pod \"network-metrics-daemon-shdh8\" (UID: \"838f4166-d0dc-478f-aa98-4e1fd7ad300a\") " pod="openshift-multus/network-metrics-daemon-shdh8" Jul 06 22:09:13.157806 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:13.157691 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:09:13.157806 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:13.157760 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/838f4166-d0dc-478f-aa98-4e1fd7ad300a-metrics-certs podName:838f4166-d0dc-478f-aa98-4e1fd7ad300a nodeName:}" failed. No retries permitted until 2026-07-06 22:09:15.157738443 +0000 UTC m=+6.457450298 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/838f4166-d0dc-478f-aa98-4e1fd7ad300a-metrics-certs") pod "network-metrics-daemon-shdh8" (UID: "838f4166-d0dc-478f-aa98-4e1fd7ad300a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:09:13.158258 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:13.158010 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 22:09:13.158258 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:13.158066 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d681bb3e-9bb1-4705-98cb-fd2cd5dc7063-original-pull-secret podName:d681bb3e-9bb1-4705-98cb-fd2cd5dc7063 nodeName:}" failed. No retries permitted until 2026-07-06 22:09:15.158048962 +0000 UTC m=+6.457760796 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d681bb3e-9bb1-4705-98cb-fd2cd5dc7063-original-pull-secret") pod "global-pull-secret-syncer-zt94q" (UID: "d681bb3e-9bb1-4705-98cb-fd2cd5dc7063") : object "kube-system"/"original-pull-secret" not registered Jul 06 22:09:13.179636 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:13.179350 2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 22:04:11 +0000 UTC" deadline="2028-02-15 02:26:32.200630106 +0000 UTC" Jul 06 22:09:13.179636 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:13.179384 2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14116h17m19.021251762s" Jul 06 22:09:13.257989 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:13.257914 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4z7z\" (UniqueName: \"kubernetes.io/projected/852d3c7b-6b13-4603-9a80-7b07a035d22f-kube-api-access-z4z7z\") pod \"network-check-target-vdbr6\" (UID: \"852d3c7b-6b13-4603-9a80-7b07a035d22f\") " pod="openshift-network-diagnostics/network-check-target-vdbr6" Jul 06 22:09:13.258213 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:13.258104 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 22:09:13.258213 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:13.258136 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 22:09:13.258213 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:13.258150 2555 projected.go:196] Error preparing data for projected volume kube-api-access-z4z7z for pod openshift-network-diagnostics/network-check-target-vdbr6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:09:13.258213 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:13.258215 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/852d3c7b-6b13-4603-9a80-7b07a035d22f-kube-api-access-z4z7z podName:852d3c7b-6b13-4603-9a80-7b07a035d22f nodeName:}" failed. No retries permitted until 2026-07-06 22:09:15.258196138 +0000 UTC m=+6.557907970 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-z4z7z" (UniqueName: "kubernetes.io/projected/852d3c7b-6b13-4603-9a80-7b07a035d22f-kube-api-access-z4z7z") pod "network-check-target-vdbr6" (UID: "852d3c7b-6b13-4603-9a80-7b07a035d22f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:09:13.334613 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:13.334576 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-shdh8" Jul 06 22:09:13.334809 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:13.334735 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-shdh8" podUID="838f4166-d0dc-478f-aa98-4e1fd7ad300a" Jul 06 22:09:13.335225 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:13.335202 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdbr6" Jul 06 22:09:13.335335 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:13.335305 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdbr6" podUID="852d3c7b-6b13-4603-9a80-7b07a035d22f" Jul 06 22:09:13.335403 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:13.335392 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zt94q" Jul 06 22:09:13.335500 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:13.335465 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zt94q" podUID="d681bb3e-9bb1-4705-98cb-fd2cd5dc7063" Jul 06 22:09:13.361525 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:13.361155 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" event={"ID":"e50d3b5f-832e-481f-8807-7774ff8d5521","Type":"ContainerStarted","Data":"d8c3d8104302cc16325758fb8518ee5111d39b7b145373007e4685fb4678269b"} Jul 06 22:09:13.379076 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:13.379032 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9tlpj" event={"ID":"7c3b8e45-4537-4010-97c9-6081a2e3edcf","Type":"ContainerStarted","Data":"768d58fa35c6388dd5d1ae7d4e7ae3ba50490e964568fa91b243dba0bbfa0a03"} Jul 06 22:09:13.388462 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:13.388421 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kwwj2" event={"ID":"327650c8-6a1d-4167-9a90-f9b24f67ffe4","Type":"ContainerStarted","Data":"2f72abdc742012e81eacde70e0e2f8612b23e740cc5a5b3de0fe20370c04a0b4"} Jul 06 22:09:13.395763 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:13.395720 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t5rgx" event={"ID":"ff073db8-8f49-4832-8b5e-928a864bd41d","Type":"ContainerStarted","Data":"1ee96b5b0632bb27a8ca711d1379bb9a0e6aa3c2726be36b11262db0b5607e91"} Jul 06 22:09:13.411114 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:13.411005 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-71.ec2.internal" event={"ID":"b05f823fcae217aa6cef90a2100bffba","Type":"ContainerStarted","Data":"0412d221168a7b42a317c1c40b5022a7c131c0a7f462dd6b4ca232d1225e0500"} Jul 06 22:09:13.424524 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:13.424481 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hqgv6" event={"ID":"dbf68764-0a98-4778-9b27-8e03df2a1c1b","Type":"ContainerStarted","Data":"baa64a27e3100ed0ff4c839d499897ec6a3d93cdcf98ec1817916f0d2b106b8f"} Jul 06 22:09:13.428610 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:13.428574 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"tuned\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5be8cdcf34f35a89a70b6bfa49e66897641871237570b3e09766f83a0a473e6a\\\": ErrImagePull: pull QPS exceeded\"" pod="openshift-cluster-node-tuning-operator/tuned-hqgv6" podUID="dbf68764-0a98-4778-9b27-8e03df2a1c1b" Jul 06 22:09:13.435722 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:13.435689 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dfd8v" event={"ID":"7139257f-7bcb-4486-8818-83a0c465eea4","Type":"ContainerStarted","Data":"5d6e70326e60dc8f0c74c54bdf5819bf79c623bbe0d1debb445bae5e51f13184"} Jul 06 22:09:13.452355 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:13.452314 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-grmjd" event={"ID":"f7d6b6f8-91a4-4341-b018-4e768abdd9a0","Type":"ContainerStarted","Data":"44570911c02e56775ec5ff03eb730537e7ab16c1c0c22675aefc6ed6abdcef36"} Jul 06 22:09:13.483358 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:13.483135 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-71.ec2.internal" event={"ID":"33df906486200f7aaa67d6b5660b85fa","Type":"ContainerStarted","Data":"78eea6a51129dce496a1e926b7817c20c052f71ceeca936b99da0206cb534cf9"} Jul 06 22:09:13.487087 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:13.487000 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhckn" event={"ID":"245656d5-d53d-4319-b3d9-364a463bfeda","Type":"ContainerStarted","Data":"4566e922b24c9ebdb9a71116f1045520e2f0fb323864fae8a3d8e8b61a2612e9"} Jul 06 22:09:13.495104 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:13.495048 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-h7bb2" event={"ID":"5147af52-3533-40a1-9797-566a9f3b2c86","Type":"ContainerStarted","Data":"90a17b13ffce6f60e1e8e2c3883c7aa611f497e2ab41bc3f9fc922e739e054de"} Jul 06 22:09:15.191173 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:15.191125 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d681bb3e-9bb1-4705-98cb-fd2cd5dc7063-original-pull-secret\") pod \"global-pull-secret-syncer-zt94q\" (UID: \"d681bb3e-9bb1-4705-98cb-fd2cd5dc7063\") " pod="kube-system/global-pull-secret-syncer-zt94q" Jul 06 22:09:15.191652 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:15.191186 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/838f4166-d0dc-478f-aa98-4e1fd7ad300a-metrics-certs\") pod \"network-metrics-daemon-shdh8\" (UID: \"838f4166-d0dc-478f-aa98-4e1fd7ad300a\") " pod="openshift-multus/network-metrics-daemon-shdh8" Jul 06 22:09:15.191652 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:15.191309 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:09:15.191652 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:15.191382 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/838f4166-d0dc-478f-aa98-4e1fd7ad300a-metrics-certs podName:838f4166-d0dc-478f-aa98-4e1fd7ad300a nodeName:}" failed. No retries permitted until 2026-07-06 22:09:19.191363288 +0000 UTC m=+10.491075129 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/838f4166-d0dc-478f-aa98-4e1fd7ad300a-metrics-certs") pod "network-metrics-daemon-shdh8" (UID: "838f4166-d0dc-478f-aa98-4e1fd7ad300a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:09:15.191824 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:15.191807 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 22:09:15.191874 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:15.191861 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d681bb3e-9bb1-4705-98cb-fd2cd5dc7063-original-pull-secret podName:d681bb3e-9bb1-4705-98cb-fd2cd5dc7063 nodeName:}" failed. No retries permitted until 2026-07-06 22:09:19.191846245 +0000 UTC m=+10.491558076 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d681bb3e-9bb1-4705-98cb-fd2cd5dc7063-original-pull-secret") pod "global-pull-secret-syncer-zt94q" (UID: "d681bb3e-9bb1-4705-98cb-fd2cd5dc7063") : object "kube-system"/"original-pull-secret" not registered Jul 06 22:09:15.291956 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:15.291916 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4z7z\" (UniqueName: \"kubernetes.io/projected/852d3c7b-6b13-4603-9a80-7b07a035d22f-kube-api-access-z4z7z\") pod \"network-check-target-vdbr6\" (UID: \"852d3c7b-6b13-4603-9a80-7b07a035d22f\") " pod="openshift-network-diagnostics/network-check-target-vdbr6" Jul 06 22:09:15.292157 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:15.292140 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 22:09:15.292233 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:15.292161 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 22:09:15.292233 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:15.292174 2555 projected.go:196] Error preparing data for projected volume kube-api-access-z4z7z for pod openshift-network-diagnostics/network-check-target-vdbr6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:09:15.292340 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:15.292241 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/852d3c7b-6b13-4603-9a80-7b07a035d22f-kube-api-access-z4z7z podName:852d3c7b-6b13-4603-9a80-7b07a035d22f nodeName:}" failed. No retries permitted until 2026-07-06 22:09:19.292223422 +0000 UTC m=+10.591935252 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-z4z7z" (UniqueName: "kubernetes.io/projected/852d3c7b-6b13-4603-9a80-7b07a035d22f-kube-api-access-z4z7z") pod "network-check-target-vdbr6" (UID: "852d3c7b-6b13-4603-9a80-7b07a035d22f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:09:15.336561 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:15.336159 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zt94q" Jul 06 22:09:15.336561 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:15.336306 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zt94q" podUID="d681bb3e-9bb1-4705-98cb-fd2cd5dc7063" Jul 06 22:09:15.336780 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:15.336671 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-shdh8" Jul 06 22:09:15.336843 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:15.336774 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-shdh8" podUID="838f4166-d0dc-478f-aa98-4e1fd7ad300a" Jul 06 22:09:15.336843 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:15.336838 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdbr6" Jul 06 22:09:15.336956 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:15.336904 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdbr6" podUID="852d3c7b-6b13-4603-9a80-7b07a035d22f" Jul 06 22:09:17.335479 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:17.334683 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zt94q" Jul 06 22:09:17.335479 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:17.334822 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zt94q" podUID="d681bb3e-9bb1-4705-98cb-fd2cd5dc7063" Jul 06 22:09:17.335479 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:17.335096 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdbr6" Jul 06 22:09:17.335479 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:17.335207 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdbr6" podUID="852d3c7b-6b13-4603-9a80-7b07a035d22f" Jul 06 22:09:17.335479 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:17.335246 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-shdh8" Jul 06 22:09:17.335479 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:17.335424 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-shdh8" podUID="838f4166-d0dc-478f-aa98-4e1fd7ad300a" Jul 06 22:09:19.225795 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:19.225752 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d681bb3e-9bb1-4705-98cb-fd2cd5dc7063-original-pull-secret\") pod \"global-pull-secret-syncer-zt94q\" (UID: \"d681bb3e-9bb1-4705-98cb-fd2cd5dc7063\") " pod="kube-system/global-pull-secret-syncer-zt94q" Jul 06 22:09:19.226470 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:19.225805 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/838f4166-d0dc-478f-aa98-4e1fd7ad300a-metrics-certs\") pod \"network-metrics-daemon-shdh8\" (UID: \"838f4166-d0dc-478f-aa98-4e1fd7ad300a\") " pod="openshift-multus/network-metrics-daemon-shdh8" Jul 06 22:09:19.226470 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:19.225915 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:09:19.226470 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:19.225978 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/838f4166-d0dc-478f-aa98-4e1fd7ad300a-metrics-certs podName:838f4166-d0dc-478f-aa98-4e1fd7ad300a nodeName:}" failed. No retries permitted until 2026-07-06 22:09:27.225960037 +0000 UTC m=+18.525671878 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/838f4166-d0dc-478f-aa98-4e1fd7ad300a-metrics-certs") pod "network-metrics-daemon-shdh8" (UID: "838f4166-d0dc-478f-aa98-4e1fd7ad300a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:09:19.226470 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:19.225916 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 22:09:19.226470 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:19.226077 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d681bb3e-9bb1-4705-98cb-fd2cd5dc7063-original-pull-secret podName:d681bb3e-9bb1-4705-98cb-fd2cd5dc7063 nodeName:}" failed. No retries permitted until 2026-07-06 22:09:27.226056565 +0000 UTC m=+18.525768394 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d681bb3e-9bb1-4705-98cb-fd2cd5dc7063-original-pull-secret") pod "global-pull-secret-syncer-zt94q" (UID: "d681bb3e-9bb1-4705-98cb-fd2cd5dc7063") : object "kube-system"/"original-pull-secret" not registered Jul 06 22:09:19.326757 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:19.326721 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4z7z\" (UniqueName: \"kubernetes.io/projected/852d3c7b-6b13-4603-9a80-7b07a035d22f-kube-api-access-z4z7z\") pod \"network-check-target-vdbr6\" (UID: \"852d3c7b-6b13-4603-9a80-7b07a035d22f\") " pod="openshift-network-diagnostics/network-check-target-vdbr6" Jul 06 22:09:19.326932 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:19.326847 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 22:09:19.326932 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:19.326871 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 22:09:19.326932 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:19.326883 2555 projected.go:196] Error preparing data for projected volume kube-api-access-z4z7z for pod openshift-network-diagnostics/network-check-target-vdbr6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:09:19.327109 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:19.326946 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/852d3c7b-6b13-4603-9a80-7b07a035d22f-kube-api-access-z4z7z podName:852d3c7b-6b13-4603-9a80-7b07a035d22f nodeName:}" failed. No retries permitted until 2026-07-06 22:09:27.326931404 +0000 UTC m=+18.626643231 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-z4z7z" (UniqueName: "kubernetes.io/projected/852d3c7b-6b13-4603-9a80-7b07a035d22f-kube-api-access-z4z7z") pod "network-check-target-vdbr6" (UID: "852d3c7b-6b13-4603-9a80-7b07a035d22f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:09:19.333861 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:19.333833 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdbr6" Jul 06 22:09:19.333861 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:19.333855 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-shdh8" Jul 06 22:09:19.334107 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:19.333834 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zt94q" Jul 06 22:09:19.334107 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:19.333956 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdbr6" podUID="852d3c7b-6b13-4603-9a80-7b07a035d22f" Jul 06 22:09:19.334107 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:19.334052 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zt94q" podUID="d681bb3e-9bb1-4705-98cb-fd2cd5dc7063" Jul 06 22:09:19.334267 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:19.334145 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-shdh8" podUID="838f4166-d0dc-478f-aa98-4e1fd7ad300a" Jul 06 22:09:21.335338 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:21.335108 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdbr6" Jul 06 22:09:21.335745 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:21.335260 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zt94q" Jul 06 22:09:21.335745 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:21.335295 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-shdh8" Jul 06 22:09:21.335745 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:21.335481 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdbr6" podUID="852d3c7b-6b13-4603-9a80-7b07a035d22f" Jul 06 22:09:21.335914 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:21.335865 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zt94q" podUID="d681bb3e-9bb1-4705-98cb-fd2cd5dc7063" Jul 06 22:09:21.335970 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:21.335954 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-shdh8" podUID="838f4166-d0dc-478f-aa98-4e1fd7ad300a" Jul 06 22:09:21.529566 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:21.528914 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-71.ec2.internal" event={"ID":"33df906486200f7aaa67d6b5660b85fa","Type":"ContainerStarted","Data":"c508efb72024c44a74beccc48fb26b72b8e50e16e7d7fac58e3204565cbf7a8e"} Jul 06 22:09:22.533077 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:22.532983 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhckn" event={"ID":"245656d5-d53d-4319-b3d9-364a463bfeda","Type":"ContainerStarted","Data":"2fe4c03be5c8e1cd28130c1ee6cb2d207385edef27be6d827b82b7e02b69579d"} Jul 06 22:09:22.534639 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:22.534608 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9tlpj" event={"ID":"7c3b8e45-4537-4010-97c9-6081a2e3edcf","Type":"ContainerStarted","Data":"73f6ecd7c5c0e04f94e3f828295ebb9bfc4ffb84b891a482246f99440691ef17"} Jul 06 22:09:22.536502 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:22.536475 2555 generic.go:356] "Generic (PLEG): container finished" podID="ff073db8-8f49-4832-8b5e-928a864bd41d" containerID="0472eae62a5ecbc0fdaff3d85ec14b4ae176561558e67e247cf47d4b575a82a4" exitCode=0 Jul 06 22:09:22.536591 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:22.536566 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t5rgx" event={"ID":"ff073db8-8f49-4832-8b5e-928a864bd41d","Type":"ContainerDied","Data":"0472eae62a5ecbc0fdaff3d85ec14b4ae176561558e67e247cf47d4b575a82a4"} Jul 06 22:09:22.538319 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:22.538216 2555 generic.go:356] "Generic (PLEG): container finished" podID="b05f823fcae217aa6cef90a2100bffba" containerID="8a2ae0a3adefcbef499481a3075f07fab3c17881ec5e5582647d9c6a3ea1dc86" exitCode=0 Jul 06 22:09:22.538319 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:22.538296 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-71.ec2.internal" event={"ID":"b05f823fcae217aa6cef90a2100bffba","Type":"ContainerDied","Data":"8a2ae0a3adefcbef499481a3075f07fab3c17881ec5e5582647d9c6a3ea1dc86"} Jul 06 22:09:22.540037 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:22.539947 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dfd8v" event={"ID":"7139257f-7bcb-4486-8818-83a0c465eea4","Type":"ContainerStarted","Data":"5f85ac792ec539db7dda59899fa81e5c31eb65e0e24b11adb0b65cb8285c500c"} Jul 06 22:09:22.542185 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:22.542128 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-grmjd" event={"ID":"f7d6b6f8-91a4-4341-b018-4e768abdd9a0","Type":"ContainerStarted","Data":"2c303a8ef89577328ae4d7318424d519e9bd8474e3baf52ed542b4bd4cc90d81"} Jul 06 22:09:22.551243 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:22.551197 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-71.ec2.internal" podStartSLOduration=11.551178405 podStartE2EDuration="11.551178405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:09:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:09:21.544639943 +0000 UTC m=+12.844351796" watchObservedRunningTime="2026-07-06 22:09:22.551178405 +0000 UTC m=+13.850890275" Jul 06 22:09:22.552163 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:22.552114 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-9tlpj" podStartSLOduration=4.75837364 podStartE2EDuration="13.552099055s" totalImagesPullingTime="8.793725415s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:09:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:09:22.550986878 +0000 UTC m=+13.850698729" watchObservedRunningTime="2026-07-06 22:09:22.552099055 +0000 UTC m=+13.851810905" Jul 06 22:09:22.602027 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:22.601952 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-grmjd" podStartSLOduration=4.758945826 podStartE2EDuration="13.601931476s" totalImagesPullingTime="8.84298565s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:09:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:09:22.60144122 +0000 UTC m=+13.901153069" watchObservedRunningTime="2026-07-06 22:09:22.601931476 +0000 UTC m=+13.901643326" Jul 06 22:09:22.615773 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:22.615714 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-dfd8v" podStartSLOduration=4.778134472 podStartE2EDuration="13.615694579s" totalImagesPullingTime="8.837560107s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:09:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:09:22.615663732 +0000 UTC m=+13.915375581" watchObservedRunningTime="2026-07-06 22:09:22.615694579 +0000 UTC m=+13.915406430" Jul 06 22:09:23.334285 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:23.334206 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zt94q" Jul 06 22:09:23.334446 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:23.334209 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-shdh8" Jul 06 22:09:23.334446 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:23.334330 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zt94q" podUID="d681bb3e-9bb1-4705-98cb-fd2cd5dc7063" Jul 06 22:09:23.334446 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:23.334348 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdbr6" Jul 06 22:09:23.334446 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:23.334430 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-shdh8" podUID="838f4166-d0dc-478f-aa98-4e1fd7ad300a" Jul 06 22:09:23.334665 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:23.334534 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdbr6" podUID="852d3c7b-6b13-4603-9a80-7b07a035d22f" Jul 06 22:09:23.547108 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:23.547061 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-h7bb2" event={"ID":"5147af52-3533-40a1-9797-566a9f3b2c86","Type":"ContainerStarted","Data":"7f1e4a82ce01896010134bd40ea3a4a4d80398885d5bc76adb2bbe6d19e4d7df"} Jul 06 22:09:23.563216 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:23.563160 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-h7bb2" podStartSLOduration=5.692870967 podStartE2EDuration="14.563140632s" totalImagesPullingTime="8.870269665s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:09:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:09:23.562957356 +0000 UTC m=+14.862669219" watchObservedRunningTime="2026-07-06 22:09:23.563140632 +0000 UTC m=+14.862852480" Jul 06 22:09:24.965192 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:24.965156 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-grmjd" Jul 06 22:09:24.965785 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:24.965760 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-grmjd" Jul 06 22:09:25.334523 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:25.334401 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zt94q" Jul 06 22:09:25.334523 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:25.334428 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdbr6" Jul 06 22:09:25.334523 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:25.334443 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-shdh8" Jul 06 22:09:25.334795 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:25.334547 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zt94q" podUID="d681bb3e-9bb1-4705-98cb-fd2cd5dc7063" Jul 06 22:09:25.334795 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:25.334696 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdbr6" podUID="852d3c7b-6b13-4603-9a80-7b07a035d22f" Jul 06 22:09:25.334930 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:25.334807 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-shdh8" podUID="838f4166-d0dc-478f-aa98-4e1fd7ad300a" Jul 06 22:09:25.551072 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:25.550822 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-grmjd" Jul 06 22:09:25.551407 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:25.551386 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-grmjd" Jul 06 22:09:27.287421 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:27.287381 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d681bb3e-9bb1-4705-98cb-fd2cd5dc7063-original-pull-secret\") pod \"global-pull-secret-syncer-zt94q\" (UID: \"d681bb3e-9bb1-4705-98cb-fd2cd5dc7063\") " pod="kube-system/global-pull-secret-syncer-zt94q" Jul 06 22:09:27.287421 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:27.287424 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/838f4166-d0dc-478f-aa98-4e1fd7ad300a-metrics-certs\") pod \"network-metrics-daemon-shdh8\" (UID: \"838f4166-d0dc-478f-aa98-4e1fd7ad300a\") " pod="openshift-multus/network-metrics-daemon-shdh8" Jul 06 22:09:27.288021 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:27.287539 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:09:27.288021 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:27.287593 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/838f4166-d0dc-478f-aa98-4e1fd7ad300a-metrics-certs podName:838f4166-d0dc-478f-aa98-4e1fd7ad300a nodeName:}" failed. No retries permitted until 2026-07-06 22:09:43.287578814 +0000 UTC m=+34.587290641 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/838f4166-d0dc-478f-aa98-4e1fd7ad300a-metrics-certs") pod "network-metrics-daemon-shdh8" (UID: "838f4166-d0dc-478f-aa98-4e1fd7ad300a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:09:27.288021 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:27.287540 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 22:09:27.288021 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:27.287680 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d681bb3e-9bb1-4705-98cb-fd2cd5dc7063-original-pull-secret podName:d681bb3e-9bb1-4705-98cb-fd2cd5dc7063 nodeName:}" failed. No retries permitted until 2026-07-06 22:09:43.287666007 +0000 UTC m=+34.587377850 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d681bb3e-9bb1-4705-98cb-fd2cd5dc7063-original-pull-secret") pod "global-pull-secret-syncer-zt94q" (UID: "d681bb3e-9bb1-4705-98cb-fd2cd5dc7063") : object "kube-system"/"original-pull-secret" not registered Jul 06 22:09:27.334217 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:27.334188 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zt94q" Jul 06 22:09:27.334385 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:27.334187 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-shdh8" Jul 06 22:09:27.334385 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:27.334318 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zt94q" podUID="d681bb3e-9bb1-4705-98cb-fd2cd5dc7063" Jul 06 22:09:27.334507 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:27.334418 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-shdh8" podUID="838f4166-d0dc-478f-aa98-4e1fd7ad300a" Jul 06 22:09:27.334507 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:27.334474 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdbr6" Jul 06 22:09:27.334618 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:27.334539 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdbr6" podUID="852d3c7b-6b13-4603-9a80-7b07a035d22f" Jul 06 22:09:27.388186 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:27.388150 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4z7z\" (UniqueName: \"kubernetes.io/projected/852d3c7b-6b13-4603-9a80-7b07a035d22f-kube-api-access-z4z7z\") pod \"network-check-target-vdbr6\" (UID: \"852d3c7b-6b13-4603-9a80-7b07a035d22f\") " pod="openshift-network-diagnostics/network-check-target-vdbr6" Jul 06 22:09:27.388367 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:27.388346 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 22:09:27.388420 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:27.388373 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 22:09:27.388420 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:27.388387 2555 projected.go:196] Error preparing data for projected volume kube-api-access-z4z7z for pod openshift-network-diagnostics/network-check-target-vdbr6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:09:27.388494 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:27.388453 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/852d3c7b-6b13-4603-9a80-7b07a035d22f-kube-api-access-z4z7z podName:852d3c7b-6b13-4603-9a80-7b07a035d22f nodeName:}" failed. No retries permitted until 2026-07-06 22:09:43.38843479 +0000 UTC m=+34.688146623 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-z4z7z" (UniqueName: "kubernetes.io/projected/852d3c7b-6b13-4603-9a80-7b07a035d22f-kube-api-access-z4z7z") pod "network-check-target-vdbr6" (UID: "852d3c7b-6b13-4603-9a80-7b07a035d22f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:09:29.333755 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:29.333713 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-shdh8" Jul 06 22:09:29.334307 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:29.333724 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zt94q" Jul 06 22:09:29.334307 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:29.333863 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-shdh8" podUID="838f4166-d0dc-478f-aa98-4e1fd7ad300a" Jul 06 22:09:29.334307 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:29.333731 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdbr6" Jul 06 22:09:29.334307 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:29.333927 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zt94q" podUID="d681bb3e-9bb1-4705-98cb-fd2cd5dc7063" Jul 06 22:09:29.334307 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:29.333995 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdbr6" podUID="852d3c7b-6b13-4603-9a80-7b07a035d22f" Jul 06 22:09:30.958867 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:30.958808 2555 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 06 22:09:31.203630 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:31.203297 2555 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-06T22:09:30.958839043Z","UUID":"fbc78f37-d7d0-43a3-986a-470154a949f9","Handler":null,"Name":"","Endpoint":""} Jul 06 22:09:31.205581 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:31.205539 2555 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 06 22:09:31.205581 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:31.205586 2555 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 06 22:09:31.335316 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:31.335289 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zt94q" Jul 06 22:09:31.335468 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:31.335405 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zt94q" podUID="d681bb3e-9bb1-4705-98cb-fd2cd5dc7063" Jul 06 22:09:31.335812 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:31.335791 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-shdh8" Jul 06 22:09:31.335928 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:31.335909 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-shdh8" podUID="838f4166-d0dc-478f-aa98-4e1fd7ad300a" Jul 06 22:09:31.336024 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:31.335974 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdbr6" Jul 06 22:09:31.336078 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:31.336060 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdbr6" podUID="852d3c7b-6b13-4603-9a80-7b07a035d22f" Jul 06 22:09:31.565127 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:31.564575 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kwwj2" event={"ID":"327650c8-6a1d-4167-9a90-f9b24f67ffe4","Type":"ContainerStarted","Data":"bb4f1f8860fae36a1c0f931b7d93719d79943fadc07d8583320df48051ffc852"} Jul 06 22:09:31.567402 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:31.567363 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-71.ec2.internal" event={"ID":"b05f823fcae217aa6cef90a2100bffba","Type":"ContainerStarted","Data":"c1a67f533072894fe518e48d47545997ca6cfe20683c4a3e7109455de2dfeeeb"} Jul 06 22:09:31.569704 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:31.569668 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhckn" event={"ID":"245656d5-d53d-4319-b3d9-364a463bfeda","Type":"ContainerStarted","Data":"2929be2a0fe87de822faf740960b5b31f3d0fcf06b28f9edefcb12bcda0487a5"} Jul 06 22:09:31.573149 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:31.573126 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7mq8_e50d3b5f-832e-481f-8807-7774ff8d5521/ovn-acl-logging/0.log" Jul 06 22:09:31.573523 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:31.573497 2555 generic.go:356] "Generic (PLEG): container finished" podID="e50d3b5f-832e-481f-8807-7774ff8d5521" containerID="b6dc702e411f5861165139c370c82a462290e6813ca6362d64973b2fc419c576" exitCode=1 Jul 06 22:09:31.573603 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:31.573541 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" event={"ID":"e50d3b5f-832e-481f-8807-7774ff8d5521","Type":"ContainerStarted","Data":"b3e39d17c2902b973a5e7563bd9fdec43561c89a88e9a233badbd628ae27352b"} Jul 06 22:09:31.573665 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:31.573613 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" event={"ID":"e50d3b5f-832e-481f-8807-7774ff8d5521","Type":"ContainerStarted","Data":"160ee4d445fc00152235cd8657eed83165a3c1d18d31f149f6d7204c3613cab2"} Jul 06 22:09:31.573718 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:31.573664 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" event={"ID":"e50d3b5f-832e-481f-8807-7774ff8d5521","Type":"ContainerStarted","Data":"56fdbd542bdc41733de826e3e404c7f9dc3ca0eeaf04902d6f6f39f3d01b97ae"} Jul 06 22:09:31.573718 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:31.573712 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" event={"ID":"e50d3b5f-832e-481f-8807-7774ff8d5521","Type":"ContainerStarted","Data":"1a7836657e12e9f7481cd0705d7c604e31d78009c8c3d9ff387abd24820ef8b9"} Jul 06 22:09:31.573821 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:31.573807 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" event={"ID":"e50d3b5f-832e-481f-8807-7774ff8d5521","Type":"ContainerDied","Data":"b6dc702e411f5861165139c370c82a462290e6813ca6362d64973b2fc419c576"} Jul 06 22:09:31.573913 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:31.573899 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" event={"ID":"e50d3b5f-832e-481f-8807-7774ff8d5521","Type":"ContainerStarted","Data":"6ace5ce2a42ece3e854d041d215d40e594ea7001a81221668a745ad1f2d3d466"} Jul 06 22:09:31.593310 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:31.593246 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-kwwj2" podStartSLOduration=3.836146286 podStartE2EDuration="22.593225286s" totalImagesPullingTime="18.757079s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:09:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:09:31.592554864 +0000 UTC m=+22.892266714" watchObservedRunningTime="2026-07-06 22:09:31.593225286 +0000 UTC m=+22.892937169" Jul 06 22:09:31.609560 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:31.609494 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-71.ec2.internal" podStartSLOduration=20.609471659 podStartE2EDuration="20.609471659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:09:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:09:31.609079154 +0000 UTC m=+22.908791003" watchObservedRunningTime="2026-07-06 22:09:31.609471659 +0000 UTC m=+22.909183510" Jul 06 22:09:32.579101 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:32.578820 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhckn" event={"ID":"245656d5-d53d-4319-b3d9-364a463bfeda","Type":"ContainerStarted","Data":"927d2b82bcc955dc5679db23eee7571b84d62c28ba45185924cda692c452a153"} Jul 06 22:09:32.601142 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:32.601086 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhckn" podStartSLOduration=4.747655159 podStartE2EDuration="23.60106376s" totalImagesPullingTime="18.853408601s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:09:09 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:09:32.600655633 +0000 UTC m=+23.900367483" watchObservedRunningTime="2026-07-06 22:09:32.60106376 +0000 UTC m=+23.900775608" Jul 06 22:09:33.333614 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:33.333570 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zt94q" Jul 06 22:09:33.333827 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:33.333588 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdbr6" Jul 06 22:09:33.333827 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:33.333716 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zt94q" podUID="d681bb3e-9bb1-4705-98cb-fd2cd5dc7063" Jul 06 22:09:33.333827 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:33.333749 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdbr6" podUID="852d3c7b-6b13-4603-9a80-7b07a035d22f" Jul 06 22:09:33.333827 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:33.333570 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-shdh8" Jul 06 22:09:33.334075 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:33.333876 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-shdh8" podUID="838f4166-d0dc-478f-aa98-4e1fd7ad300a" Jul 06 22:09:35.334076 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:35.334041 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zt94q" Jul 06 22:09:35.334076 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:35.334082 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdbr6" Jul 06 22:09:35.334621 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:35.334041 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-shdh8" Jul 06 22:09:35.334621 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:35.334179 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zt94q" podUID="d681bb3e-9bb1-4705-98cb-fd2cd5dc7063" Jul 06 22:09:35.334621 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:35.334266 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-shdh8" podUID="838f4166-d0dc-478f-aa98-4e1fd7ad300a" Jul 06 22:09:35.334621 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:35.334334 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdbr6" podUID="852d3c7b-6b13-4603-9a80-7b07a035d22f" Jul 06 22:09:37.333641 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:37.333601 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zt94q" Jul 06 22:09:37.334136 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:37.333601 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdbr6" Jul 06 22:09:37.334136 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:37.333729 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zt94q" podUID="d681bb3e-9bb1-4705-98cb-fd2cd5dc7063" Jul 06 22:09:37.334136 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:37.333617 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-shdh8" Jul 06 22:09:37.334136 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:37.333840 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdbr6" podUID="852d3c7b-6b13-4603-9a80-7b07a035d22f" Jul 06 22:09:37.334136 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:37.333923 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-shdh8" podUID="838f4166-d0dc-478f-aa98-4e1fd7ad300a" Jul 06 22:09:38.595576 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:38.595536 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hqgv6" event={"ID":"dbf68764-0a98-4778-9b27-8e03df2a1c1b","Type":"ContainerStarted","Data":"476930fc44181d14c655e37593154f51505ceff5bc8c2e720942356e4ca102d9"} Jul 06 22:09:38.598426 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:38.598407 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7mq8_e50d3b5f-832e-481f-8807-7774ff8d5521/ovn-acl-logging/0.log" Jul 06 22:09:38.598769 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:38.598751 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" event={"ID":"e50d3b5f-832e-481f-8807-7774ff8d5521","Type":"ContainerStarted","Data":"26815ac0d4c66d6eb67f100d530164152b8e0e7dfb97256bd3455dd8a727a03f"} Jul 06 22:09:38.600433 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:38.600408 2555 generic.go:356] "Generic (PLEG): container finished" podID="ff073db8-8f49-4832-8b5e-928a864bd41d" containerID="369f72407df946dd400903c7b8398de563b338c0d4f7c12ce5bf53cc118f0611" exitCode=0 Jul 06 22:09:38.600517 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:38.600439 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t5rgx" event={"ID":"ff073db8-8f49-4832-8b5e-928a864bd41d","Type":"ContainerDied","Data":"369f72407df946dd400903c7b8398de563b338c0d4f7c12ce5bf53cc118f0611"} Jul 06 22:09:38.614354 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:38.614313 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-hqgv6" podStartSLOduration=4.550743446 podStartE2EDuration="29.614299654s" totalImagesPullingTime="25.063556208s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:09:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-07-06 22:09:38.613931728 +0000 UTC m=+29.913643589" watchObservedRunningTime="2026-07-06 22:09:38.614299654 +0000 UTC m=+29.914011502" Jul 06 22:09:39.334259 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:39.334233 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdbr6" Jul 06 22:09:39.334397 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:39.334234 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-shdh8" Jul 06 22:09:39.334397 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:39.334331 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdbr6" podUID="852d3c7b-6b13-4603-9a80-7b07a035d22f" Jul 06 22:09:39.334511 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:39.334440 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-shdh8" podUID="838f4166-d0dc-478f-aa98-4e1fd7ad300a" Jul 06 22:09:39.334511 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:39.334234 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zt94q" Jul 06 22:09:39.334574 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:39.334551 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zt94q" podUID="d681bb3e-9bb1-4705-98cb-fd2cd5dc7063" Jul 06 22:09:39.604623 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:39.604536 2555 generic.go:356] "Generic (PLEG): container finished" podID="ff073db8-8f49-4832-8b5e-928a864bd41d" containerID="f4c1e020e490b829a7419ae82ab399d8f5c7613091a979a915683675b4b9d30c" exitCode=0 Jul 06 22:09:39.604623 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:39.604594 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t5rgx" event={"ID":"ff073db8-8f49-4832-8b5e-928a864bd41d","Type":"ContainerDied","Data":"f4c1e020e490b829a7419ae82ab399d8f5c7613091a979a915683675b4b9d30c"} Jul 06 22:09:40.610038 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:40.609773 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7mq8_e50d3b5f-832e-481f-8807-7774ff8d5521/ovn-acl-logging/0.log" Jul 06 22:09:40.610504 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:40.610329 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" event={"ID":"e50d3b5f-832e-481f-8807-7774ff8d5521","Type":"ContainerStarted","Data":"fdcf21a328169e4cd77229135179d806ad3c0b001bb5eff9f4ab9ed65837d344"} Jul 06 22:09:40.610574 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:40.610558 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:09:40.610817 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:40.610797 2555 scope.go:117] "RemoveContainer" containerID="b6dc702e411f5861165139c370c82a462290e6813ca6362d64973b2fc419c576" Jul 06 22:09:40.613048 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:40.613020 2555 generic.go:356] "Generic (PLEG): container finished" podID="ff073db8-8f49-4832-8b5e-928a864bd41d" containerID="6ac6c5833df85eb869966a649f0b6d119dff6a8e6e840c05ad1bee5d01cac6f8" exitCode=0 Jul 06 22:09:40.613139 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:40.613070 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t5rgx" event={"ID":"ff073db8-8f49-4832-8b5e-928a864bd41d","Type":"ContainerDied","Data":"6ac6c5833df85eb869966a649f0b6d119dff6a8e6e840c05ad1bee5d01cac6f8"} Jul 06 22:09:40.628133 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:40.628107 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:09:41.334426 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:41.334395 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-shdh8" Jul 06 22:09:41.334589 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:41.334505 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-shdh8" podUID="838f4166-d0dc-478f-aa98-4e1fd7ad300a" Jul 06 22:09:41.334700 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:41.334679 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdbr6" Jul 06 22:09:41.334792 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:41.334757 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zt94q" Jul 06 22:09:41.334904 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:41.334857 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zt94q" podUID="d681bb3e-9bb1-4705-98cb-fd2cd5dc7063" Jul 06 22:09:41.334970 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:41.334948 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdbr6" podUID="852d3c7b-6b13-4603-9a80-7b07a035d22f" Jul 06 22:09:41.620643 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:41.620580 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7mq8_e50d3b5f-832e-481f-8807-7774ff8d5521/ovn-acl-logging/0.log" Jul 06 22:09:41.621113 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:41.621082 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" event={"ID":"e50d3b5f-832e-481f-8807-7774ff8d5521","Type":"ContainerStarted","Data":"1c8993282b98b291a74f837ad8e473f3c0a1d6b4f120a6d8610d694f9942828b"} Jul 06 22:09:41.621518 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:41.621496 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:09:41.621645 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:41.621576 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:09:41.640128 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:41.640093 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:09:41.650305 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:41.650252 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" podStartSLOduration=14.328101738 podStartE2EDuration="32.650232437s" totalImagesPullingTime="18.322130699s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:09:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:09:41.648918932 +0000 UTC m=+32.948630792" watchObservedRunningTime="2026-07-06 22:09:41.650232437 +0000 UTC m=+32.949944287" Jul 06 22:09:42.024644 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:42.024349 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zt94q"] Jul 06 22:09:42.024805 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:42.024684 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zt94q" Jul 06 22:09:42.024865 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:42.024795 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zt94q" podUID="d681bb3e-9bb1-4705-98cb-fd2cd5dc7063" Jul 06 22:09:42.027436 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:42.027376 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vdbr6"] Jul 06 22:09:42.027589 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:42.027487 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdbr6" Jul 06 22:09:42.027656 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:42.027585 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdbr6" podUID="852d3c7b-6b13-4603-9a80-7b07a035d22f" Jul 06 22:09:42.027989 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:42.027963 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-shdh8"] Jul 06 22:09:42.028125 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:42.028079 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-shdh8" Jul 06 22:09:42.028213 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:42.028188 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-shdh8" podUID="838f4166-d0dc-478f-aa98-4e1fd7ad300a" Jul 06 22:09:43.309894 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:43.309853 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d681bb3e-9bb1-4705-98cb-fd2cd5dc7063-original-pull-secret\") pod \"global-pull-secret-syncer-zt94q\" (UID: \"d681bb3e-9bb1-4705-98cb-fd2cd5dc7063\") " pod="kube-system/global-pull-secret-syncer-zt94q" Jul 06 22:09:43.310365 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:43.309905 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/838f4166-d0dc-478f-aa98-4e1fd7ad300a-metrics-certs\") pod \"network-metrics-daemon-shdh8\" (UID: \"838f4166-d0dc-478f-aa98-4e1fd7ad300a\") " pod="openshift-multus/network-metrics-daemon-shdh8" Jul 06 22:09:43.310365 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:43.310039 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:09:43.310365 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:43.310042 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 22:09:43.310365 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:43.310107 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/838f4166-d0dc-478f-aa98-4e1fd7ad300a-metrics-certs podName:838f4166-d0dc-478f-aa98-4e1fd7ad300a nodeName:}" failed. No retries permitted until 2026-07-06 22:10:15.31008736 +0000 UTC m=+66.609799193 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/838f4166-d0dc-478f-aa98-4e1fd7ad300a-metrics-certs") pod "network-metrics-daemon-shdh8" (UID: "838f4166-d0dc-478f-aa98-4e1fd7ad300a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:09:43.310365 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:43.310125 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d681bb3e-9bb1-4705-98cb-fd2cd5dc7063-original-pull-secret podName:d681bb3e-9bb1-4705-98cb-fd2cd5dc7063 nodeName:}" failed. No retries permitted until 2026-07-06 22:10:15.310115573 +0000 UTC m=+66.609827407 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d681bb3e-9bb1-4705-98cb-fd2cd5dc7063-original-pull-secret") pod "global-pull-secret-syncer-zt94q" (UID: "d681bb3e-9bb1-4705-98cb-fd2cd5dc7063") : object "kube-system"/"original-pull-secret" not registered Jul 06 22:09:43.333936 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:43.333900 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-shdh8" Jul 06 22:09:43.334084 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:43.333958 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdbr6" Jul 06 22:09:43.334084 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:43.333969 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zt94q" Jul 06 22:09:43.334148 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:43.334075 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-shdh8" podUID="838f4166-d0dc-478f-aa98-4e1fd7ad300a" Jul 06 22:09:43.334148 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:43.334118 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdbr6" podUID="852d3c7b-6b13-4603-9a80-7b07a035d22f" Jul 06 22:09:43.334224 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:43.334190 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zt94q" podUID="d681bb3e-9bb1-4705-98cb-fd2cd5dc7063" Jul 06 22:09:43.411189 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:43.411142 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4z7z\" (UniqueName: \"kubernetes.io/projected/852d3c7b-6b13-4603-9a80-7b07a035d22f-kube-api-access-z4z7z\") pod \"network-check-target-vdbr6\" (UID: \"852d3c7b-6b13-4603-9a80-7b07a035d22f\") " pod="openshift-network-diagnostics/network-check-target-vdbr6" Jul 06 22:09:43.411346 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:43.411313 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 22:09:43.411346 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:43.411339 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 22:09:43.411432 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:43.411353 2555 projected.go:196] Error preparing data for projected volume kube-api-access-z4z7z for pod openshift-network-diagnostics/network-check-target-vdbr6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:09:43.411432 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:43.411421 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/852d3c7b-6b13-4603-9a80-7b07a035d22f-kube-api-access-z4z7z podName:852d3c7b-6b13-4603-9a80-7b07a035d22f nodeName:}" failed. No retries permitted until 2026-07-06 22:10:15.41140179 +0000 UTC m=+66.711113617 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-z4z7z" (UniqueName: "kubernetes.io/projected/852d3c7b-6b13-4603-9a80-7b07a035d22f-kube-api-access-z4z7z") pod "network-check-target-vdbr6" (UID: "852d3c7b-6b13-4603-9a80-7b07a035d22f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:09:44.097061 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:44.097033 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dfd8v_7139257f-7bcb-4486-8818-83a0c465eea4/dns-node-resolver/0.log" Jul 06 22:09:44.875710 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:44.875682 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9tlpj_7c3b8e45-4537-4010-97c9-6081a2e3edcf/node-ca/0.log" Jul 06 22:09:45.334143 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:45.334051 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-shdh8" Jul 06 22:09:45.334143 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:45.334100 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zt94q" Jul 06 22:09:45.334345 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:45.334064 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdbr6" Jul 06 22:09:45.334345 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:45.334204 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-shdh8" podUID="838f4166-d0dc-478f-aa98-4e1fd7ad300a" Jul 06 22:09:45.334345 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:45.334278 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zt94q" podUID="d681bb3e-9bb1-4705-98cb-fd2cd5dc7063" Jul 06 22:09:45.334476 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:45.334360 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdbr6" podUID="852d3c7b-6b13-4603-9a80-7b07a035d22f" Jul 06 22:09:46.635871 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:46.635842 2555 generic.go:356] "Generic (PLEG): container finished" podID="ff073db8-8f49-4832-8b5e-928a864bd41d" containerID="df46a9561e9317b06623c8bdbae7cf52545755eabd9a2302fecfaba5706d2657" exitCode=0 Jul 06 22:09:46.636447 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:46.635890 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t5rgx" event={"ID":"ff073db8-8f49-4832-8b5e-928a864bd41d","Type":"ContainerDied","Data":"df46a9561e9317b06623c8bdbae7cf52545755eabd9a2302fecfaba5706d2657"} Jul 06 22:09:47.333851 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:47.333814 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdbr6" Jul 06 22:09:47.334039 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:47.333861 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-shdh8" Jul 06 22:09:47.334039 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:47.333919 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zt94q" Jul 06 22:09:47.334132 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:47.334057 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-shdh8" podUID="838f4166-d0dc-478f-aa98-4e1fd7ad300a" Jul 06 22:09:47.334179 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:47.334158 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdbr6" podUID="852d3c7b-6b13-4603-9a80-7b07a035d22f" Jul 06 22:09:47.334260 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:47.334242 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zt94q" podUID="d681bb3e-9bb1-4705-98cb-fd2cd5dc7063" Jul 06 22:09:47.640919 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:47.640843 2555 generic.go:356] "Generic (PLEG): container finished" podID="ff073db8-8f49-4832-8b5e-928a864bd41d" containerID="73252f68155f091b7dd5661aade8e45ecc6ce544408088359b1b1551d03a10ff" exitCode=0 Jul 06 22:09:47.640919 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:47.640883 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t5rgx" event={"ID":"ff073db8-8f49-4832-8b5e-928a864bd41d","Type":"ContainerDied","Data":"73252f68155f091b7dd5661aade8e45ecc6ce544408088359b1b1551d03a10ff"} Jul 06 22:09:48.646933 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:48.646896 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t5rgx" event={"ID":"ff073db8-8f49-4832-8b5e-928a864bd41d","Type":"ContainerStarted","Data":"df3ccd651585571cc6197a8e6d9c7b6a223db1b19dde814f350bec8026d17b30"} Jul 06 22:09:48.673564 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:48.673510 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-t5rgx" podStartSLOduration=8.977891736 podStartE2EDuration="39.673494089s" totalImagesPullingTime="30.695602353s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:09:09 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:09:48.673466714 +0000 UTC m=+39.973178564" watchObservedRunningTime="2026-07-06 22:09:48.673494089 +0000 UTC m=+39.973205929" Jul 06 22:09:49.333531 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:49.333341 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zt94q" Jul 06 22:09:49.333703 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:49.333341 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdbr6" Jul 06 22:09:49.333703 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:49.333609 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zt94q" podUID="d681bb3e-9bb1-4705-98cb-fd2cd5dc7063" Jul 06 22:09:49.333703 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:49.333349 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-shdh8" Jul 06 22:09:49.333703 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:49.333678 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdbr6" podUID="852d3c7b-6b13-4603-9a80-7b07a035d22f" Jul 06 22:09:49.333853 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:49.333750 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-shdh8" podUID="838f4166-d0dc-478f-aa98-4e1fd7ad300a" Jul 06 22:09:51.335061 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:51.335002 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdbr6" Jul 06 22:09:51.335061 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:51.335065 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zt94q" Jul 06 22:09:51.335504 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:51.335129 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdbr6" podUID="852d3c7b-6b13-4603-9a80-7b07a035d22f" Jul 06 22:09:51.335504 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:51.335217 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zt94q" podUID="d681bb3e-9bb1-4705-98cb-fd2cd5dc7063" Jul 06 22:09:51.335504 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:51.335260 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-shdh8" Jul 06 22:09:51.335504 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:51.335350 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-shdh8" podUID="838f4166-d0dc-478f-aa98-4e1fd7ad300a" Jul 06 22:09:53.334162 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:53.334125 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdbr6" Jul 06 22:09:53.334541 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:53.334125 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-shdh8" Jul 06 22:09:53.334541 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:53.334238 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdbr6" podUID="852d3c7b-6b13-4603-9a80-7b07a035d22f" Jul 06 22:09:53.334541 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:53.334137 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zt94q" Jul 06 22:09:53.334541 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:53.334314 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-shdh8" podUID="838f4166-d0dc-478f-aa98-4e1fd7ad300a" Jul 06 22:09:53.334541 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:53.334390 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zt94q" podUID="d681bb3e-9bb1-4705-98cb-fd2cd5dc7063" Jul 06 22:09:55.334230 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:55.334187 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdbr6" Jul 06 22:09:55.334672 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:55.334298 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdbr6" podUID="852d3c7b-6b13-4603-9a80-7b07a035d22f" Jul 06 22:09:55.334672 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:55.334194 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zt94q" Jul 06 22:09:55.334672 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:55.334348 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-shdh8" Jul 06 22:09:55.334672 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:55.334409 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zt94q" podUID="d681bb3e-9bb1-4705-98cb-fd2cd5dc7063" Jul 06 22:09:55.334672 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:55.334476 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-shdh8" podUID="838f4166-d0dc-478f-aa98-4e1fd7ad300a" Jul 06 22:09:57.334331 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:57.334292 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-shdh8" Jul 06 22:09:57.334830 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:57.334296 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zt94q" Jul 06 22:09:57.334830 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:57.334441 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-shdh8" podUID="838f4166-d0dc-478f-aa98-4e1fd7ad300a" Jul 06 22:09:57.334830 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:57.334296 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdbr6" Jul 06 22:09:57.334830 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:57.334477 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zt94q" podUID="d681bb3e-9bb1-4705-98cb-fd2cd5dc7063" Jul 06 22:09:57.334830 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:57.334551 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdbr6" podUID="852d3c7b-6b13-4603-9a80-7b07a035d22f" Jul 06 22:09:59.333989 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:59.333952 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-shdh8" Jul 06 22:09:59.334430 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:59.333956 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdbr6" Jul 06 22:09:59.334430 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:59.334083 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-shdh8" podUID="838f4166-d0dc-478f-aa98-4e1fd7ad300a" Jul 06 22:09:59.334430 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:09:59.333952 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zt94q" Jul 06 22:09:59.334430 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:59.334129 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdbr6" podUID="852d3c7b-6b13-4603-9a80-7b07a035d22f" Jul 06 22:09:59.334430 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:09:59.334230 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zt94q" podUID="d681bb3e-9bb1-4705-98cb-fd2cd5dc7063" Jul 06 22:10:01.335317 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:01.335279 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-shdh8" Jul 06 22:10:01.335699 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:01.335421 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zt94q" Jul 06 22:10:01.335699 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:01.335455 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdbr6" Jul 06 22:10:01.335699 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:10:01.335485 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-shdh8" podUID="838f4166-d0dc-478f-aa98-4e1fd7ad300a" Jul 06 22:10:01.335699 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:10:01.335536 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdbr6" podUID="852d3c7b-6b13-4603-9a80-7b07a035d22f" Jul 06 22:10:01.335699 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:10:01.335619 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zt94q" podUID="d681bb3e-9bb1-4705-98cb-fd2cd5dc7063" Jul 06 22:10:03.334159 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:03.334114 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdbr6" Jul 06 22:10:03.334159 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:03.334155 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zt94q" Jul 06 22:10:03.334819 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:03.334231 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-shdh8" Jul 06 22:10:03.334819 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:10:03.334253 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdbr6" podUID="852d3c7b-6b13-4603-9a80-7b07a035d22f" Jul 06 22:10:03.334819 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:10:03.334305 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-shdh8" podUID="838f4166-d0dc-478f-aa98-4e1fd7ad300a" Jul 06 22:10:03.334819 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:10:03.334355 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zt94q" podUID="d681bb3e-9bb1-4705-98cb-fd2cd5dc7063" Jul 06 22:10:05.334119 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:05.334089 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdbr6" Jul 06 22:10:05.334503 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:05.334089 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-shdh8" Jul 06 22:10:05.334503 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:10:05.334215 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdbr6" podUID="852d3c7b-6b13-4603-9a80-7b07a035d22f" Jul 06 22:10:05.334503 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:05.334089 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zt94q" Jul 06 22:10:05.334503 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:10:05.334275 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-shdh8" podUID="838f4166-d0dc-478f-aa98-4e1fd7ad300a" Jul 06 22:10:05.334503 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:10:05.334326 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zt94q" podUID="d681bb3e-9bb1-4705-98cb-fd2cd5dc7063" Jul 06 22:10:06.911053 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:06.911024 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-71.ec2.internal" event="NodeReady" Jul 06 22:10:06.911449 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:06.911168 2555 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 06 22:10:06.961666 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:06.961630 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xj686"] Jul 06 22:10:06.982761 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:06.982720 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9xdn4"] Jul 06 22:10:06.982936 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:06.982857 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xj686" Jul 06 22:10:06.985484 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:06.985464 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ddjw7\"" Jul 06 22:10:06.985855 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:06.985840 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 06 22:10:06.986058 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:06.986032 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 06 22:10:06.986175 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:06.986111 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xj686"] Jul 06 22:10:06.986175 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:06.986150 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9xdn4"] Jul 06 22:10:06.986284 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:06.986229 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9xdn4" Jul 06 22:10:06.989426 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:06.989408 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 06 22:10:06.989642 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:06.989620 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 06 22:10:06.989722 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:06.989679 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 06 22:10:06.990031 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:06.989998 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-fbkk6\"" Jul 06 22:10:07.069701 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:07.069660 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1dbc58da-974a-43a5-8106-008bccb0da53-cert\") pod \"ingress-canary-9xdn4\" (UID: \"1dbc58da-974a-43a5-8106-008bccb0da53\") " pod="openshift-ingress-canary/ingress-canary-9xdn4" Jul 06 22:10:07.069701 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:07.069702 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0dd52ea-c96a-480a-8c24-629502b393a3-tmp-dir\") pod \"dns-default-xj686\" (UID: \"d0dd52ea-c96a-480a-8c24-629502b393a3\") " pod="openshift-dns/dns-default-xj686" Jul 06 22:10:07.069972 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:07.069765 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8svsw\" (UniqueName: \"kubernetes.io/projected/d0dd52ea-c96a-480a-8c24-629502b393a3-kube-api-access-8svsw\") pod \"dns-default-xj686\" (UID: \"d0dd52ea-c96a-480a-8c24-629502b393a3\") " pod="openshift-dns/dns-default-xj686" Jul 06 22:10:07.069972 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:07.069849 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d0dd52ea-c96a-480a-8c24-629502b393a3-metrics-tls\") pod \"dns-default-xj686\" (UID: \"d0dd52ea-c96a-480a-8c24-629502b393a3\") " pod="openshift-dns/dns-default-xj686" Jul 06 22:10:07.069972 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:07.069875 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45xhj\" (UniqueName: \"kubernetes.io/projected/1dbc58da-974a-43a5-8106-008bccb0da53-kube-api-access-45xhj\") pod \"ingress-canary-9xdn4\" (UID: \"1dbc58da-974a-43a5-8106-008bccb0da53\") " pod="openshift-ingress-canary/ingress-canary-9xdn4" Jul 06 22:10:07.069972 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:07.069911 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d0dd52ea-c96a-480a-8c24-629502b393a3-config-volume\") pod \"dns-default-xj686\" (UID: \"d0dd52ea-c96a-480a-8c24-629502b393a3\") " pod="openshift-dns/dns-default-xj686" Jul 06 22:10:07.070799 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:07.070777 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-m4k2h"] Jul 06 22:10:07.113609 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:07.113574 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-m4k2h"] Jul 06 22:10:07.113774 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:07.113696 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-m4k2h" Jul 06 22:10:07.116486 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:07.116465 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 06 22:10:07.116486 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:07.116466 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 06 22:10:07.116709 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:07.116505 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 06 22:10:07.116709 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:07.116660 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-66l9r\"" Jul 06 22:10:07.116807 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:07.116714 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 06 22:10:07.170556 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:07.170452 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8eb86aa0-0a0b-4de6-8302-1e0f557df2ba-data-volume\") pod \"insights-runtime-extractor-m4k2h\" (UID: \"8eb86aa0-0a0b-4de6-8302-1e0f557df2ba\") " pod="openshift-insights/insights-runtime-extractor-m4k2h" Jul 06 22:10:07.170556 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:07.170497 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d0dd52ea-c96a-480a-8c24-629502b393a3-metrics-tls\") pod \"dns-default-xj686\" (UID: \"d0dd52ea-c96a-480a-8c24-629502b393a3\") " pod="openshift-dns/dns-default-xj686" Jul 06 22:10:07.170556 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:07.170517 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45xhj\" (UniqueName: \"kubernetes.io/projected/1dbc58da-974a-43a5-8106-008bccb0da53-kube-api-access-45xhj\") pod \"ingress-canary-9xdn4\" (UID: \"1dbc58da-974a-43a5-8106-008bccb0da53\") " pod="openshift-ingress-canary/ingress-canary-9xdn4" Jul 06 22:10:07.170556 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:07.170553 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8eb86aa0-0a0b-4de6-8302-1e0f557df2ba-crio-socket\") pod \"insights-runtime-extractor-m4k2h\" (UID: \"8eb86aa0-0a0b-4de6-8302-1e0f557df2ba\") " pod="openshift-insights/insights-runtime-extractor-m4k2h" Jul 06 22:10:07.170880 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:07.170572 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d0dd52ea-c96a-480a-8c24-629502b393a3-config-volume\") pod \"dns-default-xj686\" (UID: \"d0dd52ea-c96a-480a-8c24-629502b393a3\") " pod="openshift-dns/dns-default-xj686" Jul 06 22:10:07.170880 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:07.170590 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8eb86aa0-0a0b-4de6-8302-1e0f557df2ba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m4k2h\" (UID: \"8eb86aa0-0a0b-4de6-8302-1e0f557df2ba\") " pod="openshift-insights/insights-runtime-extractor-m4k2h" Jul 06 22:10:07.170880 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:07.170634 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1dbc58da-974a-43a5-8106-008bccb0da53-cert\") pod \"ingress-canary-9xdn4\" (UID: \"1dbc58da-974a-43a5-8106-008bccb0da53\") " pod="openshift-ingress-canary/ingress-canary-9xdn4" Jul 06 22:10:07.170880 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:07.170651 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0dd52ea-c96a-480a-8c24-629502b393a3-tmp-dir\") pod \"dns-default-xj686\" (UID: \"d0dd52ea-c96a-480a-8c24-629502b393a3\") " pod="openshift-dns/dns-default-xj686" Jul 06 22:10:07.170880 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:07.170673 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8svsw\" (UniqueName: \"kubernetes.io/projected/d0dd52ea-c96a-480a-8c24-629502b393a3-kube-api-access-8svsw\") pod \"dns-default-xj686\" (UID: \"d0dd52ea-c96a-480a-8c24-629502b393a3\") " pod="openshift-dns/dns-default-xj686" Jul 06 22:10:07.170880 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:07.170699 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8eb86aa0-0a0b-4de6-8302-1e0f557df2ba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m4k2h\" (UID: \"8eb86aa0-0a0b-4de6-8302-1e0f557df2ba\") " pod="openshift-insights/insights-runtime-extractor-m4k2h" Jul 06 22:10:07.170880 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:07.170723 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stgc5\" (UniqueName: \"kubernetes.io/projected/8eb86aa0-0a0b-4de6-8302-1e0f557df2ba-kube-api-access-stgc5\") pod \"insights-runtime-extractor-m4k2h\" (UID: \"8eb86aa0-0a0b-4de6-8302-1e0f557df2ba\") " pod="openshift-insights/insights-runtime-extractor-m4k2h" Jul 06 22:10:07.171142 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:07.171000 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0dd52ea-c96a-480a-8c24-629502b393a3-tmp-dir\") pod \"dns-default-xj686\" (UID: \"d0dd52ea-c96a-480a-8c24-629502b393a3\") " pod="openshift-dns/dns-default-xj686" Jul 06 22:10:07.171254 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:07.171233 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d0dd52ea-c96a-480a-8c24-629502b393a3-config-volume\") pod \"dns-default-xj686\" (UID: \"d0dd52ea-c96a-480a-8c24-629502b393a3\") " pod="openshift-dns/dns-default-xj686" Jul 06 22:10:07.174507 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:07.174481 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d0dd52ea-c96a-480a-8c24-629502b393a3-metrics-tls\") pod \"dns-default-xj686\" (UID: \"d0dd52ea-c96a-480a-8c24-629502b393a3\") " pod="openshift-dns/dns-default-xj686" Jul 06 22:10:07.174507 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:07.174507 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1dbc58da-974a-43a5-8106-008bccb0da53-cert\") pod \"ingress-canary-9xdn4\" (UID: \"1dbc58da-974a-43a5-8106-008bccb0da53\") " pod="openshift-ingress-canary/ingress-canary-9xdn4" Jul 06 22:10:07.179079 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:07.179054 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45xhj\" (UniqueName: \"kubernetes.io/projected/1dbc58da-974a-43a5-8106-008bccb0da53-kube-api-access-45xhj\") pod \"ingress-canary-9xdn4\" (UID: \"1dbc58da-974a-43a5-8106-008bccb0da53\") " pod="openshift-ingress-canary/ingress-canary-9xdn4" Jul 06 22:10:07.179189 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:07.179106 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8svsw\" (UniqueName: \"kubernetes.io/projected/d0dd52ea-c96a-480a-8c24-629502b393a3-kube-api-access-8svsw\") pod \"dns-default-xj686\" (UID: \"d0dd52ea-c96a-480a-8c24-629502b393a3\") " pod="openshift-dns/dns-default-xj686" Jul 06 22:10:07.271527 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:07.271490 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8eb86aa0-0a0b-4de6-8302-1e0f557df2ba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m4k2h\" (UID: \"8eb86aa0-0a0b-4de6-8302-1e0f557df2ba\") " pod="openshift-insights/insights-runtime-extractor-m4k2h" Jul 06 22:10:07.271527 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:07.271531 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stgc5\" (UniqueName: \"kubernetes.io/projected/8eb86aa0-0a0b-4de6-8302-1e0f557df2ba-kube-api-access-stgc5\") pod \"insights-runtime-extractor-m4k2h\" (UID: \"8eb86aa0-0a0b-4de6-8302-1e0f557df2ba\") " pod="openshift-insights/insights-runtime-extractor-m4k2h" Jul 06 22:10:07.271750 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:07.271645 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8eb86aa0-0a0b-4de6-8302-1e0f557df2ba-data-volume\") pod \"insights-runtime-extractor-m4k2h\" (UID: \"8eb86aa0-0a0b-4de6-8302-1e0f557df2ba\") " pod="openshift-insights/insights-runtime-extractor-m4k2h" Jul 06 22:10:07.271750 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:07.271688 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8eb86aa0-0a0b-4de6-8302-1e0f557df2ba-crio-socket\") pod \"insights-runtime-extractor-m4k2h\" (UID: \"8eb86aa0-0a0b-4de6-8302-1e0f557df2ba\") " pod="openshift-insights/insights-runtime-extractor-m4k2h" Jul 06 22:10:07.271750 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:07.271718 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8eb86aa0-0a0b-4de6-8302-1e0f557df2ba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m4k2h\" (UID: \"8eb86aa0-0a0b-4de6-8302-1e0f557df2ba\") " pod="openshift-insights/insights-runtime-extractor-m4k2h" Jul 06 22:10:07.271933 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:07.271909 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8eb86aa0-0a0b-4de6-8302-1e0f557df2ba-data-volume\") pod \"insights-runtime-extractor-m4k2h\" (UID: \"8eb86aa0-0a0b-4de6-8302-1e0f557df2ba\") " pod="openshift-insights/insights-runtime-extractor-m4k2h" Jul 06 22:10:07.271985 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:07.271909 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8eb86aa0-0a0b-4de6-8302-1e0f557df2ba-crio-socket\") pod \"insights-runtime-extractor-m4k2h\" (UID: \"8eb86aa0-0a0b-4de6-8302-1e0f557df2ba\") " pod="openshift-insights/insights-runtime-extractor-m4k2h" Jul 06 22:10:07.272088 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:07.272064 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8eb86aa0-0a0b-4de6-8302-1e0f557df2ba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m4k2h\" (UID: \"8eb86aa0-0a0b-4de6-8302-1e0f557df2ba\") " pod="openshift-insights/insights-runtime-extractor-m4k2h" Jul 06 22:10:07.273890 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:07.273870 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8eb86aa0-0a0b-4de6-8302-1e0f557df2ba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m4k2h\" (UID: \"8eb86aa0-0a0b-4de6-8302-1e0f557df2ba\") " pod="openshift-insights/insights-runtime-extractor-m4k2h" Jul 06 22:10:07.280448 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:07.280423 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stgc5\" (UniqueName: \"kubernetes.io/projected/8eb86aa0-0a0b-4de6-8302-1e0f557df2ba-kube-api-access-stgc5\") pod \"insights-runtime-extractor-m4k2h\" (UID: \"8eb86aa0-0a0b-4de6-8302-1e0f557df2ba\") " pod="openshift-insights/insights-runtime-extractor-m4k2h" Jul 06 22:10:07.294422 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:07.294390 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xj686" Jul 06 22:10:07.299486 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:07.299468 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9xdn4" Jul 06 22:10:07.333803 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:07.333779 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zt94q" Jul 06 22:10:07.333986 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:07.333779 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdbr6" Jul 06 22:10:07.333986 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:07.333906 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-shdh8" Jul 06 22:10:07.336901 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:07.336882 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 22:10:07.337613 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:07.337594 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 22:10:07.337613 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:07.337610 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pk242\"" Jul 06 22:10:07.337613 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:07.337595 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 22:10:07.337846 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:07.337616 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dftmz\"" Jul 06 22:10:07.339443 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:07.339427 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 22:10:07.422615 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:07.422549 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-m4k2h" Jul 06 22:10:07.506264 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:07.506225 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xj686"] Jul 06 22:10:07.522899 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:07.522868 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9xdn4"] Jul 06 22:10:07.526478 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:10:07.526438 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dbc58da_974a_43a5_8106_008bccb0da53.slice/crio-c0d98b90c906a80a0e51212b480247e10d4f9518849a41c6049cbc5ed5982ff7 WatchSource:0}: Error finding container c0d98b90c906a80a0e51212b480247e10d4f9518849a41c6049cbc5ed5982ff7: Status 404 returned error can't find the container with id c0d98b90c906a80a0e51212b480247e10d4f9518849a41c6049cbc5ed5982ff7 Jul 06 22:10:07.577413 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:07.577110 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-m4k2h"] Jul 06 22:10:07.582055 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:10:07.582025 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eb86aa0_0a0b_4de6_8302_1e0f557df2ba.slice/crio-e3044977d16dbd91c63ca14a89317e3380c983f47d9d2b8042a379aa533d6424 WatchSource:0}: Error finding container e3044977d16dbd91c63ca14a89317e3380c983f47d9d2b8042a379aa533d6424: Status 404 returned error can't find the container with id e3044977d16dbd91c63ca14a89317e3380c983f47d9d2b8042a379aa533d6424 Jul 06 22:10:07.690072 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:07.689970 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xj686" event={"ID":"d0dd52ea-c96a-480a-8c24-629502b393a3","Type":"ContainerStarted","Data":"be4d0e60864bf5ee63c55034e0b2bd460a5e360581d6cbe53e66b4d950d78f11"} Jul 06 22:10:07.691223 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:07.691200 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m4k2h" event={"ID":"8eb86aa0-0a0b-4de6-8302-1e0f557df2ba","Type":"ContainerStarted","Data":"e3044977d16dbd91c63ca14a89317e3380c983f47d9d2b8042a379aa533d6424"} Jul 06 22:10:07.692312 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:07.692291 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9xdn4" event={"ID":"1dbc58da-974a-43a5-8106-008bccb0da53","Type":"ContainerStarted","Data":"c0d98b90c906a80a0e51212b480247e10d4f9518849a41c6049cbc5ed5982ff7"} Jul 06 22:10:10.189259 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:10.189223 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-dn48d"] Jul 06 22:10:10.223892 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:10.223853 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-dn48d"] Jul 06 22:10:10.224096 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:10.224035 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-dn48d" Jul 06 22:10:10.226704 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:10.226673 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 06 22:10:10.226828 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:10.226679 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-mwz4z\"" Jul 06 22:10:10.295269 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:10.295230 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a1e09957-d459-471b-a220-d04d9c960228-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-dn48d\" (UID: \"a1e09957-d459-471b-a220-d04d9c960228\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-dn48d" Jul 06 22:10:10.396192 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:10.396136 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a1e09957-d459-471b-a220-d04d9c960228-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-dn48d\" (UID: \"a1e09957-d459-471b-a220-d04d9c960228\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-dn48d" Jul 06 22:10:10.399020 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:10.398954 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a1e09957-d459-471b-a220-d04d9c960228-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-dn48d\" (UID: \"a1e09957-d459-471b-a220-d04d9c960228\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-dn48d" Jul 06 22:10:10.533887 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:10.533797 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-dn48d" Jul 06 22:10:10.571432 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:10.571398 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7cf8c5f7d7-xtpbg"] Jul 06 22:10:10.574590 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:10.574563 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cf8c5f7d7-xtpbg" Jul 06 22:10:10.577774 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:10.577744 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 06 22:10:10.577938 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:10.577824 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-wtp2d\"" Jul 06 22:10:10.579073 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:10.579034 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 06 22:10:10.579073 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:10.579050 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 06 22:10:10.579073 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:10.579073 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 06 22:10:10.579276 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:10.579050 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 06 22:10:10.579636 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:10.579463 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 06 22:10:10.584480 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:10.584458 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 06 22:10:10.598154 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:10.598124 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0-service-ca\") pod \"console-7cf8c5f7d7-xtpbg\" (UID: \"9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0\") " pod="openshift-console/console-7cf8c5f7d7-xtpbg" Jul 06 22:10:10.598154 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:10.598158 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhbxz\" (UniqueName: \"kubernetes.io/projected/9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0-kube-api-access-jhbxz\") pod \"console-7cf8c5f7d7-xtpbg\" (UID: \"9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0\") " pod="openshift-console/console-7cf8c5f7d7-xtpbg" Jul 06 22:10:10.598360 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:10.598190 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0-console-oauth-config\") pod \"console-7cf8c5f7d7-xtpbg\" (UID: \"9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0\") " pod="openshift-console/console-7cf8c5f7d7-xtpbg" Jul 06 22:10:10.598360 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:10.598237 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0-oauth-serving-cert\") pod \"console-7cf8c5f7d7-xtpbg\" (UID: \"9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0\") " pod="openshift-console/console-7cf8c5f7d7-xtpbg" Jul 06 22:10:10.598360 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:10.598276 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0-console-config\") pod \"console-7cf8c5f7d7-xtpbg\" (UID: \"9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0\") " pod="openshift-console/console-7cf8c5f7d7-xtpbg" Jul 06 22:10:10.598360 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:10.598300 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0-console-serving-cert\") pod \"console-7cf8c5f7d7-xtpbg\" (UID: \"9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0\") " pod="openshift-console/console-7cf8c5f7d7-xtpbg" Jul 06 22:10:10.604252 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:10.604223 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7cf8c5f7d7-xtpbg"] Jul 06 22:10:10.698648 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:10.698615 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0-service-ca\") pod \"console-7cf8c5f7d7-xtpbg\" (UID: \"9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0\") " pod="openshift-console/console-7cf8c5f7d7-xtpbg" Jul 06 22:10:10.698648 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:10.698647 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhbxz\" (UniqueName: \"kubernetes.io/projected/9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0-kube-api-access-jhbxz\") pod \"console-7cf8c5f7d7-xtpbg\" (UID: \"9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0\") " pod="openshift-console/console-7cf8c5f7d7-xtpbg" Jul 06 22:10:10.698648 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:10.698669 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0-console-oauth-config\") pod \"console-7cf8c5f7d7-xtpbg\" (UID: \"9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0\") " pod="openshift-console/console-7cf8c5f7d7-xtpbg" Jul 06 22:10:10.698927 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:10.698690 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0-oauth-serving-cert\") pod \"console-7cf8c5f7d7-xtpbg\" (UID: \"9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0\") " pod="openshift-console/console-7cf8c5f7d7-xtpbg" Jul 06 22:10:10.699110 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:10.699084 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0-console-config\") pod \"console-7cf8c5f7d7-xtpbg\" (UID: \"9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0\") " pod="openshift-console/console-7cf8c5f7d7-xtpbg" Jul 06 22:10:10.699246 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:10.699150 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0-console-serving-cert\") pod \"console-7cf8c5f7d7-xtpbg\" (UID: \"9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0\") " pod="openshift-console/console-7cf8c5f7d7-xtpbg" Jul 06 22:10:10.699592 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:10.699541 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0-service-ca\") pod \"console-7cf8c5f7d7-xtpbg\" (UID: \"9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0\") " pod="openshift-console/console-7cf8c5f7d7-xtpbg" Jul 06 22:10:10.699712 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:10.699584 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0-oauth-serving-cert\") pod \"console-7cf8c5f7d7-xtpbg\" (UID: \"9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0\") " pod="openshift-console/console-7cf8c5f7d7-xtpbg" Jul 06 22:10:10.699955 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:10.699929 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0-console-config\") pod \"console-7cf8c5f7d7-xtpbg\" (UID: \"9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0\") " pod="openshift-console/console-7cf8c5f7d7-xtpbg" Jul 06 22:10:10.702039 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:10.702001 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0-console-oauth-config\") pod \"console-7cf8c5f7d7-xtpbg\" (UID: \"9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0\") " pod="openshift-console/console-7cf8c5f7d7-xtpbg" Jul 06 22:10:10.702129 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:10.702089 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0-console-serving-cert\") pod \"console-7cf8c5f7d7-xtpbg\" (UID: \"9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0\") " pod="openshift-console/console-7cf8c5f7d7-xtpbg" Jul 06 22:10:10.707848 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:10.707797 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhbxz\" (UniqueName: \"kubernetes.io/projected/9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0-kube-api-access-jhbxz\") pod \"console-7cf8c5f7d7-xtpbg\" (UID: \"9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0\") " pod="openshift-console/console-7cf8c5f7d7-xtpbg" Jul 06 22:10:10.885753 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:10.885670 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cf8c5f7d7-xtpbg" Jul 06 22:10:12.772498 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:12.772394 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-dn48d"] Jul 06 22:10:12.778668 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:10:12.778632 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1e09957_d459_471b_a220_d04d9c960228.slice/crio-7c6d890963e8565016c68f2e68d10fcc62519650e4b83ca3820684505029ea2e WatchSource:0}: Error finding container 7c6d890963e8565016c68f2e68d10fcc62519650e4b83ca3820684505029ea2e: Status 404 returned error can't find the container with id 7c6d890963e8565016c68f2e68d10fcc62519650e4b83ca3820684505029ea2e Jul 06 22:10:12.791356 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:12.791277 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7cf8c5f7d7-xtpbg"] Jul 06 22:10:12.797800 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:10:12.797743 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9854c5a1_8d22_4dc2_9c9f_fa6ffb2d0bb0.slice/crio-a7fc64452237b315f7a9216b9d688d32112c141123d3d5d84ca90e51db17b6f0 WatchSource:0}: Error finding container a7fc64452237b315f7a9216b9d688d32112c141123d3d5d84ca90e51db17b6f0: Status 404 returned error can't find the container with id a7fc64452237b315f7a9216b9d688d32112c141123d3d5d84ca90e51db17b6f0 Jul 06 22:10:13.640623 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:13.639743 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-p7mq8" Jul 06 22:10:13.713389 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:13.713348 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-dn48d" event={"ID":"a1e09957-d459-471b-a220-d04d9c960228","Type":"ContainerStarted","Data":"7c6d890963e8565016c68f2e68d10fcc62519650e4b83ca3820684505029ea2e"} Jul 06 22:10:13.715209 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:13.715183 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9xdn4" event={"ID":"1dbc58da-974a-43a5-8106-008bccb0da53","Type":"ContainerStarted","Data":"0401cb43aee7e81844e3ec04dcd0db59055f9cab5f0349096c78650ba7e6b4bc"} Jul 06 22:10:13.716816 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:13.716791 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cf8c5f7d7-xtpbg" event={"ID":"9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0","Type":"ContainerStarted","Data":"a7fc64452237b315f7a9216b9d688d32112c141123d3d5d84ca90e51db17b6f0"} Jul 06 22:10:13.718943 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:13.718914 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m4k2h" event={"ID":"8eb86aa0-0a0b-4de6-8302-1e0f557df2ba","Type":"ContainerStarted","Data":"813c6f8becec40af02b7e9dc6ae42add8b89202e37343c7608c064bfff76446e"} Jul 06 22:10:13.719098 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:13.718970 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m4k2h" event={"ID":"8eb86aa0-0a0b-4de6-8302-1e0f557df2ba","Type":"ContainerStarted","Data":"5ad226b289efb7d06e592583e8610abbbb50d223ecf16a0325e8f36a38eacaf6"} Jul 06 22:10:13.721429 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:13.721346 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xj686" event={"ID":"d0dd52ea-c96a-480a-8c24-629502b393a3","Type":"ContainerStarted","Data":"522715d3d7b2dd91ebe77844cb5b00d191f1a24a6c241d867f4c3dab42750fc8"} Jul 06 22:10:13.721429 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:13.721392 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xj686" event={"ID":"d0dd52ea-c96a-480a-8c24-629502b393a3","Type":"ContainerStarted","Data":"7aafc8a5f40a8879242040a75b1d197b7dc59bdab1f256caed05acfaebcc3312"} Jul 06 22:10:13.721593 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:13.721439 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xj686" Jul 06 22:10:13.733577 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:13.732980 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9xdn4" podStartSLOduration=2.642115517 podStartE2EDuration="7.732962897s" totalImagesPullingTime="5.09084738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:10:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:10:13.73243806 +0000 UTC m=+65.032149911" watchObservedRunningTime="2026-07-06 22:10:13.732962897 +0000 UTC m=+65.032674724" Jul 06 22:10:14.727351 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:14.727311 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-dn48d" event={"ID":"a1e09957-d459-471b-a220-d04d9c960228","Type":"ContainerStarted","Data":"f2b37772a7bb39c712eab733b899dd84140135dac0bf5926244532700c8786c9"} Jul 06 22:10:14.745986 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:14.745803 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-xj686" podStartSLOduration=3.640433357 podStartE2EDuration="8.745778368s" totalImagesPullingTime="5.105345011s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:10:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:10:13.754082874 +0000 UTC m=+65.053794726" watchObservedRunningTime="2026-07-06 22:10:14.745778368 +0000 UTC m=+66.045490221" Jul 06 22:10:14.746203 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:14.746062 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-dn48d" podStartSLOduration=3.408390154 podStartE2EDuration="4.745983558s" totalImagesPullingTime="1.337593404s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:10:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:10:14.745858725 +0000 UTC m=+66.045570558" watchObservedRunningTime="2026-07-06 22:10:14.745983558 +0000 UTC m=+66.045695408" Jul 06 22:10:15.338991 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:15.338958 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d681bb3e-9bb1-4705-98cb-fd2cd5dc7063-original-pull-secret\") pod \"global-pull-secret-syncer-zt94q\" (UID: \"d681bb3e-9bb1-4705-98cb-fd2cd5dc7063\") " pod="kube-system/global-pull-secret-syncer-zt94q" Jul 06 22:10:15.339202 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:15.339000 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/838f4166-d0dc-478f-aa98-4e1fd7ad300a-metrics-certs\") pod \"network-metrics-daemon-shdh8\" (UID: \"838f4166-d0dc-478f-aa98-4e1fd7ad300a\") " pod="openshift-multus/network-metrics-daemon-shdh8" Jul 06 22:10:15.341689 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:15.341663 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 22:10:15.342819 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:15.342798 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 22:10:15.352384 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:15.352360 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d681bb3e-9bb1-4705-98cb-fd2cd5dc7063-original-pull-secret\") pod \"global-pull-secret-syncer-zt94q\" (UID: \"d681bb3e-9bb1-4705-98cb-fd2cd5dc7063\") " pod="kube-system/global-pull-secret-syncer-zt94q" Jul 06 22:10:15.352509 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:15.352367 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/838f4166-d0dc-478f-aa98-4e1fd7ad300a-metrics-certs\") pod \"network-metrics-daemon-shdh8\" (UID: \"838f4166-d0dc-478f-aa98-4e1fd7ad300a\") " pod="openshift-multus/network-metrics-daemon-shdh8" Jul 06 22:10:15.440327 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:15.440288 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4z7z\" (UniqueName: \"kubernetes.io/projected/852d3c7b-6b13-4603-9a80-7b07a035d22f-kube-api-access-z4z7z\") pod \"network-check-target-vdbr6\" (UID: \"852d3c7b-6b13-4603-9a80-7b07a035d22f\") " pod="openshift-network-diagnostics/network-check-target-vdbr6" Jul 06 22:10:15.443191 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:15.443164 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 22:10:15.444232 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:15.444204 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zt94q" Jul 06 22:10:15.453447 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:15.453426 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 22:10:15.459897 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:15.459876 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dftmz\"" Jul 06 22:10:15.464320 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:15.464293 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4z7z\" (UniqueName: \"kubernetes.io/projected/852d3c7b-6b13-4603-9a80-7b07a035d22f-kube-api-access-z4z7z\") pod \"network-check-target-vdbr6\" (UID: \"852d3c7b-6b13-4603-9a80-7b07a035d22f\") " pod="openshift-network-diagnostics/network-check-target-vdbr6" Jul 06 22:10:15.467977 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:15.467935 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-shdh8" Jul 06 22:10:15.731018 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:15.730969 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-dn48d" Jul 06 22:10:15.736293 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:15.736264 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-dn48d" Jul 06 22:10:15.755229 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:15.755191 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pk242\"" Jul 06 22:10:15.762688 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:15.762663 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdbr6" Jul 06 22:10:16.257668 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:16.257632 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-kcp9n"] Jul 06 22:10:16.279705 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:16.279670 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-kcp9n"] Jul 06 22:10:16.279879 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:16.279831 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-kcp9n" Jul 06 22:10:16.282679 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:16.282651 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 06 22:10:16.283947 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:16.283881 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-4nppw\"" Jul 06 22:10:16.283947 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:16.283902 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 06 22:10:16.283947 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:16.283931 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 06 22:10:16.283947 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:16.283954 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 06 22:10:16.284297 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:16.284117 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 06 22:10:16.347472 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:16.347437 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/acbf1ef4-43b4-413a-bbf9-49485cfa7cba-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-kcp9n\" (UID: \"acbf1ef4-43b4-413a-bbf9-49485cfa7cba\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-kcp9n" Jul 06 22:10:16.347637 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:16.347504 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/acbf1ef4-43b4-413a-bbf9-49485cfa7cba-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-kcp9n\" (UID: \"acbf1ef4-43b4-413a-bbf9-49485cfa7cba\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-kcp9n" Jul 06 22:10:16.347637 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:16.347534 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/acbf1ef4-43b4-413a-bbf9-49485cfa7cba-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-kcp9n\" (UID: \"acbf1ef4-43b4-413a-bbf9-49485cfa7cba\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-kcp9n" Jul 06 22:10:16.347637 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:16.347567 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmn7k\" (UniqueName: \"kubernetes.io/projected/acbf1ef4-43b4-413a-bbf9-49485cfa7cba-kube-api-access-gmn7k\") pod \"prometheus-operator-567f7b8b7c-kcp9n\" (UID: \"acbf1ef4-43b4-413a-bbf9-49485cfa7cba\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-kcp9n" Jul 06 22:10:16.448066 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:16.448028 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/acbf1ef4-43b4-413a-bbf9-49485cfa7cba-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-kcp9n\" (UID: \"acbf1ef4-43b4-413a-bbf9-49485cfa7cba\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-kcp9n" Jul 06 22:10:16.448066 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:16.448078 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/acbf1ef4-43b4-413a-bbf9-49485cfa7cba-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-kcp9n\" (UID: \"acbf1ef4-43b4-413a-bbf9-49485cfa7cba\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-kcp9n" Jul 06 22:10:16.448336 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:16.448133 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmn7k\" (UniqueName: \"kubernetes.io/projected/acbf1ef4-43b4-413a-bbf9-49485cfa7cba-kube-api-access-gmn7k\") pod \"prometheus-operator-567f7b8b7c-kcp9n\" (UID: \"acbf1ef4-43b4-413a-bbf9-49485cfa7cba\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-kcp9n" Jul 06 22:10:16.448336 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:16.448174 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/acbf1ef4-43b4-413a-bbf9-49485cfa7cba-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-kcp9n\" (UID: \"acbf1ef4-43b4-413a-bbf9-49485cfa7cba\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-kcp9n" Jul 06 22:10:16.448865 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:16.448839 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/acbf1ef4-43b4-413a-bbf9-49485cfa7cba-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-kcp9n\" (UID: \"acbf1ef4-43b4-413a-bbf9-49485cfa7cba\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-kcp9n" Jul 06 22:10:16.450830 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:16.450805 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/acbf1ef4-43b4-413a-bbf9-49485cfa7cba-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-kcp9n\" (UID: \"acbf1ef4-43b4-413a-bbf9-49485cfa7cba\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-kcp9n" Jul 06 22:10:16.450934 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:16.450832 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/acbf1ef4-43b4-413a-bbf9-49485cfa7cba-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-kcp9n\" (UID: \"acbf1ef4-43b4-413a-bbf9-49485cfa7cba\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-kcp9n" Jul 06 22:10:16.457629 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:16.457601 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmn7k\" (UniqueName: \"kubernetes.io/projected/acbf1ef4-43b4-413a-bbf9-49485cfa7cba-kube-api-access-gmn7k\") pod \"prometheus-operator-567f7b8b7c-kcp9n\" (UID: \"acbf1ef4-43b4-413a-bbf9-49485cfa7cba\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-kcp9n" Jul 06 22:10:16.591470 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:16.591381 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-kcp9n" Jul 06 22:10:17.534453 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:17.534421 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zt94q"] Jul 06 22:10:17.538263 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:10:17.538232 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd681bb3e_9bb1_4705_98cb_fd2cd5dc7063.slice/crio-b2485166d4df69e96a415df60d26fbfcd2b74c8b444a7621258b2e1642ccd029 WatchSource:0}: Error finding container b2485166d4df69e96a415df60d26fbfcd2b74c8b444a7621258b2e1642ccd029: Status 404 returned error can't find the container with id b2485166d4df69e96a415df60d26fbfcd2b74c8b444a7621258b2e1642ccd029 Jul 06 22:10:17.742174 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:17.742130 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m4k2h" event={"ID":"8eb86aa0-0a0b-4de6-8302-1e0f557df2ba","Type":"ContainerStarted","Data":"d02064113f292fefd66db99480b33f19cf15a1691681bca1637c10a1b3ee39ea"} Jul 06 22:10:17.744214 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:17.744184 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cf8c5f7d7-xtpbg" event={"ID":"9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0","Type":"ContainerStarted","Data":"346568f475d3dfc81965dfa5d37850e11930980584f85a2aadf6c158bc5c0105"} Jul 06 22:10:17.745618 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:17.745589 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zt94q" event={"ID":"d681bb3e-9bb1-4705-98cb-fd2cd5dc7063","Type":"ContainerStarted","Data":"b2485166d4df69e96a415df60d26fbfcd2b74c8b444a7621258b2e1642ccd029"} Jul 06 22:10:17.755381 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:17.755217 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-shdh8"] Jul 06 22:10:17.757951 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:10:17.757920 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod838f4166_d0dc_478f_aa98_4e1fd7ad300a.slice/crio-603b7f9a27228301e09a060794426d362fee0b959f452ac0cb4c742943d16b5c WatchSource:0}: Error finding container 603b7f9a27228301e09a060794426d362fee0b959f452ac0cb4c742943d16b5c: Status 404 returned error can't find the container with id 603b7f9a27228301e09a060794426d362fee0b959f452ac0cb4c742943d16b5c Jul 06 22:10:17.758222 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:17.758203 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vdbr6"] Jul 06 22:10:17.760989 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:17.760461 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-m4k2h" podStartSLOduration=1.129227969 podStartE2EDuration="10.760445333s" totalImagesPullingTime="9.631217364s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:10:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:10:17.759989851 +0000 UTC m=+69.059701701" watchObservedRunningTime="2026-07-06 22:10:17.760445333 +0000 UTC m=+69.060157183" Jul 06 22:10:17.761431 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:10:17.761404 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod852d3c7b_6b13_4603_9a80_7b07a035d22f.slice/crio-87f7426a6e921df02267d5193aeb9517153b9e343539580cc7eb4a9e010e0ae9 WatchSource:0}: Error finding container 87f7426a6e921df02267d5193aeb9517153b9e343539580cc7eb4a9e010e0ae9: Status 404 returned error can't find the container with id 87f7426a6e921df02267d5193aeb9517153b9e343539580cc7eb4a9e010e0ae9 Jul 06 22:10:17.762574 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:17.762444 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-kcp9n"] Jul 06 22:10:17.765049 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:10:17.765024 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacbf1ef4_43b4_413a_bbf9_49485cfa7cba.slice/crio-bccee668c20bc3cdf809148f59f2e46666d0780728235625a8d381aaf1c408e1 WatchSource:0}: Error finding container bccee668c20bc3cdf809148f59f2e46666d0780728235625a8d381aaf1c408e1: Status 404 returned error can't find the container with id bccee668c20bc3cdf809148f59f2e46666d0780728235625a8d381aaf1c408e1 Jul 06 22:10:17.778552 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:17.778506 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7cf8c5f7d7-xtpbg" podStartSLOduration=3.210997556 podStartE2EDuration="7.778489487s" totalImagesPullingTime="4.567491931s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:10:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:10:17.777725187 +0000 UTC m=+69.077437051" watchObservedRunningTime="2026-07-06 22:10:17.778489487 +0000 UTC m=+69.078201335" Jul 06 22:10:18.353183 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:18.352424 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-c4946bb79-24mzf"] Jul 06 22:10:18.355085 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:18.355059 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c4946bb79-24mzf" Jul 06 22:10:18.376765 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:18.376731 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c4946bb79-24mzf"] Jul 06 22:10:18.379329 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:18.379288 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 06 22:10:18.465382 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:18.465331 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3d9c3b90-9586-4687-a6ee-5424470f92d5-console-config\") pod \"console-c4946bb79-24mzf\" (UID: \"3d9c3b90-9586-4687-a6ee-5424470f92d5\") " pod="openshift-console/console-c4946bb79-24mzf" Jul 06 22:10:18.465560 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:18.465389 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3d9c3b90-9586-4687-a6ee-5424470f92d5-oauth-serving-cert\") pod \"console-c4946bb79-24mzf\" (UID: \"3d9c3b90-9586-4687-a6ee-5424470f92d5\") " pod="openshift-console/console-c4946bb79-24mzf" Jul 06 22:10:18.465560 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:18.465423 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3d9c3b90-9586-4687-a6ee-5424470f92d5-console-serving-cert\") pod \"console-c4946bb79-24mzf\" (UID: \"3d9c3b90-9586-4687-a6ee-5424470f92d5\") " pod="openshift-console/console-c4946bb79-24mzf" Jul 06 22:10:18.465560 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:18.465461 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwxj7\" (UniqueName: \"kubernetes.io/projected/3d9c3b90-9586-4687-a6ee-5424470f92d5-kube-api-access-hwxj7\") pod \"console-c4946bb79-24mzf\" (UID: \"3d9c3b90-9586-4687-a6ee-5424470f92d5\") " pod="openshift-console/console-c4946bb79-24mzf" Jul 06 22:10:18.465560 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:18.465503 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3d9c3b90-9586-4687-a6ee-5424470f92d5-console-oauth-config\") pod \"console-c4946bb79-24mzf\" (UID: \"3d9c3b90-9586-4687-a6ee-5424470f92d5\") " pod="openshift-console/console-c4946bb79-24mzf" Jul 06 22:10:18.465560 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:18.465546 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d9c3b90-9586-4687-a6ee-5424470f92d5-trusted-ca-bundle\") pod \"console-c4946bb79-24mzf\" (UID: \"3d9c3b90-9586-4687-a6ee-5424470f92d5\") " pod="openshift-console/console-c4946bb79-24mzf" Jul 06 22:10:18.465832 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:18.465587 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3d9c3b90-9586-4687-a6ee-5424470f92d5-service-ca\") pod \"console-c4946bb79-24mzf\" (UID: \"3d9c3b90-9586-4687-a6ee-5424470f92d5\") " pod="openshift-console/console-c4946bb79-24mzf" Jul 06 22:10:18.567066 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:18.566659 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3d9c3b90-9586-4687-a6ee-5424470f92d5-console-oauth-config\") pod \"console-c4946bb79-24mzf\" (UID: \"3d9c3b90-9586-4687-a6ee-5424470f92d5\") " pod="openshift-console/console-c4946bb79-24mzf" Jul 06 22:10:18.567066 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:18.566723 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d9c3b90-9586-4687-a6ee-5424470f92d5-trusted-ca-bundle\") pod \"console-c4946bb79-24mzf\" (UID: \"3d9c3b90-9586-4687-a6ee-5424470f92d5\") " pod="openshift-console/console-c4946bb79-24mzf" Jul 06 22:10:18.567066 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:18.566760 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3d9c3b90-9586-4687-a6ee-5424470f92d5-service-ca\") pod \"console-c4946bb79-24mzf\" (UID: \"3d9c3b90-9586-4687-a6ee-5424470f92d5\") " pod="openshift-console/console-c4946bb79-24mzf" Jul 06 22:10:18.567066 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:18.566786 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3d9c3b90-9586-4687-a6ee-5424470f92d5-console-config\") pod \"console-c4946bb79-24mzf\" (UID: \"3d9c3b90-9586-4687-a6ee-5424470f92d5\") " pod="openshift-console/console-c4946bb79-24mzf" Jul 06 22:10:18.567066 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:18.566813 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3d9c3b90-9586-4687-a6ee-5424470f92d5-oauth-serving-cert\") pod \"console-c4946bb79-24mzf\" (UID: \"3d9c3b90-9586-4687-a6ee-5424470f92d5\") " pod="openshift-console/console-c4946bb79-24mzf" Jul 06 22:10:18.567066 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:18.566843 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3d9c3b90-9586-4687-a6ee-5424470f92d5-console-serving-cert\") pod \"console-c4946bb79-24mzf\" (UID: \"3d9c3b90-9586-4687-a6ee-5424470f92d5\") " pod="openshift-console/console-c4946bb79-24mzf" Jul 06 22:10:18.567066 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:18.566876 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwxj7\" (UniqueName: \"kubernetes.io/projected/3d9c3b90-9586-4687-a6ee-5424470f92d5-kube-api-access-hwxj7\") pod \"console-c4946bb79-24mzf\" (UID: \"3d9c3b90-9586-4687-a6ee-5424470f92d5\") " pod="openshift-console/console-c4946bb79-24mzf" Jul 06 22:10:18.567842 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:18.567615 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3d9c3b90-9586-4687-a6ee-5424470f92d5-service-ca\") pod \"console-c4946bb79-24mzf\" (UID: \"3d9c3b90-9586-4687-a6ee-5424470f92d5\") " pod="openshift-console/console-c4946bb79-24mzf" Jul 06 22:10:18.567842 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:18.567802 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3d9c3b90-9586-4687-a6ee-5424470f92d5-console-config\") pod \"console-c4946bb79-24mzf\" (UID: \"3d9c3b90-9586-4687-a6ee-5424470f92d5\") " pod="openshift-console/console-c4946bb79-24mzf" Jul 06 22:10:18.569254 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:18.568573 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3d9c3b90-9586-4687-a6ee-5424470f92d5-oauth-serving-cert\") pod \"console-c4946bb79-24mzf\" (UID: \"3d9c3b90-9586-4687-a6ee-5424470f92d5\") " pod="openshift-console/console-c4946bb79-24mzf" Jul 06 22:10:18.569254 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:18.569215 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d9c3b90-9586-4687-a6ee-5424470f92d5-trusted-ca-bundle\") pod \"console-c4946bb79-24mzf\" (UID: \"3d9c3b90-9586-4687-a6ee-5424470f92d5\") " pod="openshift-console/console-c4946bb79-24mzf" Jul 06 22:10:18.571303 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:18.570577 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3d9c3b90-9586-4687-a6ee-5424470f92d5-console-oauth-config\") pod \"console-c4946bb79-24mzf\" (UID: \"3d9c3b90-9586-4687-a6ee-5424470f92d5\") " pod="openshift-console/console-c4946bb79-24mzf" Jul 06 22:10:18.571303 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:18.570743 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3d9c3b90-9586-4687-a6ee-5424470f92d5-console-serving-cert\") pod \"console-c4946bb79-24mzf\" (UID: \"3d9c3b90-9586-4687-a6ee-5424470f92d5\") " pod="openshift-console/console-c4946bb79-24mzf" Jul 06 22:10:18.577687 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:18.577619 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwxj7\" (UniqueName: \"kubernetes.io/projected/3d9c3b90-9586-4687-a6ee-5424470f92d5-kube-api-access-hwxj7\") pod \"console-c4946bb79-24mzf\" (UID: \"3d9c3b90-9586-4687-a6ee-5424470f92d5\") " pod="openshift-console/console-c4946bb79-24mzf" Jul 06 22:10:18.669430 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:18.669153 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c4946bb79-24mzf" Jul 06 22:10:18.753207 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:18.753154 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vdbr6" event={"ID":"852d3c7b-6b13-4603-9a80-7b07a035d22f","Type":"ContainerStarted","Data":"87f7426a6e921df02267d5193aeb9517153b9e343539580cc7eb4a9e010e0ae9"} Jul 06 22:10:18.754879 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:18.754843 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-kcp9n" event={"ID":"acbf1ef4-43b4-413a-bbf9-49485cfa7cba","Type":"ContainerStarted","Data":"bccee668c20bc3cdf809148f59f2e46666d0780728235625a8d381aaf1c408e1"} Jul 06 22:10:18.757167 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:18.757135 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-shdh8" event={"ID":"838f4166-d0dc-478f-aa98-4e1fd7ad300a","Type":"ContainerStarted","Data":"603b7f9a27228301e09a060794426d362fee0b959f452ac0cb4c742943d16b5c"} Jul 06 22:10:19.070852 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:19.070773 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c4946bb79-24mzf"] Jul 06 22:10:19.077727 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:10:19.077672 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d9c3b90_9586_4687_a6ee_5424470f92d5.slice/crio-fa28e70ce44beb4944b3099a9cb3df27d3e0da336885c5700977dbbf0cb08034 WatchSource:0}: Error finding container fa28e70ce44beb4944b3099a9cb3df27d3e0da336885c5700977dbbf0cb08034: Status 404 returned error can't find the container with id fa28e70ce44beb4944b3099a9cb3df27d3e0da336885c5700977dbbf0cb08034 Jul 06 22:10:19.765105 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:19.765062 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-kcp9n" event={"ID":"acbf1ef4-43b4-413a-bbf9-49485cfa7cba","Type":"ContainerStarted","Data":"f3e06c55b67d0aa4d16ab13884348acbc5f783290907a1715a08cbf60b0c1743"} Jul 06 22:10:19.768585 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:19.768551 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-shdh8" event={"ID":"838f4166-d0dc-478f-aa98-4e1fd7ad300a","Type":"ContainerStarted","Data":"0f2a156ea8dc60c62cbc358cd5aa351efc9daee59ed6588f4c1ad35c87df7129"} Jul 06 22:10:19.768738 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:19.768616 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-shdh8" event={"ID":"838f4166-d0dc-478f-aa98-4e1fd7ad300a","Type":"ContainerStarted","Data":"a57664496f766884552a97563652ef7279a2e9e1af46be440f9cb2e07f753a3a"} Jul 06 22:10:19.773884 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:19.773333 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c4946bb79-24mzf" event={"ID":"3d9c3b90-9586-4687-a6ee-5424470f92d5","Type":"ContainerStarted","Data":"c4ab27e490c205cfb57b00c58233a9cf93985bee23815eb1110da29e7043094f"} Jul 06 22:10:19.773884 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:19.773389 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c4946bb79-24mzf" event={"ID":"3d9c3b90-9586-4687-a6ee-5424470f92d5","Type":"ContainerStarted","Data":"fa28e70ce44beb4944b3099a9cb3df27d3e0da336885c5700977dbbf0cb08034"} Jul 06 22:10:19.787309 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:19.787245 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-shdh8" podStartSLOduration=69.646357376 podStartE2EDuration="1m10.787229435s" totalImagesPullingTime="1.140872059s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:09:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:10:19.786354274 +0000 UTC m=+71.086066125" watchObservedRunningTime="2026-07-06 22:10:19.787229435 +0000 UTC m=+71.086941284" Jul 06 22:10:19.807805 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:19.806750 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-c4946bb79-24mzf" podStartSLOduration=1.806728294 podStartE2EDuration="1.806728294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:10:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:10:19.80607676 +0000 UTC m=+71.105788608" watchObservedRunningTime="2026-07-06 22:10:19.806728294 +0000 UTC m=+71.106440148" Jul 06 22:10:20.780512 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:20.780464 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-kcp9n" event={"ID":"acbf1ef4-43b4-413a-bbf9-49485cfa7cba","Type":"ContainerStarted","Data":"f92f9d29f2e7fe9885d64eb8180905c8612a002201cd0945e085df8c25d3d627"} Jul 06 22:10:20.800473 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:20.800411 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-kcp9n" podStartSLOduration=2.904394563 podStartE2EDuration="4.800391585s" totalImagesPullingTime="1.895997022s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:10:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:10:20.800082678 +0000 UTC m=+72.099794512" watchObservedRunningTime="2026-07-06 22:10:20.800391585 +0000 UTC m=+72.100103440" Jul 06 22:10:20.886155 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:20.886116 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7cf8c5f7d7-xtpbg" Jul 06 22:10:20.886335 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:20.886200 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7cf8c5f7d7-xtpbg" Jul 06 22:10:20.891950 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:20.891921 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7cf8c5f7d7-xtpbg" Jul 06 22:10:21.789281 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:21.789232 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7cf8c5f7d7-xtpbg" Jul 06 22:10:22.634311 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.634281 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-st8th"] Jul 06 22:10:22.637273 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.637242 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-st8th" Jul 06 22:10:22.641090 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.641064 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 06 22:10:22.641574 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.641555 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-n2t56\"" Jul 06 22:10:22.641669 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.641555 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 06 22:10:22.654877 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.654783 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-st8th"] Jul 06 22:10:22.656046 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.655993 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-xfb9f"] Jul 06 22:10:22.658645 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.658622 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-xfb9f" Jul 06 22:10:22.661497 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.661473 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 06 22:10:22.662207 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.661971 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-dftpw\"" Jul 06 22:10:22.662301 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.662217 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 06 22:10:22.662595 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.662577 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 06 22:10:22.671211 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.671160 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-xfb9f"] Jul 06 22:10:22.700313 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.700278 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea64e052-1c86-42b5-a8fd-b3f7e2f042cb-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-st8th\" (UID: \"ea64e052-1c86-42b5-a8fd-b3f7e2f042cb\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-st8th" Jul 06 22:10:22.700459 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.700322 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2c5f15ee-59ba-4a1a-9e8b-5826acf7a438-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-xfb9f\" (UID: \"2c5f15ee-59ba-4a1a-9e8b-5826acf7a438\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-xfb9f" Jul 06 22:10:22.700459 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.700342 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ea64e052-1c86-42b5-a8fd-b3f7e2f042cb-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-st8th\" (UID: \"ea64e052-1c86-42b5-a8fd-b3f7e2f042cb\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-st8th" Jul 06 22:10:22.700459 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.700426 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7k9m\" (UniqueName: \"kubernetes.io/projected/2c5f15ee-59ba-4a1a-9e8b-5826acf7a438-kube-api-access-p7k9m\") pod \"kube-state-metrics-7c9dbb8b7d-xfb9f\" (UID: \"2c5f15ee-59ba-4a1a-9e8b-5826acf7a438\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-xfb9f" Jul 06 22:10:22.700583 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.700464 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c5f15ee-59ba-4a1a-9e8b-5826acf7a438-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-xfb9f\" (UID: \"2c5f15ee-59ba-4a1a-9e8b-5826acf7a438\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-xfb9f" Jul 06 22:10:22.700583 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.700492 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzlzs\" (UniqueName: \"kubernetes.io/projected/ea64e052-1c86-42b5-a8fd-b3f7e2f042cb-kube-api-access-rzlzs\") pod \"openshift-state-metrics-54cf96dfcf-st8th\" (UID: \"ea64e052-1c86-42b5-a8fd-b3f7e2f042cb\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-st8th" Jul 06 22:10:22.700583 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.700520 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea64e052-1c86-42b5-a8fd-b3f7e2f042cb-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-st8th\" (UID: \"ea64e052-1c86-42b5-a8fd-b3f7e2f042cb\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-st8th" Jul 06 22:10:22.700785 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.700583 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2c5f15ee-59ba-4a1a-9e8b-5826acf7a438-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-xfb9f\" (UID: \"2c5f15ee-59ba-4a1a-9e8b-5826acf7a438\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-xfb9f" Jul 06 22:10:22.700785 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.700644 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2c5f15ee-59ba-4a1a-9e8b-5826acf7a438-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-xfb9f\" (UID: \"2c5f15ee-59ba-4a1a-9e8b-5826acf7a438\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-xfb9f" Jul 06 22:10:22.700785 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.700678 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2c5f15ee-59ba-4a1a-9e8b-5826acf7a438-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-xfb9f\" (UID: \"2c5f15ee-59ba-4a1a-9e8b-5826acf7a438\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-xfb9f" Jul 06 22:10:22.703650 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.703626 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-p5xwh"] Jul 06 22:10:22.705643 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.705628 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-p5xwh" Jul 06 22:10:22.708307 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.708289 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 06 22:10:22.708490 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.708470 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 06 22:10:22.708574 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.708499 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-g2mfw\"" Jul 06 22:10:22.708642 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.708574 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 06 22:10:22.801109 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.801001 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7k9m\" (UniqueName: \"kubernetes.io/projected/2c5f15ee-59ba-4a1a-9e8b-5826acf7a438-kube-api-access-p7k9m\") pod \"kube-state-metrics-7c9dbb8b7d-xfb9f\" (UID: \"2c5f15ee-59ba-4a1a-9e8b-5826acf7a438\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-xfb9f" Jul 06 22:10:22.801736 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.801127 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c5f15ee-59ba-4a1a-9e8b-5826acf7a438-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-xfb9f\" (UID: \"2c5f15ee-59ba-4a1a-9e8b-5826acf7a438\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-xfb9f" Jul 06 22:10:22.801736 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.801156 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzlzs\" (UniqueName: \"kubernetes.io/projected/ea64e052-1c86-42b5-a8fd-b3f7e2f042cb-kube-api-access-rzlzs\") pod \"openshift-state-metrics-54cf96dfcf-st8th\" (UID: \"ea64e052-1c86-42b5-a8fd-b3f7e2f042cb\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-st8th" Jul 06 22:10:22.801736 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.801191 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea64e052-1c86-42b5-a8fd-b3f7e2f042cb-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-st8th\" (UID: \"ea64e052-1c86-42b5-a8fd-b3f7e2f042cb\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-st8th" Jul 06 22:10:22.801736 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.801221 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2c5f15ee-59ba-4a1a-9e8b-5826acf7a438-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-xfb9f\" (UID: \"2c5f15ee-59ba-4a1a-9e8b-5826acf7a438\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-xfb9f" Jul 06 22:10:22.801736 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.801262 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2c5f15ee-59ba-4a1a-9e8b-5826acf7a438-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-xfb9f\" (UID: \"2c5f15ee-59ba-4a1a-9e8b-5826acf7a438\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-xfb9f" Jul 06 22:10:22.801736 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.801292 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2c5f15ee-59ba-4a1a-9e8b-5826acf7a438-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-xfb9f\" (UID: \"2c5f15ee-59ba-4a1a-9e8b-5826acf7a438\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-xfb9f" Jul 06 22:10:22.801736 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.801322 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e1a16d17-2367-4bd9-834a-c2da55613ab4-node-exporter-tls\") pod \"node-exporter-p5xwh\" (UID: \"e1a16d17-2367-4bd9-834a-c2da55613ab4\") " pod="openshift-monitoring/node-exporter-p5xwh" Jul 06 22:10:22.801736 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.801353 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e1a16d17-2367-4bd9-834a-c2da55613ab4-node-exporter-wtmp\") pod \"node-exporter-p5xwh\" (UID: \"e1a16d17-2367-4bd9-834a-c2da55613ab4\") " pod="openshift-monitoring/node-exporter-p5xwh" Jul 06 22:10:22.801736 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.801397 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea64e052-1c86-42b5-a8fd-b3f7e2f042cb-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-st8th\" (UID: \"ea64e052-1c86-42b5-a8fd-b3f7e2f042cb\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-st8th" Jul 06 22:10:22.801736 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.801422 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e1a16d17-2367-4bd9-834a-c2da55613ab4-node-exporter-accelerators-collector-config\") pod \"node-exporter-p5xwh\" (UID: \"e1a16d17-2367-4bd9-834a-c2da55613ab4\") " pod="openshift-monitoring/node-exporter-p5xwh" Jul 06 22:10:22.801736 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.801450 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e1a16d17-2367-4bd9-834a-c2da55613ab4-metrics-client-ca\") pod \"node-exporter-p5xwh\" (UID: \"e1a16d17-2367-4bd9-834a-c2da55613ab4\") " pod="openshift-monitoring/node-exporter-p5xwh" Jul 06 22:10:22.801736 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.801507 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2c5f15ee-59ba-4a1a-9e8b-5826acf7a438-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-xfb9f\" (UID: \"2c5f15ee-59ba-4a1a-9e8b-5826acf7a438\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-xfb9f" Jul 06 22:10:22.801736 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.801533 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e1a16d17-2367-4bd9-834a-c2da55613ab4-node-exporter-textfile\") pod \"node-exporter-p5xwh\" (UID: \"e1a16d17-2367-4bd9-834a-c2da55613ab4\") " pod="openshift-monitoring/node-exporter-p5xwh" Jul 06 22:10:22.801736 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.801559 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srql2\" (UniqueName: \"kubernetes.io/projected/e1a16d17-2367-4bd9-834a-c2da55613ab4-kube-api-access-srql2\") pod \"node-exporter-p5xwh\" (UID: \"e1a16d17-2367-4bd9-834a-c2da55613ab4\") " pod="openshift-monitoring/node-exporter-p5xwh" Jul 06 22:10:22.801736 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.801587 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ea64e052-1c86-42b5-a8fd-b3f7e2f042cb-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-st8th\" (UID: \"ea64e052-1c86-42b5-a8fd-b3f7e2f042cb\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-st8th" Jul 06 22:10:22.802629 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.801613 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e1a16d17-2367-4bd9-834a-c2da55613ab4-root\") pod \"node-exporter-p5xwh\" (UID: \"e1a16d17-2367-4bd9-834a-c2da55613ab4\") " pod="openshift-monitoring/node-exporter-p5xwh" Jul 06 22:10:22.802629 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.801642 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e1a16d17-2367-4bd9-834a-c2da55613ab4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p5xwh\" (UID: \"e1a16d17-2367-4bd9-834a-c2da55613ab4\") " pod="openshift-monitoring/node-exporter-p5xwh" Jul 06 22:10:22.802629 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.801666 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e1a16d17-2367-4bd9-834a-c2da55613ab4-sys\") pod \"node-exporter-p5xwh\" (UID: \"e1a16d17-2367-4bd9-834a-c2da55613ab4\") " pod="openshift-monitoring/node-exporter-p5xwh" Jul 06 22:10:22.802629 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.801973 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea64e052-1c86-42b5-a8fd-b3f7e2f042cb-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-st8th\" (UID: \"ea64e052-1c86-42b5-a8fd-b3f7e2f042cb\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-st8th" Jul 06 22:10:22.803478 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.802711 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2c5f15ee-59ba-4a1a-9e8b-5826acf7a438-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-xfb9f\" (UID: \"2c5f15ee-59ba-4a1a-9e8b-5826acf7a438\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-xfb9f" Jul 06 22:10:22.803478 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.802950 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2c5f15ee-59ba-4a1a-9e8b-5826acf7a438-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-xfb9f\" (UID: \"2c5f15ee-59ba-4a1a-9e8b-5826acf7a438\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-xfb9f" Jul 06 22:10:22.803478 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.803000 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2c5f15ee-59ba-4a1a-9e8b-5826acf7a438-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-xfb9f\" (UID: \"2c5f15ee-59ba-4a1a-9e8b-5826acf7a438\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-xfb9f" Jul 06 22:10:22.803478 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:10:22.803110 2555 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 06 22:10:22.803478 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:10:22.803178 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea64e052-1c86-42b5-a8fd-b3f7e2f042cb-openshift-state-metrics-tls podName:ea64e052-1c86-42b5-a8fd-b3f7e2f042cb nodeName:}" failed. No retries permitted until 2026-07-06 22:10:23.303157336 +0000 UTC m=+74.602869179 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/ea64e052-1c86-42b5-a8fd-b3f7e2f042cb-openshift-state-metrics-tls") pod "openshift-state-metrics-54cf96dfcf-st8th" (UID: "ea64e052-1c86-42b5-a8fd-b3f7e2f042cb") : secret "openshift-state-metrics-tls" not found Jul 06 22:10:22.804463 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.804440 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2c5f15ee-59ba-4a1a-9e8b-5826acf7a438-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-xfb9f\" (UID: \"2c5f15ee-59ba-4a1a-9e8b-5826acf7a438\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-xfb9f" Jul 06 22:10:22.804540 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.804493 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c5f15ee-59ba-4a1a-9e8b-5826acf7a438-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-xfb9f\" (UID: \"2c5f15ee-59ba-4a1a-9e8b-5826acf7a438\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-xfb9f" Jul 06 22:10:22.805233 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.805213 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ea64e052-1c86-42b5-a8fd-b3f7e2f042cb-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-st8th\" (UID: \"ea64e052-1c86-42b5-a8fd-b3f7e2f042cb\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-st8th" Jul 06 22:10:22.810293 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.810266 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzlzs\" (UniqueName: \"kubernetes.io/projected/ea64e052-1c86-42b5-a8fd-b3f7e2f042cb-kube-api-access-rzlzs\") pod \"openshift-state-metrics-54cf96dfcf-st8th\" (UID: \"ea64e052-1c86-42b5-a8fd-b3f7e2f042cb\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-st8th" Jul 06 22:10:22.810293 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.810280 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7k9m\" (UniqueName: \"kubernetes.io/projected/2c5f15ee-59ba-4a1a-9e8b-5826acf7a438-kube-api-access-p7k9m\") pod \"kube-state-metrics-7c9dbb8b7d-xfb9f\" (UID: \"2c5f15ee-59ba-4a1a-9e8b-5826acf7a438\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-xfb9f" Jul 06 22:10:22.902450 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.902418 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e1a16d17-2367-4bd9-834a-c2da55613ab4-node-exporter-tls\") pod \"node-exporter-p5xwh\" (UID: \"e1a16d17-2367-4bd9-834a-c2da55613ab4\") " pod="openshift-monitoring/node-exporter-p5xwh" Jul 06 22:10:22.902591 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:10:22.902527 2555 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 06 22:10:22.902591 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:10:22.902583 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1a16d17-2367-4bd9-834a-c2da55613ab4-node-exporter-tls podName:e1a16d17-2367-4bd9-834a-c2da55613ab4 nodeName:}" failed. No retries permitted until 2026-07-06 22:10:23.402567338 +0000 UTC m=+74.702279178 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/e1a16d17-2367-4bd9-834a-c2da55613ab4-node-exporter-tls") pod "node-exporter-p5xwh" (UID: "e1a16d17-2367-4bd9-834a-c2da55613ab4") : secret "node-exporter-tls" not found Jul 06 22:10:22.902742 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.902722 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e1a16d17-2367-4bd9-834a-c2da55613ab4-node-exporter-wtmp\") pod \"node-exporter-p5xwh\" (UID: \"e1a16d17-2367-4bd9-834a-c2da55613ab4\") " pod="openshift-monitoring/node-exporter-p5xwh" Jul 06 22:10:22.902813 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.902799 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e1a16d17-2367-4bd9-834a-c2da55613ab4-node-exporter-accelerators-collector-config\") pod \"node-exporter-p5xwh\" (UID: \"e1a16d17-2367-4bd9-834a-c2da55613ab4\") " pod="openshift-monitoring/node-exporter-p5xwh" Jul 06 22:10:22.902864 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.902831 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e1a16d17-2367-4bd9-834a-c2da55613ab4-metrics-client-ca\") pod \"node-exporter-p5xwh\" (UID: \"e1a16d17-2367-4bd9-834a-c2da55613ab4\") " pod="openshift-monitoring/node-exporter-p5xwh" Jul 06 22:10:22.902919 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.902865 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e1a16d17-2367-4bd9-834a-c2da55613ab4-node-exporter-textfile\") pod \"node-exporter-p5xwh\" (UID: \"e1a16d17-2367-4bd9-834a-c2da55613ab4\") " pod="openshift-monitoring/node-exporter-p5xwh" Jul 06 22:10:22.902919 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.902892 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e1a16d17-2367-4bd9-834a-c2da55613ab4-node-exporter-wtmp\") pod \"node-exporter-p5xwh\" (UID: \"e1a16d17-2367-4bd9-834a-c2da55613ab4\") " pod="openshift-monitoring/node-exporter-p5xwh" Jul 06 22:10:22.903045 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.903024 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srql2\" (UniqueName: \"kubernetes.io/projected/e1a16d17-2367-4bd9-834a-c2da55613ab4-kube-api-access-srql2\") pod \"node-exporter-p5xwh\" (UID: \"e1a16d17-2367-4bd9-834a-c2da55613ab4\") " pod="openshift-monitoring/node-exporter-p5xwh" Jul 06 22:10:22.903098 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.903047 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e1a16d17-2367-4bd9-834a-c2da55613ab4-root\") pod \"node-exporter-p5xwh\" (UID: \"e1a16d17-2367-4bd9-834a-c2da55613ab4\") " pod="openshift-monitoring/node-exporter-p5xwh" Jul 06 22:10:22.903098 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.903065 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e1a16d17-2367-4bd9-834a-c2da55613ab4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p5xwh\" (UID: \"e1a16d17-2367-4bd9-834a-c2da55613ab4\") " pod="openshift-monitoring/node-exporter-p5xwh" Jul 06 22:10:22.903098 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.903083 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e1a16d17-2367-4bd9-834a-c2da55613ab4-sys\") pod \"node-exporter-p5xwh\" (UID: \"e1a16d17-2367-4bd9-834a-c2da55613ab4\") " pod="openshift-monitoring/node-exporter-p5xwh" Jul 06 22:10:22.903313 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.903286 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e1a16d17-2367-4bd9-834a-c2da55613ab4-node-exporter-textfile\") pod \"node-exporter-p5xwh\" (UID: \"e1a16d17-2367-4bd9-834a-c2da55613ab4\") " pod="openshift-monitoring/node-exporter-p5xwh" Jul 06 22:10:22.903504 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.903485 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e1a16d17-2367-4bd9-834a-c2da55613ab4-sys\") pod \"node-exporter-p5xwh\" (UID: \"e1a16d17-2367-4bd9-834a-c2da55613ab4\") " pod="openshift-monitoring/node-exporter-p5xwh" Jul 06 22:10:22.903577 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.903531 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e1a16d17-2367-4bd9-834a-c2da55613ab4-node-exporter-accelerators-collector-config\") pod \"node-exporter-p5xwh\" (UID: \"e1a16d17-2367-4bd9-834a-c2da55613ab4\") " pod="openshift-monitoring/node-exporter-p5xwh" Jul 06 22:10:22.903577 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.903552 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e1a16d17-2367-4bd9-834a-c2da55613ab4-metrics-client-ca\") pod \"node-exporter-p5xwh\" (UID: \"e1a16d17-2367-4bd9-834a-c2da55613ab4\") " pod="openshift-monitoring/node-exporter-p5xwh" Jul 06 22:10:22.903647 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.903555 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e1a16d17-2367-4bd9-834a-c2da55613ab4-root\") pod \"node-exporter-p5xwh\" (UID: \"e1a16d17-2367-4bd9-834a-c2da55613ab4\") " pod="openshift-monitoring/node-exporter-p5xwh" Jul 06 22:10:22.905399 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.905373 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e1a16d17-2367-4bd9-834a-c2da55613ab4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p5xwh\" (UID: \"e1a16d17-2367-4bd9-834a-c2da55613ab4\") " pod="openshift-monitoring/node-exporter-p5xwh" Jul 06 22:10:22.911382 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.911358 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srql2\" (UniqueName: \"kubernetes.io/projected/e1a16d17-2367-4bd9-834a-c2da55613ab4-kube-api-access-srql2\") pod \"node-exporter-p5xwh\" (UID: \"e1a16d17-2367-4bd9-834a-c2da55613ab4\") " pod="openshift-monitoring/node-exporter-p5xwh" Jul 06 22:10:22.971483 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:22.971447 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-xfb9f" Jul 06 22:10:23.104844 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:23.104811 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-xfb9f"] Jul 06 22:10:23.107738 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:10:23.107712 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c5f15ee_59ba_4a1a_9e8b_5826acf7a438.slice/crio-b25cde6ca82b2d73c61aef7d39898071a829d77472cd8e8a04732003da731e7a WatchSource:0}: Error finding container b25cde6ca82b2d73c61aef7d39898071a829d77472cd8e8a04732003da731e7a: Status 404 returned error can't find the container with id b25cde6ca82b2d73c61aef7d39898071a829d77472cd8e8a04732003da731e7a Jul 06 22:10:23.306621 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:23.306520 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea64e052-1c86-42b5-a8fd-b3f7e2f042cb-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-st8th\" (UID: \"ea64e052-1c86-42b5-a8fd-b3f7e2f042cb\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-st8th" Jul 06 22:10:23.309054 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:23.309028 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea64e052-1c86-42b5-a8fd-b3f7e2f042cb-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-st8th\" (UID: \"ea64e052-1c86-42b5-a8fd-b3f7e2f042cb\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-st8th" Jul 06 22:10:23.406970 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:23.406938 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e1a16d17-2367-4bd9-834a-c2da55613ab4-node-exporter-tls\") pod \"node-exporter-p5xwh\" (UID: \"e1a16d17-2367-4bd9-834a-c2da55613ab4\") " pod="openshift-monitoring/node-exporter-p5xwh" Jul 06 22:10:23.409242 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:23.409224 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e1a16d17-2367-4bd9-834a-c2da55613ab4-node-exporter-tls\") pod \"node-exporter-p5xwh\" (UID: \"e1a16d17-2367-4bd9-834a-c2da55613ab4\") " pod="openshift-monitoring/node-exporter-p5xwh" Jul 06 22:10:23.549156 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:23.549118 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-st8th" Jul 06 22:10:23.614174 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:23.614133 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-p5xwh" Jul 06 22:10:23.626431 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:10:23.626403 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1a16d17_2367_4bd9_834a_c2da55613ab4.slice/crio-ef18194d54779fd80c58901523f70dae5a156d126ccb2455859cfc9e72739fef WatchSource:0}: Error finding container ef18194d54779fd80c58901523f70dae5a156d126ccb2455859cfc9e72739fef: Status 404 returned error can't find the container with id ef18194d54779fd80c58901523f70dae5a156d126ccb2455859cfc9e72739fef Jul 06 22:10:23.691719 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:23.691685 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-st8th"] Jul 06 22:10:23.694381 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:10:23.694356 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea64e052_1c86_42b5_a8fd_b3f7e2f042cb.slice/crio-792ebfd43b422e1a6cf4b1d1cc87a92ee420689a740dff876f0467bc70286335 WatchSource:0}: Error finding container 792ebfd43b422e1a6cf4b1d1cc87a92ee420689a740dff876f0467bc70286335: Status 404 returned error can't find the container with id 792ebfd43b422e1a6cf4b1d1cc87a92ee420689a740dff876f0467bc70286335 Jul 06 22:10:23.730681 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:23.730649 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xj686" Jul 06 22:10:23.797150 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:23.796998 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-st8th" event={"ID":"ea64e052-1c86-42b5-a8fd-b3f7e2f042cb","Type":"ContainerStarted","Data":"912f97d8d85b4276aec8d7171fd8fe6a26b68e40180768283aaa588f39103883"} Jul 06 22:10:23.797150 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:23.797091 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-st8th" event={"ID":"ea64e052-1c86-42b5-a8fd-b3f7e2f042cb","Type":"ContainerStarted","Data":"792ebfd43b422e1a6cf4b1d1cc87a92ee420689a740dff876f0467bc70286335"} Jul 06 22:10:23.799085 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:23.799051 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-xfb9f" event={"ID":"2c5f15ee-59ba-4a1a-9e8b-5826acf7a438","Type":"ContainerStarted","Data":"b25cde6ca82b2d73c61aef7d39898071a829d77472cd8e8a04732003da731e7a"} Jul 06 22:10:23.800975 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:23.800944 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vdbr6" event={"ID":"852d3c7b-6b13-4603-9a80-7b07a035d22f","Type":"ContainerStarted","Data":"a429201497215a5fce96aee715b35bf429204a5401200e8e36649985df217284"} Jul 06 22:10:23.801217 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:23.801166 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vdbr6" Jul 06 22:10:23.803090 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:23.803042 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p5xwh" event={"ID":"e1a16d17-2367-4bd9-834a-c2da55613ab4","Type":"ContainerStarted","Data":"ef18194d54779fd80c58901523f70dae5a156d126ccb2455859cfc9e72739fef"} Jul 06 22:10:23.804854 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:23.804832 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zt94q" event={"ID":"d681bb3e-9bb1-4705-98cb-fd2cd5dc7063","Type":"ContainerStarted","Data":"b9f661f93a4440c56c848ff90dea10e053aa48a1ac8a2b7408f5f84a4c812aab"} Jul 06 22:10:23.820596 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:23.820572 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 22:10:23.823305 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:23.823257 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-vdbr6" podStartSLOduration=69.854030632 podStartE2EDuration="1m14.823238045s" totalImagesPullingTime="4.969207413s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:09:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:10:23.822778678 +0000 UTC m=+75.122490525" watchObservedRunningTime="2026-07-06 22:10:23.823238045 +0000 UTC m=+75.122949895" Jul 06 22:10:23.824980 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:23.824866 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:10:23.830326 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:23.830293 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 06 22:10:23.830728 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:23.830622 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 06 22:10:23.830728 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:23.830691 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 06 22:10:23.830897 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:23.830743 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 06 22:10:23.830897 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:23.830752 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-ll7rl\"" Jul 06 22:10:23.830897 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:23.830699 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 06 22:10:23.831127 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:23.830954 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 06 22:10:23.831127 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:23.831002 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 06 22:10:23.831127 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:23.831099 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 06 22:10:23.831127 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:23.831106 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 06 22:10:23.846478 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:23.846456 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 22:10:23.881441 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:23.881381 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-zt94q" podStartSLOduration=69.686001716 podStartE2EDuration="1m14.881358008s" totalImagesPullingTime="5.195356292s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:09:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:10:23.849277738 +0000 UTC m=+75.148989589" watchObservedRunningTime="2026-07-06 22:10:23.881358008 +0000 UTC m=+75.181069858" Jul 06 22:10:23.913032 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:23.912973 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/50356448-5321-4011-ba4c-fab5b999a2c2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"50356448-5321-4011-ba4c-fab5b999a2c2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:10:23.913183 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:23.913035 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/50356448-5321-4011-ba4c-fab5b999a2c2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"50356448-5321-4011-ba4c-fab5b999a2c2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:10:23.913183 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:23.913092 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/50356448-5321-4011-ba4c-fab5b999a2c2-config-volume\") pod \"alertmanager-main-0\" (UID: \"50356448-5321-4011-ba4c-fab5b999a2c2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:10:23.913284 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:23.913180 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/50356448-5321-4011-ba4c-fab5b999a2c2-web-config\") pod \"alertmanager-main-0\" (UID: \"50356448-5321-4011-ba4c-fab5b999a2c2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:10:23.913284 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:23.913243 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/50356448-5321-4011-ba4c-fab5b999a2c2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"50356448-5321-4011-ba4c-fab5b999a2c2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:10:23.913392 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:23.913295 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50356448-5321-4011-ba4c-fab5b999a2c2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"50356448-5321-4011-ba4c-fab5b999a2c2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:10:23.913392 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:23.913354 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/50356448-5321-4011-ba4c-fab5b999a2c2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"50356448-5321-4011-ba4c-fab5b999a2c2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:10:23.913480 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:23.913406 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/50356448-5321-4011-ba4c-fab5b999a2c2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"50356448-5321-4011-ba4c-fab5b999a2c2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:10:23.913480 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:23.913433 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/50356448-5321-4011-ba4c-fab5b999a2c2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"50356448-5321-4011-ba4c-fab5b999a2c2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:10:23.913480 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:23.913475 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/50356448-5321-4011-ba4c-fab5b999a2c2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"50356448-5321-4011-ba4c-fab5b999a2c2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:10:23.913637 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:23.913499 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8tqc\" (UniqueName: \"kubernetes.io/projected/50356448-5321-4011-ba4c-fab5b999a2c2-kube-api-access-b8tqc\") pod \"alertmanager-main-0\" (UID: \"50356448-5321-4011-ba4c-fab5b999a2c2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:10:23.913637 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:23.913578 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/50356448-5321-4011-ba4c-fab5b999a2c2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"50356448-5321-4011-ba4c-fab5b999a2c2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:10:23.913637 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:23.913622 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/50356448-5321-4011-ba4c-fab5b999a2c2-config-out\") pod \"alertmanager-main-0\" (UID: \"50356448-5321-4011-ba4c-fab5b999a2c2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:10:24.014903 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:24.014858 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/50356448-5321-4011-ba4c-fab5b999a2c2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"50356448-5321-4011-ba4c-fab5b999a2c2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:10:24.015117 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:24.014918 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/50356448-5321-4011-ba4c-fab5b999a2c2-config-out\") pod \"alertmanager-main-0\" (UID: \"50356448-5321-4011-ba4c-fab5b999a2c2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:10:24.015117 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:24.014950 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/50356448-5321-4011-ba4c-fab5b999a2c2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"50356448-5321-4011-ba4c-fab5b999a2c2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:10:24.015117 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:24.014976 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/50356448-5321-4011-ba4c-fab5b999a2c2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"50356448-5321-4011-ba4c-fab5b999a2c2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:10:24.015117 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:24.015023 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/50356448-5321-4011-ba4c-fab5b999a2c2-config-volume\") pod \"alertmanager-main-0\" (UID: \"50356448-5321-4011-ba4c-fab5b999a2c2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:10:24.015117 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:24.015058 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/50356448-5321-4011-ba4c-fab5b999a2c2-web-config\") pod \"alertmanager-main-0\" (UID: \"50356448-5321-4011-ba4c-fab5b999a2c2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:10:24.015117 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:24.015085 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/50356448-5321-4011-ba4c-fab5b999a2c2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"50356448-5321-4011-ba4c-fab5b999a2c2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:10:24.015117 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:24.015110 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50356448-5321-4011-ba4c-fab5b999a2c2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"50356448-5321-4011-ba4c-fab5b999a2c2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:10:24.015452 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:24.015158 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/50356448-5321-4011-ba4c-fab5b999a2c2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"50356448-5321-4011-ba4c-fab5b999a2c2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:10:24.015452 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:24.015205 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/50356448-5321-4011-ba4c-fab5b999a2c2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"50356448-5321-4011-ba4c-fab5b999a2c2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:10:24.015452 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:24.015234 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/50356448-5321-4011-ba4c-fab5b999a2c2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"50356448-5321-4011-ba4c-fab5b999a2c2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:10:24.015452 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:24.015272 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/50356448-5321-4011-ba4c-fab5b999a2c2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"50356448-5321-4011-ba4c-fab5b999a2c2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:10:24.015452 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:24.015292 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8tqc\" (UniqueName: \"kubernetes.io/projected/50356448-5321-4011-ba4c-fab5b999a2c2-kube-api-access-b8tqc\") pod \"alertmanager-main-0\" (UID: \"50356448-5321-4011-ba4c-fab5b999a2c2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:10:24.016535 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:24.016385 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/50356448-5321-4011-ba4c-fab5b999a2c2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"50356448-5321-4011-ba4c-fab5b999a2c2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:10:24.016535 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:10:24.016491 2555 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 06 22:10:24.016700 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:10:24.016555 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50356448-5321-4011-ba4c-fab5b999a2c2-secret-alertmanager-main-tls podName:50356448-5321-4011-ba4c-fab5b999a2c2 nodeName:}" failed. No retries permitted until 2026-07-06 22:10:24.516536422 +0000 UTC m=+75.816248251 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/50356448-5321-4011-ba4c-fab5b999a2c2-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "50356448-5321-4011-ba4c-fab5b999a2c2") : secret "alertmanager-main-tls" not found Jul 06 22:10:24.016766 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:24.016394 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/50356448-5321-4011-ba4c-fab5b999a2c2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"50356448-5321-4011-ba4c-fab5b999a2c2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:10:24.017498 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:24.017451 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50356448-5321-4011-ba4c-fab5b999a2c2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"50356448-5321-4011-ba4c-fab5b999a2c2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:10:24.018500 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:24.018459 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/50356448-5321-4011-ba4c-fab5b999a2c2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"50356448-5321-4011-ba4c-fab5b999a2c2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:10:24.018500 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:24.018467 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/50356448-5321-4011-ba4c-fab5b999a2c2-config-out\") pod \"alertmanager-main-0\" (UID: \"50356448-5321-4011-ba4c-fab5b999a2c2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:10:24.018640 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:24.018590 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/50356448-5321-4011-ba4c-fab5b999a2c2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"50356448-5321-4011-ba4c-fab5b999a2c2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:10:24.018640 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:24.018622 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/50356448-5321-4011-ba4c-fab5b999a2c2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"50356448-5321-4011-ba4c-fab5b999a2c2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:10:24.018914 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:24.018891 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/50356448-5321-4011-ba4c-fab5b999a2c2-web-config\") pod \"alertmanager-main-0\" (UID: \"50356448-5321-4011-ba4c-fab5b999a2c2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:10:24.019658 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:24.019635 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/50356448-5321-4011-ba4c-fab5b999a2c2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"50356448-5321-4011-ba4c-fab5b999a2c2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:10:24.020360 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:24.020334 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/50356448-5321-4011-ba4c-fab5b999a2c2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"50356448-5321-4011-ba4c-fab5b999a2c2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:10:24.021130 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:24.021106 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/50356448-5321-4011-ba4c-fab5b999a2c2-config-volume\") pod \"alertmanager-main-0\" (UID: \"50356448-5321-4011-ba4c-fab5b999a2c2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:10:24.026860 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:24.026835 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8tqc\" (UniqueName: \"kubernetes.io/projected/50356448-5321-4011-ba4c-fab5b999a2c2-kube-api-access-b8tqc\") pod \"alertmanager-main-0\" (UID: \"50356448-5321-4011-ba4c-fab5b999a2c2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:10:24.519966 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:24.519856 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/50356448-5321-4011-ba4c-fab5b999a2c2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"50356448-5321-4011-ba4c-fab5b999a2c2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:10:24.523078 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:24.523044 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/50356448-5321-4011-ba4c-fab5b999a2c2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"50356448-5321-4011-ba4c-fab5b999a2c2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:10:24.636681 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:24.636645 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-75c6b47584-6bdk2"] Jul 06 22:10:24.640474 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:24.640445 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-75c6b47584-6bdk2" Jul 06 22:10:24.644111 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:24.643831 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-62grt\"" Jul 06 22:10:24.644111 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:24.643886 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 06 22:10:24.644111 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:24.643964 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 06 22:10:24.644111 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:24.643973 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 06 22:10:24.644420 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:24.644220 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-5jk2095fssbf7\"" Jul 06 22:10:24.644420 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:24.644248 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 06 22:10:24.644720 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:24.644699 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 06 22:10:24.656877 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:24.656849 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-75c6b47584-6bdk2"] Jul 06 22:10:24.721728 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:24.721688 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d1a7e001-36db-4a34-ae02-b8c05ef42d77-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-75c6b47584-6bdk2\" (UID: \"d1a7e001-36db-4a34-ae02-b8c05ef42d77\") " pod="openshift-monitoring/thanos-querier-75c6b47584-6bdk2" Jul 06 22:10:24.721728 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:24.721739 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d1a7e001-36db-4a34-ae02-b8c05ef42d77-metrics-client-ca\") pod \"thanos-querier-75c6b47584-6bdk2\" (UID: \"d1a7e001-36db-4a34-ae02-b8c05ef42d77\") " pod="openshift-monitoring/thanos-querier-75c6b47584-6bdk2" Jul 06 22:10:24.721941 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:24.721829 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d1a7e001-36db-4a34-ae02-b8c05ef42d77-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-75c6b47584-6bdk2\" (UID: \"d1a7e001-36db-4a34-ae02-b8c05ef42d77\") " pod="openshift-monitoring/thanos-querier-75c6b47584-6bdk2" Jul 06 22:10:24.721941 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:24.721848 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d1a7e001-36db-4a34-ae02-b8c05ef42d77-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-75c6b47584-6bdk2\" (UID: \"d1a7e001-36db-4a34-ae02-b8c05ef42d77\") " pod="openshift-monitoring/thanos-querier-75c6b47584-6bdk2" Jul 06 22:10:24.721941 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:24.721868 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d1a7e001-36db-4a34-ae02-b8c05ef42d77-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-75c6b47584-6bdk2\" (UID: \"d1a7e001-36db-4a34-ae02-b8c05ef42d77\") " pod="openshift-monitoring/thanos-querier-75c6b47584-6bdk2" Jul 06 22:10:24.721941 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:24.721883 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d1a7e001-36db-4a34-ae02-b8c05ef42d77-secret-grpc-tls\") pod \"thanos-querier-75c6b47584-6bdk2\" (UID: \"d1a7e001-36db-4a34-ae02-b8c05ef42d77\") " pod="openshift-monitoring/thanos-querier-75c6b47584-6bdk2" Jul 06 22:10:24.721941 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:24.721909 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d68xp\" (UniqueName: \"kubernetes.io/projected/d1a7e001-36db-4a34-ae02-b8c05ef42d77-kube-api-access-d68xp\") pod \"thanos-querier-75c6b47584-6bdk2\" (UID: \"d1a7e001-36db-4a34-ae02-b8c05ef42d77\") " pod="openshift-monitoring/thanos-querier-75c6b47584-6bdk2" Jul 06 22:10:24.722150 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:24.721972 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d1a7e001-36db-4a34-ae02-b8c05ef42d77-secret-thanos-querier-tls\") pod \"thanos-querier-75c6b47584-6bdk2\" (UID: \"d1a7e001-36db-4a34-ae02-b8c05ef42d77\") " pod="openshift-monitoring/thanos-querier-75c6b47584-6bdk2" Jul 06 22:10:24.743249 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:24.743211 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:10:24.811632 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:24.811553 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-st8th" event={"ID":"ea64e052-1c86-42b5-a8fd-b3f7e2f042cb","Type":"ContainerStarted","Data":"9431f2caabdc0d072d1694a2ababf7bac492f1f464b199f3f82b863ceb832de3"} Jul 06 22:10:24.823346 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:24.823317 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d1a7e001-36db-4a34-ae02-b8c05ef42d77-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-75c6b47584-6bdk2\" (UID: \"d1a7e001-36db-4a34-ae02-b8c05ef42d77\") " pod="openshift-monitoring/thanos-querier-75c6b47584-6bdk2" Jul 06 22:10:24.823532 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:24.823355 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d1a7e001-36db-4a34-ae02-b8c05ef42d77-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-75c6b47584-6bdk2\" (UID: \"d1a7e001-36db-4a34-ae02-b8c05ef42d77\") " pod="openshift-monitoring/thanos-querier-75c6b47584-6bdk2" Jul 06 22:10:24.823532 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:24.823378 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d1a7e001-36db-4a34-ae02-b8c05ef42d77-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-75c6b47584-6bdk2\" (UID: \"d1a7e001-36db-4a34-ae02-b8c05ef42d77\") " pod="openshift-monitoring/thanos-querier-75c6b47584-6bdk2" Jul 06 22:10:24.823532 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:24.823401 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d1a7e001-36db-4a34-ae02-b8c05ef42d77-secret-grpc-tls\") pod \"thanos-querier-75c6b47584-6bdk2\" (UID: \"d1a7e001-36db-4a34-ae02-b8c05ef42d77\") " pod="openshift-monitoring/thanos-querier-75c6b47584-6bdk2" Jul 06 22:10:24.823532 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:24.823427 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d68xp\" (UniqueName: \"kubernetes.io/projected/d1a7e001-36db-4a34-ae02-b8c05ef42d77-kube-api-access-d68xp\") pod \"thanos-querier-75c6b47584-6bdk2\" (UID: \"d1a7e001-36db-4a34-ae02-b8c05ef42d77\") " pod="openshift-monitoring/thanos-querier-75c6b47584-6bdk2" Jul 06 22:10:24.823532 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:24.823482 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d1a7e001-36db-4a34-ae02-b8c05ef42d77-secret-thanos-querier-tls\") pod \"thanos-querier-75c6b47584-6bdk2\" (UID: \"d1a7e001-36db-4a34-ae02-b8c05ef42d77\") " pod="openshift-monitoring/thanos-querier-75c6b47584-6bdk2" Jul 06 22:10:24.823532 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:24.823522 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d1a7e001-36db-4a34-ae02-b8c05ef42d77-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-75c6b47584-6bdk2\" (UID: \"d1a7e001-36db-4a34-ae02-b8c05ef42d77\") " pod="openshift-monitoring/thanos-querier-75c6b47584-6bdk2" Jul 06 22:10:24.823840 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:24.823562 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d1a7e001-36db-4a34-ae02-b8c05ef42d77-metrics-client-ca\") pod \"thanos-querier-75c6b47584-6bdk2\" (UID: \"d1a7e001-36db-4a34-ae02-b8c05ef42d77\") " pod="openshift-monitoring/thanos-querier-75c6b47584-6bdk2" Jul 06 22:10:24.824355 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:24.824327 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d1a7e001-36db-4a34-ae02-b8c05ef42d77-metrics-client-ca\") pod \"thanos-querier-75c6b47584-6bdk2\" (UID: \"d1a7e001-36db-4a34-ae02-b8c05ef42d77\") " pod="openshift-monitoring/thanos-querier-75c6b47584-6bdk2" Jul 06 22:10:24.826425 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:24.826394 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d1a7e001-36db-4a34-ae02-b8c05ef42d77-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-75c6b47584-6bdk2\" (UID: \"d1a7e001-36db-4a34-ae02-b8c05ef42d77\") " pod="openshift-monitoring/thanos-querier-75c6b47584-6bdk2" Jul 06 22:10:24.826825 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:24.826805 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d1a7e001-36db-4a34-ae02-b8c05ef42d77-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-75c6b47584-6bdk2\" (UID: \"d1a7e001-36db-4a34-ae02-b8c05ef42d77\") " pod="openshift-monitoring/thanos-querier-75c6b47584-6bdk2" Jul 06 22:10:24.826929 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:24.826912 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d1a7e001-36db-4a34-ae02-b8c05ef42d77-secret-thanos-querier-tls\") pod \"thanos-querier-75c6b47584-6bdk2\" (UID: \"d1a7e001-36db-4a34-ae02-b8c05ef42d77\") " pod="openshift-monitoring/thanos-querier-75c6b47584-6bdk2" Jul 06 22:10:24.826993 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:24.826969 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d1a7e001-36db-4a34-ae02-b8c05ef42d77-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-75c6b47584-6bdk2\" (UID: \"d1a7e001-36db-4a34-ae02-b8c05ef42d77\") " pod="openshift-monitoring/thanos-querier-75c6b47584-6bdk2" Jul 06 22:10:24.827135 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:24.827111 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d1a7e001-36db-4a34-ae02-b8c05ef42d77-secret-grpc-tls\") pod \"thanos-querier-75c6b47584-6bdk2\" (UID: \"d1a7e001-36db-4a34-ae02-b8c05ef42d77\") " pod="openshift-monitoring/thanos-querier-75c6b47584-6bdk2" Jul 06 22:10:24.827456 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:24.827437 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d1a7e001-36db-4a34-ae02-b8c05ef42d77-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-75c6b47584-6bdk2\" (UID: \"d1a7e001-36db-4a34-ae02-b8c05ef42d77\") " pod="openshift-monitoring/thanos-querier-75c6b47584-6bdk2" Jul 06 22:10:24.834050 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:24.834022 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d68xp\" (UniqueName: \"kubernetes.io/projected/d1a7e001-36db-4a34-ae02-b8c05ef42d77-kube-api-access-d68xp\") pod \"thanos-querier-75c6b47584-6bdk2\" (UID: \"d1a7e001-36db-4a34-ae02-b8c05ef42d77\") " pod="openshift-monitoring/thanos-querier-75c6b47584-6bdk2" Jul 06 22:10:24.953147 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:24.953111 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-75c6b47584-6bdk2" Jul 06 22:10:25.325242 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:10:25.325136 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50356448_5321_4011_ba4c_fab5b999a2c2.slice/crio-0956d64254ede532ed68755a31a1ecae2f4da8468cfaff3e53ef7d39f80d8897 WatchSource:0}: Error finding container 0956d64254ede532ed68755a31a1ecae2f4da8468cfaff3e53ef7d39f80d8897: Status 404 returned error can't find the container with id 0956d64254ede532ed68755a31a1ecae2f4da8468cfaff3e53ef7d39f80d8897 Jul 06 22:10:25.326459 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:25.326353 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 22:10:25.355622 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:25.355591 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-75c6b47584-6bdk2"] Jul 06 22:10:25.398000 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:10:25.397962 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1a7e001_36db_4a34_ae02_b8c05ef42d77.slice/crio-09c314285a722ae24121463abadeb8f7a6d9a9b0da19914f5cc6b7735fa70cb5 WatchSource:0}: Error finding container 09c314285a722ae24121463abadeb8f7a6d9a9b0da19914f5cc6b7735fa70cb5: Status 404 returned error can't find the container with id 09c314285a722ae24121463abadeb8f7a6d9a9b0da19914f5cc6b7735fa70cb5 Jul 06 22:10:25.817124 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:25.817084 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-st8th" event={"ID":"ea64e052-1c86-42b5-a8fd-b3f7e2f042cb","Type":"ContainerStarted","Data":"e845b8723bbf12ee10e8ae8ff6a62c4ce5c20be3b808499631a93e9a04de8220"} Jul 06 22:10:25.819314 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:25.819285 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-xfb9f" event={"ID":"2c5f15ee-59ba-4a1a-9e8b-5826acf7a438","Type":"ContainerStarted","Data":"60ab39297f7dfe8dfdb3e5a0bd7c118630a50e2ccf1f3d913ae402475b44dfd1"} Jul 06 22:10:25.819441 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:25.819347 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-xfb9f" event={"ID":"2c5f15ee-59ba-4a1a-9e8b-5826acf7a438","Type":"ContainerStarted","Data":"07b4fba5808dc20efe547842f7bc9878f94abbab29c07726932444c80224c070"} Jul 06 22:10:25.819441 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:25.819376 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-xfb9f" event={"ID":"2c5f15ee-59ba-4a1a-9e8b-5826acf7a438","Type":"ContainerStarted","Data":"339021bb496b0e099038e17e51b1aec068b0366f93069275d319388f1837ad2d"} Jul 06 22:10:25.820892 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:25.820868 2555 generic.go:356] "Generic (PLEG): container finished" podID="e1a16d17-2367-4bd9-834a-c2da55613ab4" containerID="b5d038a4ed511f45b9209e8d79e17706df2477c366fc86d6928365cfe9a033bd" exitCode=0 Jul 06 22:10:25.821033 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:25.820936 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p5xwh" event={"ID":"e1a16d17-2367-4bd9-834a-c2da55613ab4","Type":"ContainerDied","Data":"b5d038a4ed511f45b9209e8d79e17706df2477c366fc86d6928365cfe9a033bd"} Jul 06 22:10:25.822199 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:25.822178 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"50356448-5321-4011-ba4c-fab5b999a2c2","Type":"ContainerStarted","Data":"0956d64254ede532ed68755a31a1ecae2f4da8468cfaff3e53ef7d39f80d8897"} Jul 06 22:10:25.823363 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:25.823343 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75c6b47584-6bdk2" event={"ID":"d1a7e001-36db-4a34-ae02-b8c05ef42d77","Type":"ContainerStarted","Data":"09c314285a722ae24121463abadeb8f7a6d9a9b0da19914f5cc6b7735fa70cb5"} Jul 06 22:10:25.837302 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:25.837246 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-st8th" podStartSLOduration=2.510880427 podStartE2EDuration="3.837227679s" totalImagesPullingTime="1.326347252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:10:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:10:25.835266697 +0000 UTC m=+77.134978548" watchObservedRunningTime="2026-07-06 22:10:25.837227679 +0000 UTC m=+77.136939531" Jul 06 22:10:25.858894 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:25.858835 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-xfb9f" podStartSLOduration=1.8266782259999998 podStartE2EDuration="3.858812722s" totalImagesPullingTime="2.032134496s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:10:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:10:25.853739178 +0000 UTC m=+77.153451032" watchObservedRunningTime="2026-07-06 22:10:25.858812722 +0000 UTC m=+77.158524573" Jul 06 22:10:26.830698 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:26.830666 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p5xwh" event={"ID":"e1a16d17-2367-4bd9-834a-c2da55613ab4","Type":"ContainerStarted","Data":"72e77a94026df72e9e2b27f5f9fd2878e9d7e7982c499abd33a0727ad05277c3"} Jul 06 22:10:26.831081 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:26.830722 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p5xwh" event={"ID":"e1a16d17-2367-4bd9-834a-c2da55613ab4","Type":"ContainerStarted","Data":"e862fe851a30c25f68b6ee30c96e0937f1d812ec9e6d423ec3895de956911596"} Jul 06 22:10:26.832288 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:26.832256 2555 generic.go:356] "Generic (PLEG): container finished" podID="50356448-5321-4011-ba4c-fab5b999a2c2" containerID="6cae586c0576001244e53c8541065728cdd103287c5923d6d80462be87691f3b" exitCode=0 Jul 06 22:10:26.832384 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:26.832339 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"50356448-5321-4011-ba4c-fab5b999a2c2","Type":"ContainerDied","Data":"6cae586c0576001244e53c8541065728cdd103287c5923d6d80462be87691f3b"} Jul 06 22:10:26.852414 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:26.852353 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-p5xwh" podStartSLOduration=3.225566018 podStartE2EDuration="4.852334977s" totalImagesPullingTime="1.513319547s" totalInitContainerRuntime="113.449412ms" isStatefulPod=true podCreationTimestamp="2026-07-06 22:10:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:10:26.850471268 +0000 UTC m=+78.150183139" watchObservedRunningTime="2026-07-06 22:10:26.852334977 +0000 UTC m=+78.152046827" Jul 06 22:10:27.066793 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:27.066715 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5678f86565-bhz5r"] Jul 06 22:10:27.071039 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:27.070989 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5678f86565-bhz5r" Jul 06 22:10:27.075160 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:27.075133 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 06 22:10:27.075288 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:27.075264 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-20lutpva8i1dn\"" Jul 06 22:10:27.075350 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:27.075322 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 06 22:10:27.075413 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:27.075199 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 06 22:10:27.075413 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:27.075215 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-grzk7\"" Jul 06 22:10:27.075532 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:27.075522 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 06 22:10:27.081603 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:27.081581 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5678f86565-bhz5r"] Jul 06 22:10:27.145406 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:27.145372 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0bd2d8d-b4a6-456b-9f88-fb699e4ed523-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5678f86565-bhz5r\" (UID: \"a0bd2d8d-b4a6-456b-9f88-fb699e4ed523\") " pod="openshift-monitoring/metrics-server-5678f86565-bhz5r" Jul 06 22:10:27.145601 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:27.145418 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a0bd2d8d-b4a6-456b-9f88-fb699e4ed523-secret-metrics-server-tls\") pod \"metrics-server-5678f86565-bhz5r\" (UID: \"a0bd2d8d-b4a6-456b-9f88-fb699e4ed523\") " pod="openshift-monitoring/metrics-server-5678f86565-bhz5r" Jul 06 22:10:27.145601 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:27.145444 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a0bd2d8d-b4a6-456b-9f88-fb699e4ed523-client-ca-bundle\") pod \"metrics-server-5678f86565-bhz5r\" (UID: \"a0bd2d8d-b4a6-456b-9f88-fb699e4ed523\") " pod="openshift-monitoring/metrics-server-5678f86565-bhz5r" Jul 06 22:10:27.145601 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:27.145578 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a0bd2d8d-b4a6-456b-9f88-fb699e4ed523-audit-log\") pod \"metrics-server-5678f86565-bhz5r\" (UID: \"a0bd2d8d-b4a6-456b-9f88-fb699e4ed523\") " pod="openshift-monitoring/metrics-server-5678f86565-bhz5r" Jul 06 22:10:27.145763 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:27.145607 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a0bd2d8d-b4a6-456b-9f88-fb699e4ed523-metrics-server-audit-profiles\") pod \"metrics-server-5678f86565-bhz5r\" (UID: \"a0bd2d8d-b4a6-456b-9f88-fb699e4ed523\") " pod="openshift-monitoring/metrics-server-5678f86565-bhz5r" Jul 06 22:10:27.145763 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:27.145658 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlz2b\" (UniqueName: \"kubernetes.io/projected/a0bd2d8d-b4a6-456b-9f88-fb699e4ed523-kube-api-access-jlz2b\") pod \"metrics-server-5678f86565-bhz5r\" (UID: \"a0bd2d8d-b4a6-456b-9f88-fb699e4ed523\") " pod="openshift-monitoring/metrics-server-5678f86565-bhz5r" Jul 06 22:10:27.145763 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:27.145722 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a0bd2d8d-b4a6-456b-9f88-fb699e4ed523-secret-metrics-server-client-certs\") pod \"metrics-server-5678f86565-bhz5r\" (UID: \"a0bd2d8d-b4a6-456b-9f88-fb699e4ed523\") " pod="openshift-monitoring/metrics-server-5678f86565-bhz5r" Jul 06 22:10:27.247047 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:27.246983 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a0bd2d8d-b4a6-456b-9f88-fb699e4ed523-audit-log\") pod \"metrics-server-5678f86565-bhz5r\" (UID: \"a0bd2d8d-b4a6-456b-9f88-fb699e4ed523\") " pod="openshift-monitoring/metrics-server-5678f86565-bhz5r" Jul 06 22:10:27.247047 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:27.247045 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a0bd2d8d-b4a6-456b-9f88-fb699e4ed523-metrics-server-audit-profiles\") pod \"metrics-server-5678f86565-bhz5r\" (UID: \"a0bd2d8d-b4a6-456b-9f88-fb699e4ed523\") " pod="openshift-monitoring/metrics-server-5678f86565-bhz5r" Jul 06 22:10:27.247297 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:27.247090 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlz2b\" (UniqueName: \"kubernetes.io/projected/a0bd2d8d-b4a6-456b-9f88-fb699e4ed523-kube-api-access-jlz2b\") pod \"metrics-server-5678f86565-bhz5r\" (UID: \"a0bd2d8d-b4a6-456b-9f88-fb699e4ed523\") " pod="openshift-monitoring/metrics-server-5678f86565-bhz5r" Jul 06 22:10:27.247297 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:27.247142 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a0bd2d8d-b4a6-456b-9f88-fb699e4ed523-secret-metrics-server-client-certs\") pod \"metrics-server-5678f86565-bhz5r\" (UID: \"a0bd2d8d-b4a6-456b-9f88-fb699e4ed523\") " pod="openshift-monitoring/metrics-server-5678f86565-bhz5r" Jul 06 22:10:27.247297 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:27.247231 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0bd2d8d-b4a6-456b-9f88-fb699e4ed523-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5678f86565-bhz5r\" (UID: \"a0bd2d8d-b4a6-456b-9f88-fb699e4ed523\") " pod="openshift-monitoring/metrics-server-5678f86565-bhz5r" Jul 06 22:10:27.247297 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:27.247280 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a0bd2d8d-b4a6-456b-9f88-fb699e4ed523-secret-metrics-server-tls\") pod \"metrics-server-5678f86565-bhz5r\" (UID: \"a0bd2d8d-b4a6-456b-9f88-fb699e4ed523\") " pod="openshift-monitoring/metrics-server-5678f86565-bhz5r" Jul 06 22:10:27.247438 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:27.247310 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a0bd2d8d-b4a6-456b-9f88-fb699e4ed523-client-ca-bundle\") pod \"metrics-server-5678f86565-bhz5r\" (UID: \"a0bd2d8d-b4a6-456b-9f88-fb699e4ed523\") " pod="openshift-monitoring/metrics-server-5678f86565-bhz5r" Jul 06 22:10:27.247499 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:27.247467 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a0bd2d8d-b4a6-456b-9f88-fb699e4ed523-audit-log\") pod \"metrics-server-5678f86565-bhz5r\" (UID: \"a0bd2d8d-b4a6-456b-9f88-fb699e4ed523\") " pod="openshift-monitoring/metrics-server-5678f86565-bhz5r" Jul 06 22:10:27.248317 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:27.248259 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0bd2d8d-b4a6-456b-9f88-fb699e4ed523-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5678f86565-bhz5r\" (UID: \"a0bd2d8d-b4a6-456b-9f88-fb699e4ed523\") " pod="openshift-monitoring/metrics-server-5678f86565-bhz5r" Jul 06 22:10:27.248450 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:27.248402 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a0bd2d8d-b4a6-456b-9f88-fb699e4ed523-metrics-server-audit-profiles\") pod \"metrics-server-5678f86565-bhz5r\" (UID: \"a0bd2d8d-b4a6-456b-9f88-fb699e4ed523\") " pod="openshift-monitoring/metrics-server-5678f86565-bhz5r" Jul 06 22:10:27.250437 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:27.250412 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a0bd2d8d-b4a6-456b-9f88-fb699e4ed523-client-ca-bundle\") pod \"metrics-server-5678f86565-bhz5r\" (UID: \"a0bd2d8d-b4a6-456b-9f88-fb699e4ed523\") " pod="openshift-monitoring/metrics-server-5678f86565-bhz5r" Jul 06 22:10:27.250589 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:27.250561 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a0bd2d8d-b4a6-456b-9f88-fb699e4ed523-secret-metrics-server-tls\") pod \"metrics-server-5678f86565-bhz5r\" (UID: \"a0bd2d8d-b4a6-456b-9f88-fb699e4ed523\") " pod="openshift-monitoring/metrics-server-5678f86565-bhz5r" Jul 06 22:10:27.250677 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:27.250592 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a0bd2d8d-b4a6-456b-9f88-fb699e4ed523-secret-metrics-server-client-certs\") pod \"metrics-server-5678f86565-bhz5r\" (UID: \"a0bd2d8d-b4a6-456b-9f88-fb699e4ed523\") " pod="openshift-monitoring/metrics-server-5678f86565-bhz5r" Jul 06 22:10:27.257137 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:27.257101 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlz2b\" (UniqueName: \"kubernetes.io/projected/a0bd2d8d-b4a6-456b-9f88-fb699e4ed523-kube-api-access-jlz2b\") pod \"metrics-server-5678f86565-bhz5r\" (UID: \"a0bd2d8d-b4a6-456b-9f88-fb699e4ed523\") " pod="openshift-monitoring/metrics-server-5678f86565-bhz5r" Jul 06 22:10:27.383749 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:27.383662 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5678f86565-bhz5r" Jul 06 22:10:27.482436 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:27.482397 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-swjnf"] Jul 06 22:10:27.485610 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:27.485590 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-swjnf" Jul 06 22:10:27.490586 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:27.490562 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 06 22:10:27.490708 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:27.490603 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-wj899\"" Jul 06 22:10:27.500666 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:27.500639 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-swjnf"] Jul 06 22:10:27.550351 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:27.550313 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs9rh\" (UniqueName: \"kubernetes.io/projected/74152ba8-c71c-42ad-8109-60c71839ae56-kube-api-access-vs9rh\") pod \"monitoring-plugin-d5dcff7cf-swjnf\" (UID: \"74152ba8-c71c-42ad-8109-60c71839ae56\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-swjnf" Jul 06 22:10:27.550546 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:27.550374 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/74152ba8-c71c-42ad-8109-60c71839ae56-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-swjnf\" (UID: \"74152ba8-c71c-42ad-8109-60c71839ae56\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-swjnf" Jul 06 22:10:27.651525 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:27.651485 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs9rh\" (UniqueName: \"kubernetes.io/projected/74152ba8-c71c-42ad-8109-60c71839ae56-kube-api-access-vs9rh\") pod \"monitoring-plugin-d5dcff7cf-swjnf\" (UID: \"74152ba8-c71c-42ad-8109-60c71839ae56\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-swjnf" Jul 06 22:10:27.651704 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:27.651535 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/74152ba8-c71c-42ad-8109-60c71839ae56-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-swjnf\" (UID: \"74152ba8-c71c-42ad-8109-60c71839ae56\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-swjnf" Jul 06 22:10:27.654182 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:27.654154 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/74152ba8-c71c-42ad-8109-60c71839ae56-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-swjnf\" (UID: \"74152ba8-c71c-42ad-8109-60c71839ae56\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-swjnf" Jul 06 22:10:27.660839 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:27.660817 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs9rh\" (UniqueName: \"kubernetes.io/projected/74152ba8-c71c-42ad-8109-60c71839ae56-kube-api-access-vs9rh\") pod \"monitoring-plugin-d5dcff7cf-swjnf\" (UID: \"74152ba8-c71c-42ad-8109-60c71839ae56\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-swjnf" Jul 06 22:10:27.756176 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:27.756137 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5678f86565-bhz5r"] Jul 06 22:10:27.763711 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:10:27.763684 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0bd2d8d_b4a6_456b_9f88_fb699e4ed523.slice/crio-40818e5ee4b7e1e03b9f8262cfd0938a6db49b20cf726384828f7d374d2432cc WatchSource:0}: Error finding container 40818e5ee4b7e1e03b9f8262cfd0938a6db49b20cf726384828f7d374d2432cc: Status 404 returned error can't find the container with id 40818e5ee4b7e1e03b9f8262cfd0938a6db49b20cf726384828f7d374d2432cc Jul 06 22:10:27.796659 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:27.796611 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-swjnf" Jul 06 22:10:27.840928 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:27.840896 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75c6b47584-6bdk2" event={"ID":"d1a7e001-36db-4a34-ae02-b8c05ef42d77","Type":"ContainerStarted","Data":"a441bb86f403ffaa0c5e9f613af48d6bafeb030d13718e07f4d5c43cee03ebb7"} Jul 06 22:10:27.841363 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:27.840962 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75c6b47584-6bdk2" event={"ID":"d1a7e001-36db-4a34-ae02-b8c05ef42d77","Type":"ContainerStarted","Data":"4849944e7a2c419ead2399f8bcf6639b9dd2f865738e1ade20c2f6c47e46c27a"} Jul 06 22:10:27.842974 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:27.842948 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5678f86565-bhz5r" event={"ID":"a0bd2d8d-b4a6-456b-9f88-fb699e4ed523","Type":"ContainerStarted","Data":"40818e5ee4b7e1e03b9f8262cfd0938a6db49b20cf726384828f7d374d2432cc"} Jul 06 22:10:27.874282 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:27.874220 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-67d899d7c8-vlbqf"] Jul 06 22:10:27.878104 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:27.878072 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-67d899d7c8-vlbqf" Jul 06 22:10:27.881536 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:27.881323 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 06 22:10:27.881765 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:27.881750 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 06 22:10:27.881943 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:27.881928 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 06 22:10:27.882698 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:27.882263 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 06 22:10:27.882698 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:27.882364 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-6zlvs\"" Jul 06 22:10:27.882698 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:27.882563 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 06 22:10:27.887483 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:27.887298 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 06 22:10:27.898598 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:27.898432 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-67d899d7c8-vlbqf"] Jul 06 22:10:27.951697 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:27.951528 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-swjnf"] Jul 06 22:10:27.953831 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:27.953792 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/491f519b-6854-4770-876e-df3f7fe5bb84-telemeter-trusted-ca-bundle\") pod \"telemeter-client-67d899d7c8-vlbqf\" (UID: \"491f519b-6854-4770-876e-df3f7fe5bb84\") " pod="openshift-monitoring/telemeter-client-67d899d7c8-vlbqf" Jul 06 22:10:27.953944 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:27.953914 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/491f519b-6854-4770-876e-df3f7fe5bb84-serving-certs-ca-bundle\") pod \"telemeter-client-67d899d7c8-vlbqf\" (UID: \"491f519b-6854-4770-876e-df3f7fe5bb84\") " pod="openshift-monitoring/telemeter-client-67d899d7c8-vlbqf" Jul 06 22:10:27.954037 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:27.953975 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/491f519b-6854-4770-876e-df3f7fe5bb84-secret-telemeter-client\") pod \"telemeter-client-67d899d7c8-vlbqf\" (UID: \"491f519b-6854-4770-876e-df3f7fe5bb84\") " pod="openshift-monitoring/telemeter-client-67d899d7c8-vlbqf" Jul 06 22:10:27.954037 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:27.954002 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/491f519b-6854-4770-876e-df3f7fe5bb84-federate-client-tls\") pod \"telemeter-client-67d899d7c8-vlbqf\" (UID: \"491f519b-6854-4770-876e-df3f7fe5bb84\") " pod="openshift-monitoring/telemeter-client-67d899d7c8-vlbqf" Jul 06 22:10:27.954132 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:27.954060 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/491f519b-6854-4770-876e-df3f7fe5bb84-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-67d899d7c8-vlbqf\" (UID: \"491f519b-6854-4770-876e-df3f7fe5bb84\") " pod="openshift-monitoring/telemeter-client-67d899d7c8-vlbqf" Jul 06 22:10:27.954132 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:27.954087 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/491f519b-6854-4770-876e-df3f7fe5bb84-metrics-client-ca\") pod \"telemeter-client-67d899d7c8-vlbqf\" (UID: \"491f519b-6854-4770-876e-df3f7fe5bb84\") " pod="openshift-monitoring/telemeter-client-67d899d7c8-vlbqf" Jul 06 22:10:27.954132 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:27.954120 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/491f519b-6854-4770-876e-df3f7fe5bb84-telemeter-client-tls\") pod \"telemeter-client-67d899d7c8-vlbqf\" (UID: \"491f519b-6854-4770-876e-df3f7fe5bb84\") " pod="openshift-monitoring/telemeter-client-67d899d7c8-vlbqf" Jul 06 22:10:27.954273 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:27.954144 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfqwj\" (UniqueName: \"kubernetes.io/projected/491f519b-6854-4770-876e-df3f7fe5bb84-kube-api-access-hfqwj\") pod \"telemeter-client-67d899d7c8-vlbqf\" (UID: \"491f519b-6854-4770-876e-df3f7fe5bb84\") " pod="openshift-monitoring/telemeter-client-67d899d7c8-vlbqf" Jul 06 22:10:27.954709 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:10:27.954684 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74152ba8_c71c_42ad_8109_60c71839ae56.slice/crio-ef8373d0909d52923554e8c7762941b1d7a6dbc34facaf0869432828f6203c63 WatchSource:0}: Error finding container ef8373d0909d52923554e8c7762941b1d7a6dbc34facaf0869432828f6203c63: Status 404 returned error can't find the container with id ef8373d0909d52923554e8c7762941b1d7a6dbc34facaf0869432828f6203c63 Jul 06 22:10:28.055397 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:28.055350 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/491f519b-6854-4770-876e-df3f7fe5bb84-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-67d899d7c8-vlbqf\" (UID: \"491f519b-6854-4770-876e-df3f7fe5bb84\") " pod="openshift-monitoring/telemeter-client-67d899d7c8-vlbqf" Jul 06 22:10:28.055397 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:28.055393 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/491f519b-6854-4770-876e-df3f7fe5bb84-metrics-client-ca\") pod \"telemeter-client-67d899d7c8-vlbqf\" (UID: \"491f519b-6854-4770-876e-df3f7fe5bb84\") " pod="openshift-monitoring/telemeter-client-67d899d7c8-vlbqf" Jul 06 22:10:28.055621 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:28.055418 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/491f519b-6854-4770-876e-df3f7fe5bb84-telemeter-client-tls\") pod \"telemeter-client-67d899d7c8-vlbqf\" (UID: \"491f519b-6854-4770-876e-df3f7fe5bb84\") " pod="openshift-monitoring/telemeter-client-67d899d7c8-vlbqf" Jul 06 22:10:28.055621 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:28.055444 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfqwj\" (UniqueName: \"kubernetes.io/projected/491f519b-6854-4770-876e-df3f7fe5bb84-kube-api-access-hfqwj\") pod \"telemeter-client-67d899d7c8-vlbqf\" (UID: \"491f519b-6854-4770-876e-df3f7fe5bb84\") " pod="openshift-monitoring/telemeter-client-67d899d7c8-vlbqf" Jul 06 22:10:28.055621 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:28.055488 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/491f519b-6854-4770-876e-df3f7fe5bb84-telemeter-trusted-ca-bundle\") pod \"telemeter-client-67d899d7c8-vlbqf\" (UID: \"491f519b-6854-4770-876e-df3f7fe5bb84\") " pod="openshift-monitoring/telemeter-client-67d899d7c8-vlbqf" Jul 06 22:10:28.056381 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:28.055802 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/491f519b-6854-4770-876e-df3f7fe5bb84-serving-certs-ca-bundle\") pod \"telemeter-client-67d899d7c8-vlbqf\" (UID: \"491f519b-6854-4770-876e-df3f7fe5bb84\") " pod="openshift-monitoring/telemeter-client-67d899d7c8-vlbqf" Jul 06 22:10:28.056381 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:28.055899 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/491f519b-6854-4770-876e-df3f7fe5bb84-secret-telemeter-client\") pod \"telemeter-client-67d899d7c8-vlbqf\" (UID: \"491f519b-6854-4770-876e-df3f7fe5bb84\") " pod="openshift-monitoring/telemeter-client-67d899d7c8-vlbqf" Jul 06 22:10:28.056381 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:28.055931 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/491f519b-6854-4770-876e-df3f7fe5bb84-federate-client-tls\") pod \"telemeter-client-67d899d7c8-vlbqf\" (UID: \"491f519b-6854-4770-876e-df3f7fe5bb84\") " pod="openshift-monitoring/telemeter-client-67d899d7c8-vlbqf" Jul 06 22:10:28.056642 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:28.056620 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/491f519b-6854-4770-876e-df3f7fe5bb84-metrics-client-ca\") pod \"telemeter-client-67d899d7c8-vlbqf\" (UID: \"491f519b-6854-4770-876e-df3f7fe5bb84\") " pod="openshift-monitoring/telemeter-client-67d899d7c8-vlbqf" Jul 06 22:10:28.059168 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:28.057101 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/491f519b-6854-4770-876e-df3f7fe5bb84-serving-certs-ca-bundle\") pod \"telemeter-client-67d899d7c8-vlbqf\" (UID: \"491f519b-6854-4770-876e-df3f7fe5bb84\") " pod="openshift-monitoring/telemeter-client-67d899d7c8-vlbqf" Jul 06 22:10:28.059168 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:28.057608 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/491f519b-6854-4770-876e-df3f7fe5bb84-telemeter-trusted-ca-bundle\") pod \"telemeter-client-67d899d7c8-vlbqf\" (UID: \"491f519b-6854-4770-876e-df3f7fe5bb84\") " pod="openshift-monitoring/telemeter-client-67d899d7c8-vlbqf" Jul 06 22:10:28.059168 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:28.058950 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/491f519b-6854-4770-876e-df3f7fe5bb84-secret-telemeter-client\") pod \"telemeter-client-67d899d7c8-vlbqf\" (UID: \"491f519b-6854-4770-876e-df3f7fe5bb84\") " pod="openshift-monitoring/telemeter-client-67d899d7c8-vlbqf" Jul 06 22:10:28.059168 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:28.059111 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/491f519b-6854-4770-876e-df3f7fe5bb84-federate-client-tls\") pod \"telemeter-client-67d899d7c8-vlbqf\" (UID: \"491f519b-6854-4770-876e-df3f7fe5bb84\") " pod="openshift-monitoring/telemeter-client-67d899d7c8-vlbqf" Jul 06 22:10:28.059168 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:28.059117 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/491f519b-6854-4770-876e-df3f7fe5bb84-telemeter-client-tls\") pod \"telemeter-client-67d899d7c8-vlbqf\" (UID: \"491f519b-6854-4770-876e-df3f7fe5bb84\") " pod="openshift-monitoring/telemeter-client-67d899d7c8-vlbqf" Jul 06 22:10:28.059460 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:28.059209 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/491f519b-6854-4770-876e-df3f7fe5bb84-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-67d899d7c8-vlbqf\" (UID: \"491f519b-6854-4770-876e-df3f7fe5bb84\") " pod="openshift-monitoring/telemeter-client-67d899d7c8-vlbqf" Jul 06 22:10:28.065340 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:28.065315 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfqwj\" (UniqueName: \"kubernetes.io/projected/491f519b-6854-4770-876e-df3f7fe5bb84-kube-api-access-hfqwj\") pod \"telemeter-client-67d899d7c8-vlbqf\" (UID: \"491f519b-6854-4770-876e-df3f7fe5bb84\") " pod="openshift-monitoring/telemeter-client-67d899d7c8-vlbqf" Jul 06 22:10:28.195169 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:28.195128 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-67d899d7c8-vlbqf" Jul 06 22:10:28.358292 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:28.358253 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-67d899d7c8-vlbqf"] Jul 06 22:10:28.362882 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:10:28.362850 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod491f519b_6854_4770_876e_df3f7fe5bb84.slice/crio-d23416a61172fd311a62ebf6ca468fecbd152554d4c182ba69c12469a4a79d22 WatchSource:0}: Error finding container d23416a61172fd311a62ebf6ca468fecbd152554d4c182ba69c12469a4a79d22: Status 404 returned error can't find the container with id d23416a61172fd311a62ebf6ca468fecbd152554d4c182ba69c12469a4a79d22 Jul 06 22:10:28.581597 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:28.581502 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c4946bb79-24mzf"] Jul 06 22:10:28.611646 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:28.610929 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-86b5c4948c-cf5nj"] Jul 06 22:10:28.615879 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:28.615525 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86b5c4948c-cf5nj" Jul 06 22:10:28.628598 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:28.628567 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86b5c4948c-cf5nj"] Jul 06 22:10:28.662753 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:28.662532 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2f54d0d8-ac21-478b-b687-976f9732aba7-console-config\") pod \"console-86b5c4948c-cf5nj\" (UID: \"2f54d0d8-ac21-478b-b687-976f9732aba7\") " pod="openshift-console/console-86b5c4948c-cf5nj" Jul 06 22:10:28.662753 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:28.662583 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2f54d0d8-ac21-478b-b687-976f9732aba7-oauth-serving-cert\") pod \"console-86b5c4948c-cf5nj\" (UID: \"2f54d0d8-ac21-478b-b687-976f9732aba7\") " pod="openshift-console/console-86b5c4948c-cf5nj" Jul 06 22:10:28.662753 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:28.662652 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-679wx\" (UniqueName: \"kubernetes.io/projected/2f54d0d8-ac21-478b-b687-976f9732aba7-kube-api-access-679wx\") pod \"console-86b5c4948c-cf5nj\" (UID: \"2f54d0d8-ac21-478b-b687-976f9732aba7\") " pod="openshift-console/console-86b5c4948c-cf5nj" Jul 06 22:10:28.662753 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:28.662686 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f54d0d8-ac21-478b-b687-976f9732aba7-console-serving-cert\") pod \"console-86b5c4948c-cf5nj\" (UID: \"2f54d0d8-ac21-478b-b687-976f9732aba7\") " pod="openshift-console/console-86b5c4948c-cf5nj" Jul 06 22:10:28.662753 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:28.662725 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2f54d0d8-ac21-478b-b687-976f9732aba7-console-oauth-config\") pod \"console-86b5c4948c-cf5nj\" (UID: \"2f54d0d8-ac21-478b-b687-976f9732aba7\") " pod="openshift-console/console-86b5c4948c-cf5nj" Jul 06 22:10:28.663168 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:28.662819 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2f54d0d8-ac21-478b-b687-976f9732aba7-service-ca\") pod \"console-86b5c4948c-cf5nj\" (UID: \"2f54d0d8-ac21-478b-b687-976f9732aba7\") " pod="openshift-console/console-86b5c4948c-cf5nj" Jul 06 22:10:28.663168 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:28.662851 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f54d0d8-ac21-478b-b687-976f9732aba7-trusted-ca-bundle\") pod \"console-86b5c4948c-cf5nj\" (UID: \"2f54d0d8-ac21-478b-b687-976f9732aba7\") " pod="openshift-console/console-86b5c4948c-cf5nj" Jul 06 22:10:28.670568 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:28.670535 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-c4946bb79-24mzf" Jul 06 22:10:28.764138 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:28.764096 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-679wx\" (UniqueName: \"kubernetes.io/projected/2f54d0d8-ac21-478b-b687-976f9732aba7-kube-api-access-679wx\") pod \"console-86b5c4948c-cf5nj\" (UID: \"2f54d0d8-ac21-478b-b687-976f9732aba7\") " pod="openshift-console/console-86b5c4948c-cf5nj" Jul 06 22:10:28.764399 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:28.764159 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f54d0d8-ac21-478b-b687-976f9732aba7-console-serving-cert\") pod \"console-86b5c4948c-cf5nj\" (UID: \"2f54d0d8-ac21-478b-b687-976f9732aba7\") " pod="openshift-console/console-86b5c4948c-cf5nj" Jul 06 22:10:28.764399 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:28.764204 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2f54d0d8-ac21-478b-b687-976f9732aba7-console-oauth-config\") pod \"console-86b5c4948c-cf5nj\" (UID: \"2f54d0d8-ac21-478b-b687-976f9732aba7\") " pod="openshift-console/console-86b5c4948c-cf5nj" Jul 06 22:10:28.764399 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:28.764235 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2f54d0d8-ac21-478b-b687-976f9732aba7-service-ca\") pod \"console-86b5c4948c-cf5nj\" (UID: \"2f54d0d8-ac21-478b-b687-976f9732aba7\") " pod="openshift-console/console-86b5c4948c-cf5nj" Jul 06 22:10:28.764399 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:28.764262 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f54d0d8-ac21-478b-b687-976f9732aba7-trusted-ca-bundle\") pod \"console-86b5c4948c-cf5nj\" (UID: \"2f54d0d8-ac21-478b-b687-976f9732aba7\") " pod="openshift-console/console-86b5c4948c-cf5nj" Jul 06 22:10:28.764399 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:28.764327 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2f54d0d8-ac21-478b-b687-976f9732aba7-console-config\") pod \"console-86b5c4948c-cf5nj\" (UID: \"2f54d0d8-ac21-478b-b687-976f9732aba7\") " pod="openshift-console/console-86b5c4948c-cf5nj" Jul 06 22:10:28.764399 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:28.764357 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2f54d0d8-ac21-478b-b687-976f9732aba7-oauth-serving-cert\") pod \"console-86b5c4948c-cf5nj\" (UID: \"2f54d0d8-ac21-478b-b687-976f9732aba7\") " pod="openshift-console/console-86b5c4948c-cf5nj" Jul 06 22:10:28.766938 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:28.765139 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2f54d0d8-ac21-478b-b687-976f9732aba7-oauth-serving-cert\") pod \"console-86b5c4948c-cf5nj\" (UID: \"2f54d0d8-ac21-478b-b687-976f9732aba7\") " pod="openshift-console/console-86b5c4948c-cf5nj" Jul 06 22:10:28.766938 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:28.766335 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2f54d0d8-ac21-478b-b687-976f9732aba7-service-ca\") pod \"console-86b5c4948c-cf5nj\" (UID: \"2f54d0d8-ac21-478b-b687-976f9732aba7\") " pod="openshift-console/console-86b5c4948c-cf5nj" Jul 06 22:10:28.766938 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:28.766894 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f54d0d8-ac21-478b-b687-976f9732aba7-trusted-ca-bundle\") pod \"console-86b5c4948c-cf5nj\" (UID: \"2f54d0d8-ac21-478b-b687-976f9732aba7\") " pod="openshift-console/console-86b5c4948c-cf5nj" Jul 06 22:10:28.767289 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:28.767264 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2f54d0d8-ac21-478b-b687-976f9732aba7-console-config\") pod \"console-86b5c4948c-cf5nj\" (UID: \"2f54d0d8-ac21-478b-b687-976f9732aba7\") " pod="openshift-console/console-86b5c4948c-cf5nj" Jul 06 22:10:28.774040 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:28.773992 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f54d0d8-ac21-478b-b687-976f9732aba7-console-serving-cert\") pod \"console-86b5c4948c-cf5nj\" (UID: \"2f54d0d8-ac21-478b-b687-976f9732aba7\") " pod="openshift-console/console-86b5c4948c-cf5nj" Jul 06 22:10:28.775285 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:28.775230 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2f54d0d8-ac21-478b-b687-976f9732aba7-console-oauth-config\") pod \"console-86b5c4948c-cf5nj\" (UID: \"2f54d0d8-ac21-478b-b687-976f9732aba7\") " pod="openshift-console/console-86b5c4948c-cf5nj" Jul 06 22:10:28.777285 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:28.777238 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-679wx\" (UniqueName: \"kubernetes.io/projected/2f54d0d8-ac21-478b-b687-976f9732aba7-kube-api-access-679wx\") pod \"console-86b5c4948c-cf5nj\" (UID: \"2f54d0d8-ac21-478b-b687-976f9732aba7\") " pod="openshift-console/console-86b5c4948c-cf5nj" Jul 06 22:10:28.853578 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:28.853464 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75c6b47584-6bdk2" event={"ID":"d1a7e001-36db-4a34-ae02-b8c05ef42d77","Type":"ContainerStarted","Data":"db88182948c5f172282f7224f2dbcb55bf8e79f1c87556feab5e5e619f372876"} Jul 06 22:10:28.855052 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:28.855025 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-67d899d7c8-vlbqf" event={"ID":"491f519b-6854-4770-876e-df3f7fe5bb84","Type":"ContainerStarted","Data":"d23416a61172fd311a62ebf6ca468fecbd152554d4c182ba69c12469a4a79d22"} Jul 06 22:10:28.856451 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:28.856368 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-swjnf" event={"ID":"74152ba8-c71c-42ad-8109-60c71839ae56","Type":"ContainerStarted","Data":"ef8373d0909d52923554e8c7762941b1d7a6dbc34facaf0869432828f6203c63"} Jul 06 22:10:28.933373 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:28.932107 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86b5c4948c-cf5nj" Jul 06 22:10:28.937966 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:28.936491 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 22:10:28.941224 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:28.941192 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:10:28.944879 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:28.944245 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 06 22:10:28.944879 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:28.944325 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 06 22:10:28.944879 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:28.944485 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 06 22:10:28.944879 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:28.944559 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-s6xbw\"" Jul 06 22:10:28.944879 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:28.944583 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 06 22:10:28.944879 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:28.944245 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 06 22:10:28.944879 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:28.944762 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 06 22:10:28.944879 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:28.944781 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 06 22:10:28.944879 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:28.944788 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 06 22:10:28.945395 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:28.945189 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 06 22:10:28.947835 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:28.947674 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 06 22:10:28.951960 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:28.950736 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 06 22:10:28.951960 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:28.950923 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 06 22:10:28.951960 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:28.951126 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3ure0b77nbuid\"" Jul 06 22:10:28.951960 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:28.951306 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 06 22:10:28.952955 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:28.952725 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 06 22:10:28.960620 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:28.958311 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 22:10:29.068338 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:29.067403 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:10:29.068338 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:29.067463 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:10:29.068338 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:29.067507 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:10:29.068338 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:29.067528 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:10:29.068338 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:29.067552 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:10:29.068338 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:29.067595 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-config-out\") pod \"prometheus-k8s-0\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:10:29.068338 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:29.067640 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:10:29.068338 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:29.067661 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlv7q\" (UniqueName: \"kubernetes.io/projected/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-kube-api-access-rlv7q\") pod \"prometheus-k8s-0\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:10:29.068338 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:29.067687 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-config\") pod \"prometheus-k8s-0\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:10:29.068338 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:29.067710 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:10:29.068338 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:29.067731 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:10:29.068338 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:29.067762 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:10:29.068338 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:29.067784 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:10:29.068338 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:29.067834 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:10:29.068338 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:29.067876 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:10:29.068338 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:29.067900 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:10:29.069215 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:29.067921 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:10:29.069215 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:29.067968 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:10:29.069215 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:29.068049 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-web-config\") pod \"prometheus-k8s-0\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:10:29.069215 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:29.068099 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:10:29.169411 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:29.169370 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-config-out\") pod \"prometheus-k8s-0\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:10:29.169578 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:29.169434 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:10:29.169578 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:29.169464 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlv7q\" (UniqueName: \"kubernetes.io/projected/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-kube-api-access-rlv7q\") pod \"prometheus-k8s-0\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:10:29.169707 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:29.169634 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-config\") pod \"prometheus-k8s-0\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:10:29.169707 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:29.169676 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:10:29.169707 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:29.169704 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:10:29.169846 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:29.169738 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:10:29.169846 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:29.169761 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:10:29.169846 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:29.169826 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:10:29.169982 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:29.169861 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:10:29.169982 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:29.169885 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:10:29.169982 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:29.169910 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:10:29.169982 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:29.169950 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:10:29.170200 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:29.170022 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-web-config\") pod \"prometheus-k8s-0\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:10:29.170200 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:29.170065 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:10:29.170200 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:29.170101 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:10:29.170200 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:29.170130 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:10:29.170200 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:29.170139 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:10:29.170200 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:29.170178 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:10:29.171124 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:29.170839 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:10:29.171124 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:29.170894 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:10:29.171124 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:29.170925 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:10:29.171370 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:29.171242 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:10:29.171435 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:29.171371 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:10:29.174586 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:29.171513 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:10:29.174586 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:29.174518 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:10:29.177459 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:29.177367 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:10:29.177835 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:29.177809 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:10:29.177969 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:29.177949 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:10:29.178272 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:29.178249 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:10:29.178340 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:29.178319 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-config\") pod \"prometheus-k8s-0\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:10:29.178626 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:29.178596 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-config-out\") pod \"prometheus-k8s-0\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:10:29.178689 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:29.178595 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:10:29.180105 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:29.178894 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:10:29.180105 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:29.179023 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:10:29.180105 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:29.179451 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:10:29.180105 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:29.179620 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:10:29.187628 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:29.187572 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-web-config\") pod \"prometheus-k8s-0\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:10:29.187804 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:29.187679 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:10:29.190117 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:29.190090 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlv7q\" (UniqueName: \"kubernetes.io/projected/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-kube-api-access-rlv7q\") pod \"prometheus-k8s-0\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:10:29.258508 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:29.258234 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:10:30.507055 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:30.506690 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86b5c4948c-cf5nj"] Jul 06 22:10:30.565916 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:30.565529 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 22:10:30.568903 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:10:30.568871 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3e80f4a_8b86_4fd5_9a7f_2e20f93e884d.slice/crio-be3ba65842b888422e36e96971b10624d71a9b5136c1f6ec39a08357ac1afd5a WatchSource:0}: Error finding container be3ba65842b888422e36e96971b10624d71a9b5136c1f6ec39a08357ac1afd5a: Status 404 returned error can't find the container with id be3ba65842b888422e36e96971b10624d71a9b5136c1f6ec39a08357ac1afd5a Jul 06 22:10:30.868332 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:30.868299 2555 generic.go:356] "Generic (PLEG): container finished" podID="a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d" containerID="d48a4ada40084d249f040e237b4daac841bd3fd6fae77bcfcfa8300ba443025e" exitCode=0 Jul 06 22:10:30.868500 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:30.868394 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d","Type":"ContainerDied","Data":"d48a4ada40084d249f040e237b4daac841bd3fd6fae77bcfcfa8300ba443025e"} Jul 06 22:10:30.868500 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:30.868492 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d","Type":"ContainerStarted","Data":"be3ba65842b888422e36e96971b10624d71a9b5136c1f6ec39a08357ac1afd5a"} Jul 06 22:10:30.872624 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:30.872600 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75c6b47584-6bdk2" event={"ID":"d1a7e001-36db-4a34-ae02-b8c05ef42d77","Type":"ContainerStarted","Data":"459e06b8c1f42ea1377f6ad1ce0a524bb2905d901a2edc9e1debc479e67b1063"} Jul 06 22:10:30.872708 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:30.872678 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75c6b47584-6bdk2" event={"ID":"d1a7e001-36db-4a34-ae02-b8c05ef42d77","Type":"ContainerStarted","Data":"c943cd2968097597819ad3e3bde8835a7555aaf808131ce0576061c7418a4962"} Jul 06 22:10:30.872763 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:30.872727 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-75c6b47584-6bdk2" Jul 06 22:10:30.872817 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:30.872770 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75c6b47584-6bdk2" event={"ID":"d1a7e001-36db-4a34-ae02-b8c05ef42d77","Type":"ContainerStarted","Data":"83f98ff4c348fd2e057bbc9e728979e27d7974332908f1008450edf7f7f4dc1f"} Jul 06 22:10:30.874676 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:30.874651 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86b5c4948c-cf5nj" event={"ID":"2f54d0d8-ac21-478b-b687-976f9732aba7","Type":"ContainerStarted","Data":"ca11e46517c42f05ee2f40b808ce0bcbf4fda271a3139a95a60ede25b67c4e4f"} Jul 06 22:10:30.874767 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:30.874694 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86b5c4948c-cf5nj" event={"ID":"2f54d0d8-ac21-478b-b687-976f9732aba7","Type":"ContainerStarted","Data":"fb3b0a094a8b7a0a93000c76a5ce290e254a7f54d09bb849f10ed85c0932b406"} Jul 06 22:10:30.878058 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:30.878031 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"50356448-5321-4011-ba4c-fab5b999a2c2","Type":"ContainerStarted","Data":"3af79b72081672f7cfdfacf59ab47451dd941da25ff09e98a371de77939acfb7"} Jul 06 22:10:30.878160 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:30.878098 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"50356448-5321-4011-ba4c-fab5b999a2c2","Type":"ContainerStarted","Data":"23318c1d5d9b9e1aee7f37b4c63df21115f1ee1c22c53d3901f6e1841acd1db8"} Jul 06 22:10:30.878160 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:30.878140 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"50356448-5321-4011-ba4c-fab5b999a2c2","Type":"ContainerStarted","Data":"161bec23d11f8d8aec191d7562f1da1a5995d49eaa35965d18193b118e3467e5"} Jul 06 22:10:30.878266 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:30.878179 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"50356448-5321-4011-ba4c-fab5b999a2c2","Type":"ContainerStarted","Data":"161b1fc3163c7d9902c1c8b23d4c77ed27bde86a77d51230b20d67dd1bafa222"} Jul 06 22:10:30.879799 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:30.879775 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5678f86565-bhz5r" event={"ID":"a0bd2d8d-b4a6-456b-9f88-fb699e4ed523","Type":"ContainerStarted","Data":"2cf5e127720847e85f19a4998fc60ddbfa02ad678f236428fbd9758c7952ed35"} Jul 06 22:10:30.881593 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:30.881568 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-swjnf" event={"ID":"74152ba8-c71c-42ad-8109-60c71839ae56","Type":"ContainerStarted","Data":"7f776627c92516d6f360057572b5ea0c2dc760988dbf1db3236a7185c4b25377"} Jul 06 22:10:30.881701 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:30.881685 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-swjnf" Jul 06 22:10:30.887442 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:30.887419 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-swjnf" Jul 06 22:10:30.915275 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:30.915213 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-swjnf" podStartSLOduration=1.5440194 podStartE2EDuration="3.915191411s" totalImagesPullingTime="2.371172011s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:10:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:10:30.913817743 +0000 UTC m=+82.213529615" watchObservedRunningTime="2026-07-06 22:10:30.915191411 +0000 UTC m=+82.214903260" Jul 06 22:10:30.956265 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:30.956202 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-75c6b47584-6bdk2" podStartSLOduration=2.233766246 podStartE2EDuration="6.956174897s" totalImagesPullingTime="4.722408651s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:10:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:10:30.940118005 +0000 UTC m=+82.239829855" watchObservedRunningTime="2026-07-06 22:10:30.956174897 +0000 UTC m=+82.255886749" Jul 06 22:10:30.978213 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:30.978157 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-86b5c4948c-cf5nj" podStartSLOduration=2.978135043 podStartE2EDuration="2.978135043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:10:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:10:30.975584874 +0000 UTC m=+82.275296758" watchObservedRunningTime="2026-07-06 22:10:30.978135043 +0000 UTC m=+82.277846903" Jul 06 22:10:31.004164 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:31.004112 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5678f86565-bhz5r" podStartSLOduration=1.4438210329999999 podStartE2EDuration="4.004096394s" totalImagesPullingTime="2.560275361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:10:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:10:31.002806818 +0000 UTC m=+82.302518705" watchObservedRunningTime="2026-07-06 22:10:31.004096394 +0000 UTC m=+82.303808243" Jul 06 22:10:31.890242 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:31.890209 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"50356448-5321-4011-ba4c-fab5b999a2c2","Type":"ContainerStarted","Data":"4bb7a61ab7f0b688773a39e8ca4a87853fce4c24962a3f489c657a2671caf087"} Jul 06 22:10:31.890643 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:31.890316 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"50356448-5321-4011-ba4c-fab5b999a2c2","Type":"ContainerStarted","Data":"91dd121750579d37564423c9038631255115f82564e95872e706e49faa57dc57"} Jul 06 22:10:31.892439 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:31.892408 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-67d899d7c8-vlbqf" event={"ID":"491f519b-6854-4770-876e-df3f7fe5bb84","Type":"ContainerStarted","Data":"5250366adaf69a4b3661bc0d9eaa6cf7e7a97bb6f4043e815e5160181ff2a3e7"} Jul 06 22:10:31.892575 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:31.892470 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-67d899d7c8-vlbqf" event={"ID":"491f519b-6854-4770-876e-df3f7fe5bb84","Type":"ContainerStarted","Data":"26abb207b3a46008c168d19cb290ec5c1668d042c967ff66a5dcffe5ce4018b2"} Jul 06 22:10:31.892575 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:31.892502 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-67d899d7c8-vlbqf" event={"ID":"491f519b-6854-4770-876e-df3f7fe5bb84","Type":"ContainerStarted","Data":"0d9091cd5dd689974b504dd606c0aa51be099574f828df97dbea7a942dd95774"} Jul 06 22:10:31.919452 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:31.919398 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.086981354 podStartE2EDuration="8.919382312s" totalImagesPullingTime="4.810325806s" totalInitContainerRuntime="22.075152ms" isStatefulPod=false podCreationTimestamp="2026-07-06 22:10:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:10:31.918959466 +0000 UTC m=+83.218671354" watchObservedRunningTime="2026-07-06 22:10:31.919382312 +0000 UTC m=+83.219094160" Jul 06 22:10:31.944441 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:31.944389 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-67d899d7c8-vlbqf" podStartSLOduration=2.036686172 podStartE2EDuration="4.944365031s" totalImagesPullingTime="2.907678859s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:10:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:10:31.942766755 +0000 UTC m=+83.242478606" watchObservedRunningTime="2026-07-06 22:10:31.944365031 +0000 UTC m=+83.244076882" Jul 06 22:10:32.816297 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:32.816265 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86b5c4948c-cf5nj"] Jul 06 22:10:34.913751 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:34.913703 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d","Type":"ContainerStarted","Data":"868f0e2e2832f412be2280dceca76d6075b832497bd7017e0cdbd6c7709d4a8e"} Jul 06 22:10:34.914112 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:34.913791 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d","Type":"ContainerStarted","Data":"70c341aec9d02c2801bd2464b7fdd403607f98f4669b5a840b5d396ea57352c2"} Jul 06 22:10:34.914112 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:34.913835 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d","Type":"ContainerStarted","Data":"7d22f9d9188e3d530d12c1f8c8f9534cde4bab8897f3fba95c2c73cd6e9d95e2"} Jul 06 22:10:35.921343 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:35.921305 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d","Type":"ContainerStarted","Data":"dce1ad64f7310c0e3f6de2bb630af3cd4c4168932a432f48be33cde6c84ea430"} Jul 06 22:10:35.921805 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:35.921374 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d","Type":"ContainerStarted","Data":"90c27314e532c2b8d3dce39f5fd09e6928ef39e4fb97a65d2fb8850cd5049e25"} Jul 06 22:10:35.921805 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:35.921403 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d","Type":"ContainerStarted","Data":"b76c28e5cf3ded5c5d3a2bbddc03ad092e65235b16f6341a2d45dffe87074a17"} Jul 06 22:10:35.957678 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:35.957621 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.149980994 podStartE2EDuration="7.957604092s" totalImagesPullingTime="3.78126756s" totalInitContainerRuntime="26.355538ms" isStatefulPod=false podCreationTimestamp="2026-07-06 22:10:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:10:35.954756258 +0000 UTC m=+87.254468108" watchObservedRunningTime="2026-07-06 22:10:35.957604092 +0000 UTC m=+87.257315940" Jul 06 22:10:36.899769 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:36.899742 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-75c6b47584-6bdk2" Jul 06 22:10:38.933070 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:38.933027 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-86b5c4948c-cf5nj" Jul 06 22:10:39.259747 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:39.259660 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:10:44.248221 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:44.248187 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7cf8c5f7d7-xtpbg"] Jul 06 22:10:47.383807 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:47.383764 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5678f86565-bhz5r" Jul 06 22:10:47.384470 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:47.383846 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5678f86565-bhz5r" Jul 06 22:10:53.611600 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:53.611523 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-c4946bb79-24mzf" podUID="3d9c3b90-9586-4687-a6ee-5424470f92d5" containerName="console" containerID="cri-o://c4ab27e490c205cfb57b00c58233a9cf93985bee23815eb1110da29e7043094f" gracePeriod=15 Jul 06 22:10:54.006505 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:54.006474 2555 generic.go:356] "Generic (PLEG): container finished" podID="3d9c3b90-9586-4687-a6ee-5424470f92d5" containerID="c4ab27e490c205cfb57b00c58233a9cf93985bee23815eb1110da29e7043094f" exitCode=0 Jul 06 22:10:54.006677 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:54.006554 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c4946bb79-24mzf" event={"ID":"3d9c3b90-9586-4687-a6ee-5424470f92d5","Type":"ContainerDied","Data":"c4ab27e490c205cfb57b00c58233a9cf93985bee23815eb1110da29e7043094f"} Jul 06 22:10:54.374436 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:54.374407 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c4946bb79-24mzf" Jul 06 22:10:54.415285 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:54.415248 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3d9c3b90-9586-4687-a6ee-5424470f92d5-console-config\") pod \"3d9c3b90-9586-4687-a6ee-5424470f92d5\" (UID: \"3d9c3b90-9586-4687-a6ee-5424470f92d5\") " Jul 06 22:10:54.415285 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:54.415289 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3d9c3b90-9586-4687-a6ee-5424470f92d5-console-oauth-config\") pod \"3d9c3b90-9586-4687-a6ee-5424470f92d5\" (UID: \"3d9c3b90-9586-4687-a6ee-5424470f92d5\") " Jul 06 22:10:54.415548 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:54.415348 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d9c3b90-9586-4687-a6ee-5424470f92d5-trusted-ca-bundle\") pod \"3d9c3b90-9586-4687-a6ee-5424470f92d5\" (UID: \"3d9c3b90-9586-4687-a6ee-5424470f92d5\") " Jul 06 22:10:54.415548 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:54.415384 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3d9c3b90-9586-4687-a6ee-5424470f92d5-console-serving-cert\") pod \"3d9c3b90-9586-4687-a6ee-5424470f92d5\" (UID: \"3d9c3b90-9586-4687-a6ee-5424470f92d5\") " Jul 06 22:10:54.415548 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:54.415407 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hwxj7\" (UniqueName: \"kubernetes.io/projected/3d9c3b90-9586-4687-a6ee-5424470f92d5-kube-api-access-hwxj7\") pod \"3d9c3b90-9586-4687-a6ee-5424470f92d5\" (UID: \"3d9c3b90-9586-4687-a6ee-5424470f92d5\") " Jul 06 22:10:54.415548 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:54.415442 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3d9c3b90-9586-4687-a6ee-5424470f92d5-service-ca\") pod \"3d9c3b90-9586-4687-a6ee-5424470f92d5\" (UID: \"3d9c3b90-9586-4687-a6ee-5424470f92d5\") " Jul 06 22:10:54.415548 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:54.415475 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3d9c3b90-9586-4687-a6ee-5424470f92d5-oauth-serving-cert\") pod \"3d9c3b90-9586-4687-a6ee-5424470f92d5\" (UID: \"3d9c3b90-9586-4687-a6ee-5424470f92d5\") " Jul 06 22:10:54.415810 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:54.415672 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d9c3b90-9586-4687-a6ee-5424470f92d5-console-config" (OuterVolumeSpecName: "console-config") pod "3d9c3b90-9586-4687-a6ee-5424470f92d5" (UID: "3d9c3b90-9586-4687-a6ee-5424470f92d5"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:10:54.415810 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:54.415750 2555 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3d9c3b90-9586-4687-a6ee-5424470f92d5-console-config\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:10:54.416042 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:54.415988 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d9c3b90-9586-4687-a6ee-5424470f92d5-service-ca" (OuterVolumeSpecName: "service-ca") pod "3d9c3b90-9586-4687-a6ee-5424470f92d5" (UID: "3d9c3b90-9586-4687-a6ee-5424470f92d5"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:10:54.416042 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:54.416020 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d9c3b90-9586-4687-a6ee-5424470f92d5-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3d9c3b90-9586-4687-a6ee-5424470f92d5" (UID: "3d9c3b90-9586-4687-a6ee-5424470f92d5"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:10:54.416226 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:54.416196 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d9c3b90-9586-4687-a6ee-5424470f92d5-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "3d9c3b90-9586-4687-a6ee-5424470f92d5" (UID: "3d9c3b90-9586-4687-a6ee-5424470f92d5"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:10:54.417795 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:54.417772 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d9c3b90-9586-4687-a6ee-5424470f92d5-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3d9c3b90-9586-4687-a6ee-5424470f92d5" (UID: "3d9c3b90-9586-4687-a6ee-5424470f92d5"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:10:54.418042 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:54.417997 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d9c3b90-9586-4687-a6ee-5424470f92d5-kube-api-access-hwxj7" (OuterVolumeSpecName: "kube-api-access-hwxj7") pod "3d9c3b90-9586-4687-a6ee-5424470f92d5" (UID: "3d9c3b90-9586-4687-a6ee-5424470f92d5"). InnerVolumeSpecName "kube-api-access-hwxj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:10:54.418124 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:54.418053 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d9c3b90-9586-4687-a6ee-5424470f92d5-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3d9c3b90-9586-4687-a6ee-5424470f92d5" (UID: "3d9c3b90-9586-4687-a6ee-5424470f92d5"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:10:54.516299 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:54.516210 2555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d9c3b90-9586-4687-a6ee-5424470f92d5-trusted-ca-bundle\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:10:54.516299 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:54.516239 2555 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3d9c3b90-9586-4687-a6ee-5424470f92d5-console-serving-cert\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:10:54.516299 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:54.516249 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwxj7\" (UniqueName: \"kubernetes.io/projected/3d9c3b90-9586-4687-a6ee-5424470f92d5-kube-api-access-hwxj7\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:10:54.516299 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:54.516260 2555 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3d9c3b90-9586-4687-a6ee-5424470f92d5-service-ca\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:10:54.516299 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:54.516268 2555 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3d9c3b90-9586-4687-a6ee-5424470f92d5-oauth-serving-cert\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:10:54.516299 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:54.516276 2555 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3d9c3b90-9586-4687-a6ee-5424470f92d5-console-oauth-config\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:10:54.814045 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:54.813938 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-vdbr6" Jul 06 22:10:55.012699 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:55.012671 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c4946bb79-24mzf" Jul 06 22:10:55.012887 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:55.012677 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c4946bb79-24mzf" event={"ID":"3d9c3b90-9586-4687-a6ee-5424470f92d5","Type":"ContainerDied","Data":"fa28e70ce44beb4944b3099a9cb3df27d3e0da336885c5700977dbbf0cb08034"} Jul 06 22:10:55.012887 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:55.012797 2555 scope.go:117] "RemoveContainer" containerID="c4ab27e490c205cfb57b00c58233a9cf93985bee23815eb1110da29e7043094f" Jul 06 22:10:55.036314 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:55.036284 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c4946bb79-24mzf"] Jul 06 22:10:55.040597 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:55.040570 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-c4946bb79-24mzf"] Jul 06 22:10:55.338965 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:55.338934 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d9c3b90-9586-4687-a6ee-5424470f92d5" path="/var/lib/kubelet/pods/3d9c3b90-9586-4687-a6ee-5424470f92d5/volumes" Jul 06 22:10:57.924726 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:57.924687 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-86b5c4948c-cf5nj" podUID="2f54d0d8-ac21-478b-b687-976f9732aba7" containerName="console" containerID="cri-o://ca11e46517c42f05ee2f40b808ce0bcbf4fda271a3139a95a60ede25b67c4e4f" gracePeriod=15 Jul 06 22:10:58.223607 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:58.223571 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86b5c4948c-cf5nj" Jul 06 22:10:58.250317 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:58.250287 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2f54d0d8-ac21-478b-b687-976f9732aba7-oauth-serving-cert\") pod \"2f54d0d8-ac21-478b-b687-976f9732aba7\" (UID: \"2f54d0d8-ac21-478b-b687-976f9732aba7\") " Jul 06 22:10:58.250482 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:58.250335 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2f54d0d8-ac21-478b-b687-976f9732aba7-console-config\") pod \"2f54d0d8-ac21-478b-b687-976f9732aba7\" (UID: \"2f54d0d8-ac21-478b-b687-976f9732aba7\") " Jul 06 22:10:58.250482 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:58.250365 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f54d0d8-ac21-478b-b687-976f9732aba7-console-serving-cert\") pod \"2f54d0d8-ac21-478b-b687-976f9732aba7\" (UID: \"2f54d0d8-ac21-478b-b687-976f9732aba7\") " Jul 06 22:10:58.250482 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:58.250430 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f54d0d8-ac21-478b-b687-976f9732aba7-trusted-ca-bundle\") pod \"2f54d0d8-ac21-478b-b687-976f9732aba7\" (UID: \"2f54d0d8-ac21-478b-b687-976f9732aba7\") " Jul 06 22:10:58.250482 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:58.250471 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-679wx\" (UniqueName: \"kubernetes.io/projected/2f54d0d8-ac21-478b-b687-976f9732aba7-kube-api-access-679wx\") pod \"2f54d0d8-ac21-478b-b687-976f9732aba7\" (UID: \"2f54d0d8-ac21-478b-b687-976f9732aba7\") " Jul 06 22:10:58.250678 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:58.250494 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2f54d0d8-ac21-478b-b687-976f9732aba7-console-oauth-config\") pod \"2f54d0d8-ac21-478b-b687-976f9732aba7\" (UID: \"2f54d0d8-ac21-478b-b687-976f9732aba7\") " Jul 06 22:10:58.250678 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:58.250530 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2f54d0d8-ac21-478b-b687-976f9732aba7-service-ca\") pod \"2f54d0d8-ac21-478b-b687-976f9732aba7\" (UID: \"2f54d0d8-ac21-478b-b687-976f9732aba7\") " Jul 06 22:10:58.250773 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:58.250749 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f54d0d8-ac21-478b-b687-976f9732aba7-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2f54d0d8-ac21-478b-b687-976f9732aba7" (UID: "2f54d0d8-ac21-478b-b687-976f9732aba7"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:10:58.250826 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:58.250788 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f54d0d8-ac21-478b-b687-976f9732aba7-console-config" (OuterVolumeSpecName: "console-config") pod "2f54d0d8-ac21-478b-b687-976f9732aba7" (UID: "2f54d0d8-ac21-478b-b687-976f9732aba7"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:10:58.251201 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:58.251176 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f54d0d8-ac21-478b-b687-976f9732aba7-service-ca" (OuterVolumeSpecName: "service-ca") pod "2f54d0d8-ac21-478b-b687-976f9732aba7" (UID: "2f54d0d8-ac21-478b-b687-976f9732aba7"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:10:58.251328 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:58.251180 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f54d0d8-ac21-478b-b687-976f9732aba7-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "2f54d0d8-ac21-478b-b687-976f9732aba7" (UID: "2f54d0d8-ac21-478b-b687-976f9732aba7"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:10:58.252836 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:58.252816 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f54d0d8-ac21-478b-b687-976f9732aba7-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2f54d0d8-ac21-478b-b687-976f9732aba7" (UID: "2f54d0d8-ac21-478b-b687-976f9732aba7"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:10:58.253111 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:58.253082 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f54d0d8-ac21-478b-b687-976f9732aba7-kube-api-access-679wx" (OuterVolumeSpecName: "kube-api-access-679wx") pod "2f54d0d8-ac21-478b-b687-976f9732aba7" (UID: "2f54d0d8-ac21-478b-b687-976f9732aba7"). InnerVolumeSpecName "kube-api-access-679wx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:10:58.253202 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:58.253136 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f54d0d8-ac21-478b-b687-976f9732aba7-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2f54d0d8-ac21-478b-b687-976f9732aba7" (UID: "2f54d0d8-ac21-478b-b687-976f9732aba7"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:10:58.351753 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:58.351703 2555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f54d0d8-ac21-478b-b687-976f9732aba7-trusted-ca-bundle\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:10:58.351753 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:58.351743 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-679wx\" (UniqueName: \"kubernetes.io/projected/2f54d0d8-ac21-478b-b687-976f9732aba7-kube-api-access-679wx\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:10:58.351753 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:58.351756 2555 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2f54d0d8-ac21-478b-b687-976f9732aba7-console-oauth-config\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:10:58.351985 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:58.351769 2555 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2f54d0d8-ac21-478b-b687-976f9732aba7-service-ca\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:10:58.351985 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:58.351782 2555 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2f54d0d8-ac21-478b-b687-976f9732aba7-oauth-serving-cert\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:10:58.351985 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:58.351794 2555 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2f54d0d8-ac21-478b-b687-976f9732aba7-console-config\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:10:58.351985 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:58.351805 2555 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f54d0d8-ac21-478b-b687-976f9732aba7-console-serving-cert\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:10:59.033247 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:59.033201 2555 generic.go:356] "Generic (PLEG): container finished" podID="2f54d0d8-ac21-478b-b687-976f9732aba7" containerID="ca11e46517c42f05ee2f40b808ce0bcbf4fda271a3139a95a60ede25b67c4e4f" exitCode=0 Jul 06 22:10:59.033247 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:59.033247 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86b5c4948c-cf5nj" Jul 06 22:10:59.033711 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:59.033256 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86b5c4948c-cf5nj" event={"ID":"2f54d0d8-ac21-478b-b687-976f9732aba7","Type":"ContainerDied","Data":"ca11e46517c42f05ee2f40b808ce0bcbf4fda271a3139a95a60ede25b67c4e4f"} Jul 06 22:10:59.033711 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:59.033306 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86b5c4948c-cf5nj" event={"ID":"2f54d0d8-ac21-478b-b687-976f9732aba7","Type":"ContainerDied","Data":"fb3b0a094a8b7a0a93000c76a5ce290e254a7f54d09bb849f10ed85c0932b406"} Jul 06 22:10:59.033711 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:59.033334 2555 scope.go:117] "RemoveContainer" containerID="ca11e46517c42f05ee2f40b808ce0bcbf4fda271a3139a95a60ede25b67c4e4f" Jul 06 22:10:59.042260 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:59.042220 2555 scope.go:117] "RemoveContainer" containerID="ca11e46517c42f05ee2f40b808ce0bcbf4fda271a3139a95a60ede25b67c4e4f" Jul 06 22:10:59.042585 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:10:59.042557 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca11e46517c42f05ee2f40b808ce0bcbf4fda271a3139a95a60ede25b67c4e4f\": container with ID starting with ca11e46517c42f05ee2f40b808ce0bcbf4fda271a3139a95a60ede25b67c4e4f not found: ID does not exist" containerID="ca11e46517c42f05ee2f40b808ce0bcbf4fda271a3139a95a60ede25b67c4e4f" Jul 06 22:10:59.042639 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:59.042593 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca11e46517c42f05ee2f40b808ce0bcbf4fda271a3139a95a60ede25b67c4e4f"} err="failed to get container status \"ca11e46517c42f05ee2f40b808ce0bcbf4fda271a3139a95a60ede25b67c4e4f\": rpc error: code = NotFound desc = could not find container \"ca11e46517c42f05ee2f40b808ce0bcbf4fda271a3139a95a60ede25b67c4e4f\": container with ID starting with ca11e46517c42f05ee2f40b808ce0bcbf4fda271a3139a95a60ede25b67c4e4f not found: ID does not exist" Jul 06 22:10:59.056449 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:59.056418 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86b5c4948c-cf5nj"] Jul 06 22:10:59.065048 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:59.064996 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-86b5c4948c-cf5nj"] Jul 06 22:10:59.339974 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:10:59.339892 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f54d0d8-ac21-478b-b687-976f9732aba7" path="/var/lib/kubelet/pods/2f54d0d8-ac21-478b-b687-976f9732aba7/volumes" Jul 06 22:11:07.389220 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:07.389192 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5678f86565-bhz5r" Jul 06 22:11:07.393040 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:07.392992 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5678f86565-bhz5r" Jul 06 22:11:09.268396 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:09.268337 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-7cf8c5f7d7-xtpbg" podUID="9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0" containerName="console" containerID="cri-o://346568f475d3dfc81965dfa5d37850e11930980584f85a2aadf6c158bc5c0105" gracePeriod=15 Jul 06 22:11:09.523051 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:09.523025 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cf8c5f7d7-xtpbg" Jul 06 22:11:09.657929 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:09.657885 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0-service-ca\") pod \"9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0\" (UID: \"9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0\") " Jul 06 22:11:09.657929 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:09.657936 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jhbxz\" (UniqueName: \"kubernetes.io/projected/9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0-kube-api-access-jhbxz\") pod \"9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0\" (UID: \"9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0\") " Jul 06 22:11:09.658218 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:09.657981 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0-console-oauth-config\") pod \"9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0\" (UID: \"9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0\") " Jul 06 22:11:09.658218 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:09.658044 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0-console-config\") pod \"9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0\" (UID: \"9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0\") " Jul 06 22:11:09.658218 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:09.658118 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0-console-serving-cert\") pod \"9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0\" (UID: \"9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0\") " Jul 06 22:11:09.658218 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:09.658154 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0-oauth-serving-cert\") pod \"9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0\" (UID: \"9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0\") " Jul 06 22:11:09.658447 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:09.658419 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0-service-ca" (OuterVolumeSpecName: "service-ca") pod "9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0" (UID: "9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:11:09.658680 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:09.658510 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0-console-config" (OuterVolumeSpecName: "console-config") pod "9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0" (UID: "9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:11:09.658680 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:09.658579 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0" (UID: "9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:11:09.660230 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:09.660206 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0" (UID: "9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:11:09.660323 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:09.660236 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0-kube-api-access-jhbxz" (OuterVolumeSpecName: "kube-api-access-jhbxz") pod "9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0" (UID: "9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0"). InnerVolumeSpecName "kube-api-access-jhbxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:11:09.660323 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:09.660276 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0" (UID: "9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:11:09.759621 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:09.759535 2555 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0-console-serving-cert\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:11:09.759621 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:09.759566 2555 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0-oauth-serving-cert\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:11:09.759621 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:09.759576 2555 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0-service-ca\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:11:09.759621 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:09.759586 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhbxz\" (UniqueName: \"kubernetes.io/projected/9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0-kube-api-access-jhbxz\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:11:09.759621 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:09.759595 2555 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0-console-oauth-config\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:11:09.759621 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:09.759604 2555 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0-console-config\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:11:10.089665 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:10.089574 2555 generic.go:356] "Generic (PLEG): container finished" podID="9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0" containerID="346568f475d3dfc81965dfa5d37850e11930980584f85a2aadf6c158bc5c0105" exitCode=0 Jul 06 22:11:10.089665 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:10.089622 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cf8c5f7d7-xtpbg" Jul 06 22:11:10.089665 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:10.089652 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cf8c5f7d7-xtpbg" event={"ID":"9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0","Type":"ContainerDied","Data":"346568f475d3dfc81965dfa5d37850e11930980584f85a2aadf6c158bc5c0105"} Jul 06 22:11:10.089878 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:10.089703 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cf8c5f7d7-xtpbg" event={"ID":"9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0","Type":"ContainerDied","Data":"a7fc64452237b315f7a9216b9d688d32112c141123d3d5d84ca90e51db17b6f0"} Jul 06 22:11:10.089878 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:10.089727 2555 scope.go:117] "RemoveContainer" containerID="346568f475d3dfc81965dfa5d37850e11930980584f85a2aadf6c158bc5c0105" Jul 06 22:11:10.098546 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:10.098525 2555 scope.go:117] "RemoveContainer" containerID="346568f475d3dfc81965dfa5d37850e11930980584f85a2aadf6c158bc5c0105" Jul 06 22:11:10.098802 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:11:10.098780 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"346568f475d3dfc81965dfa5d37850e11930980584f85a2aadf6c158bc5c0105\": container with ID starting with 346568f475d3dfc81965dfa5d37850e11930980584f85a2aadf6c158bc5c0105 not found: ID does not exist" containerID="346568f475d3dfc81965dfa5d37850e11930980584f85a2aadf6c158bc5c0105" Jul 06 22:11:10.098847 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:10.098812 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"346568f475d3dfc81965dfa5d37850e11930980584f85a2aadf6c158bc5c0105"} err="failed to get container status \"346568f475d3dfc81965dfa5d37850e11930980584f85a2aadf6c158bc5c0105\": rpc error: code = NotFound desc = could not find container \"346568f475d3dfc81965dfa5d37850e11930980584f85a2aadf6c158bc5c0105\": container with ID starting with 346568f475d3dfc81965dfa5d37850e11930980584f85a2aadf6c158bc5c0105 not found: ID does not exist" Jul 06 22:11:10.111442 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:10.111405 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7cf8c5f7d7-xtpbg"] Jul 06 22:11:10.112767 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:10.112743 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7cf8c5f7d7-xtpbg"] Jul 06 22:11:11.338749 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:11.338717 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0" path="/var/lib/kubelet/pods/9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0/volumes" Jul 06 22:11:29.259255 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:29.259216 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:11:29.278539 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:29.278511 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:11:30.185991 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:30.185962 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:11:43.070456 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:43.070424 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 22:11:43.070999 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:43.070737 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="50356448-5321-4011-ba4c-fab5b999a2c2" containerName="alertmanager" containerID="cri-o://161b1fc3163c7d9902c1c8b23d4c77ed27bde86a77d51230b20d67dd1bafa222" gracePeriod=120 Jul 06 22:11:43.070999 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:43.070774 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="50356448-5321-4011-ba4c-fab5b999a2c2" containerName="kube-rbac-proxy-metric" containerID="cri-o://91dd121750579d37564423c9038631255115f82564e95872e706e49faa57dc57" gracePeriod=120 Jul 06 22:11:43.070999 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:43.070836 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="50356448-5321-4011-ba4c-fab5b999a2c2" containerName="kube-rbac-proxy" containerID="cri-o://3af79b72081672f7cfdfacf59ab47451dd941da25ff09e98a371de77939acfb7" gracePeriod=120 Jul 06 22:11:43.070999 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:43.070824 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="50356448-5321-4011-ba4c-fab5b999a2c2" containerName="config-reloader" containerID="cri-o://161bec23d11f8d8aec191d7562f1da1a5995d49eaa35965d18193b118e3467e5" gracePeriod=120 Jul 06 22:11:43.070999 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:43.070807 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="50356448-5321-4011-ba4c-fab5b999a2c2" containerName="kube-rbac-proxy-web" containerID="cri-o://23318c1d5d9b9e1aee7f37b4c63df21115f1ee1c22c53d3901f6e1841acd1db8" gracePeriod=120 Jul 06 22:11:43.070999 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:43.070844 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="50356448-5321-4011-ba4c-fab5b999a2c2" containerName="prom-label-proxy" containerID="cri-o://4bb7a61ab7f0b688773a39e8ca4a87853fce4c24962a3f489c657a2671caf087" gracePeriod=120 Jul 06 22:11:43.226445 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:43.226402 2555 generic.go:356] "Generic (PLEG): container finished" podID="50356448-5321-4011-ba4c-fab5b999a2c2" containerID="4bb7a61ab7f0b688773a39e8ca4a87853fce4c24962a3f489c657a2671caf087" exitCode=0 Jul 06 22:11:43.226445 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:43.226428 2555 generic.go:356] "Generic (PLEG): container finished" podID="50356448-5321-4011-ba4c-fab5b999a2c2" containerID="3af79b72081672f7cfdfacf59ab47451dd941da25ff09e98a371de77939acfb7" exitCode=0 Jul 06 22:11:43.226445 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:43.226436 2555 generic.go:356] "Generic (PLEG): container finished" podID="50356448-5321-4011-ba4c-fab5b999a2c2" containerID="161bec23d11f8d8aec191d7562f1da1a5995d49eaa35965d18193b118e3467e5" exitCode=0 Jul 06 22:11:43.226445 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:43.226442 2555 generic.go:356] "Generic (PLEG): container finished" podID="50356448-5321-4011-ba4c-fab5b999a2c2" containerID="161b1fc3163c7d9902c1c8b23d4c77ed27bde86a77d51230b20d67dd1bafa222" exitCode=0 Jul 06 22:11:43.226757 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:43.226474 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"50356448-5321-4011-ba4c-fab5b999a2c2","Type":"ContainerDied","Data":"4bb7a61ab7f0b688773a39e8ca4a87853fce4c24962a3f489c657a2671caf087"} Jul 06 22:11:43.226757 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:43.226596 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"50356448-5321-4011-ba4c-fab5b999a2c2","Type":"ContainerDied","Data":"3af79b72081672f7cfdfacf59ab47451dd941da25ff09e98a371de77939acfb7"} Jul 06 22:11:43.226757 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:43.226638 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"50356448-5321-4011-ba4c-fab5b999a2c2","Type":"ContainerDied","Data":"161bec23d11f8d8aec191d7562f1da1a5995d49eaa35965d18193b118e3467e5"} Jul 06 22:11:43.226757 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:43.226680 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"50356448-5321-4011-ba4c-fab5b999a2c2","Type":"ContainerDied","Data":"161b1fc3163c7d9902c1c8b23d4c77ed27bde86a77d51230b20d67dd1bafa222"} Jul 06 22:11:44.234950 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:44.234912 2555 generic.go:356] "Generic (PLEG): container finished" podID="50356448-5321-4011-ba4c-fab5b999a2c2" containerID="91dd121750579d37564423c9038631255115f82564e95872e706e49faa57dc57" exitCode=0 Jul 06 22:11:44.234950 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:44.234944 2555 generic.go:356] "Generic (PLEG): container finished" podID="50356448-5321-4011-ba4c-fab5b999a2c2" containerID="23318c1d5d9b9e1aee7f37b4c63df21115f1ee1c22c53d3901f6e1841acd1db8" exitCode=0 Jul 06 22:11:44.235352 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:44.234972 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"50356448-5321-4011-ba4c-fab5b999a2c2","Type":"ContainerDied","Data":"91dd121750579d37564423c9038631255115f82564e95872e706e49faa57dc57"} Jul 06 22:11:44.235352 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:44.235068 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"50356448-5321-4011-ba4c-fab5b999a2c2","Type":"ContainerDied","Data":"23318c1d5d9b9e1aee7f37b4c63df21115f1ee1c22c53d3901f6e1841acd1db8"} Jul 06 22:11:44.322124 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:44.322098 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:11:44.371256 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:44.371147 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/50356448-5321-4011-ba4c-fab5b999a2c2-secret-alertmanager-kube-rbac-proxy\") pod \"50356448-5321-4011-ba4c-fab5b999a2c2\" (UID: \"50356448-5321-4011-ba4c-fab5b999a2c2\") " Jul 06 22:11:44.371256 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:44.371212 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/50356448-5321-4011-ba4c-fab5b999a2c2-web-config\") pod \"50356448-5321-4011-ba4c-fab5b999a2c2\" (UID: \"50356448-5321-4011-ba4c-fab5b999a2c2\") " Jul 06 22:11:44.371256 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:44.371252 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/50356448-5321-4011-ba4c-fab5b999a2c2-metrics-client-ca\") pod \"50356448-5321-4011-ba4c-fab5b999a2c2\" (UID: \"50356448-5321-4011-ba4c-fab5b999a2c2\") " Jul 06 22:11:44.371539 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:44.371306 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/50356448-5321-4011-ba4c-fab5b999a2c2-config-out\") pod \"50356448-5321-4011-ba4c-fab5b999a2c2\" (UID: \"50356448-5321-4011-ba4c-fab5b999a2c2\") " Jul 06 22:11:44.371539 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:44.371331 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/50356448-5321-4011-ba4c-fab5b999a2c2-cluster-tls-config\") pod \"50356448-5321-4011-ba4c-fab5b999a2c2\" (UID: \"50356448-5321-4011-ba4c-fab5b999a2c2\") " Jul 06 22:11:44.371539 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:44.371363 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/50356448-5321-4011-ba4c-fab5b999a2c2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"50356448-5321-4011-ba4c-fab5b999a2c2\" (UID: \"50356448-5321-4011-ba4c-fab5b999a2c2\") " Jul 06 22:11:44.371539 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:44.371402 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b8tqc\" (UniqueName: \"kubernetes.io/projected/50356448-5321-4011-ba4c-fab5b999a2c2-kube-api-access-b8tqc\") pod \"50356448-5321-4011-ba4c-fab5b999a2c2\" (UID: \"50356448-5321-4011-ba4c-fab5b999a2c2\") " Jul 06 22:11:44.371539 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:44.371449 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/50356448-5321-4011-ba4c-fab5b999a2c2-secret-alertmanager-kube-rbac-proxy-web\") pod \"50356448-5321-4011-ba4c-fab5b999a2c2\" (UID: \"50356448-5321-4011-ba4c-fab5b999a2c2\") " Jul 06 22:11:44.371539 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:44.371481 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/50356448-5321-4011-ba4c-fab5b999a2c2-alertmanager-main-db\") pod \"50356448-5321-4011-ba4c-fab5b999a2c2\" (UID: \"50356448-5321-4011-ba4c-fab5b999a2c2\") " Jul 06 22:11:44.371539 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:44.371507 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/50356448-5321-4011-ba4c-fab5b999a2c2-secret-alertmanager-main-tls\") pod \"50356448-5321-4011-ba4c-fab5b999a2c2\" (UID: \"50356448-5321-4011-ba4c-fab5b999a2c2\") " Jul 06 22:11:44.371896 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:44.371582 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/50356448-5321-4011-ba4c-fab5b999a2c2-config-volume\") pod \"50356448-5321-4011-ba4c-fab5b999a2c2\" (UID: \"50356448-5321-4011-ba4c-fab5b999a2c2\") " Jul 06 22:11:44.371896 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:44.371607 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50356448-5321-4011-ba4c-fab5b999a2c2-alertmanager-trusted-ca-bundle\") pod \"50356448-5321-4011-ba4c-fab5b999a2c2\" (UID: \"50356448-5321-4011-ba4c-fab5b999a2c2\") " Jul 06 22:11:44.371896 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:44.371630 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/50356448-5321-4011-ba4c-fab5b999a2c2-tls-assets\") pod \"50356448-5321-4011-ba4c-fab5b999a2c2\" (UID: \"50356448-5321-4011-ba4c-fab5b999a2c2\") " Jul 06 22:11:44.374336 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:44.374076 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50356448-5321-4011-ba4c-fab5b999a2c2-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "50356448-5321-4011-ba4c-fab5b999a2c2" (UID: "50356448-5321-4011-ba4c-fab5b999a2c2"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:11:44.374465 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:44.374364 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50356448-5321-4011-ba4c-fab5b999a2c2-kube-api-access-b8tqc" (OuterVolumeSpecName: "kube-api-access-b8tqc") pod "50356448-5321-4011-ba4c-fab5b999a2c2" (UID: "50356448-5321-4011-ba4c-fab5b999a2c2"). InnerVolumeSpecName "kube-api-access-b8tqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:11:44.375029 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:44.374751 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50356448-5321-4011-ba4c-fab5b999a2c2-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "50356448-5321-4011-ba4c-fab5b999a2c2" (UID: "50356448-5321-4011-ba4c-fab5b999a2c2"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:11:44.375029 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:44.374908 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50356448-5321-4011-ba4c-fab5b999a2c2-config-out" (OuterVolumeSpecName: "config-out") pod "50356448-5321-4011-ba4c-fab5b999a2c2" (UID: "50356448-5321-4011-ba4c-fab5b999a2c2"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:11:44.375299 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:44.375282 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50356448-5321-4011-ba4c-fab5b999a2c2-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "50356448-5321-4011-ba4c-fab5b999a2c2" (UID: "50356448-5321-4011-ba4c-fab5b999a2c2"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:11:44.375627 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:44.375564 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50356448-5321-4011-ba4c-fab5b999a2c2-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "50356448-5321-4011-ba4c-fab5b999a2c2" (UID: "50356448-5321-4011-ba4c-fab5b999a2c2"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:11:44.376464 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:44.376422 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50356448-5321-4011-ba4c-fab5b999a2c2-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "50356448-5321-4011-ba4c-fab5b999a2c2" (UID: "50356448-5321-4011-ba4c-fab5b999a2c2"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:11:44.377656 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:44.377626 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50356448-5321-4011-ba4c-fab5b999a2c2-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "50356448-5321-4011-ba4c-fab5b999a2c2" (UID: "50356448-5321-4011-ba4c-fab5b999a2c2"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:11:44.377756 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:44.377649 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50356448-5321-4011-ba4c-fab5b999a2c2-config-volume" (OuterVolumeSpecName: "config-volume") pod "50356448-5321-4011-ba4c-fab5b999a2c2" (UID: "50356448-5321-4011-ba4c-fab5b999a2c2"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:11:44.377756 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:44.377665 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50356448-5321-4011-ba4c-fab5b999a2c2-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "50356448-5321-4011-ba4c-fab5b999a2c2" (UID: "50356448-5321-4011-ba4c-fab5b999a2c2"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:11:44.378567 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:44.378519 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50356448-5321-4011-ba4c-fab5b999a2c2-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "50356448-5321-4011-ba4c-fab5b999a2c2" (UID: "50356448-5321-4011-ba4c-fab5b999a2c2"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:11:44.382165 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:44.382139 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50356448-5321-4011-ba4c-fab5b999a2c2-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "50356448-5321-4011-ba4c-fab5b999a2c2" (UID: "50356448-5321-4011-ba4c-fab5b999a2c2"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:11:44.386934 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:44.386909 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50356448-5321-4011-ba4c-fab5b999a2c2-web-config" (OuterVolumeSpecName: "web-config") pod "50356448-5321-4011-ba4c-fab5b999a2c2" (UID: "50356448-5321-4011-ba4c-fab5b999a2c2"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:11:44.472546 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:44.472510 2555 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/50356448-5321-4011-ba4c-fab5b999a2c2-config-out\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:11:44.472546 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:44.472545 2555 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/50356448-5321-4011-ba4c-fab5b999a2c2-cluster-tls-config\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:11:44.472746 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:44.472558 2555 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/50356448-5321-4011-ba4c-fab5b999a2c2-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:11:44.472746 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:44.472568 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8tqc\" (UniqueName: \"kubernetes.io/projected/50356448-5321-4011-ba4c-fab5b999a2c2-kube-api-access-b8tqc\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:11:44.472746 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:44.472579 2555 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/50356448-5321-4011-ba4c-fab5b999a2c2-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:11:44.472746 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:44.472589 2555 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/50356448-5321-4011-ba4c-fab5b999a2c2-alertmanager-main-db\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:11:44.472746 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:44.472598 2555 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/50356448-5321-4011-ba4c-fab5b999a2c2-secret-alertmanager-main-tls\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:11:44.472746 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:44.472606 2555 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/50356448-5321-4011-ba4c-fab5b999a2c2-config-volume\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:11:44.472746 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:44.472615 2555 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50356448-5321-4011-ba4c-fab5b999a2c2-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:11:44.472746 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:44.472623 2555 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/50356448-5321-4011-ba4c-fab5b999a2c2-tls-assets\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:11:44.472746 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:44.472632 2555 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/50356448-5321-4011-ba4c-fab5b999a2c2-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:11:44.472746 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:44.472664 2555 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/50356448-5321-4011-ba4c-fab5b999a2c2-web-config\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:11:44.472746 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:44.472672 2555 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/50356448-5321-4011-ba4c-fab5b999a2c2-metrics-client-ca\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:11:45.242249 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.242213 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"50356448-5321-4011-ba4c-fab5b999a2c2","Type":"ContainerDied","Data":"0956d64254ede532ed68755a31a1ecae2f4da8468cfaff3e53ef7d39f80d8897"} Jul 06 22:11:45.242631 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.242257 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:11:45.242631 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.242292 2555 scope.go:117] "RemoveContainer" containerID="4bb7a61ab7f0b688773a39e8ca4a87853fce4c24962a3f489c657a2671caf087" Jul 06 22:11:45.250370 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.250250 2555 scope.go:117] "RemoveContainer" containerID="91dd121750579d37564423c9038631255115f82564e95872e706e49faa57dc57" Jul 06 22:11:45.257323 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.257304 2555 scope.go:117] "RemoveContainer" containerID="3af79b72081672f7cfdfacf59ab47451dd941da25ff09e98a371de77939acfb7" Jul 06 22:11:45.264462 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.264445 2555 scope.go:117] "RemoveContainer" containerID="23318c1d5d9b9e1aee7f37b4c63df21115f1ee1c22c53d3901f6e1841acd1db8" Jul 06 22:11:45.268363 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.268340 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 22:11:45.271632 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.271609 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 22:11:45.272720 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.272707 2555 scope.go:117] "RemoveContainer" containerID="161bec23d11f8d8aec191d7562f1da1a5995d49eaa35965d18193b118e3467e5" Jul 06 22:11:45.280263 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.280246 2555 scope.go:117] "RemoveContainer" containerID="161b1fc3163c7d9902c1c8b23d4c77ed27bde86a77d51230b20d67dd1bafa222" Jul 06 22:11:45.287578 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.287556 2555 scope.go:117] "RemoveContainer" containerID="6cae586c0576001244e53c8541065728cdd103287c5923d6d80462be87691f3b" Jul 06 22:11:45.298702 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.298680 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 22:11:45.298981 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.298968 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50356448-5321-4011-ba4c-fab5b999a2c2" containerName="kube-rbac-proxy-metric" Jul 06 22:11:45.299044 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.298982 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="50356448-5321-4011-ba4c-fab5b999a2c2" containerName="kube-rbac-proxy-metric" Jul 06 22:11:45.299044 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.298992 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50356448-5321-4011-ba4c-fab5b999a2c2" containerName="prom-label-proxy" Jul 06 22:11:45.299044 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.298998 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="50356448-5321-4011-ba4c-fab5b999a2c2" containerName="prom-label-proxy" Jul 06 22:11:45.299044 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.299038 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50356448-5321-4011-ba4c-fab5b999a2c2" containerName="alertmanager" Jul 06 22:11:45.299186 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.299047 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="50356448-5321-4011-ba4c-fab5b999a2c2" containerName="alertmanager" Jul 06 22:11:45.299186 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.299054 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50356448-5321-4011-ba4c-fab5b999a2c2" containerName="kube-rbac-proxy-web" Jul 06 22:11:45.299186 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.299059 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="50356448-5321-4011-ba4c-fab5b999a2c2" containerName="kube-rbac-proxy-web" Jul 06 22:11:45.299186 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.299066 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50356448-5321-4011-ba4c-fab5b999a2c2" containerName="init-config-reloader" Jul 06 22:11:45.299186 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.299071 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="50356448-5321-4011-ba4c-fab5b999a2c2" containerName="init-config-reloader" Jul 06 22:11:45.299186 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.299082 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50356448-5321-4011-ba4c-fab5b999a2c2" containerName="kube-rbac-proxy" Jul 06 22:11:45.299186 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.299087 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="50356448-5321-4011-ba4c-fab5b999a2c2" containerName="kube-rbac-proxy" Jul 06 22:11:45.299186 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.299095 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d9c3b90-9586-4687-a6ee-5424470f92d5" containerName="console" Jul 06 22:11:45.299186 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.299100 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d9c3b90-9586-4687-a6ee-5424470f92d5" containerName="console" Jul 06 22:11:45.299186 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.299109 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f54d0d8-ac21-478b-b687-976f9732aba7" containerName="console" Jul 06 22:11:45.299186 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.299114 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f54d0d8-ac21-478b-b687-976f9732aba7" containerName="console" Jul 06 22:11:45.299186 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.299147 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50356448-5321-4011-ba4c-fab5b999a2c2" containerName="config-reloader" Jul 06 22:11:45.299186 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.299152 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="50356448-5321-4011-ba4c-fab5b999a2c2" containerName="config-reloader" Jul 06 22:11:45.299186 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.299160 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0" containerName="console" Jul 06 22:11:45.299186 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.299164 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0" containerName="console" Jul 06 22:11:45.299592 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.299219 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f54d0d8-ac21-478b-b687-976f9732aba7" containerName="console" Jul 06 22:11:45.299592 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.299230 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="50356448-5321-4011-ba4c-fab5b999a2c2" containerName="config-reloader" Jul 06 22:11:45.299592 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.299236 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="50356448-5321-4011-ba4c-fab5b999a2c2" containerName="kube-rbac-proxy-metric" Jul 06 22:11:45.299592 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.299242 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="50356448-5321-4011-ba4c-fab5b999a2c2" containerName="alertmanager" Jul 06 22:11:45.299592 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.299246 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="50356448-5321-4011-ba4c-fab5b999a2c2" containerName="prom-label-proxy" Jul 06 22:11:45.299592 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.299252 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d9c3b90-9586-4687-a6ee-5424470f92d5" containerName="console" Jul 06 22:11:45.299592 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.299257 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="50356448-5321-4011-ba4c-fab5b999a2c2" containerName="kube-rbac-proxy" Jul 06 22:11:45.299592 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.299262 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="9854c5a1-8d22-4dc2-9c9f-fa6ffb2d0bb0" containerName="console" Jul 06 22:11:45.299592 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.299268 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="50356448-5321-4011-ba4c-fab5b999a2c2" containerName="kube-rbac-proxy-web" Jul 06 22:11:45.304695 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.304675 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:11:45.308135 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.308103 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 06 22:11:45.308464 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.308135 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 06 22:11:45.308464 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.308109 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 06 22:11:45.308464 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.308117 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 06 22:11:45.308464 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.308301 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 06 22:11:45.308464 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.308338 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 06 22:11:45.308771 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.308568 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 06 22:11:45.308771 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.308622 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 06 22:11:45.308906 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.308888 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-ll7rl\"" Jul 06 22:11:45.313293 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.313269 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 06 22:11:45.318918 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.318894 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 22:11:45.340145 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.340110 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50356448-5321-4011-ba4c-fab5b999a2c2" path="/var/lib/kubelet/pods/50356448-5321-4011-ba4c-fab5b999a2c2/volumes" Jul 06 22:11:45.380806 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.380719 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38ffd73f-16ff-4301-9084-a4dc29c0cdef-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"38ffd73f-16ff-4301-9084-a4dc29c0cdef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:11:45.380806 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.380758 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/38ffd73f-16ff-4301-9084-a4dc29c0cdef-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"38ffd73f-16ff-4301-9084-a4dc29c0cdef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:11:45.380806 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.380781 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/38ffd73f-16ff-4301-9084-a4dc29c0cdef-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"38ffd73f-16ff-4301-9084-a4dc29c0cdef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:11:45.381025 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.380838 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/38ffd73f-16ff-4301-9084-a4dc29c0cdef-tls-assets\") pod \"alertmanager-main-0\" (UID: \"38ffd73f-16ff-4301-9084-a4dc29c0cdef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:11:45.381025 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.380874 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/38ffd73f-16ff-4301-9084-a4dc29c0cdef-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"38ffd73f-16ff-4301-9084-a4dc29c0cdef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:11:45.381025 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.380896 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/38ffd73f-16ff-4301-9084-a4dc29c0cdef-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"38ffd73f-16ff-4301-9084-a4dc29c0cdef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:11:45.381025 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.380951 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/38ffd73f-16ff-4301-9084-a4dc29c0cdef-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"38ffd73f-16ff-4301-9084-a4dc29c0cdef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:11:45.381025 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.380983 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfr2w\" (UniqueName: \"kubernetes.io/projected/38ffd73f-16ff-4301-9084-a4dc29c0cdef-kube-api-access-gfr2w\") pod \"alertmanager-main-0\" (UID: \"38ffd73f-16ff-4301-9084-a4dc29c0cdef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:11:45.381198 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.381002 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/38ffd73f-16ff-4301-9084-a4dc29c0cdef-config-out\") pod \"alertmanager-main-0\" (UID: \"38ffd73f-16ff-4301-9084-a4dc29c0cdef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:11:45.381198 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.381051 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/38ffd73f-16ff-4301-9084-a4dc29c0cdef-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"38ffd73f-16ff-4301-9084-a4dc29c0cdef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:11:45.381198 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.381087 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/38ffd73f-16ff-4301-9084-a4dc29c0cdef-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"38ffd73f-16ff-4301-9084-a4dc29c0cdef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:11:45.381198 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.381113 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/38ffd73f-16ff-4301-9084-a4dc29c0cdef-config-volume\") pod \"alertmanager-main-0\" (UID: \"38ffd73f-16ff-4301-9084-a4dc29c0cdef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:11:45.381198 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.381130 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/38ffd73f-16ff-4301-9084-a4dc29c0cdef-web-config\") pod \"alertmanager-main-0\" (UID: \"38ffd73f-16ff-4301-9084-a4dc29c0cdef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:11:45.481696 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.481643 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfr2w\" (UniqueName: \"kubernetes.io/projected/38ffd73f-16ff-4301-9084-a4dc29c0cdef-kube-api-access-gfr2w\") pod \"alertmanager-main-0\" (UID: \"38ffd73f-16ff-4301-9084-a4dc29c0cdef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:11:45.481696 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.481704 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/38ffd73f-16ff-4301-9084-a4dc29c0cdef-config-out\") pod \"alertmanager-main-0\" (UID: \"38ffd73f-16ff-4301-9084-a4dc29c0cdef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:11:45.481931 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.481729 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/38ffd73f-16ff-4301-9084-a4dc29c0cdef-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"38ffd73f-16ff-4301-9084-a4dc29c0cdef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:11:45.481931 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.481760 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/38ffd73f-16ff-4301-9084-a4dc29c0cdef-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"38ffd73f-16ff-4301-9084-a4dc29c0cdef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:11:45.481931 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.481786 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/38ffd73f-16ff-4301-9084-a4dc29c0cdef-config-volume\") pod \"alertmanager-main-0\" (UID: \"38ffd73f-16ff-4301-9084-a4dc29c0cdef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:11:45.481931 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.481803 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/38ffd73f-16ff-4301-9084-a4dc29c0cdef-web-config\") pod \"alertmanager-main-0\" (UID: \"38ffd73f-16ff-4301-9084-a4dc29c0cdef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:11:45.481931 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.481826 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38ffd73f-16ff-4301-9084-a4dc29c0cdef-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"38ffd73f-16ff-4301-9084-a4dc29c0cdef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:11:45.481931 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.481890 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/38ffd73f-16ff-4301-9084-a4dc29c0cdef-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"38ffd73f-16ff-4301-9084-a4dc29c0cdef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:11:45.482267 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.481940 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/38ffd73f-16ff-4301-9084-a4dc29c0cdef-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"38ffd73f-16ff-4301-9084-a4dc29c0cdef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:11:45.482267 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.481978 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/38ffd73f-16ff-4301-9084-a4dc29c0cdef-tls-assets\") pod \"alertmanager-main-0\" (UID: \"38ffd73f-16ff-4301-9084-a4dc29c0cdef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:11:45.482267 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.482002 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/38ffd73f-16ff-4301-9084-a4dc29c0cdef-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"38ffd73f-16ff-4301-9084-a4dc29c0cdef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:11:45.482954 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.482658 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/38ffd73f-16ff-4301-9084-a4dc29c0cdef-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"38ffd73f-16ff-4301-9084-a4dc29c0cdef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:11:45.482954 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.482716 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/38ffd73f-16ff-4301-9084-a4dc29c0cdef-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"38ffd73f-16ff-4301-9084-a4dc29c0cdef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:11:45.482954 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.482802 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/38ffd73f-16ff-4301-9084-a4dc29c0cdef-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"38ffd73f-16ff-4301-9084-a4dc29c0cdef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:11:45.482954 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.482834 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38ffd73f-16ff-4301-9084-a4dc29c0cdef-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"38ffd73f-16ff-4301-9084-a4dc29c0cdef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:11:45.483394 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.483279 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/38ffd73f-16ff-4301-9084-a4dc29c0cdef-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"38ffd73f-16ff-4301-9084-a4dc29c0cdef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:11:45.484844 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.484800 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/38ffd73f-16ff-4301-9084-a4dc29c0cdef-config-out\") pod \"alertmanager-main-0\" (UID: \"38ffd73f-16ff-4301-9084-a4dc29c0cdef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:11:45.485085 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.485062 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/38ffd73f-16ff-4301-9084-a4dc29c0cdef-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"38ffd73f-16ff-4301-9084-a4dc29c0cdef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:11:45.485542 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.485501 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/38ffd73f-16ff-4301-9084-a4dc29c0cdef-config-volume\") pod \"alertmanager-main-0\" (UID: \"38ffd73f-16ff-4301-9084-a4dc29c0cdef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:11:45.485542 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.485533 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/38ffd73f-16ff-4301-9084-a4dc29c0cdef-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"38ffd73f-16ff-4301-9084-a4dc29c0cdef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:11:45.485694 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.485609 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/38ffd73f-16ff-4301-9084-a4dc29c0cdef-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"38ffd73f-16ff-4301-9084-a4dc29c0cdef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:11:45.485694 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.485634 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/38ffd73f-16ff-4301-9084-a4dc29c0cdef-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"38ffd73f-16ff-4301-9084-a4dc29c0cdef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:11:45.485909 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.485889 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/38ffd73f-16ff-4301-9084-a4dc29c0cdef-web-config\") pod \"alertmanager-main-0\" (UID: \"38ffd73f-16ff-4301-9084-a4dc29c0cdef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:11:45.486085 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.486067 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/38ffd73f-16ff-4301-9084-a4dc29c0cdef-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"38ffd73f-16ff-4301-9084-a4dc29c0cdef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:11:45.486557 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.486539 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/38ffd73f-16ff-4301-9084-a4dc29c0cdef-tls-assets\") pod \"alertmanager-main-0\" (UID: \"38ffd73f-16ff-4301-9084-a4dc29c0cdef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:11:45.490001 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.489982 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfr2w\" (UniqueName: \"kubernetes.io/projected/38ffd73f-16ff-4301-9084-a4dc29c0cdef-kube-api-access-gfr2w\") pod \"alertmanager-main-0\" (UID: \"38ffd73f-16ff-4301-9084-a4dc29c0cdef\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:11:45.614683 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.614644 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:11:45.755215 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:45.755183 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 22:11:45.758047 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:11:45.758016 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38ffd73f_16ff_4301_9084_a4dc29c0cdef.slice/crio-c6e9ff320facb3c303e414f9c8e081b7b527e6d486d9889b30e180fbf8bec7db WatchSource:0}: Error finding container c6e9ff320facb3c303e414f9c8e081b7b527e6d486d9889b30e180fbf8bec7db: Status 404 returned error can't find the container with id c6e9ff320facb3c303e414f9c8e081b7b527e6d486d9889b30e180fbf8bec7db Jul 06 22:11:46.248107 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:46.248056 2555 generic.go:356] "Generic (PLEG): container finished" podID="38ffd73f-16ff-4301-9084-a4dc29c0cdef" containerID="d42e52ff94d358a02ff893223fc3960fd957cee73cf36417a31c9f856b23d67a" exitCode=0 Jul 06 22:11:46.248572 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:46.248156 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"38ffd73f-16ff-4301-9084-a4dc29c0cdef","Type":"ContainerDied","Data":"d42e52ff94d358a02ff893223fc3960fd957cee73cf36417a31c9f856b23d67a"} Jul 06 22:11:46.248572 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:46.248254 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"38ffd73f-16ff-4301-9084-a4dc29c0cdef","Type":"ContainerStarted","Data":"c6e9ff320facb3c303e414f9c8e081b7b527e6d486d9889b30e180fbf8bec7db"} Jul 06 22:11:47.258783 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.258743 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"38ffd73f-16ff-4301-9084-a4dc29c0cdef","Type":"ContainerStarted","Data":"4c315eac97d1a9817004d19682df6e8105a0ea4dffe5682270ae25b6a09c8da7"} Jul 06 22:11:47.259277 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.258822 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"38ffd73f-16ff-4301-9084-a4dc29c0cdef","Type":"ContainerStarted","Data":"87f0409939152202ea8c7e17d27e833a00b2fec014ff026eaa4dd669c751179b"} Jul 06 22:11:47.259277 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.258868 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"38ffd73f-16ff-4301-9084-a4dc29c0cdef","Type":"ContainerStarted","Data":"93a9e4103e5c1d2041c1e6648982d53c000d43aaf194cb36db7b8c4aa538aa75"} Jul 06 22:11:47.259277 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.258908 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"38ffd73f-16ff-4301-9084-a4dc29c0cdef","Type":"ContainerStarted","Data":"1fe3791124743776ee4829bbb391ee8abd3608a18f6832abb20b987c41a7b69e"} Jul 06 22:11:47.259277 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.258949 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"38ffd73f-16ff-4301-9084-a4dc29c0cdef","Type":"ContainerStarted","Data":"89f6eb1bedc22ae1b532dd8f125d4e0c4dd92ed93a0929c7a55112196405b72b"} Jul 06 22:11:47.259277 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.258986 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"38ffd73f-16ff-4301-9084-a4dc29c0cdef","Type":"ContainerStarted","Data":"548e4691875a78eb96b3de3f58fb0cf40d10dcbb8e55bb1dec4f54467272e114"} Jul 06 22:11:47.289405 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.289342 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.274313471 podStartE2EDuration="2.289321797s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.008326ms" isStatefulPod=false podCreationTimestamp="2026-07-06 22:11:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:11:47.287327762 +0000 UTC m=+158.587039611" watchObservedRunningTime="2026-07-06 22:11:47.289321797 +0000 UTC m=+158.589033648" Jul 06 22:11:47.346835 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.346804 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 22:11:47.347157 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.347127 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d" containerName="prometheus" containerID="cri-o://7d22f9d9188e3d530d12c1f8c8f9534cde4bab8897f3fba95c2c73cd6e9d95e2" gracePeriod=600 Jul 06 22:11:47.347248 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.347149 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d" containerName="thanos-sidecar" containerID="cri-o://868f0e2e2832f412be2280dceca76d6075b832497bd7017e0cdbd6c7709d4a8e" gracePeriod=600 Jul 06 22:11:47.347341 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.347283 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d" containerName="kube-rbac-proxy-web" containerID="cri-o://b76c28e5cf3ded5c5d3a2bbddc03ad092e65235b16f6341a2d45dffe87074a17" gracePeriod=600 Jul 06 22:11:47.347341 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.347297 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d" containerName="config-reloader" containerID="cri-o://70c341aec9d02c2801bd2464b7fdd403607f98f4669b5a840b5d396ea57352c2" gracePeriod=600 Jul 06 22:11:47.347461 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.347404 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d" containerName="kube-rbac-proxy-thanos" containerID="cri-o://dce1ad64f7310c0e3f6de2bb630af3cd4c4168932a432f48be33cde6c84ea430" gracePeriod=600 Jul 06 22:11:47.347562 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.347149 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d" containerName="kube-rbac-proxy" containerID="cri-o://90c27314e532c2b8d3dce39f5fd09e6928ef39e4fb97a65d2fb8850cd5049e25" gracePeriod=600 Jul 06 22:11:47.731259 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.731234 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:11:47.816634 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.816553 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-prometheus-k8s-rulefiles-1\") pod \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " Jul 06 22:11:47.816634 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.816583 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-secret-prometheus-k8s-tls\") pod \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " Jul 06 22:11:47.816634 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.816601 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-secret-kube-rbac-proxy\") pod \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " Jul 06 22:11:47.816634 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.816628 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-config-out\") pod \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " Jul 06 22:11:47.816948 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.816654 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-tls-assets\") pod \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " Jul 06 22:11:47.816948 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.816685 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-secret-metrics-client-certs\") pod \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " Jul 06 22:11:47.816948 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.816721 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-configmap-kubelet-serving-ca-bundle\") pod \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " Jul 06 22:11:47.816948 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.816749 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-prometheus-trusted-ca-bundle\") pod \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " Jul 06 22:11:47.816948 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.816781 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-prometheus-k8s-rulefiles-2\") pod \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " Jul 06 22:11:47.816948 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.816806 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-web-config\") pod \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " Jul 06 22:11:47.816948 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.816850 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-secret-grpc-tls\") pod \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " Jul 06 22:11:47.816948 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.816884 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-configmap-serving-certs-ca-bundle\") pod \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " Jul 06 22:11:47.816948 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.816913 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " Jul 06 22:11:47.817430 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.816942 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d" (UID: "a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:11:47.817430 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.816954 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " Jul 06 22:11:47.817430 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.817134 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rlv7q\" (UniqueName: \"kubernetes.io/projected/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-kube-api-access-rlv7q\") pod \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " Jul 06 22:11:47.817430 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.817179 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-prometheus-k8s-db\") pod \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " Jul 06 22:11:47.817430 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.817207 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-prometheus-k8s-rulefiles-0\") pod \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " Jul 06 22:11:47.817430 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.817245 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-thanos-prometheus-http-client-file\") pod \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " Jul 06 22:11:47.817430 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.817273 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-configmap-metrics-client-ca\") pod \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " Jul 06 22:11:47.817430 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.817328 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-config\") pod \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\" (UID: \"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d\") " Jul 06 22:11:47.817430 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.817406 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d" (UID: "a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:11:47.817893 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.817661 2555 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:11:47.817893 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.817682 2555 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-prometheus-trusted-ca-bundle\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:11:47.817893 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.817847 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d" (UID: "a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:11:47.819267 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.819241 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d" (UID: "a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:11:47.819529 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.819507 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d" (UID: "a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:11:47.819653 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.819538 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-config-out" (OuterVolumeSpecName: "config-out") pod "a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d" (UID: "a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:11:47.819653 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.819619 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d" (UID: "a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:11:47.819833 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.819803 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d" (UID: "a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:11:47.819998 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.819973 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d" (UID: "a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:11:47.819998 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.819987 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-config" (OuterVolumeSpecName: "config") pod "a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d" (UID: "a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:11:47.820211 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.820188 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d" (UID: "a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:11:47.820343 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.820316 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d" (UID: "a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:11:47.820421 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.820399 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d" (UID: "a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:11:47.820691 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.820667 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d" (UID: "a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:11:47.821159 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.821133 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d" (UID: "a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:11:47.821366 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.821345 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d" (UID: "a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:11:47.822071 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.822056 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d" (UID: "a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:11:47.822212 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.822183 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-kube-api-access-rlv7q" (OuterVolumeSpecName: "kube-api-access-rlv7q") pod "a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d" (UID: "a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d"). InnerVolumeSpecName "kube-api-access-rlv7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:11:47.822373 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.822353 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d" (UID: "a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:11:47.832604 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.832575 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-web-config" (OuterVolumeSpecName: "web-config") pod "a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d" (UID: "a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:11:47.918231 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.918187 2555 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:11:47.918231 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.918225 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlv7q\" (UniqueName: \"kubernetes.io/projected/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-kube-api-access-rlv7q\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:11:47.918231 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.918237 2555 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-prometheus-k8s-db\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:11:47.918469 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.918246 2555 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:11:47.918469 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.918256 2555 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-thanos-prometheus-http-client-file\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:11:47.918469 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.918266 2555 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-configmap-metrics-client-ca\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:11:47.918469 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.918275 2555 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-config\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:11:47.918469 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.918284 2555 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-secret-prometheus-k8s-tls\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:11:47.918469 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.918292 2555 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-secret-kube-rbac-proxy\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:11:47.918469 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.918302 2555 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-config-out\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:11:47.918469 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.918310 2555 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-tls-assets\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:11:47.918469 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.918317 2555 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-secret-metrics-client-certs\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:11:47.918469 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.918326 2555 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:11:47.918469 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.918334 2555 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:11:47.918469 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.918342 2555 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-web-config\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:11:47.918469 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.918350 2555 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-secret-grpc-tls\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:11:47.918469 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.918357 2555 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:11:47.918469 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:47.918365 2555 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:11:48.266085 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.266040 2555 generic.go:356] "Generic (PLEG): container finished" podID="a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d" containerID="dce1ad64f7310c0e3f6de2bb630af3cd4c4168932a432f48be33cde6c84ea430" exitCode=0 Jul 06 22:11:48.266085 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.266077 2555 generic.go:356] "Generic (PLEG): container finished" podID="a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d" containerID="90c27314e532c2b8d3dce39f5fd09e6928ef39e4fb97a65d2fb8850cd5049e25" exitCode=0 Jul 06 22:11:48.266085 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.266083 2555 generic.go:356] "Generic (PLEG): container finished" podID="a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d" containerID="b76c28e5cf3ded5c5d3a2bbddc03ad092e65235b16f6341a2d45dffe87074a17" exitCode=0 Jul 06 22:11:48.266085 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.266088 2555 generic.go:356] "Generic (PLEG): container finished" podID="a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d" containerID="868f0e2e2832f412be2280dceca76d6075b832497bd7017e0cdbd6c7709d4a8e" exitCode=0 Jul 06 22:11:48.266085 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.266094 2555 generic.go:356] "Generic (PLEG): container finished" podID="a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d" containerID="70c341aec9d02c2801bd2464b7fdd403607f98f4669b5a840b5d396ea57352c2" exitCode=0 Jul 06 22:11:48.266085 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.266098 2555 generic.go:356] "Generic (PLEG): container finished" podID="a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d" containerID="7d22f9d9188e3d530d12c1f8c8f9534cde4bab8897f3fba95c2c73cd6e9d95e2" exitCode=0 Jul 06 22:11:48.266801 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.266113 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:11:48.266801 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.266124 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d","Type":"ContainerDied","Data":"dce1ad64f7310c0e3f6de2bb630af3cd4c4168932a432f48be33cde6c84ea430"} Jul 06 22:11:48.266801 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.266227 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d","Type":"ContainerDied","Data":"90c27314e532c2b8d3dce39f5fd09e6928ef39e4fb97a65d2fb8850cd5049e25"} Jul 06 22:11:48.266801 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.266257 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d","Type":"ContainerDied","Data":"b76c28e5cf3ded5c5d3a2bbddc03ad092e65235b16f6341a2d45dffe87074a17"} Jul 06 22:11:48.266801 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.266284 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d","Type":"ContainerDied","Data":"868f0e2e2832f412be2280dceca76d6075b832497bd7017e0cdbd6c7709d4a8e"} Jul 06 22:11:48.266801 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.266291 2555 scope.go:117] "RemoveContainer" containerID="dce1ad64f7310c0e3f6de2bb630af3cd4c4168932a432f48be33cde6c84ea430" Jul 06 22:11:48.266801 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.266314 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d","Type":"ContainerDied","Data":"70c341aec9d02c2801bd2464b7fdd403607f98f4669b5a840b5d396ea57352c2"} Jul 06 22:11:48.266801 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.266343 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d","Type":"ContainerDied","Data":"7d22f9d9188e3d530d12c1f8c8f9534cde4bab8897f3fba95c2c73cd6e9d95e2"} Jul 06 22:11:48.266801 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.266381 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d","Type":"ContainerDied","Data":"be3ba65842b888422e36e96971b10624d71a9b5136c1f6ec39a08357ac1afd5a"} Jul 06 22:11:48.274529 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.274509 2555 scope.go:117] "RemoveContainer" containerID="90c27314e532c2b8d3dce39f5fd09e6928ef39e4fb97a65d2fb8850cd5049e25" Jul 06 22:11:48.281774 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.281754 2555 scope.go:117] "RemoveContainer" containerID="b76c28e5cf3ded5c5d3a2bbddc03ad092e65235b16f6341a2d45dffe87074a17" Jul 06 22:11:48.288697 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.288677 2555 scope.go:117] "RemoveContainer" containerID="868f0e2e2832f412be2280dceca76d6075b832497bd7017e0cdbd6c7709d4a8e" Jul 06 22:11:48.290761 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.290739 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 22:11:48.295668 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.295642 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 22:11:48.297255 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.297240 2555 scope.go:117] "RemoveContainer" containerID="70c341aec9d02c2801bd2464b7fdd403607f98f4669b5a840b5d396ea57352c2" Jul 06 22:11:48.304409 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.304393 2555 scope.go:117] "RemoveContainer" containerID="7d22f9d9188e3d530d12c1f8c8f9534cde4bab8897f3fba95c2c73cd6e9d95e2" Jul 06 22:11:48.312051 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.312033 2555 scope.go:117] "RemoveContainer" containerID="d48a4ada40084d249f040e237b4daac841bd3fd6fae77bcfcfa8300ba443025e" Jul 06 22:11:48.318819 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.318740 2555 scope.go:117] "RemoveContainer" containerID="dce1ad64f7310c0e3f6de2bb630af3cd4c4168932a432f48be33cde6c84ea430" Jul 06 22:11:48.319121 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:11:48.319096 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dce1ad64f7310c0e3f6de2bb630af3cd4c4168932a432f48be33cde6c84ea430\": container with ID starting with dce1ad64f7310c0e3f6de2bb630af3cd4c4168932a432f48be33cde6c84ea430 not found: ID does not exist" containerID="dce1ad64f7310c0e3f6de2bb630af3cd4c4168932a432f48be33cde6c84ea430" Jul 06 22:11:48.319236 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.319136 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dce1ad64f7310c0e3f6de2bb630af3cd4c4168932a432f48be33cde6c84ea430"} err="failed to get container status \"dce1ad64f7310c0e3f6de2bb630af3cd4c4168932a432f48be33cde6c84ea430\": rpc error: code = NotFound desc = could not find container \"dce1ad64f7310c0e3f6de2bb630af3cd4c4168932a432f48be33cde6c84ea430\": container with ID starting with dce1ad64f7310c0e3f6de2bb630af3cd4c4168932a432f48be33cde6c84ea430 not found: ID does not exist" Jul 06 22:11:48.319236 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.319163 2555 scope.go:117] "RemoveContainer" containerID="90c27314e532c2b8d3dce39f5fd09e6928ef39e4fb97a65d2fb8850cd5049e25" Jul 06 22:11:48.319647 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:11:48.319621 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90c27314e532c2b8d3dce39f5fd09e6928ef39e4fb97a65d2fb8850cd5049e25\": container with ID starting with 90c27314e532c2b8d3dce39f5fd09e6928ef39e4fb97a65d2fb8850cd5049e25 not found: ID does not exist" containerID="90c27314e532c2b8d3dce39f5fd09e6928ef39e4fb97a65d2fb8850cd5049e25" Jul 06 22:11:48.319712 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.319657 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90c27314e532c2b8d3dce39f5fd09e6928ef39e4fb97a65d2fb8850cd5049e25"} err="failed to get container status \"90c27314e532c2b8d3dce39f5fd09e6928ef39e4fb97a65d2fb8850cd5049e25\": rpc error: code = NotFound desc = could not find container \"90c27314e532c2b8d3dce39f5fd09e6928ef39e4fb97a65d2fb8850cd5049e25\": container with ID starting with 90c27314e532c2b8d3dce39f5fd09e6928ef39e4fb97a65d2fb8850cd5049e25 not found: ID does not exist" Jul 06 22:11:48.319712 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.319680 2555 scope.go:117] "RemoveContainer" containerID="b76c28e5cf3ded5c5d3a2bbddc03ad092e65235b16f6341a2d45dffe87074a17" Jul 06 22:11:48.319958 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:11:48.319940 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b76c28e5cf3ded5c5d3a2bbddc03ad092e65235b16f6341a2d45dffe87074a17\": container with ID starting with b76c28e5cf3ded5c5d3a2bbddc03ad092e65235b16f6341a2d45dffe87074a17 not found: ID does not exist" containerID="b76c28e5cf3ded5c5d3a2bbddc03ad092e65235b16f6341a2d45dffe87074a17" Jul 06 22:11:48.320043 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.319965 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b76c28e5cf3ded5c5d3a2bbddc03ad092e65235b16f6341a2d45dffe87074a17"} err="failed to get container status \"b76c28e5cf3ded5c5d3a2bbddc03ad092e65235b16f6341a2d45dffe87074a17\": rpc error: code = NotFound desc = could not find container \"b76c28e5cf3ded5c5d3a2bbddc03ad092e65235b16f6341a2d45dffe87074a17\": container with ID starting with b76c28e5cf3ded5c5d3a2bbddc03ad092e65235b16f6341a2d45dffe87074a17 not found: ID does not exist" Jul 06 22:11:48.320043 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.319982 2555 scope.go:117] "RemoveContainer" containerID="868f0e2e2832f412be2280dceca76d6075b832497bd7017e0cdbd6c7709d4a8e" Jul 06 22:11:48.320260 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:11:48.320240 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"868f0e2e2832f412be2280dceca76d6075b832497bd7017e0cdbd6c7709d4a8e\": container with ID starting with 868f0e2e2832f412be2280dceca76d6075b832497bd7017e0cdbd6c7709d4a8e not found: ID does not exist" containerID="868f0e2e2832f412be2280dceca76d6075b832497bd7017e0cdbd6c7709d4a8e" Jul 06 22:11:48.320332 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.320267 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"868f0e2e2832f412be2280dceca76d6075b832497bd7017e0cdbd6c7709d4a8e"} err="failed to get container status \"868f0e2e2832f412be2280dceca76d6075b832497bd7017e0cdbd6c7709d4a8e\": rpc error: code = NotFound desc = could not find container \"868f0e2e2832f412be2280dceca76d6075b832497bd7017e0cdbd6c7709d4a8e\": container with ID starting with 868f0e2e2832f412be2280dceca76d6075b832497bd7017e0cdbd6c7709d4a8e not found: ID does not exist" Jul 06 22:11:48.320332 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.320286 2555 scope.go:117] "RemoveContainer" containerID="70c341aec9d02c2801bd2464b7fdd403607f98f4669b5a840b5d396ea57352c2" Jul 06 22:11:48.320523 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:11:48.320507 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70c341aec9d02c2801bd2464b7fdd403607f98f4669b5a840b5d396ea57352c2\": container with ID starting with 70c341aec9d02c2801bd2464b7fdd403607f98f4669b5a840b5d396ea57352c2 not found: ID does not exist" containerID="70c341aec9d02c2801bd2464b7fdd403607f98f4669b5a840b5d396ea57352c2" Jul 06 22:11:48.320569 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.320528 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70c341aec9d02c2801bd2464b7fdd403607f98f4669b5a840b5d396ea57352c2"} err="failed to get container status \"70c341aec9d02c2801bd2464b7fdd403607f98f4669b5a840b5d396ea57352c2\": rpc error: code = NotFound desc = could not find container \"70c341aec9d02c2801bd2464b7fdd403607f98f4669b5a840b5d396ea57352c2\": container with ID starting with 70c341aec9d02c2801bd2464b7fdd403607f98f4669b5a840b5d396ea57352c2 not found: ID does not exist" Jul 06 22:11:48.320569 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.320541 2555 scope.go:117] "RemoveContainer" containerID="7d22f9d9188e3d530d12c1f8c8f9534cde4bab8897f3fba95c2c73cd6e9d95e2" Jul 06 22:11:48.320643 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.320585 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 22:11:48.320773 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:11:48.320756 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d22f9d9188e3d530d12c1f8c8f9534cde4bab8897f3fba95c2c73cd6e9d95e2\": container with ID starting with 7d22f9d9188e3d530d12c1f8c8f9534cde4bab8897f3fba95c2c73cd6e9d95e2 not found: ID does not exist" containerID="7d22f9d9188e3d530d12c1f8c8f9534cde4bab8897f3fba95c2c73cd6e9d95e2" Jul 06 22:11:48.320814 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.320777 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d22f9d9188e3d530d12c1f8c8f9534cde4bab8897f3fba95c2c73cd6e9d95e2"} err="failed to get container status \"7d22f9d9188e3d530d12c1f8c8f9534cde4bab8897f3fba95c2c73cd6e9d95e2\": rpc error: code = NotFound desc = could not find container \"7d22f9d9188e3d530d12c1f8c8f9534cde4bab8897f3fba95c2c73cd6e9d95e2\": container with ID starting with 7d22f9d9188e3d530d12c1f8c8f9534cde4bab8897f3fba95c2c73cd6e9d95e2 not found: ID does not exist" Jul 06 22:11:48.320814 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.320791 2555 scope.go:117] "RemoveContainer" containerID="d48a4ada40084d249f040e237b4daac841bd3fd6fae77bcfcfa8300ba443025e" Jul 06 22:11:48.320909 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.320891 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d" containerName="kube-rbac-proxy-thanos" Jul 06 22:11:48.320909 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.320909 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d" containerName="kube-rbac-proxy-thanos" Jul 06 22:11:48.321119 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.320919 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d" containerName="config-reloader" Jul 06 22:11:48.321119 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.320925 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d" containerName="config-reloader" Jul 06 22:11:48.321119 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.320933 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d" containerName="thanos-sidecar" Jul 06 22:11:48.321119 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.320943 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d" containerName="thanos-sidecar" Jul 06 22:11:48.321119 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.320952 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d" containerName="kube-rbac-proxy" Jul 06 22:11:48.321119 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.320960 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d" containerName="kube-rbac-proxy" Jul 06 22:11:48.321119 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.320972 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d" containerName="init-config-reloader" Jul 06 22:11:48.321119 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.320979 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d" containerName="init-config-reloader" Jul 06 22:11:48.321119 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.320989 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d" containerName="kube-rbac-proxy-web" Jul 06 22:11:48.321119 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.320997 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d" containerName="kube-rbac-proxy-web" Jul 06 22:11:48.321119 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.321031 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d" containerName="prometheus" Jul 06 22:11:48.321119 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.321038 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d" containerName="prometheus" Jul 06 22:11:48.321119 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:11:48.321060 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d48a4ada40084d249f040e237b4daac841bd3fd6fae77bcfcfa8300ba443025e\": container with ID starting with d48a4ada40084d249f040e237b4daac841bd3fd6fae77bcfcfa8300ba443025e not found: ID does not exist" containerID="d48a4ada40084d249f040e237b4daac841bd3fd6fae77bcfcfa8300ba443025e" Jul 06 22:11:48.321119 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.321084 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d48a4ada40084d249f040e237b4daac841bd3fd6fae77bcfcfa8300ba443025e"} err="failed to get container status \"d48a4ada40084d249f040e237b4daac841bd3fd6fae77bcfcfa8300ba443025e\": rpc error: code = NotFound desc = could not find container \"d48a4ada40084d249f040e237b4daac841bd3fd6fae77bcfcfa8300ba443025e\": container with ID starting with d48a4ada40084d249f040e237b4daac841bd3fd6fae77bcfcfa8300ba443025e not found: ID does not exist" Jul 06 22:11:48.321119 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.321104 2555 scope.go:117] "RemoveContainer" containerID="dce1ad64f7310c0e3f6de2bb630af3cd4c4168932a432f48be33cde6c84ea430" Jul 06 22:11:48.321119 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.321109 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d" containerName="kube-rbac-proxy-thanos" Jul 06 22:11:48.321119 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.321119 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d" containerName="thanos-sidecar" Jul 06 22:11:48.321119 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.321125 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d" containerName="kube-rbac-proxy" Jul 06 22:11:48.321119 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.321132 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d" containerName="prometheus" Jul 06 22:11:48.321909 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.321138 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d" containerName="kube-rbac-proxy-web" Jul 06 22:11:48.321909 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.321145 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d" containerName="config-reloader" Jul 06 22:11:48.321909 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.321366 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dce1ad64f7310c0e3f6de2bb630af3cd4c4168932a432f48be33cde6c84ea430"} err="failed to get container status \"dce1ad64f7310c0e3f6de2bb630af3cd4c4168932a432f48be33cde6c84ea430\": rpc error: code = NotFound desc = could not find container \"dce1ad64f7310c0e3f6de2bb630af3cd4c4168932a432f48be33cde6c84ea430\": container with ID starting with dce1ad64f7310c0e3f6de2bb630af3cd4c4168932a432f48be33cde6c84ea430 not found: ID does not exist" Jul 06 22:11:48.321909 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.321388 2555 scope.go:117] "RemoveContainer" containerID="90c27314e532c2b8d3dce39f5fd09e6928ef39e4fb97a65d2fb8850cd5049e25" Jul 06 22:11:48.321909 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.321618 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90c27314e532c2b8d3dce39f5fd09e6928ef39e4fb97a65d2fb8850cd5049e25"} err="failed to get container status \"90c27314e532c2b8d3dce39f5fd09e6928ef39e4fb97a65d2fb8850cd5049e25\": rpc error: code = NotFound desc = could not find container \"90c27314e532c2b8d3dce39f5fd09e6928ef39e4fb97a65d2fb8850cd5049e25\": container with ID starting with 90c27314e532c2b8d3dce39f5fd09e6928ef39e4fb97a65d2fb8850cd5049e25 not found: ID does not exist" Jul 06 22:11:48.321909 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.321636 2555 scope.go:117] "RemoveContainer" containerID="b76c28e5cf3ded5c5d3a2bbddc03ad092e65235b16f6341a2d45dffe87074a17" Jul 06 22:11:48.322149 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.321915 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b76c28e5cf3ded5c5d3a2bbddc03ad092e65235b16f6341a2d45dffe87074a17"} err="failed to get container status \"b76c28e5cf3ded5c5d3a2bbddc03ad092e65235b16f6341a2d45dffe87074a17\": rpc error: code = NotFound desc = could not find container \"b76c28e5cf3ded5c5d3a2bbddc03ad092e65235b16f6341a2d45dffe87074a17\": container with ID starting with b76c28e5cf3ded5c5d3a2bbddc03ad092e65235b16f6341a2d45dffe87074a17 not found: ID does not exist" Jul 06 22:11:48.322149 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.321943 2555 scope.go:117] "RemoveContainer" containerID="868f0e2e2832f412be2280dceca76d6075b832497bd7017e0cdbd6c7709d4a8e" Jul 06 22:11:48.322602 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.322584 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"868f0e2e2832f412be2280dceca76d6075b832497bd7017e0cdbd6c7709d4a8e"} err="failed to get container status \"868f0e2e2832f412be2280dceca76d6075b832497bd7017e0cdbd6c7709d4a8e\": rpc error: code = NotFound desc = could not find container \"868f0e2e2832f412be2280dceca76d6075b832497bd7017e0cdbd6c7709d4a8e\": container with ID starting with 868f0e2e2832f412be2280dceca76d6075b832497bd7017e0cdbd6c7709d4a8e not found: ID does not exist" Jul 06 22:11:48.322602 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.322602 2555 scope.go:117] "RemoveContainer" containerID="70c341aec9d02c2801bd2464b7fdd403607f98f4669b5a840b5d396ea57352c2" Jul 06 22:11:48.322842 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.322824 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70c341aec9d02c2801bd2464b7fdd403607f98f4669b5a840b5d396ea57352c2"} err="failed to get container status \"70c341aec9d02c2801bd2464b7fdd403607f98f4669b5a840b5d396ea57352c2\": rpc error: code = NotFound desc = could not find container \"70c341aec9d02c2801bd2464b7fdd403607f98f4669b5a840b5d396ea57352c2\": container with ID starting with 70c341aec9d02c2801bd2464b7fdd403607f98f4669b5a840b5d396ea57352c2 not found: ID does not exist" Jul 06 22:11:48.322893 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.322843 2555 scope.go:117] "RemoveContainer" containerID="7d22f9d9188e3d530d12c1f8c8f9534cde4bab8897f3fba95c2c73cd6e9d95e2" Jul 06 22:11:48.323158 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.323137 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d22f9d9188e3d530d12c1f8c8f9534cde4bab8897f3fba95c2c73cd6e9d95e2"} err="failed to get container status \"7d22f9d9188e3d530d12c1f8c8f9534cde4bab8897f3fba95c2c73cd6e9d95e2\": rpc error: code = NotFound desc = could not find container \"7d22f9d9188e3d530d12c1f8c8f9534cde4bab8897f3fba95c2c73cd6e9d95e2\": container with ID starting with 7d22f9d9188e3d530d12c1f8c8f9534cde4bab8897f3fba95c2c73cd6e9d95e2 not found: ID does not exist" Jul 06 22:11:48.323158 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.323157 2555 scope.go:117] "RemoveContainer" containerID="d48a4ada40084d249f040e237b4daac841bd3fd6fae77bcfcfa8300ba443025e" Jul 06 22:11:48.323417 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.323399 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d48a4ada40084d249f040e237b4daac841bd3fd6fae77bcfcfa8300ba443025e"} err="failed to get container status \"d48a4ada40084d249f040e237b4daac841bd3fd6fae77bcfcfa8300ba443025e\": rpc error: code = NotFound desc = could not find container \"d48a4ada40084d249f040e237b4daac841bd3fd6fae77bcfcfa8300ba443025e\": container with ID starting with d48a4ada40084d249f040e237b4daac841bd3fd6fae77bcfcfa8300ba443025e not found: ID does not exist" Jul 06 22:11:48.323479 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.323417 2555 scope.go:117] "RemoveContainer" containerID="dce1ad64f7310c0e3f6de2bb630af3cd4c4168932a432f48be33cde6c84ea430" Jul 06 22:11:48.323641 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.323621 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dce1ad64f7310c0e3f6de2bb630af3cd4c4168932a432f48be33cde6c84ea430"} err="failed to get container status \"dce1ad64f7310c0e3f6de2bb630af3cd4c4168932a432f48be33cde6c84ea430\": rpc error: code = NotFound desc = could not find container \"dce1ad64f7310c0e3f6de2bb630af3cd4c4168932a432f48be33cde6c84ea430\": container with ID starting with dce1ad64f7310c0e3f6de2bb630af3cd4c4168932a432f48be33cde6c84ea430 not found: ID does not exist" Jul 06 22:11:48.323641 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.323640 2555 scope.go:117] "RemoveContainer" containerID="90c27314e532c2b8d3dce39f5fd09e6928ef39e4fb97a65d2fb8850cd5049e25" Jul 06 22:11:48.323847 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.323832 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90c27314e532c2b8d3dce39f5fd09e6928ef39e4fb97a65d2fb8850cd5049e25"} err="failed to get container status \"90c27314e532c2b8d3dce39f5fd09e6928ef39e4fb97a65d2fb8850cd5049e25\": rpc error: code = NotFound desc = could not find container \"90c27314e532c2b8d3dce39f5fd09e6928ef39e4fb97a65d2fb8850cd5049e25\": container with ID starting with 90c27314e532c2b8d3dce39f5fd09e6928ef39e4fb97a65d2fb8850cd5049e25 not found: ID does not exist" Jul 06 22:11:48.323847 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.323846 2555 scope.go:117] "RemoveContainer" containerID="b76c28e5cf3ded5c5d3a2bbddc03ad092e65235b16f6341a2d45dffe87074a17" Jul 06 22:11:48.324086 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.324067 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b76c28e5cf3ded5c5d3a2bbddc03ad092e65235b16f6341a2d45dffe87074a17"} err="failed to get container status \"b76c28e5cf3ded5c5d3a2bbddc03ad092e65235b16f6341a2d45dffe87074a17\": rpc error: code = NotFound desc = could not find container \"b76c28e5cf3ded5c5d3a2bbddc03ad092e65235b16f6341a2d45dffe87074a17\": container with ID starting with b76c28e5cf3ded5c5d3a2bbddc03ad092e65235b16f6341a2d45dffe87074a17 not found: ID does not exist" Jul 06 22:11:48.324143 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.324086 2555 scope.go:117] "RemoveContainer" containerID="868f0e2e2832f412be2280dceca76d6075b832497bd7017e0cdbd6c7709d4a8e" Jul 06 22:11:48.324312 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.324290 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"868f0e2e2832f412be2280dceca76d6075b832497bd7017e0cdbd6c7709d4a8e"} err="failed to get container status \"868f0e2e2832f412be2280dceca76d6075b832497bd7017e0cdbd6c7709d4a8e\": rpc error: code = NotFound desc = could not find container \"868f0e2e2832f412be2280dceca76d6075b832497bd7017e0cdbd6c7709d4a8e\": container with ID starting with 868f0e2e2832f412be2280dceca76d6075b832497bd7017e0cdbd6c7709d4a8e not found: ID does not exist" Jul 06 22:11:48.324355 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.324314 2555 scope.go:117] "RemoveContainer" containerID="70c341aec9d02c2801bd2464b7fdd403607f98f4669b5a840b5d396ea57352c2" Jul 06 22:11:48.324519 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.324504 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70c341aec9d02c2801bd2464b7fdd403607f98f4669b5a840b5d396ea57352c2"} err="failed to get container status \"70c341aec9d02c2801bd2464b7fdd403607f98f4669b5a840b5d396ea57352c2\": rpc error: code = NotFound desc = could not find container \"70c341aec9d02c2801bd2464b7fdd403607f98f4669b5a840b5d396ea57352c2\": container with ID starting with 70c341aec9d02c2801bd2464b7fdd403607f98f4669b5a840b5d396ea57352c2 not found: ID does not exist" Jul 06 22:11:48.324562 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.324520 2555 scope.go:117] "RemoveContainer" containerID="7d22f9d9188e3d530d12c1f8c8f9534cde4bab8897f3fba95c2c73cd6e9d95e2" Jul 06 22:11:48.324790 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.324763 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d22f9d9188e3d530d12c1f8c8f9534cde4bab8897f3fba95c2c73cd6e9d95e2"} err="failed to get container status \"7d22f9d9188e3d530d12c1f8c8f9534cde4bab8897f3fba95c2c73cd6e9d95e2\": rpc error: code = NotFound desc = could not find container \"7d22f9d9188e3d530d12c1f8c8f9534cde4bab8897f3fba95c2c73cd6e9d95e2\": container with ID starting with 7d22f9d9188e3d530d12c1f8c8f9534cde4bab8897f3fba95c2c73cd6e9d95e2 not found: ID does not exist" Jul 06 22:11:48.324790 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.324790 2555 scope.go:117] "RemoveContainer" containerID="d48a4ada40084d249f040e237b4daac841bd3fd6fae77bcfcfa8300ba443025e" Jul 06 22:11:48.325023 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.324992 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:11:48.325104 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.325080 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d48a4ada40084d249f040e237b4daac841bd3fd6fae77bcfcfa8300ba443025e"} err="failed to get container status \"d48a4ada40084d249f040e237b4daac841bd3fd6fae77bcfcfa8300ba443025e\": rpc error: code = NotFound desc = could not find container \"d48a4ada40084d249f040e237b4daac841bd3fd6fae77bcfcfa8300ba443025e\": container with ID starting with d48a4ada40084d249f040e237b4daac841bd3fd6fae77bcfcfa8300ba443025e not found: ID does not exist" Jul 06 22:11:48.325173 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.325105 2555 scope.go:117] "RemoveContainer" containerID="dce1ad64f7310c0e3f6de2bb630af3cd4c4168932a432f48be33cde6c84ea430" Jul 06 22:11:48.325559 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.325438 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dce1ad64f7310c0e3f6de2bb630af3cd4c4168932a432f48be33cde6c84ea430"} err="failed to get container status \"dce1ad64f7310c0e3f6de2bb630af3cd4c4168932a432f48be33cde6c84ea430\": rpc error: code = NotFound desc = could not find container \"dce1ad64f7310c0e3f6de2bb630af3cd4c4168932a432f48be33cde6c84ea430\": container with ID starting with dce1ad64f7310c0e3f6de2bb630af3cd4c4168932a432f48be33cde6c84ea430 not found: ID does not exist" Jul 06 22:11:48.325559 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.325561 2555 scope.go:117] "RemoveContainer" containerID="90c27314e532c2b8d3dce39f5fd09e6928ef39e4fb97a65d2fb8850cd5049e25" Jul 06 22:11:48.325950 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.325927 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90c27314e532c2b8d3dce39f5fd09e6928ef39e4fb97a65d2fb8850cd5049e25"} err="failed to get container status \"90c27314e532c2b8d3dce39f5fd09e6928ef39e4fb97a65d2fb8850cd5049e25\": rpc error: code = NotFound desc = could not find container \"90c27314e532c2b8d3dce39f5fd09e6928ef39e4fb97a65d2fb8850cd5049e25\": container with ID starting with 90c27314e532c2b8d3dce39f5fd09e6928ef39e4fb97a65d2fb8850cd5049e25 not found: ID does not exist" Jul 06 22:11:48.326079 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.325954 2555 scope.go:117] "RemoveContainer" containerID="b76c28e5cf3ded5c5d3a2bbddc03ad092e65235b16f6341a2d45dffe87074a17" Jul 06 22:11:48.326360 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.326341 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b76c28e5cf3ded5c5d3a2bbddc03ad092e65235b16f6341a2d45dffe87074a17"} err="failed to get container status \"b76c28e5cf3ded5c5d3a2bbddc03ad092e65235b16f6341a2d45dffe87074a17\": rpc error: code = NotFound desc = could not find container \"b76c28e5cf3ded5c5d3a2bbddc03ad092e65235b16f6341a2d45dffe87074a17\": container with ID starting with b76c28e5cf3ded5c5d3a2bbddc03ad092e65235b16f6341a2d45dffe87074a17 not found: ID does not exist" Jul 06 22:11:48.326498 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.326362 2555 scope.go:117] "RemoveContainer" containerID="868f0e2e2832f412be2280dceca76d6075b832497bd7017e0cdbd6c7709d4a8e" Jul 06 22:11:48.326656 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.326638 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"868f0e2e2832f412be2280dceca76d6075b832497bd7017e0cdbd6c7709d4a8e"} err="failed to get container status \"868f0e2e2832f412be2280dceca76d6075b832497bd7017e0cdbd6c7709d4a8e\": rpc error: code = NotFound desc = could not find container \"868f0e2e2832f412be2280dceca76d6075b832497bd7017e0cdbd6c7709d4a8e\": container with ID starting with 868f0e2e2832f412be2280dceca76d6075b832497bd7017e0cdbd6c7709d4a8e not found: ID does not exist" Jul 06 22:11:48.326703 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.326656 2555 scope.go:117] "RemoveContainer" containerID="70c341aec9d02c2801bd2464b7fdd403607f98f4669b5a840b5d396ea57352c2" Jul 06 22:11:48.326892 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.326873 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70c341aec9d02c2801bd2464b7fdd403607f98f4669b5a840b5d396ea57352c2"} err="failed to get container status \"70c341aec9d02c2801bd2464b7fdd403607f98f4669b5a840b5d396ea57352c2\": rpc error: code = NotFound desc = could not find container \"70c341aec9d02c2801bd2464b7fdd403607f98f4669b5a840b5d396ea57352c2\": container with ID starting with 70c341aec9d02c2801bd2464b7fdd403607f98f4669b5a840b5d396ea57352c2 not found: ID does not exist" Jul 06 22:11:48.326966 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.326892 2555 scope.go:117] "RemoveContainer" containerID="7d22f9d9188e3d530d12c1f8c8f9534cde4bab8897f3fba95c2c73cd6e9d95e2" Jul 06 22:11:48.327175 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.327147 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d22f9d9188e3d530d12c1f8c8f9534cde4bab8897f3fba95c2c73cd6e9d95e2"} err="failed to get container status \"7d22f9d9188e3d530d12c1f8c8f9534cde4bab8897f3fba95c2c73cd6e9d95e2\": rpc error: code = NotFound desc = could not find container \"7d22f9d9188e3d530d12c1f8c8f9534cde4bab8897f3fba95c2c73cd6e9d95e2\": container with ID starting with 7d22f9d9188e3d530d12c1f8c8f9534cde4bab8897f3fba95c2c73cd6e9d95e2 not found: ID does not exist" Jul 06 22:11:48.327276 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.327176 2555 scope.go:117] "RemoveContainer" containerID="d48a4ada40084d249f040e237b4daac841bd3fd6fae77bcfcfa8300ba443025e" Jul 06 22:11:48.327470 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.327453 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d48a4ada40084d249f040e237b4daac841bd3fd6fae77bcfcfa8300ba443025e"} err="failed to get container status \"d48a4ada40084d249f040e237b4daac841bd3fd6fae77bcfcfa8300ba443025e\": rpc error: code = NotFound desc = could not find container \"d48a4ada40084d249f040e237b4daac841bd3fd6fae77bcfcfa8300ba443025e\": container with ID starting with d48a4ada40084d249f040e237b4daac841bd3fd6fae77bcfcfa8300ba443025e not found: ID does not exist" Jul 06 22:11:48.327542 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.327471 2555 scope.go:117] "RemoveContainer" containerID="dce1ad64f7310c0e3f6de2bb630af3cd4c4168932a432f48be33cde6c84ea430" Jul 06 22:11:48.327753 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.327730 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dce1ad64f7310c0e3f6de2bb630af3cd4c4168932a432f48be33cde6c84ea430"} err="failed to get container status \"dce1ad64f7310c0e3f6de2bb630af3cd4c4168932a432f48be33cde6c84ea430\": rpc error: code = NotFound desc = could not find container \"dce1ad64f7310c0e3f6de2bb630af3cd4c4168932a432f48be33cde6c84ea430\": container with ID starting with dce1ad64f7310c0e3f6de2bb630af3cd4c4168932a432f48be33cde6c84ea430 not found: ID does not exist" Jul 06 22:11:48.327820 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.327754 2555 scope.go:117] "RemoveContainer" containerID="90c27314e532c2b8d3dce39f5fd09e6928ef39e4fb97a65d2fb8850cd5049e25" Jul 06 22:11:48.327820 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.327783 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 06 22:11:48.327956 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.327941 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 06 22:11:48.328041 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.327967 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90c27314e532c2b8d3dce39f5fd09e6928ef39e4fb97a65d2fb8850cd5049e25"} err="failed to get container status \"90c27314e532c2b8d3dce39f5fd09e6928ef39e4fb97a65d2fb8850cd5049e25\": rpc error: code = NotFound desc = could not find container \"90c27314e532c2b8d3dce39f5fd09e6928ef39e4fb97a65d2fb8850cd5049e25\": container with ID starting with 90c27314e532c2b8d3dce39f5fd09e6928ef39e4fb97a65d2fb8850cd5049e25 not found: ID does not exist" Jul 06 22:11:48.328041 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.327989 2555 scope.go:117] "RemoveContainer" containerID="b76c28e5cf3ded5c5d3a2bbddc03ad092e65235b16f6341a2d45dffe87074a17" Jul 06 22:11:48.328157 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.327995 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 06 22:11:48.328209 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.328045 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 06 22:11:48.328261 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.328210 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 06 22:11:48.328327 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.328311 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3ure0b77nbuid\"" Jul 06 22:11:48.328403 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.328373 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b76c28e5cf3ded5c5d3a2bbddc03ad092e65235b16f6341a2d45dffe87074a17"} err="failed to get container status \"b76c28e5cf3ded5c5d3a2bbddc03ad092e65235b16f6341a2d45dffe87074a17\": rpc error: code = NotFound desc = could not find container \"b76c28e5cf3ded5c5d3a2bbddc03ad092e65235b16f6341a2d45dffe87074a17\": container with ID starting with b76c28e5cf3ded5c5d3a2bbddc03ad092e65235b16f6341a2d45dffe87074a17 not found: ID does not exist" Jul 06 22:11:48.328465 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.328406 2555 scope.go:117] "RemoveContainer" containerID="868f0e2e2832f412be2280dceca76d6075b832497bd7017e0cdbd6c7709d4a8e" Jul 06 22:11:48.328515 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.328503 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 06 22:11:48.328734 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.328704 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"868f0e2e2832f412be2280dceca76d6075b832497bd7017e0cdbd6c7709d4a8e"} err="failed to get container status \"868f0e2e2832f412be2280dceca76d6075b832497bd7017e0cdbd6c7709d4a8e\": rpc error: code = NotFound desc = could not find container \"868f0e2e2832f412be2280dceca76d6075b832497bd7017e0cdbd6c7709d4a8e\": container with ID starting with 868f0e2e2832f412be2280dceca76d6075b832497bd7017e0cdbd6c7709d4a8e not found: ID does not exist" Jul 06 22:11:48.328734 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.328731 2555 scope.go:117] "RemoveContainer" containerID="70c341aec9d02c2801bd2464b7fdd403607f98f4669b5a840b5d396ea57352c2" Jul 06 22:11:48.328967 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.328939 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70c341aec9d02c2801bd2464b7fdd403607f98f4669b5a840b5d396ea57352c2"} err="failed to get container status \"70c341aec9d02c2801bd2464b7fdd403607f98f4669b5a840b5d396ea57352c2\": rpc error: code = NotFound desc = could not find container \"70c341aec9d02c2801bd2464b7fdd403607f98f4669b5a840b5d396ea57352c2\": container with ID starting with 70c341aec9d02c2801bd2464b7fdd403607f98f4669b5a840b5d396ea57352c2 not found: ID does not exist" Jul 06 22:11:48.328967 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.328956 2555 scope.go:117] "RemoveContainer" containerID="7d22f9d9188e3d530d12c1f8c8f9534cde4bab8897f3fba95c2c73cd6e9d95e2" Jul 06 22:11:48.329138 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.328989 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 06 22:11:48.329138 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.329029 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 06 22:11:48.329138 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.329098 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 06 22:11:48.329442 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.329397 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 06 22:11:48.329534 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.329462 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d22f9d9188e3d530d12c1f8c8f9534cde4bab8897f3fba95c2c73cd6e9d95e2"} err="failed to get container status \"7d22f9d9188e3d530d12c1f8c8f9534cde4bab8897f3fba95c2c73cd6e9d95e2\": rpc error: code = NotFound desc = could not find container \"7d22f9d9188e3d530d12c1f8c8f9534cde4bab8897f3fba95c2c73cd6e9d95e2\": container with ID starting with 7d22f9d9188e3d530d12c1f8c8f9534cde4bab8897f3fba95c2c73cd6e9d95e2 not found: ID does not exist" Jul 06 22:11:48.329534 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.329496 2555 scope.go:117] "RemoveContainer" containerID="d48a4ada40084d249f040e237b4daac841bd3fd6fae77bcfcfa8300ba443025e" Jul 06 22:11:48.329534 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.329528 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 06 22:11:48.329695 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.329547 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 06 22:11:48.329695 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.329677 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-s6xbw\"" Jul 06 22:11:48.329796 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.329757 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d48a4ada40084d249f040e237b4daac841bd3fd6fae77bcfcfa8300ba443025e"} err="failed to get container status \"d48a4ada40084d249f040e237b4daac841bd3fd6fae77bcfcfa8300ba443025e\": rpc error: code = NotFound desc = could not find container \"d48a4ada40084d249f040e237b4daac841bd3fd6fae77bcfcfa8300ba443025e\": container with ID starting with d48a4ada40084d249f040e237b4daac841bd3fd6fae77bcfcfa8300ba443025e not found: ID does not exist" Jul 06 22:11:48.329796 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.329781 2555 scope.go:117] "RemoveContainer" containerID="dce1ad64f7310c0e3f6de2bb630af3cd4c4168932a432f48be33cde6c84ea430" Jul 06 22:11:48.330205 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.330176 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dce1ad64f7310c0e3f6de2bb630af3cd4c4168932a432f48be33cde6c84ea430"} err="failed to get container status \"dce1ad64f7310c0e3f6de2bb630af3cd4c4168932a432f48be33cde6c84ea430\": rpc error: code = NotFound desc = could not find container \"dce1ad64f7310c0e3f6de2bb630af3cd4c4168932a432f48be33cde6c84ea430\": container with ID starting with dce1ad64f7310c0e3f6de2bb630af3cd4c4168932a432f48be33cde6c84ea430 not found: ID does not exist" Jul 06 22:11:48.330300 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.330206 2555 scope.go:117] "RemoveContainer" containerID="90c27314e532c2b8d3dce39f5fd09e6928ef39e4fb97a65d2fb8850cd5049e25" Jul 06 22:11:48.330461 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.330442 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90c27314e532c2b8d3dce39f5fd09e6928ef39e4fb97a65d2fb8850cd5049e25"} err="failed to get container status \"90c27314e532c2b8d3dce39f5fd09e6928ef39e4fb97a65d2fb8850cd5049e25\": rpc error: code = NotFound desc = could not find container \"90c27314e532c2b8d3dce39f5fd09e6928ef39e4fb97a65d2fb8850cd5049e25\": container with ID starting with 90c27314e532c2b8d3dce39f5fd09e6928ef39e4fb97a65d2fb8850cd5049e25 not found: ID does not exist" Jul 06 22:11:48.330561 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.330476 2555 scope.go:117] "RemoveContainer" containerID="b76c28e5cf3ded5c5d3a2bbddc03ad092e65235b16f6341a2d45dffe87074a17" Jul 06 22:11:48.330792 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.330739 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b76c28e5cf3ded5c5d3a2bbddc03ad092e65235b16f6341a2d45dffe87074a17"} err="failed to get container status \"b76c28e5cf3ded5c5d3a2bbddc03ad092e65235b16f6341a2d45dffe87074a17\": rpc error: code = NotFound desc = could not find container \"b76c28e5cf3ded5c5d3a2bbddc03ad092e65235b16f6341a2d45dffe87074a17\": container with ID starting with b76c28e5cf3ded5c5d3a2bbddc03ad092e65235b16f6341a2d45dffe87074a17 not found: ID does not exist" Jul 06 22:11:48.330792 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.330766 2555 scope.go:117] "RemoveContainer" containerID="868f0e2e2832f412be2280dceca76d6075b832497bd7017e0cdbd6c7709d4a8e" Jul 06 22:11:48.331077 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.331048 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"868f0e2e2832f412be2280dceca76d6075b832497bd7017e0cdbd6c7709d4a8e"} err="failed to get container status \"868f0e2e2832f412be2280dceca76d6075b832497bd7017e0cdbd6c7709d4a8e\": rpc error: code = NotFound desc = could not find container \"868f0e2e2832f412be2280dceca76d6075b832497bd7017e0cdbd6c7709d4a8e\": container with ID starting with 868f0e2e2832f412be2280dceca76d6075b832497bd7017e0cdbd6c7709d4a8e not found: ID does not exist" Jul 06 22:11:48.331077 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.331075 2555 scope.go:117] "RemoveContainer" containerID="70c341aec9d02c2801bd2464b7fdd403607f98f4669b5a840b5d396ea57352c2" Jul 06 22:11:48.331647 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.331617 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70c341aec9d02c2801bd2464b7fdd403607f98f4669b5a840b5d396ea57352c2"} err="failed to get container status \"70c341aec9d02c2801bd2464b7fdd403607f98f4669b5a840b5d396ea57352c2\": rpc error: code = NotFound desc = could not find container \"70c341aec9d02c2801bd2464b7fdd403607f98f4669b5a840b5d396ea57352c2\": container with ID starting with 70c341aec9d02c2801bd2464b7fdd403607f98f4669b5a840b5d396ea57352c2 not found: ID does not exist" Jul 06 22:11:48.331748 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.331648 2555 scope.go:117] "RemoveContainer" containerID="7d22f9d9188e3d530d12c1f8c8f9534cde4bab8897f3fba95c2c73cd6e9d95e2" Jul 06 22:11:48.332024 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.331978 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d22f9d9188e3d530d12c1f8c8f9534cde4bab8897f3fba95c2c73cd6e9d95e2"} err="failed to get container status \"7d22f9d9188e3d530d12c1f8c8f9534cde4bab8897f3fba95c2c73cd6e9d95e2\": rpc error: code = NotFound desc = could not find container \"7d22f9d9188e3d530d12c1f8c8f9534cde4bab8897f3fba95c2c73cd6e9d95e2\": container with ID starting with 7d22f9d9188e3d530d12c1f8c8f9534cde4bab8897f3fba95c2c73cd6e9d95e2 not found: ID does not exist" Jul 06 22:11:48.332119 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.332039 2555 scope.go:117] "RemoveContainer" containerID="d48a4ada40084d249f040e237b4daac841bd3fd6fae77bcfcfa8300ba443025e" Jul 06 22:11:48.332353 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.332320 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 06 22:11:48.332471 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.332377 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d48a4ada40084d249f040e237b4daac841bd3fd6fae77bcfcfa8300ba443025e"} err="failed to get container status \"d48a4ada40084d249f040e237b4daac841bd3fd6fae77bcfcfa8300ba443025e\": rpc error: code = NotFound desc = could not find container \"d48a4ada40084d249f040e237b4daac841bd3fd6fae77bcfcfa8300ba443025e\": container with ID starting with d48a4ada40084d249f040e237b4daac841bd3fd6fae77bcfcfa8300ba443025e not found: ID does not exist" Jul 06 22:11:48.334469 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.334450 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 06 22:11:48.340534 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.340507 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 22:11:48.422959 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.422918 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85868a97-948b-4c4b-b5ab-30c7fbbfef4f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"85868a97-948b-4c4b-b5ab-30c7fbbfef4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:11:48.422959 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.422962 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/85868a97-948b-4c4b-b5ab-30c7fbbfef4f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"85868a97-948b-4c4b-b5ab-30c7fbbfef4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:11:48.423208 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.422988 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/85868a97-948b-4c4b-b5ab-30c7fbbfef4f-config\") pod \"prometheus-k8s-0\" (UID: \"85868a97-948b-4c4b-b5ab-30c7fbbfef4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:11:48.423208 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.423056 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/85868a97-948b-4c4b-b5ab-30c7fbbfef4f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"85868a97-948b-4c4b-b5ab-30c7fbbfef4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:11:48.423208 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.423084 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/85868a97-948b-4c4b-b5ab-30c7fbbfef4f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"85868a97-948b-4c4b-b5ab-30c7fbbfef4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:11:48.423208 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.423108 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/85868a97-948b-4c4b-b5ab-30c7fbbfef4f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"85868a97-948b-4c4b-b5ab-30c7fbbfef4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:11:48.423208 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.423166 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/85868a97-948b-4c4b-b5ab-30c7fbbfef4f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"85868a97-948b-4c4b-b5ab-30c7fbbfef4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:11:48.423208 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.423205 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/85868a97-948b-4c4b-b5ab-30c7fbbfef4f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"85868a97-948b-4c4b-b5ab-30c7fbbfef4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:11:48.423390 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.423253 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/85868a97-948b-4c4b-b5ab-30c7fbbfef4f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"85868a97-948b-4c4b-b5ab-30c7fbbfef4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:11:48.423390 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.423288 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85868a97-948b-4c4b-b5ab-30c7fbbfef4f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"85868a97-948b-4c4b-b5ab-30c7fbbfef4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:11:48.423390 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.423326 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/85868a97-948b-4c4b-b5ab-30c7fbbfef4f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"85868a97-948b-4c4b-b5ab-30c7fbbfef4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:11:48.423390 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.423354 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85868a97-948b-4c4b-b5ab-30c7fbbfef4f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"85868a97-948b-4c4b-b5ab-30c7fbbfef4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:11:48.423508 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.423404 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/85868a97-948b-4c4b-b5ab-30c7fbbfef4f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"85868a97-948b-4c4b-b5ab-30c7fbbfef4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:11:48.423508 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.423420 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/85868a97-948b-4c4b-b5ab-30c7fbbfef4f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"85868a97-948b-4c4b-b5ab-30c7fbbfef4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:11:48.423508 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.423443 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/85868a97-948b-4c4b-b5ab-30c7fbbfef4f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"85868a97-948b-4c4b-b5ab-30c7fbbfef4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:11:48.423508 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.423463 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/85868a97-948b-4c4b-b5ab-30c7fbbfef4f-web-config\") pod \"prometheus-k8s-0\" (UID: \"85868a97-948b-4c4b-b5ab-30c7fbbfef4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:11:48.423508 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.423476 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47pqw\" (UniqueName: \"kubernetes.io/projected/85868a97-948b-4c4b-b5ab-30c7fbbfef4f-kube-api-access-47pqw\") pod \"prometheus-k8s-0\" (UID: \"85868a97-948b-4c4b-b5ab-30c7fbbfef4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:11:48.423650 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.423507 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/85868a97-948b-4c4b-b5ab-30c7fbbfef4f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"85868a97-948b-4c4b-b5ab-30c7fbbfef4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:11:48.423650 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.423527 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/85868a97-948b-4c4b-b5ab-30c7fbbfef4f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"85868a97-948b-4c4b-b5ab-30c7fbbfef4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:11:48.423650 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.423546 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/85868a97-948b-4c4b-b5ab-30c7fbbfef4f-config-out\") pod \"prometheus-k8s-0\" (UID: \"85868a97-948b-4c4b-b5ab-30c7fbbfef4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:11:48.524942 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.524889 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/85868a97-948b-4c4b-b5ab-30c7fbbfef4f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"85868a97-948b-4c4b-b5ab-30c7fbbfef4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:11:48.524942 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.524943 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/85868a97-948b-4c4b-b5ab-30c7fbbfef4f-web-config\") pod \"prometheus-k8s-0\" (UID: \"85868a97-948b-4c4b-b5ab-30c7fbbfef4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:11:48.525222 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.524965 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47pqw\" (UniqueName: \"kubernetes.io/projected/85868a97-948b-4c4b-b5ab-30c7fbbfef4f-kube-api-access-47pqw\") pod \"prometheus-k8s-0\" (UID: \"85868a97-948b-4c4b-b5ab-30c7fbbfef4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:11:48.525222 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.525002 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/85868a97-948b-4c4b-b5ab-30c7fbbfef4f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"85868a97-948b-4c4b-b5ab-30c7fbbfef4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:11:48.525222 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.525046 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/85868a97-948b-4c4b-b5ab-30c7fbbfef4f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"85868a97-948b-4c4b-b5ab-30c7fbbfef4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:11:48.525222 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.525070 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/85868a97-948b-4c4b-b5ab-30c7fbbfef4f-config-out\") pod \"prometheus-k8s-0\" (UID: \"85868a97-948b-4c4b-b5ab-30c7fbbfef4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:11:48.525222 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.525124 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85868a97-948b-4c4b-b5ab-30c7fbbfef4f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"85868a97-948b-4c4b-b5ab-30c7fbbfef4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:11:48.525222 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.525151 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/85868a97-948b-4c4b-b5ab-30c7fbbfef4f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"85868a97-948b-4c4b-b5ab-30c7fbbfef4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:11:48.525222 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.525175 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/85868a97-948b-4c4b-b5ab-30c7fbbfef4f-config\") pod \"prometheus-k8s-0\" (UID: \"85868a97-948b-4c4b-b5ab-30c7fbbfef4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:11:48.525568 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.525411 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/85868a97-948b-4c4b-b5ab-30c7fbbfef4f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"85868a97-948b-4c4b-b5ab-30c7fbbfef4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:11:48.525568 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.525452 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/85868a97-948b-4c4b-b5ab-30c7fbbfef4f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"85868a97-948b-4c4b-b5ab-30c7fbbfef4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:11:48.525568 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.525481 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/85868a97-948b-4c4b-b5ab-30c7fbbfef4f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"85868a97-948b-4c4b-b5ab-30c7fbbfef4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:11:48.525568 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.525534 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/85868a97-948b-4c4b-b5ab-30c7fbbfef4f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"85868a97-948b-4c4b-b5ab-30c7fbbfef4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:11:48.525568 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.525566 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/85868a97-948b-4c4b-b5ab-30c7fbbfef4f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"85868a97-948b-4c4b-b5ab-30c7fbbfef4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:11:48.525823 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.525597 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/85868a97-948b-4c4b-b5ab-30c7fbbfef4f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"85868a97-948b-4c4b-b5ab-30c7fbbfef4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:11:48.525823 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.525629 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85868a97-948b-4c4b-b5ab-30c7fbbfef4f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"85868a97-948b-4c4b-b5ab-30c7fbbfef4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:11:48.525823 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.525664 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/85868a97-948b-4c4b-b5ab-30c7fbbfef4f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"85868a97-948b-4c4b-b5ab-30c7fbbfef4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:11:48.525823 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.525705 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85868a97-948b-4c4b-b5ab-30c7fbbfef4f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"85868a97-948b-4c4b-b5ab-30c7fbbfef4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:11:48.525823 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.525766 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/85868a97-948b-4c4b-b5ab-30c7fbbfef4f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"85868a97-948b-4c4b-b5ab-30c7fbbfef4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:11:48.525823 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.525808 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/85868a97-948b-4c4b-b5ab-30c7fbbfef4f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"85868a97-948b-4c4b-b5ab-30c7fbbfef4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:11:48.526516 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.526490 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/85868a97-948b-4c4b-b5ab-30c7fbbfef4f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"85868a97-948b-4c4b-b5ab-30c7fbbfef4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:11:48.528163 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.528137 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/85868a97-948b-4c4b-b5ab-30c7fbbfef4f-config\") pod \"prometheus-k8s-0\" (UID: \"85868a97-948b-4c4b-b5ab-30c7fbbfef4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:11:48.528265 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.528209 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/85868a97-948b-4c4b-b5ab-30c7fbbfef4f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"85868a97-948b-4c4b-b5ab-30c7fbbfef4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:11:48.528329 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.528300 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/85868a97-948b-4c4b-b5ab-30c7fbbfef4f-web-config\") pod \"prometheus-k8s-0\" (UID: \"85868a97-948b-4c4b-b5ab-30c7fbbfef4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:11:48.528381 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.528326 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/85868a97-948b-4c4b-b5ab-30c7fbbfef4f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"85868a97-948b-4c4b-b5ab-30c7fbbfef4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:11:48.528442 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.528391 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/85868a97-948b-4c4b-b5ab-30c7fbbfef4f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"85868a97-948b-4c4b-b5ab-30c7fbbfef4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:11:48.528752 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.528725 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/85868a97-948b-4c4b-b5ab-30c7fbbfef4f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"85868a97-948b-4c4b-b5ab-30c7fbbfef4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:11:48.528904 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.528787 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/85868a97-948b-4c4b-b5ab-30c7fbbfef4f-config-out\") pod \"prometheus-k8s-0\" (UID: \"85868a97-948b-4c4b-b5ab-30c7fbbfef4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:11:48.528904 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.528846 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85868a97-948b-4c4b-b5ab-30c7fbbfef4f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"85868a97-948b-4c4b-b5ab-30c7fbbfef4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:11:48.529083 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.529049 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85868a97-948b-4c4b-b5ab-30c7fbbfef4f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"85868a97-948b-4c4b-b5ab-30c7fbbfef4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:11:48.529516 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.529349 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85868a97-948b-4c4b-b5ab-30c7fbbfef4f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"85868a97-948b-4c4b-b5ab-30c7fbbfef4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:11:48.529516 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.529447 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/85868a97-948b-4c4b-b5ab-30c7fbbfef4f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"85868a97-948b-4c4b-b5ab-30c7fbbfef4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:11:48.529668 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.529573 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/85868a97-948b-4c4b-b5ab-30c7fbbfef4f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"85868a97-948b-4c4b-b5ab-30c7fbbfef4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:11:48.529668 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.529619 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/85868a97-948b-4c4b-b5ab-30c7fbbfef4f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"85868a97-948b-4c4b-b5ab-30c7fbbfef4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:11:48.530079 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.530058 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/85868a97-948b-4c4b-b5ab-30c7fbbfef4f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"85868a97-948b-4c4b-b5ab-30c7fbbfef4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:11:48.531041 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.531019 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/85868a97-948b-4c4b-b5ab-30c7fbbfef4f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"85868a97-948b-4c4b-b5ab-30c7fbbfef4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:11:48.531642 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.531623 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/85868a97-948b-4c4b-b5ab-30c7fbbfef4f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"85868a97-948b-4c4b-b5ab-30c7fbbfef4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:11:48.531866 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.531850 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/85868a97-948b-4c4b-b5ab-30c7fbbfef4f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"85868a97-948b-4c4b-b5ab-30c7fbbfef4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:11:48.532323 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.532303 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/85868a97-948b-4c4b-b5ab-30c7fbbfef4f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"85868a97-948b-4c4b-b5ab-30c7fbbfef4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:11:48.534091 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.534075 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47pqw\" (UniqueName: \"kubernetes.io/projected/85868a97-948b-4c4b-b5ab-30c7fbbfef4f-kube-api-access-47pqw\") pod \"prometheus-k8s-0\" (UID: \"85868a97-948b-4c4b-b5ab-30c7fbbfef4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:11:48.636881 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.636836 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:11:48.776741 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:48.776715 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 22:11:48.778597 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:11:48.778570 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85868a97_948b_4c4b_b5ab_30c7fbbfef4f.slice/crio-f90f8bb42e55433e127a3dcb3aa7d06a188dd27e13213931e4c2c809b7096532 WatchSource:0}: Error finding container f90f8bb42e55433e127a3dcb3aa7d06a188dd27e13213931e4c2c809b7096532: Status 404 returned error can't find the container with id f90f8bb42e55433e127a3dcb3aa7d06a188dd27e13213931e4c2c809b7096532 Jul 06 22:11:49.271142 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:49.271106 2555 generic.go:356] "Generic (PLEG): container finished" podID="85868a97-948b-4c4b-b5ab-30c7fbbfef4f" containerID="2cc6f903a67dd3b2e402c8ff2cae466c2c0482707f745b30f1b0332d48c71493" exitCode=0 Jul 06 22:11:49.271553 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:49.271188 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"85868a97-948b-4c4b-b5ab-30c7fbbfef4f","Type":"ContainerDied","Data":"2cc6f903a67dd3b2e402c8ff2cae466c2c0482707f745b30f1b0332d48c71493"} Jul 06 22:11:49.271553 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:49.271297 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"85868a97-948b-4c4b-b5ab-30c7fbbfef4f","Type":"ContainerStarted","Data":"f90f8bb42e55433e127a3dcb3aa7d06a188dd27e13213931e4c2c809b7096532"} Jul 06 22:11:49.339757 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:49.339724 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d" path="/var/lib/kubelet/pods/a3e80f4a-8b86-4fd5-9a7f-2e20f93e884d/volumes" Jul 06 22:11:50.281073 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:50.281032 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"85868a97-948b-4c4b-b5ab-30c7fbbfef4f","Type":"ContainerStarted","Data":"5328725113fc2ab43d6081eb706a9187b23a129e5e8086f6c637918541e0ef99"} Jul 06 22:11:50.281451 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:50.281119 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"85868a97-948b-4c4b-b5ab-30c7fbbfef4f","Type":"ContainerStarted","Data":"f9de2e458a0c82b4fa17e6e91bea5b523eb48395820488377565d427e017db30"} Jul 06 22:11:50.281451 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:50.281166 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"85868a97-948b-4c4b-b5ab-30c7fbbfef4f","Type":"ContainerStarted","Data":"5e20332bd8ad583e0bb0f4a30ff62f683b0644f41ff9ae839c252d345239675b"} Jul 06 22:11:50.281451 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:50.281201 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"85868a97-948b-4c4b-b5ab-30c7fbbfef4f","Type":"ContainerStarted","Data":"71da1db307e2556275a600d69e49e10049dce2f8eb1246e5c292ae4b6b130ae5"} Jul 06 22:11:50.281451 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:50.281236 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"85868a97-948b-4c4b-b5ab-30c7fbbfef4f","Type":"ContainerStarted","Data":"2fd7619ad85e7cb4761dfce35e3e5c9dd408c8b8ee751be649881e474af0a221"} Jul 06 22:11:50.281451 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:50.281276 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"85868a97-948b-4c4b-b5ab-30c7fbbfef4f","Type":"ContainerStarted","Data":"db66f3b2ebd592070603fa954d8176be4f694f710be0c20793357cb864f777da"} Jul 06 22:11:50.315788 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:50.315718 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.293483395 podStartE2EDuration="2.315665874s" totalImagesPullingTime="0s" totalInitContainerRuntime="22.182479ms" isStatefulPod=false podCreationTimestamp="2026-07-06 22:11:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:11:50.313438182 +0000 UTC m=+161.613150069" watchObservedRunningTime="2026-07-06 22:11:50.315665874 +0000 UTC m=+161.615377724" Jul 06 22:11:53.637295 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:11:53.637254 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:12:48.637785 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:12:48.637746 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:12:48.653498 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:12:48.653472 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:12:49.534789 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:12:49.534764 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:13:31.148464 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:13:31.148425 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-rdqnr"] Jul 06 22:13:31.150723 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:13:31.150707 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-rdqnr" Jul 06 22:13:31.153440 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:13:31.153410 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 06 22:13:31.153566 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:13:31.153454 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-z2wvr\"" Jul 06 22:13:31.153566 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:13:31.153498 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 06 22:13:31.161021 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:13:31.160960 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-rdqnr"] Jul 06 22:13:31.218683 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:13:31.218650 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nz7n\" (UniqueName: \"kubernetes.io/projected/3f16d563-15f5-4453-a5d3-c4971f5e5d18-kube-api-access-5nz7n\") pod \"cert-manager-cainjector-559cdd8c87-rdqnr\" (UID: \"3f16d563-15f5-4453-a5d3-c4971f5e5d18\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-rdqnr" Jul 06 22:13:31.218871 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:13:31.218720 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3f16d563-15f5-4453-a5d3-c4971f5e5d18-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-rdqnr\" (UID: \"3f16d563-15f5-4453-a5d3-c4971f5e5d18\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-rdqnr" Jul 06 22:13:31.320092 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:13:31.320050 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3f16d563-15f5-4453-a5d3-c4971f5e5d18-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-rdqnr\" (UID: \"3f16d563-15f5-4453-a5d3-c4971f5e5d18\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-rdqnr" Jul 06 22:13:31.320298 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:13:31.320131 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nz7n\" (UniqueName: \"kubernetes.io/projected/3f16d563-15f5-4453-a5d3-c4971f5e5d18-kube-api-access-5nz7n\") pod \"cert-manager-cainjector-559cdd8c87-rdqnr\" (UID: \"3f16d563-15f5-4453-a5d3-c4971f5e5d18\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-rdqnr" Jul 06 22:13:31.331281 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:13:31.331253 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3f16d563-15f5-4453-a5d3-c4971f5e5d18-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-rdqnr\" (UID: \"3f16d563-15f5-4453-a5d3-c4971f5e5d18\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-rdqnr" Jul 06 22:13:31.331520 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:13:31.331499 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nz7n\" (UniqueName: \"kubernetes.io/projected/3f16d563-15f5-4453-a5d3-c4971f5e5d18-kube-api-access-5nz7n\") pod \"cert-manager-cainjector-559cdd8c87-rdqnr\" (UID: \"3f16d563-15f5-4453-a5d3-c4971f5e5d18\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-rdqnr" Jul 06 22:13:31.469526 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:13:31.469452 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-rdqnr" Jul 06 22:13:31.604587 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:13:31.604562 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-rdqnr"] Jul 06 22:13:31.607218 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:13:31.607192 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f16d563_15f5_4453_a5d3_c4971f5e5d18.slice/crio-48bfa2417a3bef8967e4e65091a6f1f8131ac5c24b9c04df932a8a41c9a7d7df WatchSource:0}: Error finding container 48bfa2417a3bef8967e4e65091a6f1f8131ac5c24b9c04df932a8a41c9a7d7df: Status 404 returned error can't find the container with id 48bfa2417a3bef8967e4e65091a6f1f8131ac5c24b9c04df932a8a41c9a7d7df Jul 06 22:13:31.687118 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:13:31.687080 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-rdqnr" event={"ID":"3f16d563-15f5-4453-a5d3-c4971f5e5d18","Type":"ContainerStarted","Data":"48bfa2417a3bef8967e4e65091a6f1f8131ac5c24b9c04df932a8a41c9a7d7df"} Jul 06 22:13:35.196616 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:13:35.193820 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-dldbz"] Jul 06 22:13:35.198763 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:13:35.198737 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-dldbz" Jul 06 22:13:35.202360 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:13:35.202329 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-9xc2z\"" Jul 06 22:13:35.211044 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:13:35.211018 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-dldbz"] Jul 06 22:13:35.253963 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:13:35.253929 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vqdp\" (UniqueName: \"kubernetes.io/projected/443cc820-6cbe-460c-b074-41adf461c709-kube-api-access-6vqdp\") pod \"cert-manager-webhook-6486ffd997-dldbz\" (UID: \"443cc820-6cbe-460c-b074-41adf461c709\") " pod="cert-manager/cert-manager-webhook-6486ffd997-dldbz" Jul 06 22:13:35.254123 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:13:35.253970 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/443cc820-6cbe-460c-b074-41adf461c709-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-dldbz\" (UID: \"443cc820-6cbe-460c-b074-41adf461c709\") " pod="cert-manager/cert-manager-webhook-6486ffd997-dldbz" Jul 06 22:13:35.354908 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:13:35.354872 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vqdp\" (UniqueName: \"kubernetes.io/projected/443cc820-6cbe-460c-b074-41adf461c709-kube-api-access-6vqdp\") pod \"cert-manager-webhook-6486ffd997-dldbz\" (UID: \"443cc820-6cbe-460c-b074-41adf461c709\") " pod="cert-manager/cert-manager-webhook-6486ffd997-dldbz" Jul 06 22:13:35.355146 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:13:35.354918 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/443cc820-6cbe-460c-b074-41adf461c709-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-dldbz\" (UID: \"443cc820-6cbe-460c-b074-41adf461c709\") " pod="cert-manager/cert-manager-webhook-6486ffd997-dldbz" Jul 06 22:13:35.364524 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:13:35.364501 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/443cc820-6cbe-460c-b074-41adf461c709-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-dldbz\" (UID: \"443cc820-6cbe-460c-b074-41adf461c709\") " pod="cert-manager/cert-manager-webhook-6486ffd997-dldbz" Jul 06 22:13:35.364704 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:13:35.364684 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vqdp\" (UniqueName: \"kubernetes.io/projected/443cc820-6cbe-460c-b074-41adf461c709-kube-api-access-6vqdp\") pod \"cert-manager-webhook-6486ffd997-dldbz\" (UID: \"443cc820-6cbe-460c-b074-41adf461c709\") " pod="cert-manager/cert-manager-webhook-6486ffd997-dldbz" Jul 06 22:13:35.508479 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:13:35.508387 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-dldbz" Jul 06 22:13:35.639735 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:13:35.639708 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-dldbz"] Jul 06 22:13:35.641323 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:13:35.641297 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod443cc820_6cbe_460c_b074_41adf461c709.slice/crio-05d449adfc508b4a70dd589d8eaa7a0dc8cef29beef77561d3a4bdc09706171c WatchSource:0}: Error finding container 05d449adfc508b4a70dd589d8eaa7a0dc8cef29beef77561d3a4bdc09706171c: Status 404 returned error can't find the container with id 05d449adfc508b4a70dd589d8eaa7a0dc8cef29beef77561d3a4bdc09706171c Jul 06 22:13:35.707090 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:13:35.707051 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-dldbz" event={"ID":"443cc820-6cbe-460c-b074-41adf461c709","Type":"ContainerStarted","Data":"24c2d2452378850db79812ed5acbe8f96aec32e13a120f15e8443a55e6220094"} Jul 06 22:13:35.707293 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:13:35.707107 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-dldbz" event={"ID":"443cc820-6cbe-460c-b074-41adf461c709","Type":"ContainerStarted","Data":"05d449adfc508b4a70dd589d8eaa7a0dc8cef29beef77561d3a4bdc09706171c"} Jul 06 22:13:35.707293 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:13:35.707131 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-dldbz" Jul 06 22:13:35.708747 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:13:35.708714 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-rdqnr" event={"ID":"3f16d563-15f5-4453-a5d3-c4971f5e5d18","Type":"ContainerStarted","Data":"fe595f8941a1389c17c711de35246935aa90795f9a3b9163f0bf2c9a83f2d1f1"} Jul 06 22:13:35.731859 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:13:35.731800 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-dldbz" podStartSLOduration=0.731782469 podStartE2EDuration="731.782469ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:13:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:13:35.730678841 +0000 UTC m=+267.030390692" watchObservedRunningTime="2026-07-06 22:13:35.731782469 +0000 UTC m=+267.031494318" Jul 06 22:13:35.763111 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:13:35.762986 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-rdqnr" podStartSLOduration=1.702313119 podStartE2EDuration="4.76296808s" totalImagesPullingTime="3.060654961s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:13:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:13:35.761136187 +0000 UTC m=+267.060848036" watchObservedRunningTime="2026-07-06 22:13:35.76296808 +0000 UTC m=+267.062679929" Jul 06 22:13:41.715136 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:13:41.715100 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-dldbz" Jul 06 22:14:09.144377 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:09.144343 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7mq8_e50d3b5f-832e-481f-8807-7774ff8d5521/ovn-acl-logging/0.log" Jul 06 22:14:09.151979 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:09.151955 2555 kubelet.go:1648] "Image garbage collection succeeded" Jul 06 22:14:09.152812 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:09.152793 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7mq8_e50d3b5f-832e-481f-8807-7774ff8d5521/ovn-acl-logging/0.log" Jul 06 22:14:18.024614 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:18.024575 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7bcf4db656-jlzqb"] Jul 06 22:14:18.026854 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:18.026834 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7bcf4db656-jlzqb" Jul 06 22:14:18.030573 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:18.030550 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 06 22:14:18.030672 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:18.030599 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 06 22:14:18.031698 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:18.031680 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 06 22:14:18.031698 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:18.031691 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-bdtzw\"" Jul 06 22:14:18.031837 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:18.031795 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 06 22:14:18.031964 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:18.031950 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 06 22:14:18.038028 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:18.037988 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7bcf4db656-jlzqb"] Jul 06 22:14:18.109441 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:18.109409 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/08a08704-1234-4ea2-9c3c-af2d3dad8fda-manager-config\") pod \"lws-controller-manager-7bcf4db656-jlzqb\" (UID: \"08a08704-1234-4ea2-9c3c-af2d3dad8fda\") " pod="openshift-lws-operator/lws-controller-manager-7bcf4db656-jlzqb" Jul 06 22:14:18.109624 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:18.109460 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flcz6\" (UniqueName: \"kubernetes.io/projected/08a08704-1234-4ea2-9c3c-af2d3dad8fda-kube-api-access-flcz6\") pod \"lws-controller-manager-7bcf4db656-jlzqb\" (UID: \"08a08704-1234-4ea2-9c3c-af2d3dad8fda\") " pod="openshift-lws-operator/lws-controller-manager-7bcf4db656-jlzqb" Jul 06 22:14:18.109624 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:18.109491 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/08a08704-1234-4ea2-9c3c-af2d3dad8fda-metrics-cert\") pod \"lws-controller-manager-7bcf4db656-jlzqb\" (UID: \"08a08704-1234-4ea2-9c3c-af2d3dad8fda\") " pod="openshift-lws-operator/lws-controller-manager-7bcf4db656-jlzqb" Jul 06 22:14:18.109624 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:18.109573 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/08a08704-1234-4ea2-9c3c-af2d3dad8fda-cert\") pod \"lws-controller-manager-7bcf4db656-jlzqb\" (UID: \"08a08704-1234-4ea2-9c3c-af2d3dad8fda\") " pod="openshift-lws-operator/lws-controller-manager-7bcf4db656-jlzqb" Jul 06 22:14:18.210839 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:18.210794 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/08a08704-1234-4ea2-9c3c-af2d3dad8fda-cert\") pod \"lws-controller-manager-7bcf4db656-jlzqb\" (UID: \"08a08704-1234-4ea2-9c3c-af2d3dad8fda\") " pod="openshift-lws-operator/lws-controller-manager-7bcf4db656-jlzqb" Jul 06 22:14:18.211048 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:18.210888 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/08a08704-1234-4ea2-9c3c-af2d3dad8fda-manager-config\") pod \"lws-controller-manager-7bcf4db656-jlzqb\" (UID: \"08a08704-1234-4ea2-9c3c-af2d3dad8fda\") " pod="openshift-lws-operator/lws-controller-manager-7bcf4db656-jlzqb" Jul 06 22:14:18.211048 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:18.210931 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flcz6\" (UniqueName: \"kubernetes.io/projected/08a08704-1234-4ea2-9c3c-af2d3dad8fda-kube-api-access-flcz6\") pod \"lws-controller-manager-7bcf4db656-jlzqb\" (UID: \"08a08704-1234-4ea2-9c3c-af2d3dad8fda\") " pod="openshift-lws-operator/lws-controller-manager-7bcf4db656-jlzqb" Jul 06 22:14:18.211048 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:18.210976 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/08a08704-1234-4ea2-9c3c-af2d3dad8fda-metrics-cert\") pod \"lws-controller-manager-7bcf4db656-jlzqb\" (UID: \"08a08704-1234-4ea2-9c3c-af2d3dad8fda\") " pod="openshift-lws-operator/lws-controller-manager-7bcf4db656-jlzqb" Jul 06 22:14:18.211652 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:18.211626 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/08a08704-1234-4ea2-9c3c-af2d3dad8fda-manager-config\") pod \"lws-controller-manager-7bcf4db656-jlzqb\" (UID: \"08a08704-1234-4ea2-9c3c-af2d3dad8fda\") " pod="openshift-lws-operator/lws-controller-manager-7bcf4db656-jlzqb" Jul 06 22:14:18.213312 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:18.213286 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/08a08704-1234-4ea2-9c3c-af2d3dad8fda-metrics-cert\") pod \"lws-controller-manager-7bcf4db656-jlzqb\" (UID: \"08a08704-1234-4ea2-9c3c-af2d3dad8fda\") " pod="openshift-lws-operator/lws-controller-manager-7bcf4db656-jlzqb" Jul 06 22:14:18.213401 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:18.213349 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/08a08704-1234-4ea2-9c3c-af2d3dad8fda-cert\") pod \"lws-controller-manager-7bcf4db656-jlzqb\" (UID: \"08a08704-1234-4ea2-9c3c-af2d3dad8fda\") " pod="openshift-lws-operator/lws-controller-manager-7bcf4db656-jlzqb" Jul 06 22:14:18.220664 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:18.220639 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flcz6\" (UniqueName: \"kubernetes.io/projected/08a08704-1234-4ea2-9c3c-af2d3dad8fda-kube-api-access-flcz6\") pod \"lws-controller-manager-7bcf4db656-jlzqb\" (UID: \"08a08704-1234-4ea2-9c3c-af2d3dad8fda\") " pod="openshift-lws-operator/lws-controller-manager-7bcf4db656-jlzqb" Jul 06 22:14:18.336805 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:18.336717 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7bcf4db656-jlzqb" Jul 06 22:14:18.469813 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:18.469782 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7bcf4db656-jlzqb"] Jul 06 22:14:18.472254 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:14:18.472225 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08a08704_1234_4ea2_9c3c_af2d3dad8fda.slice/crio-0098b30894045398fabdb0b5f49f2ab8bce1ce8853f6e5628fe0251cd6ac0af5 WatchSource:0}: Error finding container 0098b30894045398fabdb0b5f49f2ab8bce1ce8853f6e5628fe0251cd6ac0af5: Status 404 returned error can't find the container with id 0098b30894045398fabdb0b5f49f2ab8bce1ce8853f6e5628fe0251cd6ac0af5 Jul 06 22:14:18.474170 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:18.474154 2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 22:14:18.891346 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:18.891311 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7bcf4db656-jlzqb" event={"ID":"08a08704-1234-4ea2-9c3c-af2d3dad8fda","Type":"ContainerStarted","Data":"0098b30894045398fabdb0b5f49f2ab8bce1ce8853f6e5628fe0251cd6ac0af5"} Jul 06 22:14:21.906928 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:21.906892 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7bcf4db656-jlzqb" event={"ID":"08a08704-1234-4ea2-9c3c-af2d3dad8fda","Type":"ContainerStarted","Data":"8db5075012e12e0e7072fab23bdb5ba64529d272c7137a6d941184b2572a874e"} Jul 06 22:14:21.907328 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:21.906984 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7bcf4db656-jlzqb" Jul 06 22:14:21.930707 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:21.930651 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7bcf4db656-jlzqb" podStartSLOduration=1.4304464989999999 podStartE2EDuration="3.930635483s" totalImagesPullingTime="2.500188984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:14:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:14:21.929975391 +0000 UTC m=+313.229687241" watchObservedRunningTime="2026-07-06 22:14:21.930635483 +0000 UTC m=+313.230347401" Jul 06 22:14:23.468522 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:23.468489 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5c695b4674-75zsx"] Jul 06 22:14:23.472221 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:23.472198 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5c695b4674-75zsx" Jul 06 22:14:23.476398 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:23.476373 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 06 22:14:23.476645 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:23.476631 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 06 22:14:23.476757 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:23.476736 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-54jfp\"" Jul 06 22:14:23.477317 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:23.477301 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 06 22:14:23.479691 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:23.479674 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 06 22:14:23.491964 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:23.491930 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5c695b4674-75zsx"] Jul 06 22:14:23.563654 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:23.563618 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlpgn\" (UniqueName: \"kubernetes.io/projected/bc5a156b-600c-418e-98ca-4cb925001ce5-kube-api-access-xlpgn\") pod \"opendatahub-operator-controller-manager-5c695b4674-75zsx\" (UID: \"bc5a156b-600c-418e-98ca-4cb925001ce5\") " pod="opendatahub/opendatahub-operator-controller-manager-5c695b4674-75zsx" Jul 06 22:14:23.563866 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:23.563667 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc5a156b-600c-418e-98ca-4cb925001ce5-tmp\") pod \"opendatahub-operator-controller-manager-5c695b4674-75zsx\" (UID: \"bc5a156b-600c-418e-98ca-4cb925001ce5\") " pod="opendatahub/opendatahub-operator-controller-manager-5c695b4674-75zsx" Jul 06 22:14:23.563866 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:23.563797 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bc5a156b-600c-418e-98ca-4cb925001ce5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c695b4674-75zsx\" (UID: \"bc5a156b-600c-418e-98ca-4cb925001ce5\") " pod="opendatahub/opendatahub-operator-controller-manager-5c695b4674-75zsx" Jul 06 22:14:23.563866 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:23.563833 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bc5a156b-600c-418e-98ca-4cb925001ce5-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c695b4674-75zsx\" (UID: \"bc5a156b-600c-418e-98ca-4cb925001ce5\") " pod="opendatahub/opendatahub-operator-controller-manager-5c695b4674-75zsx" Jul 06 22:14:23.564046 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:23.563900 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bc5a156b-600c-418e-98ca-4cb925001ce5-manifests\") pod \"opendatahub-operator-controller-manager-5c695b4674-75zsx\" (UID: \"bc5a156b-600c-418e-98ca-4cb925001ce5\") " pod="opendatahub/opendatahub-operator-controller-manager-5c695b4674-75zsx" Jul 06 22:14:23.664377 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:23.664337 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc5a156b-600c-418e-98ca-4cb925001ce5-tmp\") pod \"opendatahub-operator-controller-manager-5c695b4674-75zsx\" (UID: \"bc5a156b-600c-418e-98ca-4cb925001ce5\") " pod="opendatahub/opendatahub-operator-controller-manager-5c695b4674-75zsx" Jul 06 22:14:23.664559 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:23.664414 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bc5a156b-600c-418e-98ca-4cb925001ce5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c695b4674-75zsx\" (UID: \"bc5a156b-600c-418e-98ca-4cb925001ce5\") " pod="opendatahub/opendatahub-operator-controller-manager-5c695b4674-75zsx" Jul 06 22:14:23.664559 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:23.664438 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bc5a156b-600c-418e-98ca-4cb925001ce5-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c695b4674-75zsx\" (UID: \"bc5a156b-600c-418e-98ca-4cb925001ce5\") " pod="opendatahub/opendatahub-operator-controller-manager-5c695b4674-75zsx" Jul 06 22:14:23.664559 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:23.664472 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bc5a156b-600c-418e-98ca-4cb925001ce5-manifests\") pod \"opendatahub-operator-controller-manager-5c695b4674-75zsx\" (UID: \"bc5a156b-600c-418e-98ca-4cb925001ce5\") " pod="opendatahub/opendatahub-operator-controller-manager-5c695b4674-75zsx" Jul 06 22:14:23.664716 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:23.664613 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlpgn\" (UniqueName: \"kubernetes.io/projected/bc5a156b-600c-418e-98ca-4cb925001ce5-kube-api-access-xlpgn\") pod \"opendatahub-operator-controller-manager-5c695b4674-75zsx\" (UID: \"bc5a156b-600c-418e-98ca-4cb925001ce5\") " pod="opendatahub/opendatahub-operator-controller-manager-5c695b4674-75zsx" Jul 06 22:14:23.664880 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:23.664860 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bc5a156b-600c-418e-98ca-4cb925001ce5-manifests\") pod \"opendatahub-operator-controller-manager-5c695b4674-75zsx\" (UID: \"bc5a156b-600c-418e-98ca-4cb925001ce5\") " pod="opendatahub/opendatahub-operator-controller-manager-5c695b4674-75zsx" Jul 06 22:14:23.666578 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:23.666547 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc5a156b-600c-418e-98ca-4cb925001ce5-tmp\") pod \"opendatahub-operator-controller-manager-5c695b4674-75zsx\" (UID: \"bc5a156b-600c-418e-98ca-4cb925001ce5\") " pod="opendatahub/opendatahub-operator-controller-manager-5c695b4674-75zsx" Jul 06 22:14:23.666812 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:23.666793 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bc5a156b-600c-418e-98ca-4cb925001ce5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c695b4674-75zsx\" (UID: \"bc5a156b-600c-418e-98ca-4cb925001ce5\") " pod="opendatahub/opendatahub-operator-controller-manager-5c695b4674-75zsx" Jul 06 22:14:23.666891 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:23.666873 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bc5a156b-600c-418e-98ca-4cb925001ce5-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c695b4674-75zsx\" (UID: \"bc5a156b-600c-418e-98ca-4cb925001ce5\") " pod="opendatahub/opendatahub-operator-controller-manager-5c695b4674-75zsx" Jul 06 22:14:23.675211 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:23.675185 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlpgn\" (UniqueName: \"kubernetes.io/projected/bc5a156b-600c-418e-98ca-4cb925001ce5-kube-api-access-xlpgn\") pod \"opendatahub-operator-controller-manager-5c695b4674-75zsx\" (UID: \"bc5a156b-600c-418e-98ca-4cb925001ce5\") " pod="opendatahub/opendatahub-operator-controller-manager-5c695b4674-75zsx" Jul 06 22:14:23.783038 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:23.782919 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5c695b4674-75zsx" Jul 06 22:14:23.927383 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:23.927194 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5c695b4674-75zsx"] Jul 06 22:14:23.930580 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:14:23.930550 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc5a156b_600c_418e_98ca_4cb925001ce5.slice/crio-d1dc4960d7727076e5e049b35f3e0c840333ac27f7bbd386318ade7472988e35 WatchSource:0}: Error finding container d1dc4960d7727076e5e049b35f3e0c840333ac27f7bbd386318ade7472988e35: Status 404 returned error can't find the container with id d1dc4960d7727076e5e049b35f3e0c840333ac27f7bbd386318ade7472988e35 Jul 06 22:14:24.923457 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:24.923418 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c695b4674-75zsx" event={"ID":"bc5a156b-600c-418e-98ca-4cb925001ce5","Type":"ContainerStarted","Data":"d1dc4960d7727076e5e049b35f3e0c840333ac27f7bbd386318ade7472988e35"} Jul 06 22:14:27.941185 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:27.941146 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c695b4674-75zsx" event={"ID":"bc5a156b-600c-418e-98ca-4cb925001ce5","Type":"ContainerStarted","Data":"0e4e8551cfdf3764998e8bbf7fa4f109e1f83afc9e6e4810b4aaa67347ae3b7e"} Jul 06 22:14:28.947345 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:28.947311 2555 generic.go:356] "Generic (PLEG): container finished" podID="bc5a156b-600c-418e-98ca-4cb925001ce5" containerID="0e4e8551cfdf3764998e8bbf7fa4f109e1f83afc9e6e4810b4aaa67347ae3b7e" exitCode=0 Jul 06 22:14:28.947794 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:28.947397 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c695b4674-75zsx" event={"ID":"bc5a156b-600c-418e-98ca-4cb925001ce5","Type":"ContainerDied","Data":"0e4e8551cfdf3764998e8bbf7fa4f109e1f83afc9e6e4810b4aaa67347ae3b7e"} Jul 06 22:14:29.953855 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:29.953821 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c695b4674-75zsx" event={"ID":"bc5a156b-600c-418e-98ca-4cb925001ce5","Type":"ContainerStarted","Data":"0bc99dcdffb93d17aafdc7f59d1044c4b7822e45caecc873883c276c32d2f887"} Jul 06 22:14:29.954358 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:29.953959 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5c695b4674-75zsx" Jul 06 22:14:29.976869 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:29.976810 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5c695b4674-75zsx" podStartSLOduration=2.454809146 podStartE2EDuration="6.976791654s" totalImagesPullingTime="3.495055092s" totalInitContainerRuntime="1.026927416s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:14:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:14:29.975985147 +0000 UTC m=+321.275697009" watchObservedRunningTime="2026-07-06 22:14:29.976791654 +0000 UTC m=+321.276503504" Jul 06 22:14:32.913536 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:32.913504 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7bcf4db656-jlzqb" Jul 06 22:14:40.960849 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:40.960819 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5c695b4674-75zsx" Jul 06 22:14:43.146731 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:43.146696 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-658774566b-l959z"] Jul 06 22:14:43.160775 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:43.160738 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-658774566b-l959z"] Jul 06 22:14:43.160937 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:43.160866 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-658774566b-l959z" Jul 06 22:14:43.164922 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:43.164900 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 06 22:14:43.165164 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:43.165078 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 06 22:14:43.165164 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:43.165136 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-rcbc4\"" Jul 06 22:14:43.165164 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:43.165150 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 06 22:14:43.165380 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:43.165094 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 06 22:14:43.253215 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:43.253180 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ca2aaf5-c87c-498e-aec9-da080ea88381-tls-certs\") pod \"kube-auth-proxy-658774566b-l959z\" (UID: \"2ca2aaf5-c87c-498e-aec9-da080ea88381\") " pod="openshift-ingress/kube-auth-proxy-658774566b-l959z" Jul 06 22:14:43.253215 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:43.253219 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2ca2aaf5-c87c-498e-aec9-da080ea88381-tmp\") pod \"kube-auth-proxy-658774566b-l959z\" (UID: \"2ca2aaf5-c87c-498e-aec9-da080ea88381\") " pod="openshift-ingress/kube-auth-proxy-658774566b-l959z" Jul 06 22:14:43.253435 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:43.253331 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvlzr\" (UniqueName: \"kubernetes.io/projected/2ca2aaf5-c87c-498e-aec9-da080ea88381-kube-api-access-tvlzr\") pod \"kube-auth-proxy-658774566b-l959z\" (UID: \"2ca2aaf5-c87c-498e-aec9-da080ea88381\") " pod="openshift-ingress/kube-auth-proxy-658774566b-l959z" Jul 06 22:14:43.354584 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:43.354545 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvlzr\" (UniqueName: \"kubernetes.io/projected/2ca2aaf5-c87c-498e-aec9-da080ea88381-kube-api-access-tvlzr\") pod \"kube-auth-proxy-658774566b-l959z\" (UID: \"2ca2aaf5-c87c-498e-aec9-da080ea88381\") " pod="openshift-ingress/kube-auth-proxy-658774566b-l959z" Jul 06 22:14:43.354778 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:43.354627 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ca2aaf5-c87c-498e-aec9-da080ea88381-tls-certs\") pod \"kube-auth-proxy-658774566b-l959z\" (UID: \"2ca2aaf5-c87c-498e-aec9-da080ea88381\") " pod="openshift-ingress/kube-auth-proxy-658774566b-l959z" Jul 06 22:14:43.354778 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:43.354652 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2ca2aaf5-c87c-498e-aec9-da080ea88381-tmp\") pod \"kube-auth-proxy-658774566b-l959z\" (UID: \"2ca2aaf5-c87c-498e-aec9-da080ea88381\") " pod="openshift-ingress/kube-auth-proxy-658774566b-l959z" Jul 06 22:14:43.356995 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:43.356970 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2ca2aaf5-c87c-498e-aec9-da080ea88381-tmp\") pod \"kube-auth-proxy-658774566b-l959z\" (UID: \"2ca2aaf5-c87c-498e-aec9-da080ea88381\") " pod="openshift-ingress/kube-auth-proxy-658774566b-l959z" Jul 06 22:14:43.357334 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:43.357312 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ca2aaf5-c87c-498e-aec9-da080ea88381-tls-certs\") pod \"kube-auth-proxy-658774566b-l959z\" (UID: \"2ca2aaf5-c87c-498e-aec9-da080ea88381\") " pod="openshift-ingress/kube-auth-proxy-658774566b-l959z" Jul 06 22:14:43.371965 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:43.371924 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvlzr\" (UniqueName: \"kubernetes.io/projected/2ca2aaf5-c87c-498e-aec9-da080ea88381-kube-api-access-tvlzr\") pod \"kube-auth-proxy-658774566b-l959z\" (UID: \"2ca2aaf5-c87c-498e-aec9-da080ea88381\") " pod="openshift-ingress/kube-auth-proxy-658774566b-l959z" Jul 06 22:14:43.471479 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:43.471386 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-658774566b-l959z" Jul 06 22:14:43.634574 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:43.634550 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-658774566b-l959z"] Jul 06 22:14:43.636890 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:14:43.636842 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ca2aaf5_c87c_498e_aec9_da080ea88381.slice/crio-d0a176db53e2ed212a50e2e23e7e528e2cf8ae01804549f3a8a0ab468e6af64c WatchSource:0}: Error finding container d0a176db53e2ed212a50e2e23e7e528e2cf8ae01804549f3a8a0ab468e6af64c: Status 404 returned error can't find the container with id d0a176db53e2ed212a50e2e23e7e528e2cf8ae01804549f3a8a0ab468e6af64c Jul 06 22:14:44.020984 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:44.020946 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-658774566b-l959z" event={"ID":"2ca2aaf5-c87c-498e-aec9-da080ea88381","Type":"ContainerStarted","Data":"d0a176db53e2ed212a50e2e23e7e528e2cf8ae01804549f3a8a0ab468e6af64c"} Jul 06 22:14:47.037687 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:47.037640 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-658774566b-l959z" event={"ID":"2ca2aaf5-c87c-498e-aec9-da080ea88381","Type":"ContainerStarted","Data":"e5772afe6d00a6598640c93bd5f6a7115f409a74ad4c24024dfeca2aab5e9b4b"} Jul 06 22:14:47.056923 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:14:47.056866 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-658774566b-l959z" podStartSLOduration=1.125626632 podStartE2EDuration="4.056849851s" totalImagesPullingTime="2.931223219s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:14:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:14:47.055536353 +0000 UTC m=+338.355248206" watchObservedRunningTime="2026-07-06 22:14:47.056849851 +0000 UTC m=+338.356561699" Jul 06 22:16:03.468322 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:03.468284 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-576896d655-4w5c6"] Jul 06 22:16:03.471728 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:03.471710 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-576896d655-4w5c6" Jul 06 22:16:03.475191 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:03.475154 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 06 22:16:03.475517 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:03.475494 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 06 22:16:03.476854 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:03.476376 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-wtp2d\"" Jul 06 22:16:03.476854 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:03.476435 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 06 22:16:03.476854 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:03.476515 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 06 22:16:03.476854 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:03.476680 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 06 22:16:03.476854 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:03.476734 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 06 22:16:03.477203 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:03.476909 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 06 22:16:03.484888 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:03.484859 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 06 22:16:03.485590 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:03.485569 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-576896d655-4w5c6"] Jul 06 22:16:03.587939 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:03.587896 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/27b7b97e-de46-420f-bfb2-edfe27d7301e-oauth-serving-cert\") pod \"console-576896d655-4w5c6\" (UID: \"27b7b97e-de46-420f-bfb2-edfe27d7301e\") " pod="openshift-console/console-576896d655-4w5c6" Jul 06 22:16:03.588158 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:03.588046 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27b7b97e-de46-420f-bfb2-edfe27d7301e-trusted-ca-bundle\") pod \"console-576896d655-4w5c6\" (UID: \"27b7b97e-de46-420f-bfb2-edfe27d7301e\") " pod="openshift-console/console-576896d655-4w5c6" Jul 06 22:16:03.588158 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:03.588084 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjwtv\" (UniqueName: \"kubernetes.io/projected/27b7b97e-de46-420f-bfb2-edfe27d7301e-kube-api-access-fjwtv\") pod \"console-576896d655-4w5c6\" (UID: \"27b7b97e-de46-420f-bfb2-edfe27d7301e\") " pod="openshift-console/console-576896d655-4w5c6" Jul 06 22:16:03.588158 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:03.588135 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/27b7b97e-de46-420f-bfb2-edfe27d7301e-console-serving-cert\") pod \"console-576896d655-4w5c6\" (UID: \"27b7b97e-de46-420f-bfb2-edfe27d7301e\") " pod="openshift-console/console-576896d655-4w5c6" Jul 06 22:16:03.588158 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:03.588159 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/27b7b97e-de46-420f-bfb2-edfe27d7301e-console-oauth-config\") pod \"console-576896d655-4w5c6\" (UID: \"27b7b97e-de46-420f-bfb2-edfe27d7301e\") " pod="openshift-console/console-576896d655-4w5c6" Jul 06 22:16:03.588329 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:03.588194 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/27b7b97e-de46-420f-bfb2-edfe27d7301e-console-config\") pod \"console-576896d655-4w5c6\" (UID: \"27b7b97e-de46-420f-bfb2-edfe27d7301e\") " pod="openshift-console/console-576896d655-4w5c6" Jul 06 22:16:03.588329 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:03.588212 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/27b7b97e-de46-420f-bfb2-edfe27d7301e-service-ca\") pod \"console-576896d655-4w5c6\" (UID: \"27b7b97e-de46-420f-bfb2-edfe27d7301e\") " pod="openshift-console/console-576896d655-4w5c6" Jul 06 22:16:03.688652 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:03.688609 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/27b7b97e-de46-420f-bfb2-edfe27d7301e-console-serving-cert\") pod \"console-576896d655-4w5c6\" (UID: \"27b7b97e-de46-420f-bfb2-edfe27d7301e\") " pod="openshift-console/console-576896d655-4w5c6" Jul 06 22:16:03.688859 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:03.688661 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/27b7b97e-de46-420f-bfb2-edfe27d7301e-console-oauth-config\") pod \"console-576896d655-4w5c6\" (UID: \"27b7b97e-de46-420f-bfb2-edfe27d7301e\") " pod="openshift-console/console-576896d655-4w5c6" Jul 06 22:16:03.688859 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:03.688698 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/27b7b97e-de46-420f-bfb2-edfe27d7301e-console-config\") pod \"console-576896d655-4w5c6\" (UID: \"27b7b97e-de46-420f-bfb2-edfe27d7301e\") " pod="openshift-console/console-576896d655-4w5c6" Jul 06 22:16:03.688859 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:03.688720 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/27b7b97e-de46-420f-bfb2-edfe27d7301e-service-ca\") pod \"console-576896d655-4w5c6\" (UID: \"27b7b97e-de46-420f-bfb2-edfe27d7301e\") " pod="openshift-console/console-576896d655-4w5c6" Jul 06 22:16:03.688859 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:03.688767 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/27b7b97e-de46-420f-bfb2-edfe27d7301e-oauth-serving-cert\") pod \"console-576896d655-4w5c6\" (UID: \"27b7b97e-de46-420f-bfb2-edfe27d7301e\") " pod="openshift-console/console-576896d655-4w5c6" Jul 06 22:16:03.688859 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:03.688855 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27b7b97e-de46-420f-bfb2-edfe27d7301e-trusted-ca-bundle\") pod \"console-576896d655-4w5c6\" (UID: \"27b7b97e-de46-420f-bfb2-edfe27d7301e\") " pod="openshift-console/console-576896d655-4w5c6" Jul 06 22:16:03.689153 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:03.688881 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjwtv\" (UniqueName: \"kubernetes.io/projected/27b7b97e-de46-420f-bfb2-edfe27d7301e-kube-api-access-fjwtv\") pod \"console-576896d655-4w5c6\" (UID: \"27b7b97e-de46-420f-bfb2-edfe27d7301e\") " pod="openshift-console/console-576896d655-4w5c6" Jul 06 22:16:03.689670 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:03.689642 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/27b7b97e-de46-420f-bfb2-edfe27d7301e-oauth-serving-cert\") pod \"console-576896d655-4w5c6\" (UID: \"27b7b97e-de46-420f-bfb2-edfe27d7301e\") " pod="openshift-console/console-576896d655-4w5c6" Jul 06 22:16:03.689760 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:03.689652 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/27b7b97e-de46-420f-bfb2-edfe27d7301e-service-ca\") pod \"console-576896d655-4w5c6\" (UID: \"27b7b97e-de46-420f-bfb2-edfe27d7301e\") " pod="openshift-console/console-576896d655-4w5c6" Jul 06 22:16:03.689760 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:03.689689 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/27b7b97e-de46-420f-bfb2-edfe27d7301e-console-config\") pod \"console-576896d655-4w5c6\" (UID: \"27b7b97e-de46-420f-bfb2-edfe27d7301e\") " pod="openshift-console/console-576896d655-4w5c6" Jul 06 22:16:03.689839 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:03.689777 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27b7b97e-de46-420f-bfb2-edfe27d7301e-trusted-ca-bundle\") pod \"console-576896d655-4w5c6\" (UID: \"27b7b97e-de46-420f-bfb2-edfe27d7301e\") " pod="openshift-console/console-576896d655-4w5c6" Jul 06 22:16:03.691203 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:03.691181 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/27b7b97e-de46-420f-bfb2-edfe27d7301e-console-oauth-config\") pod \"console-576896d655-4w5c6\" (UID: \"27b7b97e-de46-420f-bfb2-edfe27d7301e\") " pod="openshift-console/console-576896d655-4w5c6" Jul 06 22:16:03.691453 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:03.691434 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/27b7b97e-de46-420f-bfb2-edfe27d7301e-console-serving-cert\") pod \"console-576896d655-4w5c6\" (UID: \"27b7b97e-de46-420f-bfb2-edfe27d7301e\") " pod="openshift-console/console-576896d655-4w5c6" Jul 06 22:16:03.700635 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:03.700607 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjwtv\" (UniqueName: \"kubernetes.io/projected/27b7b97e-de46-420f-bfb2-edfe27d7301e-kube-api-access-fjwtv\") pod \"console-576896d655-4w5c6\" (UID: \"27b7b97e-de46-420f-bfb2-edfe27d7301e\") " pod="openshift-console/console-576896d655-4w5c6" Jul 06 22:16:03.784918 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:03.784835 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-576896d655-4w5c6" Jul 06 22:16:03.952113 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:03.952085 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-576896d655-4w5c6"] Jul 06 22:16:03.954075 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:16:03.954040 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27b7b97e_de46_420f_bfb2_edfe27d7301e.slice/crio-2ab3b73fb754c1be22751b32cc49f36ab8edc43fa768c07d88c3532b16cc7124 WatchSource:0}: Error finding container 2ab3b73fb754c1be22751b32cc49f36ab8edc43fa768c07d88c3532b16cc7124: Status 404 returned error can't find the container with id 2ab3b73fb754c1be22751b32cc49f36ab8edc43fa768c07d88c3532b16cc7124 Jul 06 22:16:04.404164 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:04.404131 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-576896d655-4w5c6" event={"ID":"27b7b97e-de46-420f-bfb2-edfe27d7301e","Type":"ContainerStarted","Data":"82c0db9ac4579daf21488e41c111a2299978034bf155926249d0a9f08f28e7e5"} Jul 06 22:16:04.404391 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:04.404176 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-576896d655-4w5c6" event={"ID":"27b7b97e-de46-420f-bfb2-edfe27d7301e","Type":"ContainerStarted","Data":"2ab3b73fb754c1be22751b32cc49f36ab8edc43fa768c07d88c3532b16cc7124"} Jul 06 22:16:04.425258 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:04.425200 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-576896d655-4w5c6" podStartSLOduration=1.425179736 podStartE2EDuration="1.425179736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:16:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:16:04.423553997 +0000 UTC m=+415.723265848" watchObservedRunningTime="2026-07-06 22:16:04.425179736 +0000 UTC m=+415.724891585" Jul 06 22:16:13.785174 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:13.785126 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-576896d655-4w5c6" Jul 06 22:16:13.785642 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:13.785199 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-576896d655-4w5c6" Jul 06 22:16:13.790587 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:13.790555 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-576896d655-4w5c6" Jul 06 22:16:14.461312 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:14.461286 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-576896d655-4w5c6" Jul 06 22:16:35.619755 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:35.619676 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-rgzxd"] Jul 06 22:16:35.623327 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:35.623310 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rgzxd" Jul 06 22:16:35.626506 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:35.626486 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 06 22:16:35.627663 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:35.627648 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 06 22:16:35.627776 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:35.627696 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-ppdw5\"" Jul 06 22:16:35.627776 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:35.627736 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 06 22:16:35.627891 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:35.627744 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 06 22:16:35.633040 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:35.633018 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-rgzxd"] Jul 06 22:16:35.685746 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:35.685706 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8e03dcbd-aef7-452e-a850-6331b00df90b-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-rgzxd\" (UID: \"8e03dcbd-aef7-452e-a850-6331b00df90b\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rgzxd" Jul 06 22:16:35.685746 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:35.685744 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8e03dcbd-aef7-452e-a850-6331b00df90b-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-rgzxd\" (UID: \"8e03dcbd-aef7-452e-a850-6331b00df90b\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rgzxd" Jul 06 22:16:35.685970 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:35.685777 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhs26\" (UniqueName: \"kubernetes.io/projected/8e03dcbd-aef7-452e-a850-6331b00df90b-kube-api-access-dhs26\") pod \"kuadrant-console-plugin-66b5774985-rgzxd\" (UID: \"8e03dcbd-aef7-452e-a850-6331b00df90b\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rgzxd" Jul 06 22:16:35.786767 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:35.786733 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhs26\" (UniqueName: \"kubernetes.io/projected/8e03dcbd-aef7-452e-a850-6331b00df90b-kube-api-access-dhs26\") pod \"kuadrant-console-plugin-66b5774985-rgzxd\" (UID: \"8e03dcbd-aef7-452e-a850-6331b00df90b\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rgzxd" Jul 06 22:16:35.786957 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:35.786856 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8e03dcbd-aef7-452e-a850-6331b00df90b-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-rgzxd\" (UID: \"8e03dcbd-aef7-452e-a850-6331b00df90b\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rgzxd" Jul 06 22:16:35.786957 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:35.786885 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8e03dcbd-aef7-452e-a850-6331b00df90b-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-rgzxd\" (UID: \"8e03dcbd-aef7-452e-a850-6331b00df90b\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rgzxd" Jul 06 22:16:35.787097 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:16:35.786997 2555 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 06 22:16:35.787097 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:16:35.787095 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e03dcbd-aef7-452e-a850-6331b00df90b-plugin-serving-cert podName:8e03dcbd-aef7-452e-a850-6331b00df90b nodeName:}" failed. No retries permitted until 2026-07-06 22:16:36.287077455 +0000 UTC m=+447.586789283 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/8e03dcbd-aef7-452e-a850-6331b00df90b-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-rgzxd" (UID: "8e03dcbd-aef7-452e-a850-6331b00df90b") : secret "plugin-serving-cert" not found Jul 06 22:16:35.787563 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:35.787547 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8e03dcbd-aef7-452e-a850-6331b00df90b-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-rgzxd\" (UID: \"8e03dcbd-aef7-452e-a850-6331b00df90b\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rgzxd" Jul 06 22:16:35.802975 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:35.802948 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhs26\" (UniqueName: \"kubernetes.io/projected/8e03dcbd-aef7-452e-a850-6331b00df90b-kube-api-access-dhs26\") pod \"kuadrant-console-plugin-66b5774985-rgzxd\" (UID: \"8e03dcbd-aef7-452e-a850-6331b00df90b\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rgzxd" Jul 06 22:16:36.292221 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:36.292182 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8e03dcbd-aef7-452e-a850-6331b00df90b-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-rgzxd\" (UID: \"8e03dcbd-aef7-452e-a850-6331b00df90b\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rgzxd" Jul 06 22:16:36.294606 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:36.294582 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8e03dcbd-aef7-452e-a850-6331b00df90b-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-rgzxd\" (UID: \"8e03dcbd-aef7-452e-a850-6331b00df90b\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rgzxd" Jul 06 22:16:36.533239 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:36.533205 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rgzxd" Jul 06 22:16:36.664546 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:36.664518 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-rgzxd"] Jul 06 22:16:36.666367 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:16:36.666343 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e03dcbd_aef7_452e_a850_6331b00df90b.slice/crio-37d5567d3fb7f0d12fcb9b4d04045a6ca2aee785d4bc824af2ac7d577128c7b7 WatchSource:0}: Error finding container 37d5567d3fb7f0d12fcb9b4d04045a6ca2aee785d4bc824af2ac7d577128c7b7: Status 404 returned error can't find the container with id 37d5567d3fb7f0d12fcb9b4d04045a6ca2aee785d4bc824af2ac7d577128c7b7 Jul 06 22:16:37.569799 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:37.569766 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rgzxd" event={"ID":"8e03dcbd-aef7-452e-a850-6331b00df90b","Type":"ContainerStarted","Data":"37d5567d3fb7f0d12fcb9b4d04045a6ca2aee785d4bc824af2ac7d577128c7b7"} Jul 06 22:16:45.987435 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:45.987393 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jwqrx"] Jul 06 22:16:45.991379 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:45.991351 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jwqrx" Jul 06 22:16:45.994420 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:45.994390 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-tdbf8\"" Jul 06 22:16:46.005058 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:46.004976 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jwqrx"] Jul 06 22:16:46.092382 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:46.092350 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b87nw\" (UniqueName: \"kubernetes.io/projected/4df8a97d-bc32-4790-b551-e97c65523756-kube-api-access-b87nw\") pod \"limitador-operator-controller-manager-859d7b78bd-jwqrx\" (UID: \"4df8a97d-bc32-4790-b551-e97c65523756\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jwqrx" Jul 06 22:16:46.192970 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:46.192913 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b87nw\" (UniqueName: \"kubernetes.io/projected/4df8a97d-bc32-4790-b551-e97c65523756-kube-api-access-b87nw\") pod \"limitador-operator-controller-manager-859d7b78bd-jwqrx\" (UID: \"4df8a97d-bc32-4790-b551-e97c65523756\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jwqrx" Jul 06 22:16:46.207669 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:46.207630 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b87nw\" (UniqueName: \"kubernetes.io/projected/4df8a97d-bc32-4790-b551-e97c65523756-kube-api-access-b87nw\") pod \"limitador-operator-controller-manager-859d7b78bd-jwqrx\" (UID: \"4df8a97d-bc32-4790-b551-e97c65523756\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jwqrx" Jul 06 22:16:46.305963 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:46.305866 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jwqrx" Jul 06 22:16:46.466208 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:46.464196 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jwqrx"] Jul 06 22:16:46.469081 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:16:46.469050 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4df8a97d_bc32_4790_b551_e97c65523756.slice/crio-116c983a72a5f4fc19b5efe2a9db4d476c6f547058c4e4524c544da1705c4769 WatchSource:0}: Error finding container 116c983a72a5f4fc19b5efe2a9db4d476c6f547058c4e4524c544da1705c4769: Status 404 returned error can't find the container with id 116c983a72a5f4fc19b5efe2a9db4d476c6f547058c4e4524c544da1705c4769 Jul 06 22:16:46.624402 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:16:46.624300 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jwqrx" event={"ID":"4df8a97d-bc32-4790-b551-e97c65523756","Type":"ContainerStarted","Data":"116c983a72a5f4fc19b5efe2a9db4d476c6f547058c4e4524c544da1705c4769"} Jul 06 22:17:02.714424 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:17:02.714378 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rgzxd" event={"ID":"8e03dcbd-aef7-452e-a850-6331b00df90b","Type":"ContainerStarted","Data":"dd8eccb7dae489a3270ba72887b00e345d9ce80730c9f344738b777669fbe276"} Jul 06 22:17:02.745823 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:17:02.745763 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rgzxd" podStartSLOduration=2.3419710990000002 podStartE2EDuration="27.745745704s" totalImagesPullingTime="25.403774605s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:16:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:17:02.743493149 +0000 UTC m=+474.043204999" watchObservedRunningTime="2026-07-06 22:17:02.745745704 +0000 UTC m=+474.045457553" Jul 06 22:17:05.731323 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:17:05.731282 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jwqrx" event={"ID":"4df8a97d-bc32-4790-b551-e97c65523756","Type":"ContainerStarted","Data":"8937a90e1d640c339cb4563f189db62346a02df6639a39a62cf91eb27d11d6db"} Jul 06 22:17:05.731755 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:17:05.731542 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jwqrx" Jul 06 22:17:05.750907 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:17:05.750849 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jwqrx" podStartSLOduration=2.3790791860000002 podStartE2EDuration="20.750829069s" totalImagesPullingTime="18.371749883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:17:05.749973418 +0000 UTC m=+477.049685269" watchObservedRunningTime="2026-07-06 22:17:05.750829069 +0000 UTC m=+477.050540920" Jul 06 22:17:16.738086 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:17:16.738053 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jwqrx" Jul 06 22:17:32.255881 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:17:32.255844 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-9lskn"] Jul 06 22:17:32.260016 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:17:32.259985 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-9lskn" Jul 06 22:17:32.262725 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:17:32.262699 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-sl6cf\"" Jul 06 22:17:32.268292 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:17:32.268270 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-9lskn"] Jul 06 22:17:32.443699 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:17:32.443655 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2vlt\" (UniqueName: \"kubernetes.io/projected/0519a006-1609-4619-9241-c51a40fb407b-kube-api-access-b2vlt\") pod \"authorino-75d556bd77-9lskn\" (UID: \"0519a006-1609-4619-9241-c51a40fb407b\") " pod="kuadrant-system/authorino-75d556bd77-9lskn" Jul 06 22:17:32.544283 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:17:32.544191 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2vlt\" (UniqueName: \"kubernetes.io/projected/0519a006-1609-4619-9241-c51a40fb407b-kube-api-access-b2vlt\") pod \"authorino-75d556bd77-9lskn\" (UID: \"0519a006-1609-4619-9241-c51a40fb407b\") " pod="kuadrant-system/authorino-75d556bd77-9lskn" Jul 06 22:17:32.570137 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:17:32.570111 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2vlt\" (UniqueName: \"kubernetes.io/projected/0519a006-1609-4619-9241-c51a40fb407b-kube-api-access-b2vlt\") pod \"authorino-75d556bd77-9lskn\" (UID: \"0519a006-1609-4619-9241-c51a40fb407b\") " pod="kuadrant-system/authorino-75d556bd77-9lskn" Jul 06 22:17:32.869674 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:17:32.869581 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-9lskn" Jul 06 22:17:33.005102 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:17:33.005069 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-9lskn"] Jul 06 22:17:33.015347 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:17:33.015320 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0519a006_1609_4619_9241_c51a40fb407b.slice/crio-36ecac35f7bf88a6aa3f1f4ae1c8d3b46ae16894a25b5508b0a43fa354f90b96 WatchSource:0}: Error finding container 36ecac35f7bf88a6aa3f1f4ae1c8d3b46ae16894a25b5508b0a43fa354f90b96: Status 404 returned error can't find the container with id 36ecac35f7bf88a6aa3f1f4ae1c8d3b46ae16894a25b5508b0a43fa354f90b96 Jul 06 22:17:33.870216 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:17:33.870147 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-9lskn" event={"ID":"0519a006-1609-4619-9241-c51a40fb407b","Type":"ContainerStarted","Data":"36ecac35f7bf88a6aa3f1f4ae1c8d3b46ae16894a25b5508b0a43fa354f90b96"} Jul 06 22:17:36.888030 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:17:36.887969 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-9lskn" event={"ID":"0519a006-1609-4619-9241-c51a40fb407b","Type":"ContainerStarted","Data":"ab8f5b1c2c53cfc751049482ce28fe8708c330cc142e189c9a2ddf897f1e925c"} Jul 06 22:17:36.909640 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:17:36.909579 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-9lskn" podStartSLOduration=1.701151399 podStartE2EDuration="4.909563898s" totalImagesPullingTime="3.208412499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:17:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:17:36.908437436 +0000 UTC m=+508.208149296" watchObservedRunningTime="2026-07-06 22:17:36.909563898 +0000 UTC m=+508.209275784" Jul 06 22:18:04.769447 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:18:04.769362 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-9lskn"] Jul 06 22:18:04.769919 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:18:04.769546 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-9lskn" podUID="0519a006-1609-4619-9241-c51a40fb407b" containerName="authorino" containerID="cri-o://ab8f5b1c2c53cfc751049482ce28fe8708c330cc142e189c9a2ddf897f1e925c" gracePeriod=30 Jul 06 22:18:05.020328 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:18:05.020250 2555 generic.go:356] "Generic (PLEG): container finished" podID="0519a006-1609-4619-9241-c51a40fb407b" containerID="ab8f5b1c2c53cfc751049482ce28fe8708c330cc142e189c9a2ddf897f1e925c" exitCode=0 Jul 06 22:18:05.020457 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:18:05.020325 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-9lskn" event={"ID":"0519a006-1609-4619-9241-c51a40fb407b","Type":"ContainerDied","Data":"ab8f5b1c2c53cfc751049482ce28fe8708c330cc142e189c9a2ddf897f1e925c"} Jul 06 22:18:05.020457 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:18:05.020376 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-9lskn" event={"ID":"0519a006-1609-4619-9241-c51a40fb407b","Type":"ContainerDied","Data":"36ecac35f7bf88a6aa3f1f4ae1c8d3b46ae16894a25b5508b0a43fa354f90b96"} Jul 06 22:18:05.020457 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:18:05.020393 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36ecac35f7bf88a6aa3f1f4ae1c8d3b46ae16894a25b5508b0a43fa354f90b96" Jul 06 22:18:05.024882 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:18:05.024857 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-9lskn" Jul 06 22:18:05.150823 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:18:05.150783 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b2vlt\" (UniqueName: \"kubernetes.io/projected/0519a006-1609-4619-9241-c51a40fb407b-kube-api-access-b2vlt\") pod \"0519a006-1609-4619-9241-c51a40fb407b\" (UID: \"0519a006-1609-4619-9241-c51a40fb407b\") " Jul 06 22:18:05.153064 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:18:05.153034 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0519a006-1609-4619-9241-c51a40fb407b-kube-api-access-b2vlt" (OuterVolumeSpecName: "kube-api-access-b2vlt") pod "0519a006-1609-4619-9241-c51a40fb407b" (UID: "0519a006-1609-4619-9241-c51a40fb407b"). InnerVolumeSpecName "kube-api-access-b2vlt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:18:05.252462 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:18:05.252418 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2vlt\" (UniqueName: \"kubernetes.io/projected/0519a006-1609-4619-9241-c51a40fb407b-kube-api-access-b2vlt\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:18:06.026053 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:18:06.026020 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-9lskn" Jul 06 22:18:06.042817 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:18:06.042776 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-9lskn"] Jul 06 22:18:06.046579 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:18:06.046552 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-9lskn"] Jul 06 22:18:07.341700 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:18:07.341668 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0519a006-1609-4619-9241-c51a40fb407b" path="/var/lib/kubelet/pods/0519a006-1609-4619-9241-c51a40fb407b/volumes" Jul 06 22:18:45.571852 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:18:45.571812 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-6c74cd8d45-rdzxd"] Jul 06 22:18:45.572367 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:18:45.572241 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0519a006-1609-4619-9241-c51a40fb407b" containerName="authorino" Jul 06 22:18:45.572367 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:18:45.572257 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0519a006-1609-4619-9241-c51a40fb407b" containerName="authorino" Jul 06 22:18:45.572367 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:18:45.572313 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="0519a006-1609-4619-9241-c51a40fb407b" containerName="authorino" Jul 06 22:18:45.574487 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:18:45.574463 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-6c74cd8d45-rdzxd" Jul 06 22:18:45.577272 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:18:45.577248 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-serving-cert\"" Jul 06 22:18:45.577445 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:18:45.577423 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-kw98h\"" Jul 06 22:18:45.586283 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:18:45.586252 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-6c74cd8d45-rdzxd"] Jul 06 22:18:45.713066 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:18:45.712995 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13ecb710-e391-4cd3-85a9-e508b11d6dc8-maas-api-tls\") pod \"maas-api-6c74cd8d45-rdzxd\" (UID: \"13ecb710-e391-4cd3-85a9-e508b11d6dc8\") " pod="opendatahub/maas-api-6c74cd8d45-rdzxd" Jul 06 22:18:45.713066 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:18:45.713069 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkvcp\" (UniqueName: \"kubernetes.io/projected/13ecb710-e391-4cd3-85a9-e508b11d6dc8-kube-api-access-pkvcp\") pod \"maas-api-6c74cd8d45-rdzxd\" (UID: \"13ecb710-e391-4cd3-85a9-e508b11d6dc8\") " pod="opendatahub/maas-api-6c74cd8d45-rdzxd" Jul 06 22:18:45.713281 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:18:45.713196 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13ecb710-e391-4cd3-85a9-e508b11d6dc8-openshift-service-ca\") pod \"maas-api-6c74cd8d45-rdzxd\" (UID: \"13ecb710-e391-4cd3-85a9-e508b11d6dc8\") " pod="opendatahub/maas-api-6c74cd8d45-rdzxd" Jul 06 22:18:45.814106 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:18:45.814072 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13ecb710-e391-4cd3-85a9-e508b11d6dc8-openshift-service-ca\") pod \"maas-api-6c74cd8d45-rdzxd\" (UID: \"13ecb710-e391-4cd3-85a9-e508b11d6dc8\") " pod="opendatahub/maas-api-6c74cd8d45-rdzxd" Jul 06 22:18:45.814302 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:18:45.814128 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13ecb710-e391-4cd3-85a9-e508b11d6dc8-maas-api-tls\") pod \"maas-api-6c74cd8d45-rdzxd\" (UID: \"13ecb710-e391-4cd3-85a9-e508b11d6dc8\") " pod="opendatahub/maas-api-6c74cd8d45-rdzxd" Jul 06 22:18:45.814302 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:18:45.814249 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkvcp\" (UniqueName: \"kubernetes.io/projected/13ecb710-e391-4cd3-85a9-e508b11d6dc8-kube-api-access-pkvcp\") pod \"maas-api-6c74cd8d45-rdzxd\" (UID: \"13ecb710-e391-4cd3-85a9-e508b11d6dc8\") " pod="opendatahub/maas-api-6c74cd8d45-rdzxd" Jul 06 22:18:45.814781 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:18:45.814761 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13ecb710-e391-4cd3-85a9-e508b11d6dc8-openshift-service-ca\") pod \"maas-api-6c74cd8d45-rdzxd\" (UID: \"13ecb710-e391-4cd3-85a9-e508b11d6dc8\") " pod="opendatahub/maas-api-6c74cd8d45-rdzxd" Jul 06 22:18:45.816702 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:18:45.816677 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13ecb710-e391-4cd3-85a9-e508b11d6dc8-maas-api-tls\") pod \"maas-api-6c74cd8d45-rdzxd\" (UID: \"13ecb710-e391-4cd3-85a9-e508b11d6dc8\") " pod="opendatahub/maas-api-6c74cd8d45-rdzxd" Jul 06 22:18:45.824561 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:18:45.824488 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkvcp\" (UniqueName: \"kubernetes.io/projected/13ecb710-e391-4cd3-85a9-e508b11d6dc8-kube-api-access-pkvcp\") pod \"maas-api-6c74cd8d45-rdzxd\" (UID: \"13ecb710-e391-4cd3-85a9-e508b11d6dc8\") " pod="opendatahub/maas-api-6c74cd8d45-rdzxd" Jul 06 22:18:45.885647 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:18:45.885597 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-6c74cd8d45-rdzxd" Jul 06 22:18:46.032793 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:18:46.032769 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-6c74cd8d45-rdzxd"] Jul 06 22:18:46.035167 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:18:46.035139 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13ecb710_e391_4cd3_85a9_e508b11d6dc8.slice/crio-96d45f1c9733fabff0bf9f7ddcf96cfb2873ae1dee559b87555afeada5f44218 WatchSource:0}: Error finding container 96d45f1c9733fabff0bf9f7ddcf96cfb2873ae1dee559b87555afeada5f44218: Status 404 returned error can't find the container with id 96d45f1c9733fabff0bf9f7ddcf96cfb2873ae1dee559b87555afeada5f44218 Jul 06 22:18:46.214888 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:18:46.214843 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-6c74cd8d45-rdzxd" event={"ID":"13ecb710-e391-4cd3-85a9-e508b11d6dc8","Type":"ContainerStarted","Data":"96d45f1c9733fabff0bf9f7ddcf96cfb2873ae1dee559b87555afeada5f44218"} Jul 06 22:18:48.229080 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:18:48.229031 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-6c74cd8d45-rdzxd" event={"ID":"13ecb710-e391-4cd3-85a9-e508b11d6dc8","Type":"ContainerStarted","Data":"30c3bb29e3dc22ed6080f3f5408351e014607e17eb1fe17b0b44bc99aade4808"} Jul 06 22:18:48.229665 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:18:48.229191 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-6c74cd8d45-rdzxd" Jul 06 22:18:48.249253 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:18:48.249147 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-6c74cd8d45-rdzxd" podStartSLOduration=1.290454606 podStartE2EDuration="3.249123257s" totalImagesPullingTime="1.958668651s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:18:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:18:48.248290339 +0000 UTC m=+579.548002191" watchObservedRunningTime="2026-07-06 22:18:48.249123257 +0000 UTC m=+579.548835107" Jul 06 22:18:54.240022 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:18:54.239988 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-6c74cd8d45-rdzxd" Jul 06 22:19:09.178526 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:19:09.178492 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7mq8_e50d3b5f-832e-481f-8807-7774ff8d5521/ovn-acl-logging/0.log" Jul 06 22:19:09.183340 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:19:09.183307 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7mq8_e50d3b5f-832e-481f-8807-7774ff8d5521/ovn-acl-logging/0.log" Jul 06 22:19:22.093357 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:19:22.093315 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-jxsc9"] Jul 06 22:19:22.096950 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:19:22.096930 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jxsc9" Jul 06 22:19:22.100939 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:19:22.100905 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 06 22:19:22.100939 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:19:22.100934 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-kjlnd\"" Jul 06 22:19:22.101161 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:19:22.100934 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 06 22:19:22.101161 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:19:22.100924 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 06 22:19:22.106674 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:19:22.106648 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-jxsc9"] Jul 06 22:19:22.155511 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:19:22.155480 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/522a53e6-aca2-4f21-8756-961e2a4d3c09-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jxsc9\" (UID: \"522a53e6-aca2-4f21-8756-961e2a4d3c09\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jxsc9" Jul 06 22:19:22.155676 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:19:22.155525 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkd8w\" (UniqueName: \"kubernetes.io/projected/522a53e6-aca2-4f21-8756-961e2a4d3c09-kube-api-access-gkd8w\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jxsc9\" (UID: \"522a53e6-aca2-4f21-8756-961e2a4d3c09\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jxsc9" Jul 06 22:19:22.155676 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:19:22.155622 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/522a53e6-aca2-4f21-8756-961e2a4d3c09-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jxsc9\" (UID: \"522a53e6-aca2-4f21-8756-961e2a4d3c09\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jxsc9" Jul 06 22:19:22.155676 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:19:22.155649 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/522a53e6-aca2-4f21-8756-961e2a4d3c09-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jxsc9\" (UID: \"522a53e6-aca2-4f21-8756-961e2a4d3c09\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jxsc9" Jul 06 22:19:22.155676 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:19:22.155675 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/522a53e6-aca2-4f21-8756-961e2a4d3c09-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jxsc9\" (UID: \"522a53e6-aca2-4f21-8756-961e2a4d3c09\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jxsc9" Jul 06 22:19:22.155846 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:19:22.155715 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/522a53e6-aca2-4f21-8756-961e2a4d3c09-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jxsc9\" (UID: \"522a53e6-aca2-4f21-8756-961e2a4d3c09\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jxsc9" Jul 06 22:19:22.256861 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:19:22.256814 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/522a53e6-aca2-4f21-8756-961e2a4d3c09-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jxsc9\" (UID: \"522a53e6-aca2-4f21-8756-961e2a4d3c09\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jxsc9" Jul 06 22:19:22.257096 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:19:22.256881 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/522a53e6-aca2-4f21-8756-961e2a4d3c09-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jxsc9\" (UID: \"522a53e6-aca2-4f21-8756-961e2a4d3c09\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jxsc9" Jul 06 22:19:22.257096 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:19:22.256919 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkd8w\" (UniqueName: \"kubernetes.io/projected/522a53e6-aca2-4f21-8756-961e2a4d3c09-kube-api-access-gkd8w\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jxsc9\" (UID: \"522a53e6-aca2-4f21-8756-961e2a4d3c09\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jxsc9" Jul 06 22:19:22.257096 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:19:22.256997 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/522a53e6-aca2-4f21-8756-961e2a4d3c09-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jxsc9\" (UID: \"522a53e6-aca2-4f21-8756-961e2a4d3c09\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jxsc9" Jul 06 22:19:22.257096 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:19:22.257051 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/522a53e6-aca2-4f21-8756-961e2a4d3c09-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jxsc9\" (UID: \"522a53e6-aca2-4f21-8756-961e2a4d3c09\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jxsc9" Jul 06 22:19:22.257096 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:19:22.257084 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/522a53e6-aca2-4f21-8756-961e2a4d3c09-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jxsc9\" (UID: \"522a53e6-aca2-4f21-8756-961e2a4d3c09\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jxsc9" Jul 06 22:19:22.257348 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:19:22.257317 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/522a53e6-aca2-4f21-8756-961e2a4d3c09-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jxsc9\" (UID: \"522a53e6-aca2-4f21-8756-961e2a4d3c09\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jxsc9" Jul 06 22:19:22.257406 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:19:22.257345 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/522a53e6-aca2-4f21-8756-961e2a4d3c09-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jxsc9\" (UID: \"522a53e6-aca2-4f21-8756-961e2a4d3c09\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jxsc9" Jul 06 22:19:22.257406 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:19:22.257378 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/522a53e6-aca2-4f21-8756-961e2a4d3c09-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jxsc9\" (UID: \"522a53e6-aca2-4f21-8756-961e2a4d3c09\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jxsc9" Jul 06 22:19:22.259283 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:19:22.259258 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/522a53e6-aca2-4f21-8756-961e2a4d3c09-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jxsc9\" (UID: \"522a53e6-aca2-4f21-8756-961e2a4d3c09\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jxsc9" Jul 06 22:19:22.259444 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:19:22.259428 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/522a53e6-aca2-4f21-8756-961e2a4d3c09-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jxsc9\" (UID: \"522a53e6-aca2-4f21-8756-961e2a4d3c09\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jxsc9" Jul 06 22:19:22.269552 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:19:22.269527 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkd8w\" (UniqueName: \"kubernetes.io/projected/522a53e6-aca2-4f21-8756-961e2a4d3c09-kube-api-access-gkd8w\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jxsc9\" (UID: \"522a53e6-aca2-4f21-8756-961e2a4d3c09\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jxsc9" Jul 06 22:19:22.408325 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:19:22.408288 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jxsc9" Jul 06 22:19:22.545140 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:19:22.545082 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-jxsc9"] Jul 06 22:19:22.547439 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:19:22.547412 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod522a53e6_aca2_4f21_8756_961e2a4d3c09.slice/crio-6f6631824839a4e1b9acce6820054b839f5108decb136749a538cac11baccefe WatchSource:0}: Error finding container 6f6631824839a4e1b9acce6820054b839f5108decb136749a538cac11baccefe: Status 404 returned error can't find the container with id 6f6631824839a4e1b9acce6820054b839f5108decb136749a538cac11baccefe Jul 06 22:19:22.548719 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:19:22.548699 2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 22:19:23.401031 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:19:23.400964 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jxsc9" event={"ID":"522a53e6-aca2-4f21-8756-961e2a4d3c09","Type":"ContainerStarted","Data":"6f6631824839a4e1b9acce6820054b839f5108decb136749a538cac11baccefe"} Jul 06 22:19:24.408871 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:19:24.408826 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jxsc9" event={"ID":"522a53e6-aca2-4f21-8756-961e2a4d3c09","Type":"ContainerStarted","Data":"1176f1f1bfe91d88ea6f2e64bf288896fcea4d07de2b533af62cb67b4a551349"} Jul 06 22:19:24.427978 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:19:24.427932 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jxsc9" podStartSLOduration=1.424639569 podStartE2EDuration="2.427915908s" totalImagesPullingTime="1.003276339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:19:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:19:24.427476499 +0000 UTC m=+615.727188349" watchObservedRunningTime="2026-07-06 22:19:24.427915908 +0000 UTC m=+615.727627756" Jul 06 22:19:32.409210 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:19:32.409104 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jxsc9" Jul 06 22:19:32.409210 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:19:32.409193 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jxsc9" Jul 06 22:19:32.421782 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:19:32.421750 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jxsc9" Jul 06 22:19:32.459135 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:19:32.459109 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jxsc9" Jul 06 22:21:17.151163 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:21:17.151091 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-79b4c9c9b9-7kzcr"] Jul 06 22:21:17.154531 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:21:17.154509 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79b4c9c9b9-7kzcr" Jul 06 22:21:17.158736 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:21:17.158713 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-sl6cf\"" Jul 06 22:21:17.158832 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:21:17.158741 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 06 22:21:17.164395 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:21:17.164370 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-79b4c9c9b9-7kzcr"] Jul 06 22:21:17.241084 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:21:17.241043 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vjkw\" (UniqueName: \"kubernetes.io/projected/787cb0c4-927b-4829-8813-b17cbbc6933d-kube-api-access-8vjkw\") pod \"authorino-79b4c9c9b9-7kzcr\" (UID: \"787cb0c4-927b-4829-8813-b17cbbc6933d\") " pod="kuadrant-system/authorino-79b4c9c9b9-7kzcr" Jul 06 22:21:17.241084 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:21:17.241085 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/787cb0c4-927b-4829-8813-b17cbbc6933d-tls-cert\") pod \"authorino-79b4c9c9b9-7kzcr\" (UID: \"787cb0c4-927b-4829-8813-b17cbbc6933d\") " pod="kuadrant-system/authorino-79b4c9c9b9-7kzcr" Jul 06 22:21:17.342132 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:21:17.342090 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vjkw\" (UniqueName: \"kubernetes.io/projected/787cb0c4-927b-4829-8813-b17cbbc6933d-kube-api-access-8vjkw\") pod \"authorino-79b4c9c9b9-7kzcr\" (UID: \"787cb0c4-927b-4829-8813-b17cbbc6933d\") " pod="kuadrant-system/authorino-79b4c9c9b9-7kzcr" Jul 06 22:21:17.342132 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:21:17.342126 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/787cb0c4-927b-4829-8813-b17cbbc6933d-tls-cert\") pod \"authorino-79b4c9c9b9-7kzcr\" (UID: \"787cb0c4-927b-4829-8813-b17cbbc6933d\") " pod="kuadrant-system/authorino-79b4c9c9b9-7kzcr" Jul 06 22:21:17.344493 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:21:17.344464 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/787cb0c4-927b-4829-8813-b17cbbc6933d-tls-cert\") pod \"authorino-79b4c9c9b9-7kzcr\" (UID: \"787cb0c4-927b-4829-8813-b17cbbc6933d\") " pod="kuadrant-system/authorino-79b4c9c9b9-7kzcr" Jul 06 22:21:17.351086 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:21:17.351062 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vjkw\" (UniqueName: \"kubernetes.io/projected/787cb0c4-927b-4829-8813-b17cbbc6933d-kube-api-access-8vjkw\") pod \"authorino-79b4c9c9b9-7kzcr\" (UID: \"787cb0c4-927b-4829-8813-b17cbbc6933d\") " pod="kuadrant-system/authorino-79b4c9c9b9-7kzcr" Jul 06 22:21:17.464664 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:21:17.464576 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79b4c9c9b9-7kzcr" Jul 06 22:21:17.615530 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:21:17.615503 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-79b4c9c9b9-7kzcr"] Jul 06 22:21:17.618271 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:21:17.618242 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod787cb0c4_927b_4829_8813_b17cbbc6933d.slice/crio-e2f5bcf237487a2fe34d086542b159fba6edeb7a3b047e86623a6ba53a48331f WatchSource:0}: Error finding container e2f5bcf237487a2fe34d086542b159fba6edeb7a3b047e86623a6ba53a48331f: Status 404 returned error can't find the container with id e2f5bcf237487a2fe34d086542b159fba6edeb7a3b047e86623a6ba53a48331f Jul 06 22:21:17.968663 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:21:17.968632 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79b4c9c9b9-7kzcr" event={"ID":"787cb0c4-927b-4829-8813-b17cbbc6933d","Type":"ContainerStarted","Data":"e2f5bcf237487a2fe34d086542b159fba6edeb7a3b047e86623a6ba53a48331f"} Jul 06 22:21:18.975398 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:21:18.975359 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79b4c9c9b9-7kzcr" event={"ID":"787cb0c4-927b-4829-8813-b17cbbc6933d","Type":"ContainerStarted","Data":"e22f52ee7c3caee80667d96917e89ac72f98b56208cdcc3bba94d5dea304dbde"} Jul 06 22:21:18.997314 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:21:18.997245 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-79b4c9c9b9-7kzcr" podStartSLOduration=1.603698294 podStartE2EDuration="1.997222668s" totalImagesPullingTime="393.524374ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:21:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:21:18.996476176 +0000 UTC m=+730.296188026" watchObservedRunningTime="2026-07-06 22:21:18.997222668 +0000 UTC m=+730.296934517" Jul 06 22:24:09.212001 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:24:09.211919 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7mq8_e50d3b5f-832e-481f-8807-7774ff8d5521/ovn-acl-logging/0.log" Jul 06 22:24:09.213521 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:24:09.213500 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7mq8_e50d3b5f-832e-481f-8807-7774ff8d5521/ovn-acl-logging/0.log" Jul 06 22:24:09.254798 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:24:09.254762 2555 scope.go:117] "RemoveContainer" containerID="ab8f5b1c2c53cfc751049482ce28fe8708c330cc142e189c9a2ddf897f1e925c" Jul 06 22:29:09.247513 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:29:09.247474 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7mq8_e50d3b5f-832e-481f-8807-7774ff8d5521/ovn-acl-logging/0.log" Jul 06 22:29:09.248118 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:29:09.247657 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7mq8_e50d3b5f-832e-481f-8807-7774ff8d5521/ovn-acl-logging/0.log" Jul 06 22:34:09.280163 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:09.280123 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7mq8_e50d3b5f-832e-481f-8807-7774ff8d5521/ovn-acl-logging/0.log" Jul 06 22:34:09.282716 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:09.280895 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7mq8_e50d3b5f-832e-481f-8807-7774ff8d5521/ovn-acl-logging/0.log" Jul 06 22:34:30.672711 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:30.672674 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-697d6823-c95d7989-84fxb"] Jul 06 22:34:30.676869 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:30.676847 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-697d6823-c95d7989-84fxb" Jul 06 22:34:30.679842 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:30.679806 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-697d6823-serving-cert\"" Jul 06 22:34:30.691480 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:30.691449 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-697d6823-c95d7989-84fxb"] Jul 06 22:34:30.871668 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:30.871624 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58237ada-eb13-4550-a477-b4e5d04fdfa1-openshift-service-ca\") pod \"maas-api-e2e-ait-697d6823-c95d7989-84fxb\" (UID: \"58237ada-eb13-4550-a477-b4e5d04fdfa1\") " pod="opendatahub/maas-api-e2e-ait-697d6823-c95d7989-84fxb" Jul 06 22:34:30.871865 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:30.871778 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58237ada-eb13-4550-a477-b4e5d04fdfa1-maas-api-tls\") pod \"maas-api-e2e-ait-697d6823-c95d7989-84fxb\" (UID: \"58237ada-eb13-4550-a477-b4e5d04fdfa1\") " pod="opendatahub/maas-api-e2e-ait-697d6823-c95d7989-84fxb" Jul 06 22:34:30.871865 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:30.871820 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hllc\" (UniqueName: \"kubernetes.io/projected/58237ada-eb13-4550-a477-b4e5d04fdfa1-kube-api-access-8hllc\") pod \"maas-api-e2e-ait-697d6823-c95d7989-84fxb\" (UID: \"58237ada-eb13-4550-a477-b4e5d04fdfa1\") " pod="opendatahub/maas-api-e2e-ait-697d6823-c95d7989-84fxb" Jul 06 22:34:30.973313 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:30.973228 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58237ada-eb13-4550-a477-b4e5d04fdfa1-maas-api-tls\") pod \"maas-api-e2e-ait-697d6823-c95d7989-84fxb\" (UID: \"58237ada-eb13-4550-a477-b4e5d04fdfa1\") " pod="opendatahub/maas-api-e2e-ait-697d6823-c95d7989-84fxb" Jul 06 22:34:30.973313 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:30.973269 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hllc\" (UniqueName: \"kubernetes.io/projected/58237ada-eb13-4550-a477-b4e5d04fdfa1-kube-api-access-8hllc\") pod \"maas-api-e2e-ait-697d6823-c95d7989-84fxb\" (UID: \"58237ada-eb13-4550-a477-b4e5d04fdfa1\") " pod="opendatahub/maas-api-e2e-ait-697d6823-c95d7989-84fxb" Jul 06 22:34:30.973537 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:30.973334 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58237ada-eb13-4550-a477-b4e5d04fdfa1-openshift-service-ca\") pod \"maas-api-e2e-ait-697d6823-c95d7989-84fxb\" (UID: \"58237ada-eb13-4550-a477-b4e5d04fdfa1\") " pod="opendatahub/maas-api-e2e-ait-697d6823-c95d7989-84fxb" Jul 06 22:34:30.974056 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:30.974001 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58237ada-eb13-4550-a477-b4e5d04fdfa1-openshift-service-ca\") pod \"maas-api-e2e-ait-697d6823-c95d7989-84fxb\" (UID: \"58237ada-eb13-4550-a477-b4e5d04fdfa1\") " pod="opendatahub/maas-api-e2e-ait-697d6823-c95d7989-84fxb" Jul 06 22:34:30.975684 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:30.975664 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58237ada-eb13-4550-a477-b4e5d04fdfa1-maas-api-tls\") pod \"maas-api-e2e-ait-697d6823-c95d7989-84fxb\" (UID: \"58237ada-eb13-4550-a477-b4e5d04fdfa1\") " pod="opendatahub/maas-api-e2e-ait-697d6823-c95d7989-84fxb" Jul 06 22:34:30.984085 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:30.984057 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hllc\" (UniqueName: \"kubernetes.io/projected/58237ada-eb13-4550-a477-b4e5d04fdfa1-kube-api-access-8hllc\") pod \"maas-api-e2e-ait-697d6823-c95d7989-84fxb\" (UID: \"58237ada-eb13-4550-a477-b4e5d04fdfa1\") " pod="opendatahub/maas-api-e2e-ait-697d6823-c95d7989-84fxb" Jul 06 22:34:30.987741 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:30.987718 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-697d6823-c95d7989-84fxb" Jul 06 22:34:31.123056 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:31.123032 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-697d6823-c95d7989-84fxb"] Jul 06 22:34:31.125077 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:34:31.125047 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58237ada_eb13_4550_a477_b4e5d04fdfa1.slice/crio-4e104c2f42a05f60502ae509e6c6a828e3b8bae21e95379657906a97bde0fffe WatchSource:0}: Error finding container 4e104c2f42a05f60502ae509e6c6a828e3b8bae21e95379657906a97bde0fffe: Status 404 returned error can't find the container with id 4e104c2f42a05f60502ae509e6c6a828e3b8bae21e95379657906a97bde0fffe Jul 06 22:34:31.126452 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:31.126433 2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 22:34:31.921201 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:31.921161 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-697d6823-c95d7989-84fxb" event={"ID":"58237ada-eb13-4550-a477-b4e5d04fdfa1","Type":"ContainerStarted","Data":"013ea7943765a30bea375c4bd0f5d9866b0344ba77569e168c265d369996973d"} Jul 06 22:34:31.921687 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:31.921216 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-697d6823-c95d7989-84fxb" event={"ID":"58237ada-eb13-4550-a477-b4e5d04fdfa1","Type":"ContainerStarted","Data":"4e104c2f42a05f60502ae509e6c6a828e3b8bae21e95379657906a97bde0fffe"} Jul 06 22:34:31.921687 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:31.921244 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-697d6823-c95d7989-84fxb" Jul 06 22:34:31.938645 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:31.938587 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-697d6823-c95d7989-84fxb" podStartSLOduration=1.540407694 podStartE2EDuration="1.938570285s" totalImagesPullingTime="398.162591ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:34:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:34:31.937548438 +0000 UTC m=+1523.237260288" watchObservedRunningTime="2026-07-06 22:34:31.938570285 +0000 UTC m=+1523.238282133" Jul 06 22:34:36.028700 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:36.028662 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-697d6823-c95d7989-84fxb"] Jul 06 22:34:36.029237 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:36.028924 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-697d6823-c95d7989-84fxb" podUID="58237ada-eb13-4550-a477-b4e5d04fdfa1" containerName="maas-api" containerID="cri-o://013ea7943765a30bea375c4bd0f5d9866b0344ba77569e168c265d369996973d" gracePeriod=30 Jul 06 22:34:36.034784 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:36.034763 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-ait-697d6823-c95d7989-84fxb" Jul 06 22:34:36.275246 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:36.275220 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-697d6823-c95d7989-84fxb" Jul 06 22:34:36.315252 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:36.315151 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58237ada-eb13-4550-a477-b4e5d04fdfa1-maas-api-tls\") pod \"58237ada-eb13-4550-a477-b4e5d04fdfa1\" (UID: \"58237ada-eb13-4550-a477-b4e5d04fdfa1\") " Jul 06 22:34:36.315252 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:36.315193 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8hllc\" (UniqueName: \"kubernetes.io/projected/58237ada-eb13-4550-a477-b4e5d04fdfa1-kube-api-access-8hllc\") pod \"58237ada-eb13-4550-a477-b4e5d04fdfa1\" (UID: \"58237ada-eb13-4550-a477-b4e5d04fdfa1\") " Jul 06 22:34:36.315252 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:36.315224 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58237ada-eb13-4550-a477-b4e5d04fdfa1-openshift-service-ca\") pod \"58237ada-eb13-4550-a477-b4e5d04fdfa1\" (UID: \"58237ada-eb13-4550-a477-b4e5d04fdfa1\") " Jul 06 22:34:36.315649 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:36.315623 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58237ada-eb13-4550-a477-b4e5d04fdfa1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "58237ada-eb13-4550-a477-b4e5d04fdfa1" (UID: "58237ada-eb13-4550-a477-b4e5d04fdfa1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:34:36.317568 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:36.317532 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58237ada-eb13-4550-a477-b4e5d04fdfa1-kube-api-access-8hllc" (OuterVolumeSpecName: "kube-api-access-8hllc") pod "58237ada-eb13-4550-a477-b4e5d04fdfa1" (UID: "58237ada-eb13-4550-a477-b4e5d04fdfa1"). InnerVolumeSpecName "kube-api-access-8hllc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:34:36.317697 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:36.317575 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58237ada-eb13-4550-a477-b4e5d04fdfa1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "58237ada-eb13-4550-a477-b4e5d04fdfa1" (UID: "58237ada-eb13-4550-a477-b4e5d04fdfa1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:34:36.416880 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:36.416828 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58237ada-eb13-4550-a477-b4e5d04fdfa1-maas-api-tls\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:34:36.416880 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:36.416874 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hllc\" (UniqueName: \"kubernetes.io/projected/58237ada-eb13-4550-a477-b4e5d04fdfa1-kube-api-access-8hllc\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:34:36.416880 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:36.416886 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58237ada-eb13-4550-a477-b4e5d04fdfa1-openshift-service-ca\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:34:36.949304 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:36.949267 2555 generic.go:356] "Generic (PLEG): container finished" podID="58237ada-eb13-4550-a477-b4e5d04fdfa1" containerID="013ea7943765a30bea375c4bd0f5d9866b0344ba77569e168c265d369996973d" exitCode=0 Jul 06 22:34:36.949304 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:36.949309 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-697d6823-c95d7989-84fxb" Jul 06 22:34:36.949596 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:36.949353 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-697d6823-c95d7989-84fxb" event={"ID":"58237ada-eb13-4550-a477-b4e5d04fdfa1","Type":"ContainerDied","Data":"013ea7943765a30bea375c4bd0f5d9866b0344ba77569e168c265d369996973d"} Jul 06 22:34:36.949596 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:36.949413 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-697d6823-c95d7989-84fxb" event={"ID":"58237ada-eb13-4550-a477-b4e5d04fdfa1","Type":"ContainerDied","Data":"4e104c2f42a05f60502ae509e6c6a828e3b8bae21e95379657906a97bde0fffe"} Jul 06 22:34:36.949596 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:36.949443 2555 scope.go:117] "RemoveContainer" containerID="013ea7943765a30bea375c4bd0f5d9866b0344ba77569e168c265d369996973d" Jul 06 22:34:36.958799 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:36.958777 2555 scope.go:117] "RemoveContainer" containerID="013ea7943765a30bea375c4bd0f5d9866b0344ba77569e168c265d369996973d" Jul 06 22:34:36.959157 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:34:36.959138 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"013ea7943765a30bea375c4bd0f5d9866b0344ba77569e168c265d369996973d\": container with ID starting with 013ea7943765a30bea375c4bd0f5d9866b0344ba77569e168c265d369996973d not found: ID does not exist" containerID="013ea7943765a30bea375c4bd0f5d9866b0344ba77569e168c265d369996973d" Jul 06 22:34:36.959231 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:36.959211 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"013ea7943765a30bea375c4bd0f5d9866b0344ba77569e168c265d369996973d"} err="failed to get container status \"013ea7943765a30bea375c4bd0f5d9866b0344ba77569e168c265d369996973d\": rpc error: code = NotFound desc = could not find container \"013ea7943765a30bea375c4bd0f5d9866b0344ba77569e168c265d369996973d\": container with ID starting with 013ea7943765a30bea375c4bd0f5d9866b0344ba77569e168c265d369996973d not found: ID does not exist" Jul 06 22:34:36.971322 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:36.971290 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-697d6823-c95d7989-84fxb"] Jul 06 22:34:36.975054 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:36.975027 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-697d6823-c95d7989-84fxb"] Jul 06 22:34:37.341428 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:37.341346 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58237ada-eb13-4550-a477-b4e5d04fdfa1" path="/var/lib/kubelet/pods/58237ada-eb13-4550-a477-b4e5d04fdfa1/volumes" Jul 06 22:34:43.090306 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:43.090268 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-754cf73a-7855b9cd-7bpxl"] Jul 06 22:34:43.090802 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:43.090678 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58237ada-eb13-4550-a477-b4e5d04fdfa1" containerName="maas-api" Jul 06 22:34:43.090802 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:43.090693 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="58237ada-eb13-4550-a477-b4e5d04fdfa1" containerName="maas-api" Jul 06 22:34:43.090802 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:43.090776 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="58237ada-eb13-4550-a477-b4e5d04fdfa1" containerName="maas-api" Jul 06 22:34:43.093959 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:43.093941 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-754cf73a-7855b9cd-7bpxl" Jul 06 22:34:43.096888 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:43.096866 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-754cf73a-serving-cert\"" Jul 06 22:34:43.105474 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:43.105448 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-754cf73a-7855b9cd-7bpxl"] Jul 06 22:34:43.183603 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:43.183567 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbvvz\" (UniqueName: \"kubernetes.io/projected/c61ee1b9-8b10-401e-ba37-8f44ba3d878d-kube-api-access-wbvvz\") pod \"maas-api-e2e-ait-754cf73a-7855b9cd-7bpxl\" (UID: \"c61ee1b9-8b10-401e-ba37-8f44ba3d878d\") " pod="opendatahub/maas-api-e2e-ait-754cf73a-7855b9cd-7bpxl" Jul 06 22:34:43.183782 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:43.183633 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c61ee1b9-8b10-401e-ba37-8f44ba3d878d-maas-api-tls\") pod \"maas-api-e2e-ait-754cf73a-7855b9cd-7bpxl\" (UID: \"c61ee1b9-8b10-401e-ba37-8f44ba3d878d\") " pod="opendatahub/maas-api-e2e-ait-754cf73a-7855b9cd-7bpxl" Jul 06 22:34:43.183782 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:43.183748 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c61ee1b9-8b10-401e-ba37-8f44ba3d878d-openshift-service-ca\") pod \"maas-api-e2e-ait-754cf73a-7855b9cd-7bpxl\" (UID: \"c61ee1b9-8b10-401e-ba37-8f44ba3d878d\") " pod="opendatahub/maas-api-e2e-ait-754cf73a-7855b9cd-7bpxl" Jul 06 22:34:43.285095 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:43.285047 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c61ee1b9-8b10-401e-ba37-8f44ba3d878d-openshift-service-ca\") pod \"maas-api-e2e-ait-754cf73a-7855b9cd-7bpxl\" (UID: \"c61ee1b9-8b10-401e-ba37-8f44ba3d878d\") " pod="opendatahub/maas-api-e2e-ait-754cf73a-7855b9cd-7bpxl" Jul 06 22:34:43.285095 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:43.285094 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbvvz\" (UniqueName: \"kubernetes.io/projected/c61ee1b9-8b10-401e-ba37-8f44ba3d878d-kube-api-access-wbvvz\") pod \"maas-api-e2e-ait-754cf73a-7855b9cd-7bpxl\" (UID: \"c61ee1b9-8b10-401e-ba37-8f44ba3d878d\") " pod="opendatahub/maas-api-e2e-ait-754cf73a-7855b9cd-7bpxl" Jul 06 22:34:43.285314 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:43.285279 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c61ee1b9-8b10-401e-ba37-8f44ba3d878d-maas-api-tls\") pod \"maas-api-e2e-ait-754cf73a-7855b9cd-7bpxl\" (UID: \"c61ee1b9-8b10-401e-ba37-8f44ba3d878d\") " pod="opendatahub/maas-api-e2e-ait-754cf73a-7855b9cd-7bpxl" Jul 06 22:34:43.285427 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:34:43.285410 2555 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-ait-754cf73a-serving-cert: secret "maas-api-e2e-ait-754cf73a-serving-cert" not found Jul 06 22:34:43.285491 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:34:43.285481 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c61ee1b9-8b10-401e-ba37-8f44ba3d878d-maas-api-tls podName:c61ee1b9-8b10-401e-ba37-8f44ba3d878d nodeName:}" failed. No retries permitted until 2026-07-06 22:34:43.785461341 +0000 UTC m=+1535.085173169 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c61ee1b9-8b10-401e-ba37-8f44ba3d878d-maas-api-tls") pod "maas-api-e2e-ait-754cf73a-7855b9cd-7bpxl" (UID: "c61ee1b9-8b10-401e-ba37-8f44ba3d878d") : secret "maas-api-e2e-ait-754cf73a-serving-cert" not found Jul 06 22:34:43.285684 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:43.285665 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c61ee1b9-8b10-401e-ba37-8f44ba3d878d-openshift-service-ca\") pod \"maas-api-e2e-ait-754cf73a-7855b9cd-7bpxl\" (UID: \"c61ee1b9-8b10-401e-ba37-8f44ba3d878d\") " pod="opendatahub/maas-api-e2e-ait-754cf73a-7855b9cd-7bpxl" Jul 06 22:34:43.302745 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:43.302714 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbvvz\" (UniqueName: \"kubernetes.io/projected/c61ee1b9-8b10-401e-ba37-8f44ba3d878d-kube-api-access-wbvvz\") pod \"maas-api-e2e-ait-754cf73a-7855b9cd-7bpxl\" (UID: \"c61ee1b9-8b10-401e-ba37-8f44ba3d878d\") " pod="opendatahub/maas-api-e2e-ait-754cf73a-7855b9cd-7bpxl" Jul 06 22:34:43.791052 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:43.790994 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c61ee1b9-8b10-401e-ba37-8f44ba3d878d-maas-api-tls\") pod \"maas-api-e2e-ait-754cf73a-7855b9cd-7bpxl\" (UID: \"c61ee1b9-8b10-401e-ba37-8f44ba3d878d\") " pod="opendatahub/maas-api-e2e-ait-754cf73a-7855b9cd-7bpxl" Jul 06 22:34:43.793489 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:43.793462 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c61ee1b9-8b10-401e-ba37-8f44ba3d878d-maas-api-tls\") pod \"maas-api-e2e-ait-754cf73a-7855b9cd-7bpxl\" (UID: \"c61ee1b9-8b10-401e-ba37-8f44ba3d878d\") " pod="opendatahub/maas-api-e2e-ait-754cf73a-7855b9cd-7bpxl" Jul 06 22:34:44.004240 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:44.004198 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-754cf73a-7855b9cd-7bpxl" Jul 06 22:34:44.138421 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:44.138394 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-754cf73a-7855b9cd-7bpxl"] Jul 06 22:34:44.140302 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:34:44.140271 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc61ee1b9_8b10_401e_ba37_8f44ba3d878d.slice/crio-c64ee1bcc6ab8f23ec0c9c4822eba7a5d9aa97031be0f1ac0b61583b0c23a748 WatchSource:0}: Error finding container c64ee1bcc6ab8f23ec0c9c4822eba7a5d9aa97031be0f1ac0b61583b0c23a748: Status 404 returned error can't find the container with id c64ee1bcc6ab8f23ec0c9c4822eba7a5d9aa97031be0f1ac0b61583b0c23a748 Jul 06 22:34:44.998453 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:44.998412 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-754cf73a-7855b9cd-7bpxl" event={"ID":"c61ee1b9-8b10-401e-ba37-8f44ba3d878d","Type":"ContainerStarted","Data":"e5f493cfe715fa225e15c3916e28e15aa14c476de80aadb90996890304bdc703"} Jul 06 22:34:44.998640 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:44.998477 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-754cf73a-7855b9cd-7bpxl" event={"ID":"c61ee1b9-8b10-401e-ba37-8f44ba3d878d","Type":"ContainerStarted","Data":"c64ee1bcc6ab8f23ec0c9c4822eba7a5d9aa97031be0f1ac0b61583b0c23a748"} Jul 06 22:34:44.998640 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:44.998506 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-754cf73a-7855b9cd-7bpxl" Jul 06 22:34:45.017954 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:45.017897 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-754cf73a-7855b9cd-7bpxl" podStartSLOduration=1.73800008 podStartE2EDuration="2.01788185s" totalImagesPullingTime="279.88177ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:34:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:34:45.015753194 +0000 UTC m=+1536.315465045" watchObservedRunningTime="2026-07-06 22:34:45.01788185 +0000 UTC m=+1536.317593698" Jul 06 22:34:48.490265 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:48.490228 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-754cf73a-7855b9cd-7bpxl"] Jul 06 22:34:48.490668 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:48.490454 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-754cf73a-7855b9cd-7bpxl" podUID="c61ee1b9-8b10-401e-ba37-8f44ba3d878d" containerName="maas-api" containerID="cri-o://e5f493cfe715fa225e15c3916e28e15aa14c476de80aadb90996890304bdc703" gracePeriod=30 Jul 06 22:34:48.759496 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:48.759471 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-754cf73a-7855b9cd-7bpxl" Jul 06 22:34:48.849642 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:48.849598 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c61ee1b9-8b10-401e-ba37-8f44ba3d878d-openshift-service-ca\") pod \"c61ee1b9-8b10-401e-ba37-8f44ba3d878d\" (UID: \"c61ee1b9-8b10-401e-ba37-8f44ba3d878d\") " Jul 06 22:34:48.849642 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:48.849647 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c61ee1b9-8b10-401e-ba37-8f44ba3d878d-maas-api-tls\") pod \"c61ee1b9-8b10-401e-ba37-8f44ba3d878d\" (UID: \"c61ee1b9-8b10-401e-ba37-8f44ba3d878d\") " Jul 06 22:34:48.849856 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:48.849682 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wbvvz\" (UniqueName: \"kubernetes.io/projected/c61ee1b9-8b10-401e-ba37-8f44ba3d878d-kube-api-access-wbvvz\") pod \"c61ee1b9-8b10-401e-ba37-8f44ba3d878d\" (UID: \"c61ee1b9-8b10-401e-ba37-8f44ba3d878d\") " Jul 06 22:34:48.850059 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:48.850036 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c61ee1b9-8b10-401e-ba37-8f44ba3d878d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c61ee1b9-8b10-401e-ba37-8f44ba3d878d" (UID: "c61ee1b9-8b10-401e-ba37-8f44ba3d878d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:34:48.851830 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:48.851807 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c61ee1b9-8b10-401e-ba37-8f44ba3d878d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c61ee1b9-8b10-401e-ba37-8f44ba3d878d" (UID: "c61ee1b9-8b10-401e-ba37-8f44ba3d878d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:34:48.851888 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:48.851847 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c61ee1b9-8b10-401e-ba37-8f44ba3d878d-kube-api-access-wbvvz" (OuterVolumeSpecName: "kube-api-access-wbvvz") pod "c61ee1b9-8b10-401e-ba37-8f44ba3d878d" (UID: "c61ee1b9-8b10-401e-ba37-8f44ba3d878d"). InnerVolumeSpecName "kube-api-access-wbvvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:34:48.951320 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:48.951282 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c61ee1b9-8b10-401e-ba37-8f44ba3d878d-openshift-service-ca\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:34:48.951320 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:48.951315 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c61ee1b9-8b10-401e-ba37-8f44ba3d878d-maas-api-tls\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:34:48.951320 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:48.951327 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbvvz\" (UniqueName: \"kubernetes.io/projected/c61ee1b9-8b10-401e-ba37-8f44ba3d878d-kube-api-access-wbvvz\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:34:49.018755 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:49.018658 2555 generic.go:356] "Generic (PLEG): container finished" podID="c61ee1b9-8b10-401e-ba37-8f44ba3d878d" containerID="e5f493cfe715fa225e15c3916e28e15aa14c476de80aadb90996890304bdc703" exitCode=0 Jul 06 22:34:49.018755 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:49.018702 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-754cf73a-7855b9cd-7bpxl" Jul 06 22:34:49.018755 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:49.018746 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-754cf73a-7855b9cd-7bpxl" event={"ID":"c61ee1b9-8b10-401e-ba37-8f44ba3d878d","Type":"ContainerDied","Data":"e5f493cfe715fa225e15c3916e28e15aa14c476de80aadb90996890304bdc703"} Jul 06 22:34:49.019033 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:49.018804 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-754cf73a-7855b9cd-7bpxl" event={"ID":"c61ee1b9-8b10-401e-ba37-8f44ba3d878d","Type":"ContainerDied","Data":"c64ee1bcc6ab8f23ec0c9c4822eba7a5d9aa97031be0f1ac0b61583b0c23a748"} Jul 06 22:34:49.019033 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:49.018834 2555 scope.go:117] "RemoveContainer" containerID="e5f493cfe715fa225e15c3916e28e15aa14c476de80aadb90996890304bdc703" Jul 06 22:34:49.027640 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:49.027623 2555 scope.go:117] "RemoveContainer" containerID="e5f493cfe715fa225e15c3916e28e15aa14c476de80aadb90996890304bdc703" Jul 06 22:34:49.027905 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:34:49.027887 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5f493cfe715fa225e15c3916e28e15aa14c476de80aadb90996890304bdc703\": container with ID starting with e5f493cfe715fa225e15c3916e28e15aa14c476de80aadb90996890304bdc703 not found: ID does not exist" containerID="e5f493cfe715fa225e15c3916e28e15aa14c476de80aadb90996890304bdc703" Jul 06 22:34:49.027954 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:49.027916 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5f493cfe715fa225e15c3916e28e15aa14c476de80aadb90996890304bdc703"} err="failed to get container status \"e5f493cfe715fa225e15c3916e28e15aa14c476de80aadb90996890304bdc703\": rpc error: code = NotFound desc = could not find container \"e5f493cfe715fa225e15c3916e28e15aa14c476de80aadb90996890304bdc703\": container with ID starting with e5f493cfe715fa225e15c3916e28e15aa14c476de80aadb90996890304bdc703 not found: ID does not exist" Jul 06 22:34:49.039671 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:49.039647 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-754cf73a-7855b9cd-7bpxl"] Jul 06 22:34:49.043234 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:49.043210 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-754cf73a-7855b9cd-7bpxl"] Jul 06 22:34:49.339446 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:34:49.339368 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c61ee1b9-8b10-401e-ba37-8f44ba3d878d" path="/var/lib/kubelet/pods/c61ee1b9-8b10-401e-ba37-8f44ba3d878d/volumes" Jul 06 22:35:02.196776 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:35:02.196736 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-derive-d4b08138-ff8684f69-84qgh"] Jul 06 22:35:02.197204 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:35:02.197164 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c61ee1b9-8b10-401e-ba37-8f44ba3d878d" containerName="maas-api" Jul 06 22:35:02.197204 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:35:02.197178 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c61ee1b9-8b10-401e-ba37-8f44ba3d878d" containerName="maas-api" Jul 06 22:35:02.197280 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:35:02.197268 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c61ee1b9-8b10-401e-ba37-8f44ba3d878d" containerName="maas-api" Jul 06 22:35:02.201713 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:35:02.201695 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-d4b08138-ff8684f69-84qgh" Jul 06 22:35:02.204215 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:35:02.204191 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-derive-d4b08138-serving-cert\"" Jul 06 22:35:02.209948 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:35:02.209921 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-d4b08138-ff8684f69-84qgh"] Jul 06 22:35:02.387207 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:35:02.387158 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d672713-1325-4bc2-ab0b-77350596bbe9-openshift-service-ca\") pod \"maas-api-e2e-derive-d4b08138-ff8684f69-84qgh\" (UID: \"3d672713-1325-4bc2-ab0b-77350596bbe9\") " pod="opendatahub/maas-api-e2e-derive-d4b08138-ff8684f69-84qgh" Jul 06 22:35:02.387207 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:35:02.387204 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl595\" (UniqueName: \"kubernetes.io/projected/3d672713-1325-4bc2-ab0b-77350596bbe9-kube-api-access-rl595\") pod \"maas-api-e2e-derive-d4b08138-ff8684f69-84qgh\" (UID: \"3d672713-1325-4bc2-ab0b-77350596bbe9\") " pod="opendatahub/maas-api-e2e-derive-d4b08138-ff8684f69-84qgh" Jul 06 22:35:02.387478 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:35:02.387241 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d672713-1325-4bc2-ab0b-77350596bbe9-maas-api-tls\") pod \"maas-api-e2e-derive-d4b08138-ff8684f69-84qgh\" (UID: \"3d672713-1325-4bc2-ab0b-77350596bbe9\") " pod="opendatahub/maas-api-e2e-derive-d4b08138-ff8684f69-84qgh" Jul 06 22:35:02.488595 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:35:02.488502 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d672713-1325-4bc2-ab0b-77350596bbe9-openshift-service-ca\") pod \"maas-api-e2e-derive-d4b08138-ff8684f69-84qgh\" (UID: \"3d672713-1325-4bc2-ab0b-77350596bbe9\") " pod="opendatahub/maas-api-e2e-derive-d4b08138-ff8684f69-84qgh" Jul 06 22:35:02.488595 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:35:02.488540 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl595\" (UniqueName: \"kubernetes.io/projected/3d672713-1325-4bc2-ab0b-77350596bbe9-kube-api-access-rl595\") pod \"maas-api-e2e-derive-d4b08138-ff8684f69-84qgh\" (UID: \"3d672713-1325-4bc2-ab0b-77350596bbe9\") " pod="opendatahub/maas-api-e2e-derive-d4b08138-ff8684f69-84qgh" Jul 06 22:35:02.488595 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:35:02.488578 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d672713-1325-4bc2-ab0b-77350596bbe9-maas-api-tls\") pod \"maas-api-e2e-derive-d4b08138-ff8684f69-84qgh\" (UID: \"3d672713-1325-4bc2-ab0b-77350596bbe9\") " pod="opendatahub/maas-api-e2e-derive-d4b08138-ff8684f69-84qgh" Jul 06 22:35:02.489191 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:35:02.489169 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d672713-1325-4bc2-ab0b-77350596bbe9-openshift-service-ca\") pod \"maas-api-e2e-derive-d4b08138-ff8684f69-84qgh\" (UID: \"3d672713-1325-4bc2-ab0b-77350596bbe9\") " pod="opendatahub/maas-api-e2e-derive-d4b08138-ff8684f69-84qgh" Jul 06 22:35:02.490947 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:35:02.490924 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d672713-1325-4bc2-ab0b-77350596bbe9-maas-api-tls\") pod \"maas-api-e2e-derive-d4b08138-ff8684f69-84qgh\" (UID: \"3d672713-1325-4bc2-ab0b-77350596bbe9\") " pod="opendatahub/maas-api-e2e-derive-d4b08138-ff8684f69-84qgh" Jul 06 22:35:02.497116 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:35:02.497096 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl595\" (UniqueName: \"kubernetes.io/projected/3d672713-1325-4bc2-ab0b-77350596bbe9-kube-api-access-rl595\") pod \"maas-api-e2e-derive-d4b08138-ff8684f69-84qgh\" (UID: \"3d672713-1325-4bc2-ab0b-77350596bbe9\") " pod="opendatahub/maas-api-e2e-derive-d4b08138-ff8684f69-84qgh" Jul 06 22:35:02.514116 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:35:02.514096 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-d4b08138-ff8684f69-84qgh" Jul 06 22:35:02.651041 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:35:02.650999 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-d4b08138-ff8684f69-84qgh"] Jul 06 22:35:02.653672 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:35:02.653637 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d672713_1325_4bc2_ab0b_77350596bbe9.slice/crio-273241333e220c4651ccd72445682d73053ac1969f1f7d04a8d9f6d73dc7324d WatchSource:0}: Error finding container 273241333e220c4651ccd72445682d73053ac1969f1f7d04a8d9f6d73dc7324d: Status 404 returned error can't find the container with id 273241333e220c4651ccd72445682d73053ac1969f1f7d04a8d9f6d73dc7324d Jul 06 22:35:03.091209 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:35:03.091181 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-d4b08138-ff8684f69-84qgh" event={"ID":"3d672713-1325-4bc2-ab0b-77350596bbe9","Type":"ContainerStarted","Data":"273241333e220c4651ccd72445682d73053ac1969f1f7d04a8d9f6d73dc7324d"} Jul 06 22:35:04.097853 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:35:04.097819 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-d4b08138-ff8684f69-84qgh" event={"ID":"3d672713-1325-4bc2-ab0b-77350596bbe9","Type":"ContainerStarted","Data":"a658ebc4c9bce7cc8831327bb7a4eeaed9660f20f6014e71dda8ead32bedec78"} Jul 06 22:35:04.098272 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:35:04.097898 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-derive-d4b08138-ff8684f69-84qgh" Jul 06 22:35:04.116338 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:35:04.116282 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-derive-d4b08138-ff8684f69-84qgh" podStartSLOduration=1.729757701 podStartE2EDuration="2.116266329s" totalImagesPullingTime="386.508628ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:35:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:35:04.114273813 +0000 UTC m=+1555.413985663" watchObservedRunningTime="2026-07-06 22:35:04.116266329 +0000 UTC m=+1555.415978199" Jul 06 22:35:06.851224 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:35:06.851191 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-d4b08138-ff8684f69-84qgh"] Jul 06 22:35:06.851682 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:35:06.851388 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-derive-d4b08138-ff8684f69-84qgh" podUID="3d672713-1325-4bc2-ab0b-77350596bbe9" containerName="maas-api" containerID="cri-o://a658ebc4c9bce7cc8831327bb7a4eeaed9660f20f6014e71dda8ead32bedec78" gracePeriod=30 Jul 06 22:35:07.102172 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:35:07.102100 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-d4b08138-ff8684f69-84qgh" Jul 06 22:35:07.113565 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:35:07.113532 2555 generic.go:356] "Generic (PLEG): container finished" podID="3d672713-1325-4bc2-ab0b-77350596bbe9" containerID="a658ebc4c9bce7cc8831327bb7a4eeaed9660f20f6014e71dda8ead32bedec78" exitCode=0 Jul 06 22:35:07.113727 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:35:07.113582 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-d4b08138-ff8684f69-84qgh" event={"ID":"3d672713-1325-4bc2-ab0b-77350596bbe9","Type":"ContainerDied","Data":"a658ebc4c9bce7cc8831327bb7a4eeaed9660f20f6014e71dda8ead32bedec78"} Jul 06 22:35:07.113727 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:35:07.113584 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-d4b08138-ff8684f69-84qgh" Jul 06 22:35:07.113727 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:35:07.113624 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-d4b08138-ff8684f69-84qgh" event={"ID":"3d672713-1325-4bc2-ab0b-77350596bbe9","Type":"ContainerDied","Data":"273241333e220c4651ccd72445682d73053ac1969f1f7d04a8d9f6d73dc7324d"} Jul 06 22:35:07.113727 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:35:07.113646 2555 scope.go:117] "RemoveContainer" containerID="a658ebc4c9bce7cc8831327bb7a4eeaed9660f20f6014e71dda8ead32bedec78" Jul 06 22:35:07.122931 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:35:07.122912 2555 scope.go:117] "RemoveContainer" containerID="a658ebc4c9bce7cc8831327bb7a4eeaed9660f20f6014e71dda8ead32bedec78" Jul 06 22:35:07.123267 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:35:07.123236 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a658ebc4c9bce7cc8831327bb7a4eeaed9660f20f6014e71dda8ead32bedec78\": container with ID starting with a658ebc4c9bce7cc8831327bb7a4eeaed9660f20f6014e71dda8ead32bedec78 not found: ID does not exist" containerID="a658ebc4c9bce7cc8831327bb7a4eeaed9660f20f6014e71dda8ead32bedec78" Jul 06 22:35:07.123353 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:35:07.123279 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a658ebc4c9bce7cc8831327bb7a4eeaed9660f20f6014e71dda8ead32bedec78"} err="failed to get container status \"a658ebc4c9bce7cc8831327bb7a4eeaed9660f20f6014e71dda8ead32bedec78\": rpc error: code = NotFound desc = could not find container \"a658ebc4c9bce7cc8831327bb7a4eeaed9660f20f6014e71dda8ead32bedec78\": container with ID starting with a658ebc4c9bce7cc8831327bb7a4eeaed9660f20f6014e71dda8ead32bedec78 not found: ID does not exist" Jul 06 22:35:07.234911 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:35:07.234877 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d672713-1325-4bc2-ab0b-77350596bbe9-openshift-service-ca\") pod \"3d672713-1325-4bc2-ab0b-77350596bbe9\" (UID: \"3d672713-1325-4bc2-ab0b-77350596bbe9\") " Jul 06 22:35:07.234911 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:35:07.234914 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rl595\" (UniqueName: \"kubernetes.io/projected/3d672713-1325-4bc2-ab0b-77350596bbe9-kube-api-access-rl595\") pod \"3d672713-1325-4bc2-ab0b-77350596bbe9\" (UID: \"3d672713-1325-4bc2-ab0b-77350596bbe9\") " Jul 06 22:35:07.235216 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:35:07.234939 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d672713-1325-4bc2-ab0b-77350596bbe9-maas-api-tls\") pod \"3d672713-1325-4bc2-ab0b-77350596bbe9\" (UID: \"3d672713-1325-4bc2-ab0b-77350596bbe9\") " Jul 06 22:35:07.235371 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:35:07.235342 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d672713-1325-4bc2-ab0b-77350596bbe9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3d672713-1325-4bc2-ab0b-77350596bbe9" (UID: "3d672713-1325-4bc2-ab0b-77350596bbe9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:35:07.237100 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:35:07.237075 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d672713-1325-4bc2-ab0b-77350596bbe9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3d672713-1325-4bc2-ab0b-77350596bbe9" (UID: "3d672713-1325-4bc2-ab0b-77350596bbe9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:35:07.237239 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:35:07.237136 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d672713-1325-4bc2-ab0b-77350596bbe9-kube-api-access-rl595" (OuterVolumeSpecName: "kube-api-access-rl595") pod "3d672713-1325-4bc2-ab0b-77350596bbe9" (UID: "3d672713-1325-4bc2-ab0b-77350596bbe9"). InnerVolumeSpecName "kube-api-access-rl595". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:35:07.335738 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:35:07.335705 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d672713-1325-4bc2-ab0b-77350596bbe9-openshift-service-ca\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:35:07.335738 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:35:07.335731 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl595\" (UniqueName: \"kubernetes.io/projected/3d672713-1325-4bc2-ab0b-77350596bbe9-kube-api-access-rl595\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:35:07.335738 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:35:07.335741 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d672713-1325-4bc2-ab0b-77350596bbe9-maas-api-tls\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:35:07.431268 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:35:07.431236 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-d4b08138-ff8684f69-84qgh"] Jul 06 22:35:07.439234 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:35:07.439202 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-derive-d4b08138-ff8684f69-84qgh"] Jul 06 22:35:09.340070 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:35:09.340030 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d672713-1325-4bc2-ab0b-77350596bbe9" path="/var/lib/kubelet/pods/3d672713-1325-4bc2-ab0b-77350596bbe9/volumes" Jul 06 22:36:51.576204 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:36:51.576163 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-57b5abb4-5f95dcbbf7-xps66"] Jul 06 22:36:51.576837 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:36:51.576753 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d672713-1325-4bc2-ab0b-77350596bbe9" containerName="maas-api" Jul 06 22:36:51.576837 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:36:51.576773 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d672713-1325-4bc2-ab0b-77350596bbe9" containerName="maas-api" Jul 06 22:36:51.576964 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:36:51.576875 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d672713-1325-4bc2-ab0b-77350596bbe9" containerName="maas-api" Jul 06 22:36:51.579839 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:36:51.579818 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-57b5abb4-5f95dcbbf7-xps66" Jul 06 22:36:51.590171 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:36:51.590153 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-57b5abb4-serving-cert\"" Jul 06 22:36:51.605432 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:36:51.605398 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-57b5abb4-5f95dcbbf7-xps66"] Jul 06 22:36:51.744987 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:36:51.744949 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/967debb5-8411-4af4-aebb-c33c626e015e-maas-api-tls\") pod \"maas-api-e2e-mt-57b5abb4-5f95dcbbf7-xps66\" (UID: \"967debb5-8411-4af4-aebb-c33c626e015e\") " pod="opendatahub/maas-api-e2e-mt-57b5abb4-5f95dcbbf7-xps66" Jul 06 22:36:51.744987 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:36:51.744995 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/967debb5-8411-4af4-aebb-c33c626e015e-openshift-service-ca\") pod \"maas-api-e2e-mt-57b5abb4-5f95dcbbf7-xps66\" (UID: \"967debb5-8411-4af4-aebb-c33c626e015e\") " pod="opendatahub/maas-api-e2e-mt-57b5abb4-5f95dcbbf7-xps66" Jul 06 22:36:51.745246 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:36:51.745053 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snkcp\" (UniqueName: \"kubernetes.io/projected/967debb5-8411-4af4-aebb-c33c626e015e-kube-api-access-snkcp\") pod \"maas-api-e2e-mt-57b5abb4-5f95dcbbf7-xps66\" (UID: \"967debb5-8411-4af4-aebb-c33c626e015e\") " pod="opendatahub/maas-api-e2e-mt-57b5abb4-5f95dcbbf7-xps66" Jul 06 22:36:51.845856 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:36:51.845768 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/967debb5-8411-4af4-aebb-c33c626e015e-maas-api-tls\") pod \"maas-api-e2e-mt-57b5abb4-5f95dcbbf7-xps66\" (UID: \"967debb5-8411-4af4-aebb-c33c626e015e\") " pod="opendatahub/maas-api-e2e-mt-57b5abb4-5f95dcbbf7-xps66" Jul 06 22:36:51.845856 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:36:51.845811 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/967debb5-8411-4af4-aebb-c33c626e015e-openshift-service-ca\") pod \"maas-api-e2e-mt-57b5abb4-5f95dcbbf7-xps66\" (UID: \"967debb5-8411-4af4-aebb-c33c626e015e\") " pod="opendatahub/maas-api-e2e-mt-57b5abb4-5f95dcbbf7-xps66" Jul 06 22:36:51.845856 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:36:51.845833 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snkcp\" (UniqueName: \"kubernetes.io/projected/967debb5-8411-4af4-aebb-c33c626e015e-kube-api-access-snkcp\") pod \"maas-api-e2e-mt-57b5abb4-5f95dcbbf7-xps66\" (UID: \"967debb5-8411-4af4-aebb-c33c626e015e\") " pod="opendatahub/maas-api-e2e-mt-57b5abb4-5f95dcbbf7-xps66" Jul 06 22:36:51.846125 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:36:51.845920 2555 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-mt-57b5abb4-serving-cert: secret "maas-api-e2e-mt-57b5abb4-serving-cert" not found Jul 06 22:36:51.846125 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:36:51.845997 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/967debb5-8411-4af4-aebb-c33c626e015e-maas-api-tls podName:967debb5-8411-4af4-aebb-c33c626e015e nodeName:}" failed. No retries permitted until 2026-07-06 22:36:52.345979547 +0000 UTC m=+1663.645691388 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/967debb5-8411-4af4-aebb-c33c626e015e-maas-api-tls") pod "maas-api-e2e-mt-57b5abb4-5f95dcbbf7-xps66" (UID: "967debb5-8411-4af4-aebb-c33c626e015e") : secret "maas-api-e2e-mt-57b5abb4-serving-cert" not found Jul 06 22:36:51.846514 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:36:51.846497 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/967debb5-8411-4af4-aebb-c33c626e015e-openshift-service-ca\") pod \"maas-api-e2e-mt-57b5abb4-5f95dcbbf7-xps66\" (UID: \"967debb5-8411-4af4-aebb-c33c626e015e\") " pod="opendatahub/maas-api-e2e-mt-57b5abb4-5f95dcbbf7-xps66" Jul 06 22:36:51.854513 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:36:51.854484 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snkcp\" (UniqueName: \"kubernetes.io/projected/967debb5-8411-4af4-aebb-c33c626e015e-kube-api-access-snkcp\") pod \"maas-api-e2e-mt-57b5abb4-5f95dcbbf7-xps66\" (UID: \"967debb5-8411-4af4-aebb-c33c626e015e\") " pod="opendatahub/maas-api-e2e-mt-57b5abb4-5f95dcbbf7-xps66" Jul 06 22:36:52.350179 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:36:52.350136 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/967debb5-8411-4af4-aebb-c33c626e015e-maas-api-tls\") pod \"maas-api-e2e-mt-57b5abb4-5f95dcbbf7-xps66\" (UID: \"967debb5-8411-4af4-aebb-c33c626e015e\") " pod="opendatahub/maas-api-e2e-mt-57b5abb4-5f95dcbbf7-xps66" Jul 06 22:36:52.352555 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:36:52.352527 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/967debb5-8411-4af4-aebb-c33c626e015e-maas-api-tls\") pod \"maas-api-e2e-mt-57b5abb4-5f95dcbbf7-xps66\" (UID: \"967debb5-8411-4af4-aebb-c33c626e015e\") " pod="opendatahub/maas-api-e2e-mt-57b5abb4-5f95dcbbf7-xps66" Jul 06 22:36:52.490311 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:36:52.490267 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-57b5abb4-5f95dcbbf7-xps66" Jul 06 22:36:52.628693 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:36:52.628666 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-57b5abb4-5f95dcbbf7-xps66"] Jul 06 22:36:52.630755 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:36:52.630718 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod967debb5_8411_4af4_aebb_c33c626e015e.slice/crio-00d69826eca7fe314094a9b31051eafb76baece5afa8f027c4bcad63c014cf75 WatchSource:0}: Error finding container 00d69826eca7fe314094a9b31051eafb76baece5afa8f027c4bcad63c014cf75: Status 404 returned error can't find the container with id 00d69826eca7fe314094a9b31051eafb76baece5afa8f027c4bcad63c014cf75 Jul 06 22:36:52.640881 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:36:52.640854 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-57b5abb4-5f95dcbbf7-xps66" event={"ID":"967debb5-8411-4af4-aebb-c33c626e015e","Type":"ContainerStarted","Data":"00d69826eca7fe314094a9b31051eafb76baece5afa8f027c4bcad63c014cf75"} Jul 06 22:36:53.647872 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:36:53.647824 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-57b5abb4-5f95dcbbf7-xps66" event={"ID":"967debb5-8411-4af4-aebb-c33c626e015e","Type":"ContainerStarted","Data":"b0bcd2421756f03f2612c092fee1b7971df0c6b5d95d63c1da75fea10997d5c9"} Jul 06 22:36:53.648246 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:36:53.647902 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-57b5abb4-5f95dcbbf7-xps66" Jul 06 22:36:53.665598 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:36:53.665543 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-57b5abb4-5f95dcbbf7-xps66" podStartSLOduration=2.237128726 podStartE2EDuration="2.66552633s" totalImagesPullingTime="428.397604ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:36:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:36:53.664979544 +0000 UTC m=+1664.964691394" watchObservedRunningTime="2026-07-06 22:36:53.66552633 +0000 UTC m=+1664.965238179" Jul 06 22:36:59.657806 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:36:59.657767 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-57b5abb4-5f95dcbbf7-xps66" Jul 06 22:37:03.707310 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:37:03.707272 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-57b5abb4-5f95dcbbf7-xps66"] Jul 06 22:37:03.707720 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:37:03.707527 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-57b5abb4-5f95dcbbf7-xps66" podUID="967debb5-8411-4af4-aebb-c33c626e015e" containerName="maas-api" containerID="cri-o://b0bcd2421756f03f2612c092fee1b7971df0c6b5d95d63c1da75fea10997d5c9" gracePeriod=30 Jul 06 22:37:03.954518 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:37:03.954490 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-57b5abb4-5f95dcbbf7-xps66" Jul 06 22:37:04.061949 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:37:04.061858 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/967debb5-8411-4af4-aebb-c33c626e015e-maas-api-tls\") pod \"967debb5-8411-4af4-aebb-c33c626e015e\" (UID: \"967debb5-8411-4af4-aebb-c33c626e015e\") " Jul 06 22:37:04.061949 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:37:04.061913 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-snkcp\" (UniqueName: \"kubernetes.io/projected/967debb5-8411-4af4-aebb-c33c626e015e-kube-api-access-snkcp\") pod \"967debb5-8411-4af4-aebb-c33c626e015e\" (UID: \"967debb5-8411-4af4-aebb-c33c626e015e\") " Jul 06 22:37:04.062235 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:37:04.061968 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/967debb5-8411-4af4-aebb-c33c626e015e-openshift-service-ca\") pod \"967debb5-8411-4af4-aebb-c33c626e015e\" (UID: \"967debb5-8411-4af4-aebb-c33c626e015e\") " Jul 06 22:37:04.062361 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:37:04.062333 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/967debb5-8411-4af4-aebb-c33c626e015e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "967debb5-8411-4af4-aebb-c33c626e015e" (UID: "967debb5-8411-4af4-aebb-c33c626e015e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:37:04.064149 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:37:04.064114 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/967debb5-8411-4af4-aebb-c33c626e015e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "967debb5-8411-4af4-aebb-c33c626e015e" (UID: "967debb5-8411-4af4-aebb-c33c626e015e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:37:04.064149 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:37:04.064119 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/967debb5-8411-4af4-aebb-c33c626e015e-kube-api-access-snkcp" (OuterVolumeSpecName: "kube-api-access-snkcp") pod "967debb5-8411-4af4-aebb-c33c626e015e" (UID: "967debb5-8411-4af4-aebb-c33c626e015e"). InnerVolumeSpecName "kube-api-access-snkcp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:37:04.163718 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:37:04.163679 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/967debb5-8411-4af4-aebb-c33c626e015e-openshift-service-ca\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:37:04.163718 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:37:04.163711 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/967debb5-8411-4af4-aebb-c33c626e015e-maas-api-tls\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:37:04.163718 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:37:04.163721 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snkcp\" (UniqueName: \"kubernetes.io/projected/967debb5-8411-4af4-aebb-c33c626e015e-kube-api-access-snkcp\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:37:04.706369 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:37:04.706328 2555 generic.go:356] "Generic (PLEG): container finished" podID="967debb5-8411-4af4-aebb-c33c626e015e" containerID="b0bcd2421756f03f2612c092fee1b7971df0c6b5d95d63c1da75fea10997d5c9" exitCode=0 Jul 06 22:37:04.706585 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:37:04.706379 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-57b5abb4-5f95dcbbf7-xps66" Jul 06 22:37:04.706585 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:37:04.706421 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-57b5abb4-5f95dcbbf7-xps66" event={"ID":"967debb5-8411-4af4-aebb-c33c626e015e","Type":"ContainerDied","Data":"b0bcd2421756f03f2612c092fee1b7971df0c6b5d95d63c1da75fea10997d5c9"} Jul 06 22:37:04.706585 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:37:04.706484 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-57b5abb4-5f95dcbbf7-xps66" event={"ID":"967debb5-8411-4af4-aebb-c33c626e015e","Type":"ContainerDied","Data":"00d69826eca7fe314094a9b31051eafb76baece5afa8f027c4bcad63c014cf75"} Jul 06 22:37:04.706585 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:37:04.706514 2555 scope.go:117] "RemoveContainer" containerID="b0bcd2421756f03f2612c092fee1b7971df0c6b5d95d63c1da75fea10997d5c9" Jul 06 22:37:04.716294 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:37:04.716106 2555 scope.go:117] "RemoveContainer" containerID="b0bcd2421756f03f2612c092fee1b7971df0c6b5d95d63c1da75fea10997d5c9" Jul 06 22:37:04.716505 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:37:04.716354 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0bcd2421756f03f2612c092fee1b7971df0c6b5d95d63c1da75fea10997d5c9\": container with ID starting with b0bcd2421756f03f2612c092fee1b7971df0c6b5d95d63c1da75fea10997d5c9 not found: ID does not exist" containerID="b0bcd2421756f03f2612c092fee1b7971df0c6b5d95d63c1da75fea10997d5c9" Jul 06 22:37:04.716505 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:37:04.716380 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0bcd2421756f03f2612c092fee1b7971df0c6b5d95d63c1da75fea10997d5c9"} err="failed to get container status \"b0bcd2421756f03f2612c092fee1b7971df0c6b5d95d63c1da75fea10997d5c9\": rpc error: code = NotFound desc = could not find container \"b0bcd2421756f03f2612c092fee1b7971df0c6b5d95d63c1da75fea10997d5c9\": container with ID starting with b0bcd2421756f03f2612c092fee1b7971df0c6b5d95d63c1da75fea10997d5c9 not found: ID does not exist" Jul 06 22:37:04.726972 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:37:04.726940 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-57b5abb4-5f95dcbbf7-xps66"] Jul 06 22:37:04.732896 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:37:04.732869 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-57b5abb4-5f95dcbbf7-xps66"] Jul 06 22:37:05.340395 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:37:05.340352 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="967debb5-8411-4af4-aebb-c33c626e015e" path="/var/lib/kubelet/pods/967debb5-8411-4af4-aebb-c33c626e015e/volumes" Jul 06 22:37:32.629836 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:37:32.629751 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-b6d6e159-68bc7fdcb7-cgxth"] Jul 06 22:37:32.630449 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:37:32.630428 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="967debb5-8411-4af4-aebb-c33c626e015e" containerName="maas-api" Jul 06 22:37:32.630511 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:37:32.630454 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="967debb5-8411-4af4-aebb-c33c626e015e" containerName="maas-api" Jul 06 22:37:32.630574 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:37:32.630562 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="967debb5-8411-4af4-aebb-c33c626e015e" containerName="maas-api" Jul 06 22:37:32.634226 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:37:32.634205 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-b6d6e159-68bc7fdcb7-cgxth" Jul 06 22:37:32.636876 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:37:32.636853 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-b6d6e159-serving-cert\"" Jul 06 22:37:32.644579 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:37:32.644554 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-b6d6e159-68bc7fdcb7-cgxth"] Jul 06 22:37:32.644716 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:37:32.644692 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tfbj\" (UniqueName: \"kubernetes.io/projected/33c7b0c9-4a78-496c-b3a6-b2715538a4b6-kube-api-access-5tfbj\") pod \"maas-api-e2e-mt-b6d6e159-68bc7fdcb7-cgxth\" (UID: \"33c7b0c9-4a78-496c-b3a6-b2715538a4b6\") " pod="opendatahub/maas-api-e2e-mt-b6d6e159-68bc7fdcb7-cgxth" Jul 06 22:37:32.644776 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:37:32.644736 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33c7b0c9-4a78-496c-b3a6-b2715538a4b6-openshift-service-ca\") pod \"maas-api-e2e-mt-b6d6e159-68bc7fdcb7-cgxth\" (UID: \"33c7b0c9-4a78-496c-b3a6-b2715538a4b6\") " pod="opendatahub/maas-api-e2e-mt-b6d6e159-68bc7fdcb7-cgxth" Jul 06 22:37:32.644904 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:37:32.644886 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33c7b0c9-4a78-496c-b3a6-b2715538a4b6-maas-api-tls\") pod \"maas-api-e2e-mt-b6d6e159-68bc7fdcb7-cgxth\" (UID: \"33c7b0c9-4a78-496c-b3a6-b2715538a4b6\") " pod="opendatahub/maas-api-e2e-mt-b6d6e159-68bc7fdcb7-cgxth" Jul 06 22:37:32.745516 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:37:32.745481 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tfbj\" (UniqueName: \"kubernetes.io/projected/33c7b0c9-4a78-496c-b3a6-b2715538a4b6-kube-api-access-5tfbj\") pod \"maas-api-e2e-mt-b6d6e159-68bc7fdcb7-cgxth\" (UID: \"33c7b0c9-4a78-496c-b3a6-b2715538a4b6\") " pod="opendatahub/maas-api-e2e-mt-b6d6e159-68bc7fdcb7-cgxth" Jul 06 22:37:32.745516 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:37:32.745521 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33c7b0c9-4a78-496c-b3a6-b2715538a4b6-openshift-service-ca\") pod \"maas-api-e2e-mt-b6d6e159-68bc7fdcb7-cgxth\" (UID: \"33c7b0c9-4a78-496c-b3a6-b2715538a4b6\") " pod="opendatahub/maas-api-e2e-mt-b6d6e159-68bc7fdcb7-cgxth" Jul 06 22:37:32.745813 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:37:32.745577 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33c7b0c9-4a78-496c-b3a6-b2715538a4b6-maas-api-tls\") pod \"maas-api-e2e-mt-b6d6e159-68bc7fdcb7-cgxth\" (UID: \"33c7b0c9-4a78-496c-b3a6-b2715538a4b6\") " pod="opendatahub/maas-api-e2e-mt-b6d6e159-68bc7fdcb7-cgxth" Jul 06 22:37:32.746242 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:37:32.746215 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33c7b0c9-4a78-496c-b3a6-b2715538a4b6-openshift-service-ca\") pod \"maas-api-e2e-mt-b6d6e159-68bc7fdcb7-cgxth\" (UID: \"33c7b0c9-4a78-496c-b3a6-b2715538a4b6\") " pod="opendatahub/maas-api-e2e-mt-b6d6e159-68bc7fdcb7-cgxth" Jul 06 22:37:32.748116 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:37:32.748092 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33c7b0c9-4a78-496c-b3a6-b2715538a4b6-maas-api-tls\") pod \"maas-api-e2e-mt-b6d6e159-68bc7fdcb7-cgxth\" (UID: \"33c7b0c9-4a78-496c-b3a6-b2715538a4b6\") " pod="opendatahub/maas-api-e2e-mt-b6d6e159-68bc7fdcb7-cgxth" Jul 06 22:37:32.757262 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:37:32.757234 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tfbj\" (UniqueName: \"kubernetes.io/projected/33c7b0c9-4a78-496c-b3a6-b2715538a4b6-kube-api-access-5tfbj\") pod \"maas-api-e2e-mt-b6d6e159-68bc7fdcb7-cgxth\" (UID: \"33c7b0c9-4a78-496c-b3a6-b2715538a4b6\") " pod="opendatahub/maas-api-e2e-mt-b6d6e159-68bc7fdcb7-cgxth" Jul 06 22:37:32.946814 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:37:32.946766 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-b6d6e159-68bc7fdcb7-cgxth" Jul 06 22:37:33.082025 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:37:33.081983 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-b6d6e159-68bc7fdcb7-cgxth"] Jul 06 22:37:33.084221 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:37:33.084191 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33c7b0c9_4a78_496c_b3a6_b2715538a4b6.slice/crio-55987cb12357b3509318f1ac0487660578835f89aad3f74d3658d57ede8dcd5b WatchSource:0}: Error finding container 55987cb12357b3509318f1ac0487660578835f89aad3f74d3658d57ede8dcd5b: Status 404 returned error can't find the container with id 55987cb12357b3509318f1ac0487660578835f89aad3f74d3658d57ede8dcd5b Jul 06 22:37:33.859837 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:37:33.859801 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-b6d6e159-68bc7fdcb7-cgxth" event={"ID":"33c7b0c9-4a78-496c-b3a6-b2715538a4b6","Type":"ContainerStarted","Data":"13bcfac2db0b89f688850b5d6be3faadc73430151e6faf0165bab24bfdcee555"} Jul 06 22:37:33.860276 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:37:33.859860 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-b6d6e159-68bc7fdcb7-cgxth" event={"ID":"33c7b0c9-4a78-496c-b3a6-b2715538a4b6","Type":"ContainerStarted","Data":"55987cb12357b3509318f1ac0487660578835f89aad3f74d3658d57ede8dcd5b"} Jul 06 22:37:33.860276 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:37:33.859886 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-b6d6e159-68bc7fdcb7-cgxth" Jul 06 22:37:33.878738 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:37:33.878681 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-b6d6e159-68bc7fdcb7-cgxth" podStartSLOduration=1.576799394 podStartE2EDuration="1.878664938s" totalImagesPullingTime="301.865544ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:37:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:37:33.87675911 +0000 UTC m=+1705.176470958" watchObservedRunningTime="2026-07-06 22:37:33.878664938 +0000 UTC m=+1705.178376787" Jul 06 22:37:39.869704 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:37:39.869674 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-b6d6e159-68bc7fdcb7-cgxth" Jul 06 22:38:00.589958 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:00.589917 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-b6d6e159/e2e-lifecycle-model-b6d6e159-kserve-75548fc499-mrvkl"] Jul 06 22:38:00.594713 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:00.594681 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b6d6e159/e2e-lifecycle-model-b6d6e159-kserve-75548fc499-mrvkl" Jul 06 22:38:00.599824 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:00.599799 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-b6d6e159\"/\"default-dockercfg-7xxws\"" Jul 06 22:38:00.599996 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:00.599843 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-b6d6e159\"/\"kube-root-ca.crt\"" Jul 06 22:38:00.599996 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:00.599800 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-b6d6e159\"/\"e2e-lifecycle-model-b6d6e159-kserve-self-signed-certs\"" Jul 06 22:38:00.600215 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:00.599799 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-b6d6e159\"/\"openshift-service-ca.crt\"" Jul 06 22:38:00.602956 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:00.602926 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-b6d6e159/e2e-lifecycle-model-b6d6e159-kserve-75548fc499-mrvkl"] Jul 06 22:38:00.701394 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:00.701361 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nj8k\" (UniqueName: \"kubernetes.io/projected/e773195e-b1f7-4911-abd3-99684ff05bcf-kube-api-access-4nj8k\") pod \"e2e-lifecycle-model-b6d6e159-kserve-75548fc499-mrvkl\" (UID: \"e773195e-b1f7-4911-abd3-99684ff05bcf\") " pod="ai-tenant-e2e-mt-b6d6e159/e2e-lifecycle-model-b6d6e159-kserve-75548fc499-mrvkl" Jul 06 22:38:00.701394 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:00.701408 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e773195e-b1f7-4911-abd3-99684ff05bcf-tls-certs\") pod \"e2e-lifecycle-model-b6d6e159-kserve-75548fc499-mrvkl\" (UID: \"e773195e-b1f7-4911-abd3-99684ff05bcf\") " pod="ai-tenant-e2e-mt-b6d6e159/e2e-lifecycle-model-b6d6e159-kserve-75548fc499-mrvkl" Jul 06 22:38:00.701663 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:00.701430 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e773195e-b1f7-4911-abd3-99684ff05bcf-dshm\") pod \"e2e-lifecycle-model-b6d6e159-kserve-75548fc499-mrvkl\" (UID: \"e773195e-b1f7-4911-abd3-99684ff05bcf\") " pod="ai-tenant-e2e-mt-b6d6e159/e2e-lifecycle-model-b6d6e159-kserve-75548fc499-mrvkl" Jul 06 22:38:00.701663 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:00.701480 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e773195e-b1f7-4911-abd3-99684ff05bcf-tmp-dir\") pod \"e2e-lifecycle-model-b6d6e159-kserve-75548fc499-mrvkl\" (UID: \"e773195e-b1f7-4911-abd3-99684ff05bcf\") " pod="ai-tenant-e2e-mt-b6d6e159/e2e-lifecycle-model-b6d6e159-kserve-75548fc499-mrvkl" Jul 06 22:38:00.701663 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:00.701506 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e773195e-b1f7-4911-abd3-99684ff05bcf-model-cache\") pod \"e2e-lifecycle-model-b6d6e159-kserve-75548fc499-mrvkl\" (UID: \"e773195e-b1f7-4911-abd3-99684ff05bcf\") " pod="ai-tenant-e2e-mt-b6d6e159/e2e-lifecycle-model-b6d6e159-kserve-75548fc499-mrvkl" Jul 06 22:38:00.701663 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:00.701595 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e773195e-b1f7-4911-abd3-99684ff05bcf-home\") pod \"e2e-lifecycle-model-b6d6e159-kserve-75548fc499-mrvkl\" (UID: \"e773195e-b1f7-4911-abd3-99684ff05bcf\") " pod="ai-tenant-e2e-mt-b6d6e159/e2e-lifecycle-model-b6d6e159-kserve-75548fc499-mrvkl" Jul 06 22:38:00.803152 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:00.803111 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nj8k\" (UniqueName: \"kubernetes.io/projected/e773195e-b1f7-4911-abd3-99684ff05bcf-kube-api-access-4nj8k\") pod \"e2e-lifecycle-model-b6d6e159-kserve-75548fc499-mrvkl\" (UID: \"e773195e-b1f7-4911-abd3-99684ff05bcf\") " pod="ai-tenant-e2e-mt-b6d6e159/e2e-lifecycle-model-b6d6e159-kserve-75548fc499-mrvkl" Jul 06 22:38:00.803354 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:00.803169 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e773195e-b1f7-4911-abd3-99684ff05bcf-tls-certs\") pod \"e2e-lifecycle-model-b6d6e159-kserve-75548fc499-mrvkl\" (UID: \"e773195e-b1f7-4911-abd3-99684ff05bcf\") " pod="ai-tenant-e2e-mt-b6d6e159/e2e-lifecycle-model-b6d6e159-kserve-75548fc499-mrvkl" Jul 06 22:38:00.803354 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:00.803204 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e773195e-b1f7-4911-abd3-99684ff05bcf-dshm\") pod \"e2e-lifecycle-model-b6d6e159-kserve-75548fc499-mrvkl\" (UID: \"e773195e-b1f7-4911-abd3-99684ff05bcf\") " pod="ai-tenant-e2e-mt-b6d6e159/e2e-lifecycle-model-b6d6e159-kserve-75548fc499-mrvkl" Jul 06 22:38:00.803354 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:00.803235 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e773195e-b1f7-4911-abd3-99684ff05bcf-tmp-dir\") pod \"e2e-lifecycle-model-b6d6e159-kserve-75548fc499-mrvkl\" (UID: \"e773195e-b1f7-4911-abd3-99684ff05bcf\") " pod="ai-tenant-e2e-mt-b6d6e159/e2e-lifecycle-model-b6d6e159-kserve-75548fc499-mrvkl" Jul 06 22:38:00.803354 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:00.803276 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e773195e-b1f7-4911-abd3-99684ff05bcf-model-cache\") pod \"e2e-lifecycle-model-b6d6e159-kserve-75548fc499-mrvkl\" (UID: \"e773195e-b1f7-4911-abd3-99684ff05bcf\") " pod="ai-tenant-e2e-mt-b6d6e159/e2e-lifecycle-model-b6d6e159-kserve-75548fc499-mrvkl" Jul 06 22:38:00.803354 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:00.803327 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e773195e-b1f7-4911-abd3-99684ff05bcf-home\") pod \"e2e-lifecycle-model-b6d6e159-kserve-75548fc499-mrvkl\" (UID: \"e773195e-b1f7-4911-abd3-99684ff05bcf\") " pod="ai-tenant-e2e-mt-b6d6e159/e2e-lifecycle-model-b6d6e159-kserve-75548fc499-mrvkl" Jul 06 22:38:00.803747 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:00.803718 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e773195e-b1f7-4911-abd3-99684ff05bcf-tmp-dir\") pod \"e2e-lifecycle-model-b6d6e159-kserve-75548fc499-mrvkl\" (UID: \"e773195e-b1f7-4911-abd3-99684ff05bcf\") " pod="ai-tenant-e2e-mt-b6d6e159/e2e-lifecycle-model-b6d6e159-kserve-75548fc499-mrvkl" Jul 06 22:38:00.803747 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:00.803733 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e773195e-b1f7-4911-abd3-99684ff05bcf-home\") pod \"e2e-lifecycle-model-b6d6e159-kserve-75548fc499-mrvkl\" (UID: \"e773195e-b1f7-4911-abd3-99684ff05bcf\") " pod="ai-tenant-e2e-mt-b6d6e159/e2e-lifecycle-model-b6d6e159-kserve-75548fc499-mrvkl" Jul 06 22:38:00.803955 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:00.803786 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e773195e-b1f7-4911-abd3-99684ff05bcf-model-cache\") pod \"e2e-lifecycle-model-b6d6e159-kserve-75548fc499-mrvkl\" (UID: \"e773195e-b1f7-4911-abd3-99684ff05bcf\") " pod="ai-tenant-e2e-mt-b6d6e159/e2e-lifecycle-model-b6d6e159-kserve-75548fc499-mrvkl" Jul 06 22:38:00.805546 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:00.805521 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e773195e-b1f7-4911-abd3-99684ff05bcf-dshm\") pod \"e2e-lifecycle-model-b6d6e159-kserve-75548fc499-mrvkl\" (UID: \"e773195e-b1f7-4911-abd3-99684ff05bcf\") " pod="ai-tenant-e2e-mt-b6d6e159/e2e-lifecycle-model-b6d6e159-kserve-75548fc499-mrvkl" Jul 06 22:38:00.805670 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:00.805652 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e773195e-b1f7-4911-abd3-99684ff05bcf-tls-certs\") pod \"e2e-lifecycle-model-b6d6e159-kserve-75548fc499-mrvkl\" (UID: \"e773195e-b1f7-4911-abd3-99684ff05bcf\") " pod="ai-tenant-e2e-mt-b6d6e159/e2e-lifecycle-model-b6d6e159-kserve-75548fc499-mrvkl" Jul 06 22:38:00.811261 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:00.811236 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nj8k\" (UniqueName: \"kubernetes.io/projected/e773195e-b1f7-4911-abd3-99684ff05bcf-kube-api-access-4nj8k\") pod \"e2e-lifecycle-model-b6d6e159-kserve-75548fc499-mrvkl\" (UID: \"e773195e-b1f7-4911-abd3-99684ff05bcf\") " pod="ai-tenant-e2e-mt-b6d6e159/e2e-lifecycle-model-b6d6e159-kserve-75548fc499-mrvkl" Jul 06 22:38:00.909237 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:00.909202 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b6d6e159/e2e-lifecycle-model-b6d6e159-kserve-75548fc499-mrvkl" Jul 06 22:38:01.045591 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:01.045564 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-b6d6e159/e2e-lifecycle-model-b6d6e159-kserve-75548fc499-mrvkl"] Jul 06 22:38:01.047548 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:38:01.047513 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode773195e_b1f7_4911_abd3_99684ff05bcf.slice/crio-830f40d17c14033637507b7f732091780f9e20a030896e6700092ebbeb7b8f45 WatchSource:0}: Error finding container 830f40d17c14033637507b7f732091780f9e20a030896e6700092ebbeb7b8f45: Status 404 returned error can't find the container with id 830f40d17c14033637507b7f732091780f9e20a030896e6700092ebbeb7b8f45 Jul 06 22:38:01.999085 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:01.999049 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b6d6e159/e2e-lifecycle-model-b6d6e159-kserve-75548fc499-mrvkl" event={"ID":"e773195e-b1f7-4911-abd3-99684ff05bcf","Type":"ContainerStarted","Data":"d2af114e46f8f1f838e30967cee688f1e68dc50696e81d3926557a45b4323281"} Jul 06 22:38:01.999085 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:01.999095 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b6d6e159/e2e-lifecycle-model-b6d6e159-kserve-75548fc499-mrvkl" event={"ID":"e773195e-b1f7-4911-abd3-99684ff05bcf","Type":"ContainerStarted","Data":"830f40d17c14033637507b7f732091780f9e20a030896e6700092ebbeb7b8f45"} Jul 06 22:38:02.017902 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:02.017840 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-b6d6e159/e2e-lifecycle-model-b6d6e159-kserve-75548fc499-mrvkl" podStartSLOduration=2.01781855 podStartE2EDuration="2.01781855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:38:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:38:02.01681758 +0000 UTC m=+1733.316529434" watchObservedRunningTime="2026-07-06 22:38:02.01781855 +0000 UTC m=+1733.317530401" Jul 06 22:38:10.909964 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:10.909912 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-b6d6e159/e2e-lifecycle-model-b6d6e159-kserve-75548fc499-mrvkl" Jul 06 22:38:10.909964 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:10.909962 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-b6d6e159/e2e-lifecycle-model-b6d6e159-kserve-75548fc499-mrvkl" Jul 06 22:38:10.922488 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:10.922462 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-b6d6e159/e2e-lifecycle-model-b6d6e159-kserve-75548fc499-mrvkl" Jul 06 22:38:11.054992 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:11.054960 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-b6d6e159/e2e-lifecycle-model-b6d6e159-kserve-75548fc499-mrvkl" Jul 06 22:38:30.674488 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:30.674455 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-b6d6e159-68bc7fdcb7-cgxth"] Jul 06 22:38:30.675046 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:30.674654 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-b6d6e159-68bc7fdcb7-cgxth" podUID="33c7b0c9-4a78-496c-b3a6-b2715538a4b6" containerName="maas-api" containerID="cri-o://13bcfac2db0b89f688850b5d6be3faadc73430151e6faf0165bab24bfdcee555" gracePeriod=30 Jul 06 22:38:30.926823 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:30.926760 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-b6d6e159-68bc7fdcb7-cgxth" Jul 06 22:38:30.999242 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:30.999202 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33c7b0c9-4a78-496c-b3a6-b2715538a4b6-openshift-service-ca\") pod \"33c7b0c9-4a78-496c-b3a6-b2715538a4b6\" (UID: \"33c7b0c9-4a78-496c-b3a6-b2715538a4b6\") " Jul 06 22:38:30.999444 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:30.999279 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33c7b0c9-4a78-496c-b3a6-b2715538a4b6-maas-api-tls\") pod \"33c7b0c9-4a78-496c-b3a6-b2715538a4b6\" (UID: \"33c7b0c9-4a78-496c-b3a6-b2715538a4b6\") " Jul 06 22:38:30.999444 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:30.999388 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5tfbj\" (UniqueName: \"kubernetes.io/projected/33c7b0c9-4a78-496c-b3a6-b2715538a4b6-kube-api-access-5tfbj\") pod \"33c7b0c9-4a78-496c-b3a6-b2715538a4b6\" (UID: \"33c7b0c9-4a78-496c-b3a6-b2715538a4b6\") " Jul 06 22:38:30.999601 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:30.999574 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33c7b0c9-4a78-496c-b3a6-b2715538a4b6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "33c7b0c9-4a78-496c-b3a6-b2715538a4b6" (UID: "33c7b0c9-4a78-496c-b3a6-b2715538a4b6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:38:30.999750 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:30.999732 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33c7b0c9-4a78-496c-b3a6-b2715538a4b6-openshift-service-ca\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:38:31.001384 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:31.001362 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33c7b0c9-4a78-496c-b3a6-b2715538a4b6-kube-api-access-5tfbj" (OuterVolumeSpecName: "kube-api-access-5tfbj") pod "33c7b0c9-4a78-496c-b3a6-b2715538a4b6" (UID: "33c7b0c9-4a78-496c-b3a6-b2715538a4b6"). InnerVolumeSpecName "kube-api-access-5tfbj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:38:31.001516 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:31.001422 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33c7b0c9-4a78-496c-b3a6-b2715538a4b6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "33c7b0c9-4a78-496c-b3a6-b2715538a4b6" (UID: "33c7b0c9-4a78-496c-b3a6-b2715538a4b6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:38:31.100722 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:31.100684 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tfbj\" (UniqueName: \"kubernetes.io/projected/33c7b0c9-4a78-496c-b3a6-b2715538a4b6-kube-api-access-5tfbj\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:38:31.100722 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:31.100714 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33c7b0c9-4a78-496c-b3a6-b2715538a4b6-maas-api-tls\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:38:31.144455 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:31.144422 2555 generic.go:356] "Generic (PLEG): container finished" podID="33c7b0c9-4a78-496c-b3a6-b2715538a4b6" containerID="13bcfac2db0b89f688850b5d6be3faadc73430151e6faf0165bab24bfdcee555" exitCode=0 Jul 06 22:38:31.144619 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:31.144481 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-b6d6e159-68bc7fdcb7-cgxth" Jul 06 22:38:31.144619 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:31.144491 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-b6d6e159-68bc7fdcb7-cgxth" event={"ID":"33c7b0c9-4a78-496c-b3a6-b2715538a4b6","Type":"ContainerDied","Data":"13bcfac2db0b89f688850b5d6be3faadc73430151e6faf0165bab24bfdcee555"} Jul 06 22:38:31.144619 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:31.144533 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-b6d6e159-68bc7fdcb7-cgxth" event={"ID":"33c7b0c9-4a78-496c-b3a6-b2715538a4b6","Type":"ContainerDied","Data":"55987cb12357b3509318f1ac0487660578835f89aad3f74d3658d57ede8dcd5b"} Jul 06 22:38:31.144619 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:31.144556 2555 scope.go:117] "RemoveContainer" containerID="13bcfac2db0b89f688850b5d6be3faadc73430151e6faf0165bab24bfdcee555" Jul 06 22:38:31.154458 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:31.154442 2555 scope.go:117] "RemoveContainer" containerID="13bcfac2db0b89f688850b5d6be3faadc73430151e6faf0165bab24bfdcee555" Jul 06 22:38:31.154733 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:38:31.154704 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13bcfac2db0b89f688850b5d6be3faadc73430151e6faf0165bab24bfdcee555\": container with ID starting with 13bcfac2db0b89f688850b5d6be3faadc73430151e6faf0165bab24bfdcee555 not found: ID does not exist" containerID="13bcfac2db0b89f688850b5d6be3faadc73430151e6faf0165bab24bfdcee555" Jul 06 22:38:31.154800 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:31.154739 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13bcfac2db0b89f688850b5d6be3faadc73430151e6faf0165bab24bfdcee555"} err="failed to get container status \"13bcfac2db0b89f688850b5d6be3faadc73430151e6faf0165bab24bfdcee555\": rpc error: code = NotFound desc = could not find container \"13bcfac2db0b89f688850b5d6be3faadc73430151e6faf0165bab24bfdcee555\": container with ID starting with 13bcfac2db0b89f688850b5d6be3faadc73430151e6faf0165bab24bfdcee555 not found: ID does not exist" Jul 06 22:38:31.165533 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:31.165501 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-b6d6e159-68bc7fdcb7-cgxth"] Jul 06 22:38:31.167308 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:31.167287 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-b6d6e159-68bc7fdcb7-cgxth"] Jul 06 22:38:31.339849 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:31.339761 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33c7b0c9-4a78-496c-b3a6-b2715538a4b6" path="/var/lib/kubelet/pods/33c7b0c9-4a78-496c-b3a6-b2715538a4b6/volumes" Jul 06 22:38:41.107371 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:41.107335 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-b6d6e159/e2e-lifecycle-model-b6d6e159-kserve-75548fc499-mrvkl"] Jul 06 22:38:56.127938 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:56.127874 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-mt-b6d6e159/e2e-lifecycle-model-b6d6e159-kserve-75548fc499-mrvkl" podUID="e773195e-b1f7-4911-abd3-99684ff05bcf" containerName="main" containerID="cri-o://d2af114e46f8f1f838e30967cee688f1e68dc50696e81d3926557a45b4323281" gracePeriod=45 Jul 06 22:38:56.279637 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:56.279604 2555 generic.go:356] "Generic (PLEG): container finished" podID="e773195e-b1f7-4911-abd3-99684ff05bcf" containerID="d2af114e46f8f1f838e30967cee688f1e68dc50696e81d3926557a45b4323281" exitCode=0 Jul 06 22:38:56.279818 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:56.279673 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b6d6e159/e2e-lifecycle-model-b6d6e159-kserve-75548fc499-mrvkl" event={"ID":"e773195e-b1f7-4911-abd3-99684ff05bcf","Type":"ContainerDied","Data":"d2af114e46f8f1f838e30967cee688f1e68dc50696e81d3926557a45b4323281"} Jul 06 22:38:56.379027 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:56.378949 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b6d6e159/e2e-lifecycle-model-b6d6e159-kserve-75548fc499-mrvkl" Jul 06 22:38:56.449151 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:56.449122 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e773195e-b1f7-4911-abd3-99684ff05bcf-dshm\") pod \"e773195e-b1f7-4911-abd3-99684ff05bcf\" (UID: \"e773195e-b1f7-4911-abd3-99684ff05bcf\") " Jul 06 22:38:56.449343 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:56.449193 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e773195e-b1f7-4911-abd3-99684ff05bcf-tmp-dir\") pod \"e773195e-b1f7-4911-abd3-99684ff05bcf\" (UID: \"e773195e-b1f7-4911-abd3-99684ff05bcf\") " Jul 06 22:38:56.449343 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:56.449229 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e773195e-b1f7-4911-abd3-99684ff05bcf-tls-certs\") pod \"e773195e-b1f7-4911-abd3-99684ff05bcf\" (UID: \"e773195e-b1f7-4911-abd3-99684ff05bcf\") " Jul 06 22:38:56.449343 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:56.449267 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e773195e-b1f7-4911-abd3-99684ff05bcf-home\") pod \"e773195e-b1f7-4911-abd3-99684ff05bcf\" (UID: \"e773195e-b1f7-4911-abd3-99684ff05bcf\") " Jul 06 22:38:56.449343 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:56.449328 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4nj8k\" (UniqueName: \"kubernetes.io/projected/e773195e-b1f7-4911-abd3-99684ff05bcf-kube-api-access-4nj8k\") pod \"e773195e-b1f7-4911-abd3-99684ff05bcf\" (UID: \"e773195e-b1f7-4911-abd3-99684ff05bcf\") " Jul 06 22:38:56.449560 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:56.449360 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e773195e-b1f7-4911-abd3-99684ff05bcf-model-cache\") pod \"e773195e-b1f7-4911-abd3-99684ff05bcf\" (UID: \"e773195e-b1f7-4911-abd3-99684ff05bcf\") " Jul 06 22:38:56.449801 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:56.449768 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e773195e-b1f7-4911-abd3-99684ff05bcf-home" (OuterVolumeSpecName: "home") pod "e773195e-b1f7-4911-abd3-99684ff05bcf" (UID: "e773195e-b1f7-4911-abd3-99684ff05bcf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:38:56.449801 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:56.449781 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e773195e-b1f7-4911-abd3-99684ff05bcf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e773195e-b1f7-4911-abd3-99684ff05bcf" (UID: "e773195e-b1f7-4911-abd3-99684ff05bcf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:38:56.449965 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:56.449901 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e773195e-b1f7-4911-abd3-99684ff05bcf-model-cache" (OuterVolumeSpecName: "model-cache") pod "e773195e-b1f7-4911-abd3-99684ff05bcf" (UID: "e773195e-b1f7-4911-abd3-99684ff05bcf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:38:56.451701 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:56.451671 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e773195e-b1f7-4911-abd3-99684ff05bcf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e773195e-b1f7-4911-abd3-99684ff05bcf" (UID: "e773195e-b1f7-4911-abd3-99684ff05bcf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:38:56.451802 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:56.451756 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e773195e-b1f7-4911-abd3-99684ff05bcf-dshm" (OuterVolumeSpecName: "dshm") pod "e773195e-b1f7-4911-abd3-99684ff05bcf" (UID: "e773195e-b1f7-4911-abd3-99684ff05bcf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:38:56.451866 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:56.451830 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e773195e-b1f7-4911-abd3-99684ff05bcf-kube-api-access-4nj8k" (OuterVolumeSpecName: "kube-api-access-4nj8k") pod "e773195e-b1f7-4911-abd3-99684ff05bcf" (UID: "e773195e-b1f7-4911-abd3-99684ff05bcf"). InnerVolumeSpecName "kube-api-access-4nj8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:38:56.550288 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:56.550249 2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e773195e-b1f7-4911-abd3-99684ff05bcf-dshm\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:38:56.550288 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:56.550281 2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e773195e-b1f7-4911-abd3-99684ff05bcf-tmp-dir\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:38:56.550288 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:56.550290 2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e773195e-b1f7-4911-abd3-99684ff05bcf-tls-certs\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:38:56.550288 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:56.550298 2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e773195e-b1f7-4911-abd3-99684ff05bcf-home\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:38:56.550566 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:56.550307 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nj8k\" (UniqueName: \"kubernetes.io/projected/e773195e-b1f7-4911-abd3-99684ff05bcf-kube-api-access-4nj8k\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:38:56.550566 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:56.550317 2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e773195e-b1f7-4911-abd3-99684ff05bcf-model-cache\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:38:57.285848 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:57.285810 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b6d6e159/e2e-lifecycle-model-b6d6e159-kserve-75548fc499-mrvkl" event={"ID":"e773195e-b1f7-4911-abd3-99684ff05bcf","Type":"ContainerDied","Data":"830f40d17c14033637507b7f732091780f9e20a030896e6700092ebbeb7b8f45"} Jul 06 22:38:57.285848 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:57.285836 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b6d6e159/e2e-lifecycle-model-b6d6e159-kserve-75548fc499-mrvkl" Jul 06 22:38:57.286378 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:57.285864 2555 scope.go:117] "RemoveContainer" containerID="d2af114e46f8f1f838e30967cee688f1e68dc50696e81d3926557a45b4323281" Jul 06 22:38:57.308572 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:57.308543 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-b6d6e159/e2e-lifecycle-model-b6d6e159-kserve-75548fc499-mrvkl"] Jul 06 22:38:57.313098 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:57.313074 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-b6d6e159/e2e-lifecycle-model-b6d6e159-kserve-75548fc499-mrvkl"] Jul 06 22:38:57.340113 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:38:57.340082 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e773195e-b1f7-4911-abd3-99684ff05bcf" path="/var/lib/kubelet/pods/e773195e-b1f7-4911-abd3-99684ff05bcf/volumes" Jul 06 22:39:09.312274 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:39:09.312150 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7mq8_e50d3b5f-832e-481f-8807-7774ff8d5521/ovn-acl-logging/0.log" Jul 06 22:39:09.319219 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:39:09.315100 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7mq8_e50d3b5f-832e-481f-8807-7774ff8d5521/ovn-acl-logging/0.log" Jul 06 22:40:22.595057 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:40:22.595016 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-a-17b26a-6b7c9d9dcb-jqzb8"] Jul 06 22:40:22.595562 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:40:22.595426 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33c7b0c9-4a78-496c-b3a6-b2715538a4b6" containerName="maas-api" Jul 06 22:40:22.595562 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:40:22.595440 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="33c7b0c9-4a78-496c-b3a6-b2715538a4b6" containerName="maas-api" Jul 06 22:40:22.595562 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:40:22.595447 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e773195e-b1f7-4911-abd3-99684ff05bcf" containerName="main" Jul 06 22:40:22.595562 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:40:22.595453 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e773195e-b1f7-4911-abd3-99684ff05bcf" containerName="main" Jul 06 22:40:22.595562 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:40:22.595524 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e773195e-b1f7-4911-abd3-99684ff05bcf" containerName="main" Jul 06 22:40:22.595562 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:40:22.595548 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="33c7b0c9-4a78-496c-b3a6-b2715538a4b6" containerName="maas-api" Jul 06 22:40:22.598798 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:40:22.598777 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-17b26a-6b7c9d9dcb-jqzb8" Jul 06 22:40:22.601510 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:40:22.601484 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-a-17b26a-serving-cert\"" Jul 06 22:40:22.610647 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:40:22.610615 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-17b26a-6b7c9d9dcb-jqzb8"] Jul 06 22:40:22.677749 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:40:22.677712 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c6d3a47-96ca-47bd-b0c9-6785b321f60d-maas-api-tls\") pod \"maas-api-e2e-inf-a-17b26a-6b7c9d9dcb-jqzb8\" (UID: \"3c6d3a47-96ca-47bd-b0c9-6785b321f60d\") " pod="opendatahub/maas-api-e2e-inf-a-17b26a-6b7c9d9dcb-jqzb8" Jul 06 22:40:22.677949 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:40:22.677768 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rmls\" (UniqueName: \"kubernetes.io/projected/3c6d3a47-96ca-47bd-b0c9-6785b321f60d-kube-api-access-8rmls\") pod \"maas-api-e2e-inf-a-17b26a-6b7c9d9dcb-jqzb8\" (UID: \"3c6d3a47-96ca-47bd-b0c9-6785b321f60d\") " pod="opendatahub/maas-api-e2e-inf-a-17b26a-6b7c9d9dcb-jqzb8" Jul 06 22:40:22.677949 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:40:22.677845 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c6d3a47-96ca-47bd-b0c9-6785b321f60d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-17b26a-6b7c9d9dcb-jqzb8\" (UID: \"3c6d3a47-96ca-47bd-b0c9-6785b321f60d\") " pod="opendatahub/maas-api-e2e-inf-a-17b26a-6b7c9d9dcb-jqzb8" Jul 06 22:40:22.779207 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:40:22.779171 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rmls\" (UniqueName: \"kubernetes.io/projected/3c6d3a47-96ca-47bd-b0c9-6785b321f60d-kube-api-access-8rmls\") pod \"maas-api-e2e-inf-a-17b26a-6b7c9d9dcb-jqzb8\" (UID: \"3c6d3a47-96ca-47bd-b0c9-6785b321f60d\") " pod="opendatahub/maas-api-e2e-inf-a-17b26a-6b7c9d9dcb-jqzb8" Jul 06 22:40:22.779415 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:40:22.779264 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c6d3a47-96ca-47bd-b0c9-6785b321f60d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-17b26a-6b7c9d9dcb-jqzb8\" (UID: \"3c6d3a47-96ca-47bd-b0c9-6785b321f60d\") " pod="opendatahub/maas-api-e2e-inf-a-17b26a-6b7c9d9dcb-jqzb8" Jul 06 22:40:22.779415 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:40:22.779372 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c6d3a47-96ca-47bd-b0c9-6785b321f60d-maas-api-tls\") pod \"maas-api-e2e-inf-a-17b26a-6b7c9d9dcb-jqzb8\" (UID: \"3c6d3a47-96ca-47bd-b0c9-6785b321f60d\") " pod="opendatahub/maas-api-e2e-inf-a-17b26a-6b7c9d9dcb-jqzb8" Jul 06 22:40:22.779549 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:40:22.779483 2555 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-inf-a-17b26a-serving-cert: secret "maas-api-e2e-inf-a-17b26a-serving-cert" not found Jul 06 22:40:22.779603 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:40:22.779558 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c6d3a47-96ca-47bd-b0c9-6785b321f60d-maas-api-tls podName:3c6d3a47-96ca-47bd-b0c9-6785b321f60d nodeName:}" failed. No retries permitted until 2026-07-06 22:40:23.279533876 +0000 UTC m=+1874.579245721 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3c6d3a47-96ca-47bd-b0c9-6785b321f60d-maas-api-tls") pod "maas-api-e2e-inf-a-17b26a-6b7c9d9dcb-jqzb8" (UID: "3c6d3a47-96ca-47bd-b0c9-6785b321f60d") : secret "maas-api-e2e-inf-a-17b26a-serving-cert" not found Jul 06 22:40:22.779880 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:40:22.779857 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c6d3a47-96ca-47bd-b0c9-6785b321f60d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-17b26a-6b7c9d9dcb-jqzb8\" (UID: \"3c6d3a47-96ca-47bd-b0c9-6785b321f60d\") " pod="opendatahub/maas-api-e2e-inf-a-17b26a-6b7c9d9dcb-jqzb8" Jul 06 22:40:22.788489 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:40:22.788460 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rmls\" (UniqueName: \"kubernetes.io/projected/3c6d3a47-96ca-47bd-b0c9-6785b321f60d-kube-api-access-8rmls\") pod \"maas-api-e2e-inf-a-17b26a-6b7c9d9dcb-jqzb8\" (UID: \"3c6d3a47-96ca-47bd-b0c9-6785b321f60d\") " pod="opendatahub/maas-api-e2e-inf-a-17b26a-6b7c9d9dcb-jqzb8" Jul 06 22:40:23.283838 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:40:23.283796 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c6d3a47-96ca-47bd-b0c9-6785b321f60d-maas-api-tls\") pod \"maas-api-e2e-inf-a-17b26a-6b7c9d9dcb-jqzb8\" (UID: \"3c6d3a47-96ca-47bd-b0c9-6785b321f60d\") " pod="opendatahub/maas-api-e2e-inf-a-17b26a-6b7c9d9dcb-jqzb8" Jul 06 22:40:23.286404 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:40:23.286377 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c6d3a47-96ca-47bd-b0c9-6785b321f60d-maas-api-tls\") pod \"maas-api-e2e-inf-a-17b26a-6b7c9d9dcb-jqzb8\" (UID: \"3c6d3a47-96ca-47bd-b0c9-6785b321f60d\") " pod="opendatahub/maas-api-e2e-inf-a-17b26a-6b7c9d9dcb-jqzb8" Jul 06 22:40:23.509927 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:40:23.509883 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-17b26a-6b7c9d9dcb-jqzb8" Jul 06 22:40:23.656099 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:40:23.656073 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-17b26a-6b7c9d9dcb-jqzb8"] Jul 06 22:40:23.657974 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:40:23.657947 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c6d3a47_96ca_47bd_b0c9_6785b321f60d.slice/crio-c1cd4e1077e0d15ed38db0729f24edeaf9784f15e7ae2d881d17e3a40b474a74 WatchSource:0}: Error finding container c1cd4e1077e0d15ed38db0729f24edeaf9784f15e7ae2d881d17e3a40b474a74: Status 404 returned error can't find the container with id c1cd4e1077e0d15ed38db0729f24edeaf9784f15e7ae2d881d17e3a40b474a74 Jul 06 22:40:23.659420 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:40:23.659403 2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 22:40:23.704387 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:40:23.704342 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-17b26a-6b7c9d9dcb-jqzb8" event={"ID":"3c6d3a47-96ca-47bd-b0c9-6785b321f60d","Type":"ContainerStarted","Data":"c1cd4e1077e0d15ed38db0729f24edeaf9784f15e7ae2d881d17e3a40b474a74"} Jul 06 22:40:24.710328 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:40:24.710295 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-17b26a-6b7c9d9dcb-jqzb8" event={"ID":"3c6d3a47-96ca-47bd-b0c9-6785b321f60d","Type":"ContainerStarted","Data":"0fc33abedd451b96fa6d146a4db0a3aab36398ebbe467d6b63e83bec05bdf83a"} Jul 06 22:40:24.710744 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:40:24.710406 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-a-17b26a-6b7c9d9dcb-jqzb8" Jul 06 22:40:24.728331 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:40:24.728280 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-a-17b26a-6b7c9d9dcb-jqzb8" podStartSLOduration=2.196141482 podStartE2EDuration="2.728261238s" totalImagesPullingTime="532.119756ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:40:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:40:24.726676104 +0000 UTC m=+1876.026387957" watchObservedRunningTime="2026-07-06 22:40:24.728261238 +0000 UTC m=+1876.027973088" Jul 06 22:40:30.588189 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:40:30.588150 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-b-5d913f-f4df8b596-7wwvj"] Jul 06 22:40:30.591991 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:40:30.591968 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-5d913f-f4df8b596-7wwvj" Jul 06 22:40:30.594633 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:40:30.594615 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-b-5d913f-serving-cert\"" Jul 06 22:40:30.600670 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:40:30.600645 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-5d913f-f4df8b596-7wwvj"] Jul 06 22:40:30.653261 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:40:30.653226 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ef964a3-ea0e-4e99-bfa0-ac7869e4e75c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5d913f-f4df8b596-7wwvj\" (UID: \"9ef964a3-ea0e-4e99-bfa0-ac7869e4e75c\") " pod="opendatahub/maas-api-e2e-inf-b-5d913f-f4df8b596-7wwvj" Jul 06 22:40:30.653460 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:40:30.653348 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ef964a3-ea0e-4e99-bfa0-ac7869e4e75c-maas-api-tls\") pod \"maas-api-e2e-inf-b-5d913f-f4df8b596-7wwvj\" (UID: \"9ef964a3-ea0e-4e99-bfa0-ac7869e4e75c\") " pod="opendatahub/maas-api-e2e-inf-b-5d913f-f4df8b596-7wwvj" Jul 06 22:40:30.653531 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:40:30.653468 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw5vv\" (UniqueName: \"kubernetes.io/projected/9ef964a3-ea0e-4e99-bfa0-ac7869e4e75c-kube-api-access-zw5vv\") pod \"maas-api-e2e-inf-b-5d913f-f4df8b596-7wwvj\" (UID: \"9ef964a3-ea0e-4e99-bfa0-ac7869e4e75c\") " pod="opendatahub/maas-api-e2e-inf-b-5d913f-f4df8b596-7wwvj" Jul 06 22:40:30.720556 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:40:30.720525 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-a-17b26a-6b7c9d9dcb-jqzb8" Jul 06 22:40:30.754497 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:40:30.754462 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ef964a3-ea0e-4e99-bfa0-ac7869e4e75c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5d913f-f4df8b596-7wwvj\" (UID: \"9ef964a3-ea0e-4e99-bfa0-ac7869e4e75c\") " pod="opendatahub/maas-api-e2e-inf-b-5d913f-f4df8b596-7wwvj" Jul 06 22:40:30.754669 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:40:30.754533 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ef964a3-ea0e-4e99-bfa0-ac7869e4e75c-maas-api-tls\") pod \"maas-api-e2e-inf-b-5d913f-f4df8b596-7wwvj\" (UID: \"9ef964a3-ea0e-4e99-bfa0-ac7869e4e75c\") " pod="opendatahub/maas-api-e2e-inf-b-5d913f-f4df8b596-7wwvj" Jul 06 22:40:30.754669 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:40:30.754628 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw5vv\" (UniqueName: \"kubernetes.io/projected/9ef964a3-ea0e-4e99-bfa0-ac7869e4e75c-kube-api-access-zw5vv\") pod \"maas-api-e2e-inf-b-5d913f-f4df8b596-7wwvj\" (UID: \"9ef964a3-ea0e-4e99-bfa0-ac7869e4e75c\") " pod="opendatahub/maas-api-e2e-inf-b-5d913f-f4df8b596-7wwvj" Jul 06 22:40:30.754758 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:40:30.754696 2555 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-inf-b-5d913f-serving-cert: secret "maas-api-e2e-inf-b-5d913f-serving-cert" not found Jul 06 22:40:30.754800 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:40:30.754775 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ef964a3-ea0e-4e99-bfa0-ac7869e4e75c-maas-api-tls podName:9ef964a3-ea0e-4e99-bfa0-ac7869e4e75c nodeName:}" failed. No retries permitted until 2026-07-06 22:40:31.254754115 +0000 UTC m=+1882.554465960 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9ef964a3-ea0e-4e99-bfa0-ac7869e4e75c-maas-api-tls") pod "maas-api-e2e-inf-b-5d913f-f4df8b596-7wwvj" (UID: "9ef964a3-ea0e-4e99-bfa0-ac7869e4e75c") : secret "maas-api-e2e-inf-b-5d913f-serving-cert" not found Jul 06 22:40:30.755142 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:40:30.755126 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ef964a3-ea0e-4e99-bfa0-ac7869e4e75c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5d913f-f4df8b596-7wwvj\" (UID: \"9ef964a3-ea0e-4e99-bfa0-ac7869e4e75c\") " pod="opendatahub/maas-api-e2e-inf-b-5d913f-f4df8b596-7wwvj" Jul 06 22:40:30.765231 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:40:30.765200 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw5vv\" (UniqueName: \"kubernetes.io/projected/9ef964a3-ea0e-4e99-bfa0-ac7869e4e75c-kube-api-access-zw5vv\") pod \"maas-api-e2e-inf-b-5d913f-f4df8b596-7wwvj\" (UID: \"9ef964a3-ea0e-4e99-bfa0-ac7869e4e75c\") " pod="opendatahub/maas-api-e2e-inf-b-5d913f-f4df8b596-7wwvj" Jul 06 22:40:31.259650 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:40:31.259559 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ef964a3-ea0e-4e99-bfa0-ac7869e4e75c-maas-api-tls\") pod \"maas-api-e2e-inf-b-5d913f-f4df8b596-7wwvj\" (UID: \"9ef964a3-ea0e-4e99-bfa0-ac7869e4e75c\") " pod="opendatahub/maas-api-e2e-inf-b-5d913f-f4df8b596-7wwvj" Jul 06 22:40:31.262092 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:40:31.262065 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ef964a3-ea0e-4e99-bfa0-ac7869e4e75c-maas-api-tls\") pod \"maas-api-e2e-inf-b-5d913f-f4df8b596-7wwvj\" (UID: \"9ef964a3-ea0e-4e99-bfa0-ac7869e4e75c\") " pod="opendatahub/maas-api-e2e-inf-b-5d913f-f4df8b596-7wwvj" Jul 06 22:40:31.503233 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:40:31.503191 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-5d913f-f4df8b596-7wwvj" Jul 06 22:40:31.638659 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:40:31.638627 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-5d913f-f4df8b596-7wwvj"] Jul 06 22:40:31.640166 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:40:31.640130 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ef964a3_ea0e_4e99_bfa0_ac7869e4e75c.slice/crio-dd526eaae3693c4f94d73fcba534c2a0d9dc1cd96227b27dd324aef168d3df6a WatchSource:0}: Error finding container dd526eaae3693c4f94d73fcba534c2a0d9dc1cd96227b27dd324aef168d3df6a: Status 404 returned error can't find the container with id dd526eaae3693c4f94d73fcba534c2a0d9dc1cd96227b27dd324aef168d3df6a Jul 06 22:40:31.744955 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:40:31.744917 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-5d913f-f4df8b596-7wwvj" event={"ID":"9ef964a3-ea0e-4e99-bfa0-ac7869e4e75c","Type":"ContainerStarted","Data":"dd526eaae3693c4f94d73fcba534c2a0d9dc1cd96227b27dd324aef168d3df6a"} Jul 06 22:40:32.751345 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:40:32.751308 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-5d913f-f4df8b596-7wwvj" event={"ID":"9ef964a3-ea0e-4e99-bfa0-ac7869e4e75c","Type":"ContainerStarted","Data":"3a36e067ac3dd1a8f822f21aff9bd744473643bb54ca1038e2ce319a53806899"} Jul 06 22:40:32.751840 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:40:32.751432 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-b-5d913f-f4df8b596-7wwvj" Jul 06 22:40:32.769671 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:40:32.769619 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-b-5d913f-f4df8b596-7wwvj" podStartSLOduration=2.400766168 podStartE2EDuration="2.769604255s" totalImagesPullingTime="368.838087ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:40:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:40:32.767872933 +0000 UTC m=+1884.067584782" watchObservedRunningTime="2026-07-06 22:40:32.769604255 +0000 UTC m=+1884.069316103" Jul 06 22:40:38.762287 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:40:38.762253 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-b-5d913f-f4df8b596-7wwvj" Jul 06 22:41:08.000174 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:08.000129 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-5d913f/test-model-5d913f-kserve-75654bc86f-tgw24"] Jul 06 22:41:08.003780 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:08.003760 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5d913f/test-model-5d913f-kserve-75654bc86f-tgw24" Jul 06 22:41:08.006546 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:08.006518 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5d913f\"/\"openshift-service-ca.crt\"" Jul 06 22:41:08.007856 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:08.007837 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5d913f\"/\"kube-root-ca.crt\"" Jul 06 22:41:08.007963 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:08.007938 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5d913f\"/\"default-dockercfg-bjkkq\"" Jul 06 22:41:08.008049 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:08.007950 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5d913f\"/\"test-model-5d913f-kserve-self-signed-certs\"" Jul 06 22:41:08.012605 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:08.012582 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5d913f/test-model-5d913f-kserve-75654bc86f-tgw24"] Jul 06 22:41:08.114696 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:08.114648 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqccw\" (UniqueName: \"kubernetes.io/projected/f981e751-7168-4b3f-892f-44fa4fa255c8-kube-api-access-xqccw\") pod \"test-model-5d913f-kserve-75654bc86f-tgw24\" (UID: \"f981e751-7168-4b3f-892f-44fa4fa255c8\") " pod="ai-tenant-e2e-inf-b-5d913f/test-model-5d913f-kserve-75654bc86f-tgw24" Jul 06 22:41:08.114905 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:08.114708 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f981e751-7168-4b3f-892f-44fa4fa255c8-home\") pod \"test-model-5d913f-kserve-75654bc86f-tgw24\" (UID: \"f981e751-7168-4b3f-892f-44fa4fa255c8\") " pod="ai-tenant-e2e-inf-b-5d913f/test-model-5d913f-kserve-75654bc86f-tgw24" Jul 06 22:41:08.114905 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:08.114758 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f981e751-7168-4b3f-892f-44fa4fa255c8-tmp-dir\") pod \"test-model-5d913f-kserve-75654bc86f-tgw24\" (UID: \"f981e751-7168-4b3f-892f-44fa4fa255c8\") " pod="ai-tenant-e2e-inf-b-5d913f/test-model-5d913f-kserve-75654bc86f-tgw24" Jul 06 22:41:08.115087 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:08.114903 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f981e751-7168-4b3f-892f-44fa4fa255c8-tls-certs\") pod \"test-model-5d913f-kserve-75654bc86f-tgw24\" (UID: \"f981e751-7168-4b3f-892f-44fa4fa255c8\") " pod="ai-tenant-e2e-inf-b-5d913f/test-model-5d913f-kserve-75654bc86f-tgw24" Jul 06 22:41:08.115087 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:08.114952 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f981e751-7168-4b3f-892f-44fa4fa255c8-dshm\") pod \"test-model-5d913f-kserve-75654bc86f-tgw24\" (UID: \"f981e751-7168-4b3f-892f-44fa4fa255c8\") " pod="ai-tenant-e2e-inf-b-5d913f/test-model-5d913f-kserve-75654bc86f-tgw24" Jul 06 22:41:08.115087 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:08.114982 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f981e751-7168-4b3f-892f-44fa4fa255c8-model-cache\") pod \"test-model-5d913f-kserve-75654bc86f-tgw24\" (UID: \"f981e751-7168-4b3f-892f-44fa4fa255c8\") " pod="ai-tenant-e2e-inf-b-5d913f/test-model-5d913f-kserve-75654bc86f-tgw24" Jul 06 22:41:08.215662 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:08.215614 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f981e751-7168-4b3f-892f-44fa4fa255c8-dshm\") pod \"test-model-5d913f-kserve-75654bc86f-tgw24\" (UID: \"f981e751-7168-4b3f-892f-44fa4fa255c8\") " pod="ai-tenant-e2e-inf-b-5d913f/test-model-5d913f-kserve-75654bc86f-tgw24" Jul 06 22:41:08.215662 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:08.215667 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f981e751-7168-4b3f-892f-44fa4fa255c8-model-cache\") pod \"test-model-5d913f-kserve-75654bc86f-tgw24\" (UID: \"f981e751-7168-4b3f-892f-44fa4fa255c8\") " pod="ai-tenant-e2e-inf-b-5d913f/test-model-5d913f-kserve-75654bc86f-tgw24" Jul 06 22:41:08.215939 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:08.215705 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqccw\" (UniqueName: \"kubernetes.io/projected/f981e751-7168-4b3f-892f-44fa4fa255c8-kube-api-access-xqccw\") pod \"test-model-5d913f-kserve-75654bc86f-tgw24\" (UID: \"f981e751-7168-4b3f-892f-44fa4fa255c8\") " pod="ai-tenant-e2e-inf-b-5d913f/test-model-5d913f-kserve-75654bc86f-tgw24" Jul 06 22:41:08.215939 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:08.215777 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f981e751-7168-4b3f-892f-44fa4fa255c8-home\") pod \"test-model-5d913f-kserve-75654bc86f-tgw24\" (UID: \"f981e751-7168-4b3f-892f-44fa4fa255c8\") " pod="ai-tenant-e2e-inf-b-5d913f/test-model-5d913f-kserve-75654bc86f-tgw24" Jul 06 22:41:08.215939 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:08.215808 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f981e751-7168-4b3f-892f-44fa4fa255c8-tmp-dir\") pod \"test-model-5d913f-kserve-75654bc86f-tgw24\" (UID: \"f981e751-7168-4b3f-892f-44fa4fa255c8\") " pod="ai-tenant-e2e-inf-b-5d913f/test-model-5d913f-kserve-75654bc86f-tgw24" Jul 06 22:41:08.215939 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:08.215927 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f981e751-7168-4b3f-892f-44fa4fa255c8-tls-certs\") pod \"test-model-5d913f-kserve-75654bc86f-tgw24\" (UID: \"f981e751-7168-4b3f-892f-44fa4fa255c8\") " pod="ai-tenant-e2e-inf-b-5d913f/test-model-5d913f-kserve-75654bc86f-tgw24" Jul 06 22:41:08.216235 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:08.216212 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f981e751-7168-4b3f-892f-44fa4fa255c8-model-cache\") pod \"test-model-5d913f-kserve-75654bc86f-tgw24\" (UID: \"f981e751-7168-4b3f-892f-44fa4fa255c8\") " pod="ai-tenant-e2e-inf-b-5d913f/test-model-5d913f-kserve-75654bc86f-tgw24" Jul 06 22:41:08.216281 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:08.216238 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f981e751-7168-4b3f-892f-44fa4fa255c8-home\") pod \"test-model-5d913f-kserve-75654bc86f-tgw24\" (UID: \"f981e751-7168-4b3f-892f-44fa4fa255c8\") " pod="ai-tenant-e2e-inf-b-5d913f/test-model-5d913f-kserve-75654bc86f-tgw24" Jul 06 22:41:08.216333 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:08.216299 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f981e751-7168-4b3f-892f-44fa4fa255c8-tmp-dir\") pod \"test-model-5d913f-kserve-75654bc86f-tgw24\" (UID: \"f981e751-7168-4b3f-892f-44fa4fa255c8\") " pod="ai-tenant-e2e-inf-b-5d913f/test-model-5d913f-kserve-75654bc86f-tgw24" Jul 06 22:41:08.217991 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:08.217959 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f981e751-7168-4b3f-892f-44fa4fa255c8-dshm\") pod \"test-model-5d913f-kserve-75654bc86f-tgw24\" (UID: \"f981e751-7168-4b3f-892f-44fa4fa255c8\") " pod="ai-tenant-e2e-inf-b-5d913f/test-model-5d913f-kserve-75654bc86f-tgw24" Jul 06 22:41:08.218357 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:08.218337 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f981e751-7168-4b3f-892f-44fa4fa255c8-tls-certs\") pod \"test-model-5d913f-kserve-75654bc86f-tgw24\" (UID: \"f981e751-7168-4b3f-892f-44fa4fa255c8\") " pod="ai-tenant-e2e-inf-b-5d913f/test-model-5d913f-kserve-75654bc86f-tgw24" Jul 06 22:41:08.224425 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:08.224401 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqccw\" (UniqueName: \"kubernetes.io/projected/f981e751-7168-4b3f-892f-44fa4fa255c8-kube-api-access-xqccw\") pod \"test-model-5d913f-kserve-75654bc86f-tgw24\" (UID: \"f981e751-7168-4b3f-892f-44fa4fa255c8\") " pod="ai-tenant-e2e-inf-b-5d913f/test-model-5d913f-kserve-75654bc86f-tgw24" Jul 06 22:41:08.314456 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:08.314359 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5d913f/test-model-5d913f-kserve-75654bc86f-tgw24" Jul 06 22:41:08.451021 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:08.450974 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5d913f/test-model-5d913f-kserve-75654bc86f-tgw24"] Jul 06 22:41:08.452940 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:41:08.452915 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf981e751_7168_4b3f_892f_44fa4fa255c8.slice/crio-489d0b6271b08850ee2edb6c854ec54787a1729bdb22ea618f71369f4417e649 WatchSource:0}: Error finding container 489d0b6271b08850ee2edb6c854ec54787a1729bdb22ea618f71369f4417e649: Status 404 returned error can't find the container with id 489d0b6271b08850ee2edb6c854ec54787a1729bdb22ea618f71369f4417e649 Jul 06 22:41:08.941310 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:08.941264 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5d913f/test-model-5d913f-kserve-75654bc86f-tgw24" event={"ID":"f981e751-7168-4b3f-892f-44fa4fa255c8","Type":"ContainerStarted","Data":"b665f51ebf9d04d67c4db15b0ae22cf8f9425c94a5898fb33932febd728e0831"} Jul 06 22:41:08.941310 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:08.941317 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5d913f/test-model-5d913f-kserve-75654bc86f-tgw24" event={"ID":"f981e751-7168-4b3f-892f-44fa4fa255c8","Type":"ContainerStarted","Data":"489d0b6271b08850ee2edb6c854ec54787a1729bdb22ea618f71369f4417e649"} Jul 06 22:41:08.962846 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:08.962785 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-5d913f/test-model-5d913f-kserve-75654bc86f-tgw24" podStartSLOduration=1.962766052 podStartE2EDuration="1.962766052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:41:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:41:08.961239137 +0000 UTC m=+1920.260950987" watchObservedRunningTime="2026-07-06 22:41:08.962766052 +0000 UTC m=+1920.262477900" Jul 06 22:41:18.314854 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:18.314807 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-5d913f/test-model-5d913f-kserve-75654bc86f-tgw24" Jul 06 22:41:18.315365 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:18.315086 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-5d913f/test-model-5d913f-kserve-75654bc86f-tgw24" Jul 06 22:41:18.327841 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:18.327812 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-5d913f/test-model-5d913f-kserve-75654bc86f-tgw24" Jul 06 22:41:19.002880 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:19.002845 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-5d913f/test-model-5d913f-kserve-75654bc86f-tgw24" Jul 06 22:41:34.447521 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:34.447483 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5d913f/test-model-5d913f-kserve-75654bc86f-tgw24"] Jul 06 22:41:35.270104 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:35.270061 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-17b26a-6b7c9d9dcb-jqzb8"] Jul 06 22:41:35.270383 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:35.270350 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-a-17b26a-6b7c9d9dcb-jqzb8" podUID="3c6d3a47-96ca-47bd-b0c9-6785b321f60d" containerName="maas-api" containerID="cri-o://0fc33abedd451b96fa6d146a4db0a3aab36398ebbe467d6b63e83bec05bdf83a" gracePeriod=30 Jul 06 22:41:35.517652 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:35.517624 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-17b26a-6b7c9d9dcb-jqzb8" Jul 06 22:41:35.591466 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:35.591371 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c6d3a47-96ca-47bd-b0c9-6785b321f60d-openshift-service-ca\") pod \"3c6d3a47-96ca-47bd-b0c9-6785b321f60d\" (UID: \"3c6d3a47-96ca-47bd-b0c9-6785b321f60d\") " Jul 06 22:41:35.591466 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:35.591438 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c6d3a47-96ca-47bd-b0c9-6785b321f60d-maas-api-tls\") pod \"3c6d3a47-96ca-47bd-b0c9-6785b321f60d\" (UID: \"3c6d3a47-96ca-47bd-b0c9-6785b321f60d\") " Jul 06 22:41:35.591651 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:35.591474 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8rmls\" (UniqueName: \"kubernetes.io/projected/3c6d3a47-96ca-47bd-b0c9-6785b321f60d-kube-api-access-8rmls\") pod \"3c6d3a47-96ca-47bd-b0c9-6785b321f60d\" (UID: \"3c6d3a47-96ca-47bd-b0c9-6785b321f60d\") " Jul 06 22:41:35.591819 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:35.591782 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c6d3a47-96ca-47bd-b0c9-6785b321f60d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3c6d3a47-96ca-47bd-b0c9-6785b321f60d" (UID: "3c6d3a47-96ca-47bd-b0c9-6785b321f60d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:41:35.593497 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:35.593471 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c6d3a47-96ca-47bd-b0c9-6785b321f60d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3c6d3a47-96ca-47bd-b0c9-6785b321f60d" (UID: "3c6d3a47-96ca-47bd-b0c9-6785b321f60d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:41:35.593577 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:35.593541 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c6d3a47-96ca-47bd-b0c9-6785b321f60d-kube-api-access-8rmls" (OuterVolumeSpecName: "kube-api-access-8rmls") pod "3c6d3a47-96ca-47bd-b0c9-6785b321f60d" (UID: "3c6d3a47-96ca-47bd-b0c9-6785b321f60d"). InnerVolumeSpecName "kube-api-access-8rmls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:41:35.692258 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:35.692219 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c6d3a47-96ca-47bd-b0c9-6785b321f60d-openshift-service-ca\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:41:35.692258 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:35.692252 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c6d3a47-96ca-47bd-b0c9-6785b321f60d-maas-api-tls\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:41:35.692258 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:35.692262 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rmls\" (UniqueName: \"kubernetes.io/projected/3c6d3a47-96ca-47bd-b0c9-6785b321f60d-kube-api-access-8rmls\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:41:36.074638 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:36.074605 2555 generic.go:356] "Generic (PLEG): container finished" podID="3c6d3a47-96ca-47bd-b0c9-6785b321f60d" containerID="0fc33abedd451b96fa6d146a4db0a3aab36398ebbe467d6b63e83bec05bdf83a" exitCode=0 Jul 06 22:41:36.074830 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:36.074652 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-17b26a-6b7c9d9dcb-jqzb8" Jul 06 22:41:36.074830 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:36.074693 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-17b26a-6b7c9d9dcb-jqzb8" event={"ID":"3c6d3a47-96ca-47bd-b0c9-6785b321f60d","Type":"ContainerDied","Data":"0fc33abedd451b96fa6d146a4db0a3aab36398ebbe467d6b63e83bec05bdf83a"} Jul 06 22:41:36.074830 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:36.074745 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-17b26a-6b7c9d9dcb-jqzb8" event={"ID":"3c6d3a47-96ca-47bd-b0c9-6785b321f60d","Type":"ContainerDied","Data":"c1cd4e1077e0d15ed38db0729f24edeaf9784f15e7ae2d881d17e3a40b474a74"} Jul 06 22:41:36.074830 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:36.074767 2555 scope.go:117] "RemoveContainer" containerID="0fc33abedd451b96fa6d146a4db0a3aab36398ebbe467d6b63e83bec05bdf83a" Jul 06 22:41:36.084776 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:36.084755 2555 scope.go:117] "RemoveContainer" containerID="0fc33abedd451b96fa6d146a4db0a3aab36398ebbe467d6b63e83bec05bdf83a" Jul 06 22:41:36.085056 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:41:36.085034 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fc33abedd451b96fa6d146a4db0a3aab36398ebbe467d6b63e83bec05bdf83a\": container with ID starting with 0fc33abedd451b96fa6d146a4db0a3aab36398ebbe467d6b63e83bec05bdf83a not found: ID does not exist" containerID="0fc33abedd451b96fa6d146a4db0a3aab36398ebbe467d6b63e83bec05bdf83a" Jul 06 22:41:36.085162 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:36.085062 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fc33abedd451b96fa6d146a4db0a3aab36398ebbe467d6b63e83bec05bdf83a"} err="failed to get container status \"0fc33abedd451b96fa6d146a4db0a3aab36398ebbe467d6b63e83bec05bdf83a\": rpc error: code = NotFound desc = could not find container \"0fc33abedd451b96fa6d146a4db0a3aab36398ebbe467d6b63e83bec05bdf83a\": container with ID starting with 0fc33abedd451b96fa6d146a4db0a3aab36398ebbe467d6b63e83bec05bdf83a not found: ID does not exist" Jul 06 22:41:36.096438 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:36.096408 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-17b26a-6b7c9d9dcb-jqzb8"] Jul 06 22:41:36.100096 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:36.100067 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-17b26a-6b7c9d9dcb-jqzb8"] Jul 06 22:41:37.340276 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:37.340242 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c6d3a47-96ca-47bd-b0c9-6785b321f60d" path="/var/lib/kubelet/pods/3c6d3a47-96ca-47bd-b0c9-6785b321f60d/volumes" Jul 06 22:41:49.468508 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:49.468466 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-5d913f/test-model-5d913f-kserve-75654bc86f-tgw24" podUID="f981e751-7168-4b3f-892f-44fa4fa255c8" containerName="main" containerID="cri-o://b665f51ebf9d04d67c4db15b0ae22cf8f9425c94a5898fb33932febd728e0831" gracePeriod=45 Jul 06 22:41:49.733363 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:49.733291 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5d913f/test-model-5d913f-kserve-75654bc86f-tgw24" Jul 06 22:41:49.828393 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:49.828354 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xqccw\" (UniqueName: \"kubernetes.io/projected/f981e751-7168-4b3f-892f-44fa4fa255c8-kube-api-access-xqccw\") pod \"f981e751-7168-4b3f-892f-44fa4fa255c8\" (UID: \"f981e751-7168-4b3f-892f-44fa4fa255c8\") " Jul 06 22:41:49.828578 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:49.828401 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f981e751-7168-4b3f-892f-44fa4fa255c8-tls-certs\") pod \"f981e751-7168-4b3f-892f-44fa4fa255c8\" (UID: \"f981e751-7168-4b3f-892f-44fa4fa255c8\") " Jul 06 22:41:49.828578 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:49.828425 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f981e751-7168-4b3f-892f-44fa4fa255c8-dshm\") pod \"f981e751-7168-4b3f-892f-44fa4fa255c8\" (UID: \"f981e751-7168-4b3f-892f-44fa4fa255c8\") " Jul 06 22:41:49.828578 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:49.828446 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f981e751-7168-4b3f-892f-44fa4fa255c8-model-cache\") pod \"f981e751-7168-4b3f-892f-44fa4fa255c8\" (UID: \"f981e751-7168-4b3f-892f-44fa4fa255c8\") " Jul 06 22:41:49.828578 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:49.828501 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f981e751-7168-4b3f-892f-44fa4fa255c8-tmp-dir\") pod \"f981e751-7168-4b3f-892f-44fa4fa255c8\" (UID: \"f981e751-7168-4b3f-892f-44fa4fa255c8\") " Jul 06 22:41:49.828578 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:49.828543 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f981e751-7168-4b3f-892f-44fa4fa255c8-home\") pod \"f981e751-7168-4b3f-892f-44fa4fa255c8\" (UID: \"f981e751-7168-4b3f-892f-44fa4fa255c8\") " Jul 06 22:41:49.828850 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:49.828801 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f981e751-7168-4b3f-892f-44fa4fa255c8-model-cache" (OuterVolumeSpecName: "model-cache") pod "f981e751-7168-4b3f-892f-44fa4fa255c8" (UID: "f981e751-7168-4b3f-892f-44fa4fa255c8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:41:49.828850 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:49.828831 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f981e751-7168-4b3f-892f-44fa4fa255c8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f981e751-7168-4b3f-892f-44fa4fa255c8" (UID: "f981e751-7168-4b3f-892f-44fa4fa255c8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:41:49.828951 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:49.828891 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f981e751-7168-4b3f-892f-44fa4fa255c8-home" (OuterVolumeSpecName: "home") pod "f981e751-7168-4b3f-892f-44fa4fa255c8" (UID: "f981e751-7168-4b3f-892f-44fa4fa255c8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:41:49.828951 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:49.828934 2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f981e751-7168-4b3f-892f-44fa4fa255c8-model-cache\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:41:49.829060 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:49.828952 2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f981e751-7168-4b3f-892f-44fa4fa255c8-tmp-dir\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:41:49.830828 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:49.830804 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f981e751-7168-4b3f-892f-44fa4fa255c8-kube-api-access-xqccw" (OuterVolumeSpecName: "kube-api-access-xqccw") pod "f981e751-7168-4b3f-892f-44fa4fa255c8" (UID: "f981e751-7168-4b3f-892f-44fa4fa255c8"). InnerVolumeSpecName "kube-api-access-xqccw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:41:49.830828 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:49.830816 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f981e751-7168-4b3f-892f-44fa4fa255c8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f981e751-7168-4b3f-892f-44fa4fa255c8" (UID: "f981e751-7168-4b3f-892f-44fa4fa255c8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:41:49.830828 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:49.830808 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f981e751-7168-4b3f-892f-44fa4fa255c8-dshm" (OuterVolumeSpecName: "dshm") pod "f981e751-7168-4b3f-892f-44fa4fa255c8" (UID: "f981e751-7168-4b3f-892f-44fa4fa255c8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:41:49.930077 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:49.930034 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqccw\" (UniqueName: \"kubernetes.io/projected/f981e751-7168-4b3f-892f-44fa4fa255c8-kube-api-access-xqccw\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:41:49.930077 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:49.930070 2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f981e751-7168-4b3f-892f-44fa4fa255c8-tls-certs\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:41:49.930077 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:49.930086 2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f981e751-7168-4b3f-892f-44fa4fa255c8-dshm\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:41:49.930326 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:49.930098 2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f981e751-7168-4b3f-892f-44fa4fa255c8-home\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:41:50.149732 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:50.149697 2555 generic.go:356] "Generic (PLEG): container finished" podID="f981e751-7168-4b3f-892f-44fa4fa255c8" containerID="b665f51ebf9d04d67c4db15b0ae22cf8f9425c94a5898fb33932febd728e0831" exitCode=0 Jul 06 22:41:50.149947 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:50.149743 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5d913f/test-model-5d913f-kserve-75654bc86f-tgw24" Jul 06 22:41:50.149947 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:50.149791 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5d913f/test-model-5d913f-kserve-75654bc86f-tgw24" event={"ID":"f981e751-7168-4b3f-892f-44fa4fa255c8","Type":"ContainerDied","Data":"b665f51ebf9d04d67c4db15b0ae22cf8f9425c94a5898fb33932febd728e0831"} Jul 06 22:41:50.149947 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:50.149825 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5d913f/test-model-5d913f-kserve-75654bc86f-tgw24" event={"ID":"f981e751-7168-4b3f-892f-44fa4fa255c8","Type":"ContainerDied","Data":"489d0b6271b08850ee2edb6c854ec54787a1729bdb22ea618f71369f4417e649"} Jul 06 22:41:50.149947 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:50.149842 2555 scope.go:117] "RemoveContainer" containerID="b665f51ebf9d04d67c4db15b0ae22cf8f9425c94a5898fb33932febd728e0831" Jul 06 22:41:50.159572 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:50.159534 2555 scope.go:117] "RemoveContainer" containerID="b665f51ebf9d04d67c4db15b0ae22cf8f9425c94a5898fb33932febd728e0831" Jul 06 22:41:50.159840 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:41:50.159817 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b665f51ebf9d04d67c4db15b0ae22cf8f9425c94a5898fb33932febd728e0831\": container with ID starting with b665f51ebf9d04d67c4db15b0ae22cf8f9425c94a5898fb33932febd728e0831 not found: ID does not exist" containerID="b665f51ebf9d04d67c4db15b0ae22cf8f9425c94a5898fb33932febd728e0831" Jul 06 22:41:50.159951 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:50.159846 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b665f51ebf9d04d67c4db15b0ae22cf8f9425c94a5898fb33932febd728e0831"} err="failed to get container status \"b665f51ebf9d04d67c4db15b0ae22cf8f9425c94a5898fb33932febd728e0831\": rpc error: code = NotFound desc = could not find container \"b665f51ebf9d04d67c4db15b0ae22cf8f9425c94a5898fb33932febd728e0831\": container with ID starting with b665f51ebf9d04d67c4db15b0ae22cf8f9425c94a5898fb33932febd728e0831 not found: ID does not exist" Jul 06 22:41:50.172843 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:50.172815 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5d913f/test-model-5d913f-kserve-75654bc86f-tgw24"] Jul 06 22:41:50.176566 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:50.176541 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-5d913f/test-model-5d913f-kserve-75654bc86f-tgw24"] Jul 06 22:41:51.339665 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:41:51.339628 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f981e751-7168-4b3f-892f-44fa4fa255c8" path="/var/lib/kubelet/pods/f981e751-7168-4b3f-892f-44fa4fa255c8/volumes" Jul 06 22:42:18.447481 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:18.447443 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-5d913f-f4df8b596-7wwvj"] Jul 06 22:42:18.447930 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:18.447683 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-b-5d913f-f4df8b596-7wwvj" podUID="9ef964a3-ea0e-4e99-bfa0-ac7869e4e75c" containerName="maas-api" containerID="cri-o://3a36e067ac3dd1a8f822f21aff9bd744473643bb54ca1038e2ce319a53806899" gracePeriod=30 Jul 06 22:42:18.757762 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:18.757704 2555 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-api-e2e-inf-b-5d913f-f4df8b596-7wwvj" podUID="9ef964a3-ea0e-4e99-bfa0-ac7869e4e75c" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.43:8443/health\": dial tcp 10.134.0.43:8443: connect: connection refused" Jul 06 22:42:18.894441 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:18.894408 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-5d913f-f4df8b596-7wwvj" Jul 06 22:42:19.011458 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:19.011350 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zw5vv\" (UniqueName: \"kubernetes.io/projected/9ef964a3-ea0e-4e99-bfa0-ac7869e4e75c-kube-api-access-zw5vv\") pod \"9ef964a3-ea0e-4e99-bfa0-ac7869e4e75c\" (UID: \"9ef964a3-ea0e-4e99-bfa0-ac7869e4e75c\") " Jul 06 22:42:19.011649 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:19.011477 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ef964a3-ea0e-4e99-bfa0-ac7869e4e75c-openshift-service-ca\") pod \"9ef964a3-ea0e-4e99-bfa0-ac7869e4e75c\" (UID: \"9ef964a3-ea0e-4e99-bfa0-ac7869e4e75c\") " Jul 06 22:42:19.011649 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:19.011525 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ef964a3-ea0e-4e99-bfa0-ac7869e4e75c-maas-api-tls\") pod \"9ef964a3-ea0e-4e99-bfa0-ac7869e4e75c\" (UID: \"9ef964a3-ea0e-4e99-bfa0-ac7869e4e75c\") " Jul 06 22:42:19.011907 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:19.011881 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ef964a3-ea0e-4e99-bfa0-ac7869e4e75c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9ef964a3-ea0e-4e99-bfa0-ac7869e4e75c" (UID: "9ef964a3-ea0e-4e99-bfa0-ac7869e4e75c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:42:19.013606 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:19.013568 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ef964a3-ea0e-4e99-bfa0-ac7869e4e75c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9ef964a3-ea0e-4e99-bfa0-ac7869e4e75c" (UID: "9ef964a3-ea0e-4e99-bfa0-ac7869e4e75c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:42:19.013717 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:19.013619 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ef964a3-ea0e-4e99-bfa0-ac7869e4e75c-kube-api-access-zw5vv" (OuterVolumeSpecName: "kube-api-access-zw5vv") pod "9ef964a3-ea0e-4e99-bfa0-ac7869e4e75c" (UID: "9ef964a3-ea0e-4e99-bfa0-ac7869e4e75c"). InnerVolumeSpecName "kube-api-access-zw5vv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:42:19.112563 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:19.112528 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ef964a3-ea0e-4e99-bfa0-ac7869e4e75c-maas-api-tls\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:42:19.112563 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:19.112558 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zw5vv\" (UniqueName: \"kubernetes.io/projected/9ef964a3-ea0e-4e99-bfa0-ac7869e4e75c-kube-api-access-zw5vv\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:42:19.112563 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:19.112568 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ef964a3-ea0e-4e99-bfa0-ac7869e4e75c-openshift-service-ca\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:42:19.292302 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:19.292209 2555 generic.go:356] "Generic (PLEG): container finished" podID="9ef964a3-ea0e-4e99-bfa0-ac7869e4e75c" containerID="3a36e067ac3dd1a8f822f21aff9bd744473643bb54ca1038e2ce319a53806899" exitCode=0 Jul 06 22:42:19.292302 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:19.292254 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-5d913f-f4df8b596-7wwvj" Jul 06 22:42:19.292302 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:19.292267 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-5d913f-f4df8b596-7wwvj" event={"ID":"9ef964a3-ea0e-4e99-bfa0-ac7869e4e75c","Type":"ContainerDied","Data":"3a36e067ac3dd1a8f822f21aff9bd744473643bb54ca1038e2ce319a53806899"} Jul 06 22:42:19.292578 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:19.292352 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-5d913f-f4df8b596-7wwvj" event={"ID":"9ef964a3-ea0e-4e99-bfa0-ac7869e4e75c","Type":"ContainerDied","Data":"dd526eaae3693c4f94d73fcba534c2a0d9dc1cd96227b27dd324aef168d3df6a"} Jul 06 22:42:19.292578 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:19.292382 2555 scope.go:117] "RemoveContainer" containerID="3a36e067ac3dd1a8f822f21aff9bd744473643bb54ca1038e2ce319a53806899" Jul 06 22:42:19.301767 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:19.301747 2555 scope.go:117] "RemoveContainer" containerID="3a36e067ac3dd1a8f822f21aff9bd744473643bb54ca1038e2ce319a53806899" Jul 06 22:42:19.302030 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:42:19.301994 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a36e067ac3dd1a8f822f21aff9bd744473643bb54ca1038e2ce319a53806899\": container with ID starting with 3a36e067ac3dd1a8f822f21aff9bd744473643bb54ca1038e2ce319a53806899 not found: ID does not exist" containerID="3a36e067ac3dd1a8f822f21aff9bd744473643bb54ca1038e2ce319a53806899" Jul 06 22:42:19.302106 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:19.302035 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a36e067ac3dd1a8f822f21aff9bd744473643bb54ca1038e2ce319a53806899"} err="failed to get container status \"3a36e067ac3dd1a8f822f21aff9bd744473643bb54ca1038e2ce319a53806899\": rpc error: code = NotFound desc = could not find container \"3a36e067ac3dd1a8f822f21aff9bd744473643bb54ca1038e2ce319a53806899\": container with ID starting with 3a36e067ac3dd1a8f822f21aff9bd744473643bb54ca1038e2ce319a53806899 not found: ID does not exist" Jul 06 22:42:19.314577 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:19.314550 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-5d913f-f4df8b596-7wwvj"] Jul 06 22:42:19.318421 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:19.318400 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-5d913f-f4df8b596-7wwvj"] Jul 06 22:42:19.339644 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:19.339620 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ef964a3-ea0e-4e99-bfa0-ac7869e4e75c" path="/var/lib/kubelet/pods/9ef964a3-ea0e-4e99-bfa0-ac7869e4e75c/volumes" Jul 06 22:42:28.142228 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:28.142195 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-a-fbe640-6ff4999949-6hpgp"] Jul 06 22:42:28.142700 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:28.142637 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c6d3a47-96ca-47bd-b0c9-6785b321f60d" containerName="maas-api" Jul 06 22:42:28.142700 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:28.142650 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c6d3a47-96ca-47bd-b0c9-6785b321f60d" containerName="maas-api" Jul 06 22:42:28.142700 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:28.142663 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ef964a3-ea0e-4e99-bfa0-ac7869e4e75c" containerName="maas-api" Jul 06 22:42:28.142700 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:28.142668 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ef964a3-ea0e-4e99-bfa0-ac7869e4e75c" containerName="maas-api" Jul 06 22:42:28.142700 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:28.142683 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f981e751-7168-4b3f-892f-44fa4fa255c8" containerName="main" Jul 06 22:42:28.142700 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:28.142688 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f981e751-7168-4b3f-892f-44fa4fa255c8" containerName="main" Jul 06 22:42:28.142893 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:28.142749 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c6d3a47-96ca-47bd-b0c9-6785b321f60d" containerName="maas-api" Jul 06 22:42:28.142893 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:28.142759 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ef964a3-ea0e-4e99-bfa0-ac7869e4e75c" containerName="maas-api" Jul 06 22:42:28.142893 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:28.142765 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f981e751-7168-4b3f-892f-44fa4fa255c8" containerName="main" Jul 06 22:42:28.145702 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:28.145684 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-fbe640-6ff4999949-6hpgp" Jul 06 22:42:28.148709 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:28.148685 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-a-fbe640-serving-cert\"" Jul 06 22:42:28.156056 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:28.156032 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-fbe640-6ff4999949-6hpgp"] Jul 06 22:42:28.301346 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:28.301306 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f56c14d-5607-40f6-83f9-82429e390f6e-maas-api-tls\") pod \"maas-api-e2e-api-a-fbe640-6ff4999949-6hpgp\" (UID: \"3f56c14d-5607-40f6-83f9-82429e390f6e\") " pod="opendatahub/maas-api-e2e-api-a-fbe640-6ff4999949-6hpgp" Jul 06 22:42:28.301522 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:28.301356 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f56c14d-5607-40f6-83f9-82429e390f6e-openshift-service-ca\") pod \"maas-api-e2e-api-a-fbe640-6ff4999949-6hpgp\" (UID: \"3f56c14d-5607-40f6-83f9-82429e390f6e\") " pod="opendatahub/maas-api-e2e-api-a-fbe640-6ff4999949-6hpgp" Jul 06 22:42:28.301522 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:28.301472 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95xds\" (UniqueName: \"kubernetes.io/projected/3f56c14d-5607-40f6-83f9-82429e390f6e-kube-api-access-95xds\") pod \"maas-api-e2e-api-a-fbe640-6ff4999949-6hpgp\" (UID: \"3f56c14d-5607-40f6-83f9-82429e390f6e\") " pod="opendatahub/maas-api-e2e-api-a-fbe640-6ff4999949-6hpgp" Jul 06 22:42:28.402929 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:28.402829 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95xds\" (UniqueName: \"kubernetes.io/projected/3f56c14d-5607-40f6-83f9-82429e390f6e-kube-api-access-95xds\") pod \"maas-api-e2e-api-a-fbe640-6ff4999949-6hpgp\" (UID: \"3f56c14d-5607-40f6-83f9-82429e390f6e\") " pod="opendatahub/maas-api-e2e-api-a-fbe640-6ff4999949-6hpgp" Jul 06 22:42:28.403143 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:28.402938 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f56c14d-5607-40f6-83f9-82429e390f6e-maas-api-tls\") pod \"maas-api-e2e-api-a-fbe640-6ff4999949-6hpgp\" (UID: \"3f56c14d-5607-40f6-83f9-82429e390f6e\") " pod="opendatahub/maas-api-e2e-api-a-fbe640-6ff4999949-6hpgp" Jul 06 22:42:28.403143 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:28.402976 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f56c14d-5607-40f6-83f9-82429e390f6e-openshift-service-ca\") pod \"maas-api-e2e-api-a-fbe640-6ff4999949-6hpgp\" (UID: \"3f56c14d-5607-40f6-83f9-82429e390f6e\") " pod="opendatahub/maas-api-e2e-api-a-fbe640-6ff4999949-6hpgp" Jul 06 22:42:28.403635 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:28.403612 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f56c14d-5607-40f6-83f9-82429e390f6e-openshift-service-ca\") pod \"maas-api-e2e-api-a-fbe640-6ff4999949-6hpgp\" (UID: \"3f56c14d-5607-40f6-83f9-82429e390f6e\") " pod="opendatahub/maas-api-e2e-api-a-fbe640-6ff4999949-6hpgp" Jul 06 22:42:28.405393 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:28.405370 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f56c14d-5607-40f6-83f9-82429e390f6e-maas-api-tls\") pod \"maas-api-e2e-api-a-fbe640-6ff4999949-6hpgp\" (UID: \"3f56c14d-5607-40f6-83f9-82429e390f6e\") " pod="opendatahub/maas-api-e2e-api-a-fbe640-6ff4999949-6hpgp" Jul 06 22:42:28.412000 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:28.411971 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95xds\" (UniqueName: \"kubernetes.io/projected/3f56c14d-5607-40f6-83f9-82429e390f6e-kube-api-access-95xds\") pod \"maas-api-e2e-api-a-fbe640-6ff4999949-6hpgp\" (UID: \"3f56c14d-5607-40f6-83f9-82429e390f6e\") " pod="opendatahub/maas-api-e2e-api-a-fbe640-6ff4999949-6hpgp" Jul 06 22:42:28.458359 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:28.458315 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-fbe640-6ff4999949-6hpgp" Jul 06 22:42:28.596536 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:28.596513 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-fbe640-6ff4999949-6hpgp"] Jul 06 22:42:28.598520 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:42:28.598492 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f56c14d_5607_40f6_83f9_82429e390f6e.slice/crio-47ff3a2be5bbea02c38b935d6caced2e394a3d05799eee38f76375ba505e69f9 WatchSource:0}: Error finding container 47ff3a2be5bbea02c38b935d6caced2e394a3d05799eee38f76375ba505e69f9: Status 404 returned error can't find the container with id 47ff3a2be5bbea02c38b935d6caced2e394a3d05799eee38f76375ba505e69f9 Jul 06 22:42:29.345191 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:29.345099 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-fbe640-6ff4999949-6hpgp" event={"ID":"3f56c14d-5607-40f6-83f9-82429e390f6e","Type":"ContainerStarted","Data":"c5ddcb8842d663b49c931b45eb491429d41e485ce8f57841b4b350e6b47fd916"} Jul 06 22:42:29.345191 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:29.345158 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-fbe640-6ff4999949-6hpgp" event={"ID":"3f56c14d-5607-40f6-83f9-82429e390f6e","Type":"ContainerStarted","Data":"47ff3a2be5bbea02c38b935d6caced2e394a3d05799eee38f76375ba505e69f9"} Jul 06 22:42:29.345191 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:29.345187 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-a-fbe640-6ff4999949-6hpgp" Jul 06 22:42:29.369124 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:29.369064 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-a-fbe640-6ff4999949-6hpgp" podStartSLOduration=0.883910651 podStartE2EDuration="1.36904692s" totalImagesPullingTime="485.136269ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:42:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:42:29.367411365 +0000 UTC m=+2000.667123215" watchObservedRunningTime="2026-07-06 22:42:29.36904692 +0000 UTC m=+2000.668758768" Jul 06 22:42:35.355478 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:35.355444 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-a-fbe640-6ff4999949-6hpgp" Jul 06 22:42:36.236238 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:36.236196 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-b-63b31a-6559858d4c-kx2dv"] Jul 06 22:42:36.240711 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:36.240687 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-63b31a-6559858d4c-kx2dv" Jul 06 22:42:36.243381 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:36.243355 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-b-63b31a-serving-cert\"" Jul 06 22:42:36.253648 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:36.253623 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-63b31a-6559858d4c-kx2dv"] Jul 06 22:42:36.380073 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:36.380038 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/074ca443-ba4c-4c60-92fe-d10684f320cb-openshift-service-ca\") pod \"maas-api-e2e-api-b-63b31a-6559858d4c-kx2dv\" (UID: \"074ca443-ba4c-4c60-92fe-d10684f320cb\") " pod="opendatahub/maas-api-e2e-api-b-63b31a-6559858d4c-kx2dv" Jul 06 22:42:36.380518 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:36.380118 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/074ca443-ba4c-4c60-92fe-d10684f320cb-maas-api-tls\") pod \"maas-api-e2e-api-b-63b31a-6559858d4c-kx2dv\" (UID: \"074ca443-ba4c-4c60-92fe-d10684f320cb\") " pod="opendatahub/maas-api-e2e-api-b-63b31a-6559858d4c-kx2dv" Jul 06 22:42:36.380518 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:36.380195 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9vnr\" (UniqueName: \"kubernetes.io/projected/074ca443-ba4c-4c60-92fe-d10684f320cb-kube-api-access-m9vnr\") pod \"maas-api-e2e-api-b-63b31a-6559858d4c-kx2dv\" (UID: \"074ca443-ba4c-4c60-92fe-d10684f320cb\") " pod="opendatahub/maas-api-e2e-api-b-63b31a-6559858d4c-kx2dv" Jul 06 22:42:36.481377 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:36.481339 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/074ca443-ba4c-4c60-92fe-d10684f320cb-maas-api-tls\") pod \"maas-api-e2e-api-b-63b31a-6559858d4c-kx2dv\" (UID: \"074ca443-ba4c-4c60-92fe-d10684f320cb\") " pod="opendatahub/maas-api-e2e-api-b-63b31a-6559858d4c-kx2dv" Jul 06 22:42:36.481377 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:36.481385 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9vnr\" (UniqueName: \"kubernetes.io/projected/074ca443-ba4c-4c60-92fe-d10684f320cb-kube-api-access-m9vnr\") pod \"maas-api-e2e-api-b-63b31a-6559858d4c-kx2dv\" (UID: \"074ca443-ba4c-4c60-92fe-d10684f320cb\") " pod="opendatahub/maas-api-e2e-api-b-63b31a-6559858d4c-kx2dv" Jul 06 22:42:36.481643 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:36.481617 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/074ca443-ba4c-4c60-92fe-d10684f320cb-openshift-service-ca\") pod \"maas-api-e2e-api-b-63b31a-6559858d4c-kx2dv\" (UID: \"074ca443-ba4c-4c60-92fe-d10684f320cb\") " pod="opendatahub/maas-api-e2e-api-b-63b31a-6559858d4c-kx2dv" Jul 06 22:42:36.482202 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:36.482171 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/074ca443-ba4c-4c60-92fe-d10684f320cb-openshift-service-ca\") pod \"maas-api-e2e-api-b-63b31a-6559858d4c-kx2dv\" (UID: \"074ca443-ba4c-4c60-92fe-d10684f320cb\") " pod="opendatahub/maas-api-e2e-api-b-63b31a-6559858d4c-kx2dv" Jul 06 22:42:36.483772 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:36.483750 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/074ca443-ba4c-4c60-92fe-d10684f320cb-maas-api-tls\") pod \"maas-api-e2e-api-b-63b31a-6559858d4c-kx2dv\" (UID: \"074ca443-ba4c-4c60-92fe-d10684f320cb\") " pod="opendatahub/maas-api-e2e-api-b-63b31a-6559858d4c-kx2dv" Jul 06 22:42:36.493972 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:36.493918 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9vnr\" (UniqueName: \"kubernetes.io/projected/074ca443-ba4c-4c60-92fe-d10684f320cb-kube-api-access-m9vnr\") pod \"maas-api-e2e-api-b-63b31a-6559858d4c-kx2dv\" (UID: \"074ca443-ba4c-4c60-92fe-d10684f320cb\") " pod="opendatahub/maas-api-e2e-api-b-63b31a-6559858d4c-kx2dv" Jul 06 22:42:36.552976 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:36.552939 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-63b31a-6559858d4c-kx2dv" Jul 06 22:42:36.696193 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:36.696168 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-63b31a-6559858d4c-kx2dv"] Jul 06 22:42:36.698996 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:42:36.698967 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod074ca443_ba4c_4c60_92fe_d10684f320cb.slice/crio-fc4bc40d84a889e87ab7bf3bb6873b038df18832db909f154de077bd3207cfb9 WatchSource:0}: Error finding container fc4bc40d84a889e87ab7bf3bb6873b038df18832db909f154de077bd3207cfb9: Status 404 returned error can't find the container with id fc4bc40d84a889e87ab7bf3bb6873b038df18832db909f154de077bd3207cfb9 Jul 06 22:42:37.387231 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:37.387189 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-63b31a-6559858d4c-kx2dv" event={"ID":"074ca443-ba4c-4c60-92fe-d10684f320cb","Type":"ContainerStarted","Data":"c38ad4a578c662bc2a5301a16caea0c2d93a3b2a0ceedc75bff2eee3c13ea2bd"} Jul 06 22:42:37.387637 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:37.387251 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-63b31a-6559858d4c-kx2dv" event={"ID":"074ca443-ba4c-4c60-92fe-d10684f320cb","Type":"ContainerStarted","Data":"fc4bc40d84a889e87ab7bf3bb6873b038df18832db909f154de077bd3207cfb9"} Jul 06 22:42:37.387637 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:37.387374 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-b-63b31a-6559858d4c-kx2dv" Jul 06 22:42:37.407571 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:37.407464 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-b-63b31a-6559858d4c-kx2dv" podStartSLOduration=0.962718272 podStartE2EDuration="1.407448355s" totalImagesPullingTime="444.730083ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:42:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:42:37.404508326 +0000 UTC m=+2008.704220176" watchObservedRunningTime="2026-07-06 22:42:37.407448355 +0000 UTC m=+2008.707160206" Jul 06 22:42:43.397212 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:43.397183 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-b-63b31a-6559858d4c-kx2dv" Jul 06 22:42:49.566542 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:49.566505 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-fbe640-6ff4999949-6hpgp"] Jul 06 22:42:49.567064 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:49.566747 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-a-fbe640-6ff4999949-6hpgp" podUID="3f56c14d-5607-40f6-83f9-82429e390f6e" containerName="maas-api" containerID="cri-o://c5ddcb8842d663b49c931b45eb491429d41e485ce8f57841b4b350e6b47fd916" gracePeriod=30 Jul 06 22:42:49.823555 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:49.823484 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-fbe640-6ff4999949-6hpgp" Jul 06 22:42:49.911041 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:49.910994 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-95xds\" (UniqueName: \"kubernetes.io/projected/3f56c14d-5607-40f6-83f9-82429e390f6e-kube-api-access-95xds\") pod \"3f56c14d-5607-40f6-83f9-82429e390f6e\" (UID: \"3f56c14d-5607-40f6-83f9-82429e390f6e\") " Jul 06 22:42:49.911246 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:49.911082 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f56c14d-5607-40f6-83f9-82429e390f6e-openshift-service-ca\") pod \"3f56c14d-5607-40f6-83f9-82429e390f6e\" (UID: \"3f56c14d-5607-40f6-83f9-82429e390f6e\") " Jul 06 22:42:49.911246 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:49.911108 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f56c14d-5607-40f6-83f9-82429e390f6e-maas-api-tls\") pod \"3f56c14d-5607-40f6-83f9-82429e390f6e\" (UID: \"3f56c14d-5607-40f6-83f9-82429e390f6e\") " Jul 06 22:42:49.911490 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:49.911462 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f56c14d-5607-40f6-83f9-82429e390f6e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3f56c14d-5607-40f6-83f9-82429e390f6e" (UID: "3f56c14d-5607-40f6-83f9-82429e390f6e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:42:49.913203 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:49.913168 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f56c14d-5607-40f6-83f9-82429e390f6e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3f56c14d-5607-40f6-83f9-82429e390f6e" (UID: "3f56c14d-5607-40f6-83f9-82429e390f6e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:42:49.913307 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:49.913197 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f56c14d-5607-40f6-83f9-82429e390f6e-kube-api-access-95xds" (OuterVolumeSpecName: "kube-api-access-95xds") pod "3f56c14d-5607-40f6-83f9-82429e390f6e" (UID: "3f56c14d-5607-40f6-83f9-82429e390f6e"). InnerVolumeSpecName "kube-api-access-95xds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:42:50.012110 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:50.012071 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95xds\" (UniqueName: \"kubernetes.io/projected/3f56c14d-5607-40f6-83f9-82429e390f6e-kube-api-access-95xds\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:42:50.012110 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:50.012107 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f56c14d-5607-40f6-83f9-82429e390f6e-openshift-service-ca\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:42:50.012323 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:50.012123 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f56c14d-5607-40f6-83f9-82429e390f6e-maas-api-tls\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:42:50.454209 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:50.454171 2555 generic.go:356] "Generic (PLEG): container finished" podID="3f56c14d-5607-40f6-83f9-82429e390f6e" containerID="c5ddcb8842d663b49c931b45eb491429d41e485ce8f57841b4b350e6b47fd916" exitCode=0 Jul 06 22:42:50.454402 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:50.454221 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-fbe640-6ff4999949-6hpgp" Jul 06 22:42:50.454402 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:50.454257 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-fbe640-6ff4999949-6hpgp" event={"ID":"3f56c14d-5607-40f6-83f9-82429e390f6e","Type":"ContainerDied","Data":"c5ddcb8842d663b49c931b45eb491429d41e485ce8f57841b4b350e6b47fd916"} Jul 06 22:42:50.454402 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:50.454315 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-fbe640-6ff4999949-6hpgp" event={"ID":"3f56c14d-5607-40f6-83f9-82429e390f6e","Type":"ContainerDied","Data":"47ff3a2be5bbea02c38b935d6caced2e394a3d05799eee38f76375ba505e69f9"} Jul 06 22:42:50.454402 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:50.454345 2555 scope.go:117] "RemoveContainer" containerID="c5ddcb8842d663b49c931b45eb491429d41e485ce8f57841b4b350e6b47fd916" Jul 06 22:42:50.463633 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:50.463612 2555 scope.go:117] "RemoveContainer" containerID="c5ddcb8842d663b49c931b45eb491429d41e485ce8f57841b4b350e6b47fd916" Jul 06 22:42:50.463903 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:42:50.463885 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5ddcb8842d663b49c931b45eb491429d41e485ce8f57841b4b350e6b47fd916\": container with ID starting with c5ddcb8842d663b49c931b45eb491429d41e485ce8f57841b4b350e6b47fd916 not found: ID does not exist" containerID="c5ddcb8842d663b49c931b45eb491429d41e485ce8f57841b4b350e6b47fd916" Jul 06 22:42:50.463948 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:50.463918 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5ddcb8842d663b49c931b45eb491429d41e485ce8f57841b4b350e6b47fd916"} err="failed to get container status \"c5ddcb8842d663b49c931b45eb491429d41e485ce8f57841b4b350e6b47fd916\": rpc error: code = NotFound desc = could not find container \"c5ddcb8842d663b49c931b45eb491429d41e485ce8f57841b4b350e6b47fd916\": container with ID starting with c5ddcb8842d663b49c931b45eb491429d41e485ce8f57841b4b350e6b47fd916 not found: ID does not exist" Jul 06 22:42:50.475555 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:50.475525 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-fbe640-6ff4999949-6hpgp"] Jul 06 22:42:50.482072 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:50.482046 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-a-fbe640-6ff4999949-6hpgp"] Jul 06 22:42:51.346438 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:51.346404 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f56c14d-5607-40f6-83f9-82429e390f6e" path="/var/lib/kubelet/pods/3f56c14d-5607-40f6-83f9-82429e390f6e/volumes" Jul 06 22:42:56.391671 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:56.391635 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-63b31a-6559858d4c-kx2dv"] Jul 06 22:42:56.392140 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:56.391884 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-b-63b31a-6559858d4c-kx2dv" podUID="074ca443-ba4c-4c60-92fe-d10684f320cb" containerName="maas-api" containerID="cri-o://c38ad4a578c662bc2a5301a16caea0c2d93a3b2a0ceedc75bff2eee3c13ea2bd" gracePeriod=30 Jul 06 22:42:56.644217 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:56.644153 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-63b31a-6559858d4c-kx2dv" Jul 06 22:42:56.680319 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:56.680291 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/074ca443-ba4c-4c60-92fe-d10684f320cb-openshift-service-ca\") pod \"074ca443-ba4c-4c60-92fe-d10684f320cb\" (UID: \"074ca443-ba4c-4c60-92fe-d10684f320cb\") " Jul 06 22:42:56.680511 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:56.680326 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m9vnr\" (UniqueName: \"kubernetes.io/projected/074ca443-ba4c-4c60-92fe-d10684f320cb-kube-api-access-m9vnr\") pod \"074ca443-ba4c-4c60-92fe-d10684f320cb\" (UID: \"074ca443-ba4c-4c60-92fe-d10684f320cb\") " Jul 06 22:42:56.680511 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:56.680451 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/074ca443-ba4c-4c60-92fe-d10684f320cb-maas-api-tls\") pod \"074ca443-ba4c-4c60-92fe-d10684f320cb\" (UID: \"074ca443-ba4c-4c60-92fe-d10684f320cb\") " Jul 06 22:42:56.680684 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:56.680654 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/074ca443-ba4c-4c60-92fe-d10684f320cb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "074ca443-ba4c-4c60-92fe-d10684f320cb" (UID: "074ca443-ba4c-4c60-92fe-d10684f320cb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:42:56.682550 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:56.682517 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/074ca443-ba4c-4c60-92fe-d10684f320cb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "074ca443-ba4c-4c60-92fe-d10684f320cb" (UID: "074ca443-ba4c-4c60-92fe-d10684f320cb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:42:56.682677 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:56.682600 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/074ca443-ba4c-4c60-92fe-d10684f320cb-kube-api-access-m9vnr" (OuterVolumeSpecName: "kube-api-access-m9vnr") pod "074ca443-ba4c-4c60-92fe-d10684f320cb" (UID: "074ca443-ba4c-4c60-92fe-d10684f320cb"). InnerVolumeSpecName "kube-api-access-m9vnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:42:56.781287 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:56.781232 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/074ca443-ba4c-4c60-92fe-d10684f320cb-maas-api-tls\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:42:56.781287 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:56.781279 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/074ca443-ba4c-4c60-92fe-d10684f320cb-openshift-service-ca\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:42:56.781287 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:56.781289 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9vnr\" (UniqueName: \"kubernetes.io/projected/074ca443-ba4c-4c60-92fe-d10684f320cb-kube-api-access-m9vnr\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:42:57.491656 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:57.491620 2555 generic.go:356] "Generic (PLEG): container finished" podID="074ca443-ba4c-4c60-92fe-d10684f320cb" containerID="c38ad4a578c662bc2a5301a16caea0c2d93a3b2a0ceedc75bff2eee3c13ea2bd" exitCode=0 Jul 06 22:42:57.492128 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:57.491669 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-63b31a-6559858d4c-kx2dv" Jul 06 22:42:57.492128 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:57.491681 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-63b31a-6559858d4c-kx2dv" event={"ID":"074ca443-ba4c-4c60-92fe-d10684f320cb","Type":"ContainerDied","Data":"c38ad4a578c662bc2a5301a16caea0c2d93a3b2a0ceedc75bff2eee3c13ea2bd"} Jul 06 22:42:57.492128 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:57.491725 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-63b31a-6559858d4c-kx2dv" event={"ID":"074ca443-ba4c-4c60-92fe-d10684f320cb","Type":"ContainerDied","Data":"fc4bc40d84a889e87ab7bf3bb6873b038df18832db909f154de077bd3207cfb9"} Jul 06 22:42:57.492128 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:57.491747 2555 scope.go:117] "RemoveContainer" containerID="c38ad4a578c662bc2a5301a16caea0c2d93a3b2a0ceedc75bff2eee3c13ea2bd" Jul 06 22:42:57.501071 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:57.501049 2555 scope.go:117] "RemoveContainer" containerID="c38ad4a578c662bc2a5301a16caea0c2d93a3b2a0ceedc75bff2eee3c13ea2bd" Jul 06 22:42:57.501386 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:42:57.501348 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c38ad4a578c662bc2a5301a16caea0c2d93a3b2a0ceedc75bff2eee3c13ea2bd\": container with ID starting with c38ad4a578c662bc2a5301a16caea0c2d93a3b2a0ceedc75bff2eee3c13ea2bd not found: ID does not exist" containerID="c38ad4a578c662bc2a5301a16caea0c2d93a3b2a0ceedc75bff2eee3c13ea2bd" Jul 06 22:42:57.501461 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:57.501393 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c38ad4a578c662bc2a5301a16caea0c2d93a3b2a0ceedc75bff2eee3c13ea2bd"} err="failed to get container status \"c38ad4a578c662bc2a5301a16caea0c2d93a3b2a0ceedc75bff2eee3c13ea2bd\": rpc error: code = NotFound desc = could not find container \"c38ad4a578c662bc2a5301a16caea0c2d93a3b2a0ceedc75bff2eee3c13ea2bd\": container with ID starting with c38ad4a578c662bc2a5301a16caea0c2d93a3b2a0ceedc75bff2eee3c13ea2bd not found: ID does not exist" Jul 06 22:42:57.509624 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:57.509593 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-63b31a-6559858d4c-kx2dv"] Jul 06 22:42:57.513544 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:57.513519 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-b-63b31a-6559858d4c-kx2dv"] Jul 06 22:42:59.339456 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:42:59.339419 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="074ca443-ba4c-4c60-92fe-d10684f320cb" path="/var/lib/kubelet/pods/074ca443-ba4c-4c60-92fe-d10684f320cb/volumes" Jul 06 22:43:06.404819 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:06.404784 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-a-65387d-b7947677b-5cblp"] Jul 06 22:43:06.405310 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:06.405291 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="074ca443-ba4c-4c60-92fe-d10684f320cb" containerName="maas-api" Jul 06 22:43:06.405356 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:06.405313 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="074ca443-ba4c-4c60-92fe-d10684f320cb" containerName="maas-api" Jul 06 22:43:06.405356 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:06.405333 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f56c14d-5607-40f6-83f9-82429e390f6e" containerName="maas-api" Jul 06 22:43:06.405356 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:06.405339 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f56c14d-5607-40f6-83f9-82429e390f6e" containerName="maas-api" Jul 06 22:43:06.405454 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:06.405413 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f56c14d-5607-40f6-83f9-82429e390f6e" containerName="maas-api" Jul 06 22:43:06.405454 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:06.405421 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="074ca443-ba4c-4c60-92fe-d10684f320cb" containerName="maas-api" Jul 06 22:43:06.409909 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:06.409893 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-65387d-b7947677b-5cblp" Jul 06 22:43:06.412700 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:06.412675 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-a-65387d-serving-cert\"" Jul 06 22:43:06.418662 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:06.418634 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-65387d-b7947677b-5cblp"] Jul 06 22:43:06.469632 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:06.469586 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8bc2d37-3211-464c-8393-abc51bddbdbd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-65387d-b7947677b-5cblp\" (UID: \"a8bc2d37-3211-464c-8393-abc51bddbdbd\") " pod="opendatahub/maas-api-e2e-shared-a-65387d-b7947677b-5cblp" Jul 06 22:43:06.469835 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:06.469685 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8bc2d37-3211-464c-8393-abc51bddbdbd-maas-api-tls\") pod \"maas-api-e2e-shared-a-65387d-b7947677b-5cblp\" (UID: \"a8bc2d37-3211-464c-8393-abc51bddbdbd\") " pod="opendatahub/maas-api-e2e-shared-a-65387d-b7947677b-5cblp" Jul 06 22:43:06.469835 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:06.469753 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68686\" (UniqueName: \"kubernetes.io/projected/a8bc2d37-3211-464c-8393-abc51bddbdbd-kube-api-access-68686\") pod \"maas-api-e2e-shared-a-65387d-b7947677b-5cblp\" (UID: \"a8bc2d37-3211-464c-8393-abc51bddbdbd\") " pod="opendatahub/maas-api-e2e-shared-a-65387d-b7947677b-5cblp" Jul 06 22:43:06.571400 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:06.571366 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8bc2d37-3211-464c-8393-abc51bddbdbd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-65387d-b7947677b-5cblp\" (UID: \"a8bc2d37-3211-464c-8393-abc51bddbdbd\") " pod="opendatahub/maas-api-e2e-shared-a-65387d-b7947677b-5cblp" Jul 06 22:43:06.571400 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:06.571411 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8bc2d37-3211-464c-8393-abc51bddbdbd-maas-api-tls\") pod \"maas-api-e2e-shared-a-65387d-b7947677b-5cblp\" (UID: \"a8bc2d37-3211-464c-8393-abc51bddbdbd\") " pod="opendatahub/maas-api-e2e-shared-a-65387d-b7947677b-5cblp" Jul 06 22:43:06.571639 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:43:06.571507 2555 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-shared-a-65387d-serving-cert: secret "maas-api-e2e-shared-a-65387d-serving-cert" not found Jul 06 22:43:06.571639 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:06.571532 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68686\" (UniqueName: \"kubernetes.io/projected/a8bc2d37-3211-464c-8393-abc51bddbdbd-kube-api-access-68686\") pod \"maas-api-e2e-shared-a-65387d-b7947677b-5cblp\" (UID: \"a8bc2d37-3211-464c-8393-abc51bddbdbd\") " pod="opendatahub/maas-api-e2e-shared-a-65387d-b7947677b-5cblp" Jul 06 22:43:06.571639 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:43:06.571563 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8bc2d37-3211-464c-8393-abc51bddbdbd-maas-api-tls podName:a8bc2d37-3211-464c-8393-abc51bddbdbd nodeName:}" failed. No retries permitted until 2026-07-06 22:43:07.071547943 +0000 UTC m=+2038.371259770 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a8bc2d37-3211-464c-8393-abc51bddbdbd-maas-api-tls") pod "maas-api-e2e-shared-a-65387d-b7947677b-5cblp" (UID: "a8bc2d37-3211-464c-8393-abc51bddbdbd") : secret "maas-api-e2e-shared-a-65387d-serving-cert" not found Jul 06 22:43:06.572053 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:06.572002 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8bc2d37-3211-464c-8393-abc51bddbdbd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-65387d-b7947677b-5cblp\" (UID: \"a8bc2d37-3211-464c-8393-abc51bddbdbd\") " pod="opendatahub/maas-api-e2e-shared-a-65387d-b7947677b-5cblp" Jul 06 22:43:06.592902 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:06.592862 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68686\" (UniqueName: \"kubernetes.io/projected/a8bc2d37-3211-464c-8393-abc51bddbdbd-kube-api-access-68686\") pod \"maas-api-e2e-shared-a-65387d-b7947677b-5cblp\" (UID: \"a8bc2d37-3211-464c-8393-abc51bddbdbd\") " pod="opendatahub/maas-api-e2e-shared-a-65387d-b7947677b-5cblp" Jul 06 22:43:07.076924 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:07.076879 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8bc2d37-3211-464c-8393-abc51bddbdbd-maas-api-tls\") pod \"maas-api-e2e-shared-a-65387d-b7947677b-5cblp\" (UID: \"a8bc2d37-3211-464c-8393-abc51bddbdbd\") " pod="opendatahub/maas-api-e2e-shared-a-65387d-b7947677b-5cblp" Jul 06 22:43:07.079440 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:07.079413 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8bc2d37-3211-464c-8393-abc51bddbdbd-maas-api-tls\") pod \"maas-api-e2e-shared-a-65387d-b7947677b-5cblp\" (UID: \"a8bc2d37-3211-464c-8393-abc51bddbdbd\") " pod="opendatahub/maas-api-e2e-shared-a-65387d-b7947677b-5cblp" Jul 06 22:43:07.322185 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:07.322138 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-65387d-b7947677b-5cblp" Jul 06 22:43:07.473874 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:07.473835 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-65387d-b7947677b-5cblp"] Jul 06 22:43:07.474994 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:43:07.474966 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8bc2d37_3211_464c_8393_abc51bddbdbd.slice/crio-85129520950ba99bce1fc6eb7bb44f979292e2afa145fbdea0b8e1f09a81cfa7 WatchSource:0}: Error finding container 85129520950ba99bce1fc6eb7bb44f979292e2afa145fbdea0b8e1f09a81cfa7: Status 404 returned error can't find the container with id 85129520950ba99bce1fc6eb7bb44f979292e2afa145fbdea0b8e1f09a81cfa7 Jul 06 22:43:07.543345 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:07.543309 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-65387d-b7947677b-5cblp" event={"ID":"a8bc2d37-3211-464c-8393-abc51bddbdbd","Type":"ContainerStarted","Data":"85129520950ba99bce1fc6eb7bb44f979292e2afa145fbdea0b8e1f09a81cfa7"} Jul 06 22:43:08.549541 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:08.549503 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-65387d-b7947677b-5cblp" event={"ID":"a8bc2d37-3211-464c-8393-abc51bddbdbd","Type":"ContainerStarted","Data":"4281163b9ebda52f890d6941b02f700f13b9ef97a1e228045e5f210c5529e13c"} Jul 06 22:43:08.549951 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:08.549613 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-a-65387d-b7947677b-5cblp" Jul 06 22:43:08.585705 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:08.585651 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-a-65387d-b7947677b-5cblp" podStartSLOduration=2.1409395350000002 podStartE2EDuration="2.585635195s" totalImagesPullingTime="444.69566ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:43:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:43:08.584791891 +0000 UTC m=+2039.884503742" watchObservedRunningTime="2026-07-06 22:43:08.585635195 +0000 UTC m=+2039.885347044" Jul 06 22:43:14.523915 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:14.523882 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-b-944f9f-58fdff6c46-6nfjh"] Jul 06 22:43:14.527345 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:14.527322 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-944f9f-58fdff6c46-6nfjh" Jul 06 22:43:14.530376 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:14.530348 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-b-944f9f-serving-cert\"" Jul 06 22:43:14.538248 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:14.538221 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-944f9f-58fdff6c46-6nfjh"] Jul 06 22:43:14.559462 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:14.559430 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-a-65387d-b7947677b-5cblp" Jul 06 22:43:14.651405 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:14.651364 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj4xk\" (UniqueName: \"kubernetes.io/projected/67a6cf3b-4068-4cac-a2ec-6a2bb92dc1b4-kube-api-access-zj4xk\") pod \"maas-api-e2e-shared-b-944f9f-58fdff6c46-6nfjh\" (UID: \"67a6cf3b-4068-4cac-a2ec-6a2bb92dc1b4\") " pod="opendatahub/maas-api-e2e-shared-b-944f9f-58fdff6c46-6nfjh" Jul 06 22:43:14.651590 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:14.651496 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67a6cf3b-4068-4cac-a2ec-6a2bb92dc1b4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-944f9f-58fdff6c46-6nfjh\" (UID: \"67a6cf3b-4068-4cac-a2ec-6a2bb92dc1b4\") " pod="opendatahub/maas-api-e2e-shared-b-944f9f-58fdff6c46-6nfjh" Jul 06 22:43:14.651590 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:14.651552 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67a6cf3b-4068-4cac-a2ec-6a2bb92dc1b4-maas-api-tls\") pod \"maas-api-e2e-shared-b-944f9f-58fdff6c46-6nfjh\" (UID: \"67a6cf3b-4068-4cac-a2ec-6a2bb92dc1b4\") " pod="opendatahub/maas-api-e2e-shared-b-944f9f-58fdff6c46-6nfjh" Jul 06 22:43:14.752245 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:14.752208 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67a6cf3b-4068-4cac-a2ec-6a2bb92dc1b4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-944f9f-58fdff6c46-6nfjh\" (UID: \"67a6cf3b-4068-4cac-a2ec-6a2bb92dc1b4\") " pod="opendatahub/maas-api-e2e-shared-b-944f9f-58fdff6c46-6nfjh" Jul 06 22:43:14.752443 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:14.752271 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67a6cf3b-4068-4cac-a2ec-6a2bb92dc1b4-maas-api-tls\") pod \"maas-api-e2e-shared-b-944f9f-58fdff6c46-6nfjh\" (UID: \"67a6cf3b-4068-4cac-a2ec-6a2bb92dc1b4\") " pod="opendatahub/maas-api-e2e-shared-b-944f9f-58fdff6c46-6nfjh" Jul 06 22:43:14.752443 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:14.752336 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj4xk\" (UniqueName: \"kubernetes.io/projected/67a6cf3b-4068-4cac-a2ec-6a2bb92dc1b4-kube-api-access-zj4xk\") pod \"maas-api-e2e-shared-b-944f9f-58fdff6c46-6nfjh\" (UID: \"67a6cf3b-4068-4cac-a2ec-6a2bb92dc1b4\") " pod="opendatahub/maas-api-e2e-shared-b-944f9f-58fdff6c46-6nfjh" Jul 06 22:43:14.752443 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:43:14.752427 2555 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-shared-b-944f9f-serving-cert: secret "maas-api-e2e-shared-b-944f9f-serving-cert" not found Jul 06 22:43:14.752608 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:43:14.752491 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67a6cf3b-4068-4cac-a2ec-6a2bb92dc1b4-maas-api-tls podName:67a6cf3b-4068-4cac-a2ec-6a2bb92dc1b4 nodeName:}" failed. No retries permitted until 2026-07-06 22:43:15.252473322 +0000 UTC m=+2046.552185149 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/67a6cf3b-4068-4cac-a2ec-6a2bb92dc1b4-maas-api-tls") pod "maas-api-e2e-shared-b-944f9f-58fdff6c46-6nfjh" (UID: "67a6cf3b-4068-4cac-a2ec-6a2bb92dc1b4") : secret "maas-api-e2e-shared-b-944f9f-serving-cert" not found Jul 06 22:43:14.752861 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:14.752844 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67a6cf3b-4068-4cac-a2ec-6a2bb92dc1b4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-944f9f-58fdff6c46-6nfjh\" (UID: \"67a6cf3b-4068-4cac-a2ec-6a2bb92dc1b4\") " pod="opendatahub/maas-api-e2e-shared-b-944f9f-58fdff6c46-6nfjh" Jul 06 22:43:14.781258 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:14.781179 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj4xk\" (UniqueName: \"kubernetes.io/projected/67a6cf3b-4068-4cac-a2ec-6a2bb92dc1b4-kube-api-access-zj4xk\") pod \"maas-api-e2e-shared-b-944f9f-58fdff6c46-6nfjh\" (UID: \"67a6cf3b-4068-4cac-a2ec-6a2bb92dc1b4\") " pod="opendatahub/maas-api-e2e-shared-b-944f9f-58fdff6c46-6nfjh" Jul 06 22:43:15.257358 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:15.257317 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67a6cf3b-4068-4cac-a2ec-6a2bb92dc1b4-maas-api-tls\") pod \"maas-api-e2e-shared-b-944f9f-58fdff6c46-6nfjh\" (UID: \"67a6cf3b-4068-4cac-a2ec-6a2bb92dc1b4\") " pod="opendatahub/maas-api-e2e-shared-b-944f9f-58fdff6c46-6nfjh" Jul 06 22:43:15.259778 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:15.259752 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67a6cf3b-4068-4cac-a2ec-6a2bb92dc1b4-maas-api-tls\") pod \"maas-api-e2e-shared-b-944f9f-58fdff6c46-6nfjh\" (UID: \"67a6cf3b-4068-4cac-a2ec-6a2bb92dc1b4\") " pod="opendatahub/maas-api-e2e-shared-b-944f9f-58fdff6c46-6nfjh" Jul 06 22:43:15.438526 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:15.438483 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-944f9f-58fdff6c46-6nfjh" Jul 06 22:43:15.574175 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:15.574147 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-944f9f-58fdff6c46-6nfjh"] Jul 06 22:43:15.576314 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:43:15.576284 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67a6cf3b_4068_4cac_a2ec_6a2bb92dc1b4.slice/crio-3ef3caed0d4934594494c48b2801fcf136ddefcc04fb2a155c4c27ef459b011a WatchSource:0}: Error finding container 3ef3caed0d4934594494c48b2801fcf136ddefcc04fb2a155c4c27ef459b011a: Status 404 returned error can't find the container with id 3ef3caed0d4934594494c48b2801fcf136ddefcc04fb2a155c4c27ef459b011a Jul 06 22:43:15.592195 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:15.592163 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-944f9f-58fdff6c46-6nfjh" event={"ID":"67a6cf3b-4068-4cac-a2ec-6a2bb92dc1b4","Type":"ContainerStarted","Data":"3ef3caed0d4934594494c48b2801fcf136ddefcc04fb2a155c4c27ef459b011a"} Jul 06 22:43:16.605531 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:16.605493 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-944f9f-58fdff6c46-6nfjh" event={"ID":"67a6cf3b-4068-4cac-a2ec-6a2bb92dc1b4","Type":"ContainerStarted","Data":"cec6efba97648e70c6ca19bc78f3218a8f050fada5f68f0985a0a1b5dadbeee4"} Jul 06 22:43:16.605955 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:16.605612 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-b-944f9f-58fdff6c46-6nfjh" Jul 06 22:43:16.623335 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:16.623278 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-b-944f9f-58fdff6c46-6nfjh" podStartSLOduration=2.1204650210000002 podStartE2EDuration="2.623261299s" totalImagesPullingTime="502.796278ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:43:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:43:16.621651623 +0000 UTC m=+2047.921363472" watchObservedRunningTime="2026-07-06 22:43:16.623261299 +0000 UTC m=+2047.922973148" Jul 06 22:43:22.615849 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:22.615814 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-b-944f9f-58fdff6c46-6nfjh" Jul 06 22:43:25.596508 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:25.596472 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-65387d/auth-test-model-65387d-kserve-7d9f774599-5b44l"] Jul 06 22:43:25.600339 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:25.600315 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-65387d/auth-test-model-65387d-kserve-7d9f774599-5b44l" Jul 06 22:43:25.604390 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:25.604367 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-65387d\"/\"openshift-service-ca.crt\"" Jul 06 22:43:25.604538 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:25.604410 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-65387d\"/\"default-dockercfg-6qls2\"" Jul 06 22:43:25.604538 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:25.604366 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-65387d\"/\"kube-root-ca.crt\"" Jul 06 22:43:25.604538 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:25.604472 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-65387d\"/\"auth-test-model-65387d-kserve-self-signed-certs\"" Jul 06 22:43:25.611071 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:25.611044 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-65387d/auth-test-model-65387d-kserve-7d9f774599-5b44l"] Jul 06 22:43:25.655650 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:25.655617 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd96e2e4-b951-4367-b57c-81d1fc57b00e-tls-certs\") pod \"auth-test-model-65387d-kserve-7d9f774599-5b44l\" (UID: \"cd96e2e4-b951-4367-b57c-81d1fc57b00e\") " pod="ai-tenant-e2e-shared-a-65387d/auth-test-model-65387d-kserve-7d9f774599-5b44l" Jul 06 22:43:25.655650 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:25.655653 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkhcl\" (UniqueName: \"kubernetes.io/projected/cd96e2e4-b951-4367-b57c-81d1fc57b00e-kube-api-access-mkhcl\") pod \"auth-test-model-65387d-kserve-7d9f774599-5b44l\" (UID: \"cd96e2e4-b951-4367-b57c-81d1fc57b00e\") " pod="ai-tenant-e2e-shared-a-65387d/auth-test-model-65387d-kserve-7d9f774599-5b44l" Jul 06 22:43:25.655877 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:25.655688 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd96e2e4-b951-4367-b57c-81d1fc57b00e-dshm\") pod \"auth-test-model-65387d-kserve-7d9f774599-5b44l\" (UID: \"cd96e2e4-b951-4367-b57c-81d1fc57b00e\") " pod="ai-tenant-e2e-shared-a-65387d/auth-test-model-65387d-kserve-7d9f774599-5b44l" Jul 06 22:43:25.655877 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:25.655729 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd96e2e4-b951-4367-b57c-81d1fc57b00e-home\") pod \"auth-test-model-65387d-kserve-7d9f774599-5b44l\" (UID: \"cd96e2e4-b951-4367-b57c-81d1fc57b00e\") " pod="ai-tenant-e2e-shared-a-65387d/auth-test-model-65387d-kserve-7d9f774599-5b44l" Jul 06 22:43:25.655877 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:25.655752 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd96e2e4-b951-4367-b57c-81d1fc57b00e-model-cache\") pod \"auth-test-model-65387d-kserve-7d9f774599-5b44l\" (UID: \"cd96e2e4-b951-4367-b57c-81d1fc57b00e\") " pod="ai-tenant-e2e-shared-a-65387d/auth-test-model-65387d-kserve-7d9f774599-5b44l" Jul 06 22:43:25.655877 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:25.655769 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd96e2e4-b951-4367-b57c-81d1fc57b00e-tmp-dir\") pod \"auth-test-model-65387d-kserve-7d9f774599-5b44l\" (UID: \"cd96e2e4-b951-4367-b57c-81d1fc57b00e\") " pod="ai-tenant-e2e-shared-a-65387d/auth-test-model-65387d-kserve-7d9f774599-5b44l" Jul 06 22:43:25.756406 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:25.756363 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd96e2e4-b951-4367-b57c-81d1fc57b00e-tls-certs\") pod \"auth-test-model-65387d-kserve-7d9f774599-5b44l\" (UID: \"cd96e2e4-b951-4367-b57c-81d1fc57b00e\") " pod="ai-tenant-e2e-shared-a-65387d/auth-test-model-65387d-kserve-7d9f774599-5b44l" Jul 06 22:43:25.756599 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:25.756412 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkhcl\" (UniqueName: \"kubernetes.io/projected/cd96e2e4-b951-4367-b57c-81d1fc57b00e-kube-api-access-mkhcl\") pod \"auth-test-model-65387d-kserve-7d9f774599-5b44l\" (UID: \"cd96e2e4-b951-4367-b57c-81d1fc57b00e\") " pod="ai-tenant-e2e-shared-a-65387d/auth-test-model-65387d-kserve-7d9f774599-5b44l" Jul 06 22:43:25.756599 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:25.756447 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd96e2e4-b951-4367-b57c-81d1fc57b00e-dshm\") pod \"auth-test-model-65387d-kserve-7d9f774599-5b44l\" (UID: \"cd96e2e4-b951-4367-b57c-81d1fc57b00e\") " pod="ai-tenant-e2e-shared-a-65387d/auth-test-model-65387d-kserve-7d9f774599-5b44l" Jul 06 22:43:25.756599 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:25.756468 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd96e2e4-b951-4367-b57c-81d1fc57b00e-home\") pod \"auth-test-model-65387d-kserve-7d9f774599-5b44l\" (UID: \"cd96e2e4-b951-4367-b57c-81d1fc57b00e\") " pod="ai-tenant-e2e-shared-a-65387d/auth-test-model-65387d-kserve-7d9f774599-5b44l" Jul 06 22:43:25.756599 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:25.756491 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd96e2e4-b951-4367-b57c-81d1fc57b00e-model-cache\") pod \"auth-test-model-65387d-kserve-7d9f774599-5b44l\" (UID: \"cd96e2e4-b951-4367-b57c-81d1fc57b00e\") " pod="ai-tenant-e2e-shared-a-65387d/auth-test-model-65387d-kserve-7d9f774599-5b44l" Jul 06 22:43:25.756599 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:25.756507 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd96e2e4-b951-4367-b57c-81d1fc57b00e-tmp-dir\") pod \"auth-test-model-65387d-kserve-7d9f774599-5b44l\" (UID: \"cd96e2e4-b951-4367-b57c-81d1fc57b00e\") " pod="ai-tenant-e2e-shared-a-65387d/auth-test-model-65387d-kserve-7d9f774599-5b44l" Jul 06 22:43:25.756985 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:25.756962 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd96e2e4-b951-4367-b57c-81d1fc57b00e-model-cache\") pod \"auth-test-model-65387d-kserve-7d9f774599-5b44l\" (UID: \"cd96e2e4-b951-4367-b57c-81d1fc57b00e\") " pod="ai-tenant-e2e-shared-a-65387d/auth-test-model-65387d-kserve-7d9f774599-5b44l" Jul 06 22:43:25.757095 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:25.756998 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd96e2e4-b951-4367-b57c-81d1fc57b00e-home\") pod \"auth-test-model-65387d-kserve-7d9f774599-5b44l\" (UID: \"cd96e2e4-b951-4367-b57c-81d1fc57b00e\") " pod="ai-tenant-e2e-shared-a-65387d/auth-test-model-65387d-kserve-7d9f774599-5b44l" Jul 06 22:43:25.757095 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:25.757041 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd96e2e4-b951-4367-b57c-81d1fc57b00e-tmp-dir\") pod \"auth-test-model-65387d-kserve-7d9f774599-5b44l\" (UID: \"cd96e2e4-b951-4367-b57c-81d1fc57b00e\") " pod="ai-tenant-e2e-shared-a-65387d/auth-test-model-65387d-kserve-7d9f774599-5b44l" Jul 06 22:43:25.758809 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:25.758788 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd96e2e4-b951-4367-b57c-81d1fc57b00e-dshm\") pod \"auth-test-model-65387d-kserve-7d9f774599-5b44l\" (UID: \"cd96e2e4-b951-4367-b57c-81d1fc57b00e\") " pod="ai-tenant-e2e-shared-a-65387d/auth-test-model-65387d-kserve-7d9f774599-5b44l" Jul 06 22:43:25.759075 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:25.759056 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd96e2e4-b951-4367-b57c-81d1fc57b00e-tls-certs\") pod \"auth-test-model-65387d-kserve-7d9f774599-5b44l\" (UID: \"cd96e2e4-b951-4367-b57c-81d1fc57b00e\") " pod="ai-tenant-e2e-shared-a-65387d/auth-test-model-65387d-kserve-7d9f774599-5b44l" Jul 06 22:43:25.767492 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:25.767463 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkhcl\" (UniqueName: \"kubernetes.io/projected/cd96e2e4-b951-4367-b57c-81d1fc57b00e-kube-api-access-mkhcl\") pod \"auth-test-model-65387d-kserve-7d9f774599-5b44l\" (UID: \"cd96e2e4-b951-4367-b57c-81d1fc57b00e\") " pod="ai-tenant-e2e-shared-a-65387d/auth-test-model-65387d-kserve-7d9f774599-5b44l" Jul 06 22:43:25.911524 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:25.911482 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-65387d/auth-test-model-65387d-kserve-7d9f774599-5b44l" Jul 06 22:43:26.047655 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:26.047631 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-65387d/auth-test-model-65387d-kserve-7d9f774599-5b44l"] Jul 06 22:43:26.049677 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:43:26.049647 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd96e2e4_b951_4367_b57c_81d1fc57b00e.slice/crio-d483e8ba7abcd9c124b618ba545febca0117b573b5c8b55a4bdb3eddcc3abcb7 WatchSource:0}: Error finding container d483e8ba7abcd9c124b618ba545febca0117b573b5c8b55a4bdb3eddcc3abcb7: Status 404 returned error can't find the container with id d483e8ba7abcd9c124b618ba545febca0117b573b5c8b55a4bdb3eddcc3abcb7 Jul 06 22:43:26.658108 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:26.658072 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-65387d/auth-test-model-65387d-kserve-7d9f774599-5b44l" event={"ID":"cd96e2e4-b951-4367-b57c-81d1fc57b00e","Type":"ContainerStarted","Data":"5ad226fe96e32ff0a0d01cf57d0ebb6ca682f5f1e66e4f571097fecd15d58911"} Jul 06 22:43:26.658499 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:26.658121 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-65387d/auth-test-model-65387d-kserve-7d9f774599-5b44l" event={"ID":"cd96e2e4-b951-4367-b57c-81d1fc57b00e","Type":"ContainerStarted","Data":"d483e8ba7abcd9c124b618ba545febca0117b573b5c8b55a4bdb3eddcc3abcb7"} Jul 06 22:43:26.676181 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:26.676114 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-65387d/auth-test-model-65387d-kserve-7d9f774599-5b44l" podStartSLOduration=1.676098671 podStartE2EDuration="1.676098671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:43:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:43:26.673225674 +0000 UTC m=+2057.972937547" watchObservedRunningTime="2026-07-06 22:43:26.676098671 +0000 UTC m=+2057.975810519" Jul 06 22:43:35.912602 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:35.912511 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-65387d/auth-test-model-65387d-kserve-7d9f774599-5b44l" Jul 06 22:43:35.912602 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:35.912561 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-65387d/auth-test-model-65387d-kserve-7d9f774599-5b44l" Jul 06 22:43:35.925174 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:35.925147 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-65387d/auth-test-model-65387d-kserve-7d9f774599-5b44l" Jul 06 22:43:36.718114 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:36.718082 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-65387d/auth-test-model-65387d-kserve-7d9f774599-5b44l" Jul 06 22:43:57.607245 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:57.607213 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-944f9f/auth-test-model-944f9f-kserve-7bc56b8fd7-wcfkx"] Jul 06 22:43:57.611062 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:57.611040 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-944f9f/auth-test-model-944f9f-kserve-7bc56b8fd7-wcfkx" Jul 06 22:43:57.613801 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:57.613774 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-944f9f\"/\"kube-root-ca.crt\"" Jul 06 22:43:57.615329 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:57.615303 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-944f9f\"/\"openshift-service-ca.crt\"" Jul 06 22:43:57.615448 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:57.615303 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-944f9f\"/\"auth-test-model-944f9f-kserve-self-signed-certs\"" Jul 06 22:43:57.615448 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:57.615308 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-944f9f\"/\"default-dockercfg-8tnxt\"" Jul 06 22:43:57.620284 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:57.620257 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-944f9f/auth-test-model-944f9f-kserve-7bc56b8fd7-wcfkx"] Jul 06 22:43:57.777702 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:57.777665 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b32269df-1268-4a51-94af-655b26f959b0-tmp-dir\") pod \"auth-test-model-944f9f-kserve-7bc56b8fd7-wcfkx\" (UID: \"b32269df-1268-4a51-94af-655b26f959b0\") " pod="ai-tenant-e2e-shared-b-944f9f/auth-test-model-944f9f-kserve-7bc56b8fd7-wcfkx" Jul 06 22:43:57.777878 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:57.777719 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b32269df-1268-4a51-94af-655b26f959b0-tls-certs\") pod \"auth-test-model-944f9f-kserve-7bc56b8fd7-wcfkx\" (UID: \"b32269df-1268-4a51-94af-655b26f959b0\") " pod="ai-tenant-e2e-shared-b-944f9f/auth-test-model-944f9f-kserve-7bc56b8fd7-wcfkx" Jul 06 22:43:57.777878 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:57.777771 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b32269df-1268-4a51-94af-655b26f959b0-model-cache\") pod \"auth-test-model-944f9f-kserve-7bc56b8fd7-wcfkx\" (UID: \"b32269df-1268-4a51-94af-655b26f959b0\") " pod="ai-tenant-e2e-shared-b-944f9f/auth-test-model-944f9f-kserve-7bc56b8fd7-wcfkx" Jul 06 22:43:57.777878 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:57.777806 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b32269df-1268-4a51-94af-655b26f959b0-dshm\") pod \"auth-test-model-944f9f-kserve-7bc56b8fd7-wcfkx\" (UID: \"b32269df-1268-4a51-94af-655b26f959b0\") " pod="ai-tenant-e2e-shared-b-944f9f/auth-test-model-944f9f-kserve-7bc56b8fd7-wcfkx" Jul 06 22:43:57.777878 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:57.777867 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g75q\" (UniqueName: \"kubernetes.io/projected/b32269df-1268-4a51-94af-655b26f959b0-kube-api-access-5g75q\") pod \"auth-test-model-944f9f-kserve-7bc56b8fd7-wcfkx\" (UID: \"b32269df-1268-4a51-94af-655b26f959b0\") " pod="ai-tenant-e2e-shared-b-944f9f/auth-test-model-944f9f-kserve-7bc56b8fd7-wcfkx" Jul 06 22:43:57.778040 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:57.777912 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b32269df-1268-4a51-94af-655b26f959b0-home\") pod \"auth-test-model-944f9f-kserve-7bc56b8fd7-wcfkx\" (UID: \"b32269df-1268-4a51-94af-655b26f959b0\") " pod="ai-tenant-e2e-shared-b-944f9f/auth-test-model-944f9f-kserve-7bc56b8fd7-wcfkx" Jul 06 22:43:57.878710 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:57.878618 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b32269df-1268-4a51-94af-655b26f959b0-dshm\") pod \"auth-test-model-944f9f-kserve-7bc56b8fd7-wcfkx\" (UID: \"b32269df-1268-4a51-94af-655b26f959b0\") " pod="ai-tenant-e2e-shared-b-944f9f/auth-test-model-944f9f-kserve-7bc56b8fd7-wcfkx" Jul 06 22:43:57.878710 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:57.878696 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g75q\" (UniqueName: \"kubernetes.io/projected/b32269df-1268-4a51-94af-655b26f959b0-kube-api-access-5g75q\") pod \"auth-test-model-944f9f-kserve-7bc56b8fd7-wcfkx\" (UID: \"b32269df-1268-4a51-94af-655b26f959b0\") " pod="ai-tenant-e2e-shared-b-944f9f/auth-test-model-944f9f-kserve-7bc56b8fd7-wcfkx" Jul 06 22:43:57.878926 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:57.878741 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b32269df-1268-4a51-94af-655b26f959b0-home\") pod \"auth-test-model-944f9f-kserve-7bc56b8fd7-wcfkx\" (UID: \"b32269df-1268-4a51-94af-655b26f959b0\") " pod="ai-tenant-e2e-shared-b-944f9f/auth-test-model-944f9f-kserve-7bc56b8fd7-wcfkx" Jul 06 22:43:57.878926 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:57.878762 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b32269df-1268-4a51-94af-655b26f959b0-tmp-dir\") pod \"auth-test-model-944f9f-kserve-7bc56b8fd7-wcfkx\" (UID: \"b32269df-1268-4a51-94af-655b26f959b0\") " pod="ai-tenant-e2e-shared-b-944f9f/auth-test-model-944f9f-kserve-7bc56b8fd7-wcfkx" Jul 06 22:43:57.878926 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:57.878791 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b32269df-1268-4a51-94af-655b26f959b0-tls-certs\") pod \"auth-test-model-944f9f-kserve-7bc56b8fd7-wcfkx\" (UID: \"b32269df-1268-4a51-94af-655b26f959b0\") " pod="ai-tenant-e2e-shared-b-944f9f/auth-test-model-944f9f-kserve-7bc56b8fd7-wcfkx" Jul 06 22:43:57.878926 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:57.878828 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b32269df-1268-4a51-94af-655b26f959b0-model-cache\") pod \"auth-test-model-944f9f-kserve-7bc56b8fd7-wcfkx\" (UID: \"b32269df-1268-4a51-94af-655b26f959b0\") " pod="ai-tenant-e2e-shared-b-944f9f/auth-test-model-944f9f-kserve-7bc56b8fd7-wcfkx" Jul 06 22:43:57.879192 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:57.879171 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b32269df-1268-4a51-94af-655b26f959b0-home\") pod \"auth-test-model-944f9f-kserve-7bc56b8fd7-wcfkx\" (UID: \"b32269df-1268-4a51-94af-655b26f959b0\") " pod="ai-tenant-e2e-shared-b-944f9f/auth-test-model-944f9f-kserve-7bc56b8fd7-wcfkx" Jul 06 22:43:57.879288 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:57.879228 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b32269df-1268-4a51-94af-655b26f959b0-model-cache\") pod \"auth-test-model-944f9f-kserve-7bc56b8fd7-wcfkx\" (UID: \"b32269df-1268-4a51-94af-655b26f959b0\") " pod="ai-tenant-e2e-shared-b-944f9f/auth-test-model-944f9f-kserve-7bc56b8fd7-wcfkx" Jul 06 22:43:57.879288 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:57.879260 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b32269df-1268-4a51-94af-655b26f959b0-tmp-dir\") pod \"auth-test-model-944f9f-kserve-7bc56b8fd7-wcfkx\" (UID: \"b32269df-1268-4a51-94af-655b26f959b0\") " pod="ai-tenant-e2e-shared-b-944f9f/auth-test-model-944f9f-kserve-7bc56b8fd7-wcfkx" Jul 06 22:43:57.881101 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:57.881075 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b32269df-1268-4a51-94af-655b26f959b0-dshm\") pod \"auth-test-model-944f9f-kserve-7bc56b8fd7-wcfkx\" (UID: \"b32269df-1268-4a51-94af-655b26f959b0\") " pod="ai-tenant-e2e-shared-b-944f9f/auth-test-model-944f9f-kserve-7bc56b8fd7-wcfkx" Jul 06 22:43:57.881244 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:57.881228 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b32269df-1268-4a51-94af-655b26f959b0-tls-certs\") pod \"auth-test-model-944f9f-kserve-7bc56b8fd7-wcfkx\" (UID: \"b32269df-1268-4a51-94af-655b26f959b0\") " pod="ai-tenant-e2e-shared-b-944f9f/auth-test-model-944f9f-kserve-7bc56b8fd7-wcfkx" Jul 06 22:43:57.887609 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:57.887582 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g75q\" (UniqueName: \"kubernetes.io/projected/b32269df-1268-4a51-94af-655b26f959b0-kube-api-access-5g75q\") pod \"auth-test-model-944f9f-kserve-7bc56b8fd7-wcfkx\" (UID: \"b32269df-1268-4a51-94af-655b26f959b0\") " pod="ai-tenant-e2e-shared-b-944f9f/auth-test-model-944f9f-kserve-7bc56b8fd7-wcfkx" Jul 06 22:43:57.923506 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:57.923465 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-944f9f/auth-test-model-944f9f-kserve-7bc56b8fd7-wcfkx" Jul 06 22:43:58.057644 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:58.057609 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-944f9f/auth-test-model-944f9f-kserve-7bc56b8fd7-wcfkx"] Jul 06 22:43:58.061525 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:43:58.061498 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb32269df_1268_4a51_94af_655b26f959b0.slice/crio-acd7ec8666293d3639553970dd51bd5531ffe6ee5f14dc3e2f8c0ecaffac629a WatchSource:0}: Error finding container acd7ec8666293d3639553970dd51bd5531ffe6ee5f14dc3e2f8c0ecaffac629a: Status 404 returned error can't find the container with id acd7ec8666293d3639553970dd51bd5531ffe6ee5f14dc3e2f8c0ecaffac629a Jul 06 22:43:58.819547 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:58.819448 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-944f9f/auth-test-model-944f9f-kserve-7bc56b8fd7-wcfkx" event={"ID":"b32269df-1268-4a51-94af-655b26f959b0","Type":"ContainerStarted","Data":"539d0543d40a3ed00af142512a58e166326e636721c6aff69b13389c41b8e160"} Jul 06 22:43:58.819547 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:58.819490 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-944f9f/auth-test-model-944f9f-kserve-7bc56b8fd7-wcfkx" event={"ID":"b32269df-1268-4a51-94af-655b26f959b0","Type":"ContainerStarted","Data":"acd7ec8666293d3639553970dd51bd5531ffe6ee5f14dc3e2f8c0ecaffac629a"} Jul 06 22:43:58.837831 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:43:58.837777 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-944f9f/auth-test-model-944f9f-kserve-7bc56b8fd7-wcfkx" podStartSLOduration=1.837762152 podStartE2EDuration="1.837762152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:43:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:43:58.835776822 +0000 UTC m=+2090.135488687" watchObservedRunningTime="2026-07-06 22:43:58.837762152 +0000 UTC m=+2090.137474005" Jul 06 22:44:07.923885 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:44:07.923845 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-944f9f/auth-test-model-944f9f-kserve-7bc56b8fd7-wcfkx" Jul 06 22:44:07.923885 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:44:07.923903 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-944f9f/auth-test-model-944f9f-kserve-7bc56b8fd7-wcfkx" Jul 06 22:44:07.936627 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:44:07.936598 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-944f9f/auth-test-model-944f9f-kserve-7bc56b8fd7-wcfkx" Jul 06 22:44:08.879522 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:44:08.879485 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-944f9f/auth-test-model-944f9f-kserve-7bc56b8fd7-wcfkx" Jul 06 22:44:09.346234 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:44:09.346126 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7mq8_e50d3b5f-832e-481f-8807-7774ff8d5521/ovn-acl-logging/0.log" Jul 06 22:44:09.351111 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:44:09.351086 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7mq8_e50d3b5f-832e-481f-8807-7774ff8d5521/ovn-acl-logging/0.log" Jul 06 22:45:47.581636 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:45:47.581597 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-65387d/auth-test-model-65387d-kserve-7d9f774599-5b44l"] Jul 06 22:45:48.596443 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:45:48.596401 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-944f9f/auth-test-model-944f9f-kserve-7bc56b8fd7-wcfkx"] Jul 06 22:46:02.603179 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:02.603136 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-65387d/auth-test-model-65387d-kserve-7d9f774599-5b44l" podUID="cd96e2e4-b951-4367-b57c-81d1fc57b00e" containerName="main" containerID="cri-o://5ad226fe96e32ff0a0d01cf57d0ebb6ca682f5f1e66e4f571097fecd15d58911" gracePeriod=45 Jul 06 22:46:02.851786 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:02.851757 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-65387d/auth-test-model-65387d-kserve-7d9f774599-5b44l" Jul 06 22:46:02.949577 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:02.949531 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mkhcl\" (UniqueName: \"kubernetes.io/projected/cd96e2e4-b951-4367-b57c-81d1fc57b00e-kube-api-access-mkhcl\") pod \"cd96e2e4-b951-4367-b57c-81d1fc57b00e\" (UID: \"cd96e2e4-b951-4367-b57c-81d1fc57b00e\") " Jul 06 22:46:02.949785 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:02.949635 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd96e2e4-b951-4367-b57c-81d1fc57b00e-model-cache\") pod \"cd96e2e4-b951-4367-b57c-81d1fc57b00e\" (UID: \"cd96e2e4-b951-4367-b57c-81d1fc57b00e\") " Jul 06 22:46:02.949785 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:02.949736 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd96e2e4-b951-4367-b57c-81d1fc57b00e-tmp-dir\") pod \"cd96e2e4-b951-4367-b57c-81d1fc57b00e\" (UID: \"cd96e2e4-b951-4367-b57c-81d1fc57b00e\") " Jul 06 22:46:02.949913 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:02.949815 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd96e2e4-b951-4367-b57c-81d1fc57b00e-home\") pod \"cd96e2e4-b951-4367-b57c-81d1fc57b00e\" (UID: \"cd96e2e4-b951-4367-b57c-81d1fc57b00e\") " Jul 06 22:46:02.949913 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:02.949832 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd96e2e4-b951-4367-b57c-81d1fc57b00e-tls-certs\") pod \"cd96e2e4-b951-4367-b57c-81d1fc57b00e\" (UID: \"cd96e2e4-b951-4367-b57c-81d1fc57b00e\") " Jul 06 22:46:02.949913 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:02.949872 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd96e2e4-b951-4367-b57c-81d1fc57b00e-dshm\") pod \"cd96e2e4-b951-4367-b57c-81d1fc57b00e\" (UID: \"cd96e2e4-b951-4367-b57c-81d1fc57b00e\") " Jul 06 22:46:02.950096 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:02.949892 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd96e2e4-b951-4367-b57c-81d1fc57b00e-model-cache" (OuterVolumeSpecName: "model-cache") pod "cd96e2e4-b951-4367-b57c-81d1fc57b00e" (UID: "cd96e2e4-b951-4367-b57c-81d1fc57b00e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:46:02.950096 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:02.949928 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd96e2e4-b951-4367-b57c-81d1fc57b00e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cd96e2e4-b951-4367-b57c-81d1fc57b00e" (UID: "cd96e2e4-b951-4367-b57c-81d1fc57b00e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:46:02.950172 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:02.950092 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd96e2e4-b951-4367-b57c-81d1fc57b00e-home" (OuterVolumeSpecName: "home") pod "cd96e2e4-b951-4367-b57c-81d1fc57b00e" (UID: "cd96e2e4-b951-4367-b57c-81d1fc57b00e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:46:02.950275 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:02.950253 2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd96e2e4-b951-4367-b57c-81d1fc57b00e-model-cache\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:46:02.950331 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:02.950280 2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd96e2e4-b951-4367-b57c-81d1fc57b00e-tmp-dir\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:46:02.950331 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:02.950294 2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd96e2e4-b951-4367-b57c-81d1fc57b00e-home\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:46:02.951963 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:02.951938 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd96e2e4-b951-4367-b57c-81d1fc57b00e-dshm" (OuterVolumeSpecName: "dshm") pod "cd96e2e4-b951-4367-b57c-81d1fc57b00e" (UID: "cd96e2e4-b951-4367-b57c-81d1fc57b00e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:46:02.952107 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:02.952045 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd96e2e4-b951-4367-b57c-81d1fc57b00e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cd96e2e4-b951-4367-b57c-81d1fc57b00e" (UID: "cd96e2e4-b951-4367-b57c-81d1fc57b00e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:46:02.952107 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:02.952085 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd96e2e4-b951-4367-b57c-81d1fc57b00e-kube-api-access-mkhcl" (OuterVolumeSpecName: "kube-api-access-mkhcl") pod "cd96e2e4-b951-4367-b57c-81d1fc57b00e" (UID: "cd96e2e4-b951-4367-b57c-81d1fc57b00e"). InnerVolumeSpecName "kube-api-access-mkhcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:46:03.051142 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:03.051103 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkhcl\" (UniqueName: \"kubernetes.io/projected/cd96e2e4-b951-4367-b57c-81d1fc57b00e-kube-api-access-mkhcl\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:46:03.051142 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:03.051137 2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd96e2e4-b951-4367-b57c-81d1fc57b00e-tls-certs\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:46:03.051142 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:03.051152 2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd96e2e4-b951-4367-b57c-81d1fc57b00e-dshm\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:46:03.434713 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:03.434674 2555 generic.go:356] "Generic (PLEG): container finished" podID="cd96e2e4-b951-4367-b57c-81d1fc57b00e" containerID="5ad226fe96e32ff0a0d01cf57d0ebb6ca682f5f1e66e4f571097fecd15d58911" exitCode=0 Jul 06 22:46:03.434892 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:03.434729 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-65387d/auth-test-model-65387d-kserve-7d9f774599-5b44l" Jul 06 22:46:03.434892 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:03.434752 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-65387d/auth-test-model-65387d-kserve-7d9f774599-5b44l" event={"ID":"cd96e2e4-b951-4367-b57c-81d1fc57b00e","Type":"ContainerDied","Data":"5ad226fe96e32ff0a0d01cf57d0ebb6ca682f5f1e66e4f571097fecd15d58911"} Jul 06 22:46:03.434892 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:03.434794 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-65387d/auth-test-model-65387d-kserve-7d9f774599-5b44l" event={"ID":"cd96e2e4-b951-4367-b57c-81d1fc57b00e","Type":"ContainerDied","Data":"d483e8ba7abcd9c124b618ba545febca0117b573b5c8b55a4bdb3eddcc3abcb7"} Jul 06 22:46:03.434892 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:03.434811 2555 scope.go:117] "RemoveContainer" containerID="5ad226fe96e32ff0a0d01cf57d0ebb6ca682f5f1e66e4f571097fecd15d58911" Jul 06 22:46:03.443837 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:03.443817 2555 scope.go:117] "RemoveContainer" containerID="5ad226fe96e32ff0a0d01cf57d0ebb6ca682f5f1e66e4f571097fecd15d58911" Jul 06 22:46:03.444089 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:46:03.444071 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ad226fe96e32ff0a0d01cf57d0ebb6ca682f5f1e66e4f571097fecd15d58911\": container with ID starting with 5ad226fe96e32ff0a0d01cf57d0ebb6ca682f5f1e66e4f571097fecd15d58911 not found: ID does not exist" containerID="5ad226fe96e32ff0a0d01cf57d0ebb6ca682f5f1e66e4f571097fecd15d58911" Jul 06 22:46:03.444173 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:03.444097 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ad226fe96e32ff0a0d01cf57d0ebb6ca682f5f1e66e4f571097fecd15d58911"} err="failed to get container status \"5ad226fe96e32ff0a0d01cf57d0ebb6ca682f5f1e66e4f571097fecd15d58911\": rpc error: code = NotFound desc = could not find container \"5ad226fe96e32ff0a0d01cf57d0ebb6ca682f5f1e66e4f571097fecd15d58911\": container with ID starting with 5ad226fe96e32ff0a0d01cf57d0ebb6ca682f5f1e66e4f571097fecd15d58911 not found: ID does not exist" Jul 06 22:46:03.451717 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:03.451686 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-65387d/auth-test-model-65387d-kserve-7d9f774599-5b44l"] Jul 06 22:46:03.459620 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:03.459587 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-65387d/auth-test-model-65387d-kserve-7d9f774599-5b44l"] Jul 06 22:46:03.617517 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:03.617409 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-944f9f/auth-test-model-944f9f-kserve-7bc56b8fd7-wcfkx" podUID="b32269df-1268-4a51-94af-655b26f959b0" containerName="main" containerID="cri-o://539d0543d40a3ed00af142512a58e166326e636721c6aff69b13389c41b8e160" gracePeriod=45 Jul 06 22:46:03.876842 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:03.876769 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-944f9f/auth-test-model-944f9f-kserve-7bc56b8fd7-wcfkx" Jul 06 22:46:03.958091 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:03.958059 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b32269df-1268-4a51-94af-655b26f959b0-model-cache\") pod \"b32269df-1268-4a51-94af-655b26f959b0\" (UID: \"b32269df-1268-4a51-94af-655b26f959b0\") " Jul 06 22:46:03.958091 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:03.958098 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b32269df-1268-4a51-94af-655b26f959b0-home\") pod \"b32269df-1268-4a51-94af-655b26f959b0\" (UID: \"b32269df-1268-4a51-94af-655b26f959b0\") " Jul 06 22:46:03.958360 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:03.958128 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5g75q\" (UniqueName: \"kubernetes.io/projected/b32269df-1268-4a51-94af-655b26f959b0-kube-api-access-5g75q\") pod \"b32269df-1268-4a51-94af-655b26f959b0\" (UID: \"b32269df-1268-4a51-94af-655b26f959b0\") " Jul 06 22:46:03.958360 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:03.958175 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b32269df-1268-4a51-94af-655b26f959b0-dshm\") pod \"b32269df-1268-4a51-94af-655b26f959b0\" (UID: \"b32269df-1268-4a51-94af-655b26f959b0\") " Jul 06 22:46:03.958360 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:03.958201 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b32269df-1268-4a51-94af-655b26f959b0-tls-certs\") pod \"b32269df-1268-4a51-94af-655b26f959b0\" (UID: \"b32269df-1268-4a51-94af-655b26f959b0\") " Jul 06 22:46:03.958360 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:03.958243 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b32269df-1268-4a51-94af-655b26f959b0-tmp-dir\") pod \"b32269df-1268-4a51-94af-655b26f959b0\" (UID: \"b32269df-1268-4a51-94af-655b26f959b0\") " Jul 06 22:46:03.958565 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:03.958432 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b32269df-1268-4a51-94af-655b26f959b0-home" (OuterVolumeSpecName: "home") pod "b32269df-1268-4a51-94af-655b26f959b0" (UID: "b32269df-1268-4a51-94af-655b26f959b0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:46:03.958565 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:03.958452 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b32269df-1268-4a51-94af-655b26f959b0-model-cache" (OuterVolumeSpecName: "model-cache") pod "b32269df-1268-4a51-94af-655b26f959b0" (UID: "b32269df-1268-4a51-94af-655b26f959b0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:46:03.958565 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:03.958544 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b32269df-1268-4a51-94af-655b26f959b0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b32269df-1268-4a51-94af-655b26f959b0" (UID: "b32269df-1268-4a51-94af-655b26f959b0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:46:03.958710 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:03.958666 2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b32269df-1268-4a51-94af-655b26f959b0-model-cache\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:46:03.958710 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:03.958680 2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b32269df-1268-4a51-94af-655b26f959b0-home\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:46:03.958710 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:03.958707 2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b32269df-1268-4a51-94af-655b26f959b0-tmp-dir\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:46:03.960625 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:03.960601 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b32269df-1268-4a51-94af-655b26f959b0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b32269df-1268-4a51-94af-655b26f959b0" (UID: "b32269df-1268-4a51-94af-655b26f959b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:46:03.960734 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:03.960601 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b32269df-1268-4a51-94af-655b26f959b0-dshm" (OuterVolumeSpecName: "dshm") pod "b32269df-1268-4a51-94af-655b26f959b0" (UID: "b32269df-1268-4a51-94af-655b26f959b0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:46:03.960734 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:03.960602 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b32269df-1268-4a51-94af-655b26f959b0-kube-api-access-5g75q" (OuterVolumeSpecName: "kube-api-access-5g75q") pod "b32269df-1268-4a51-94af-655b26f959b0" (UID: "b32269df-1268-4a51-94af-655b26f959b0"). InnerVolumeSpecName "kube-api-access-5g75q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:46:04.059733 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:04.059692 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5g75q\" (UniqueName: \"kubernetes.io/projected/b32269df-1268-4a51-94af-655b26f959b0-kube-api-access-5g75q\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:46:04.059733 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:04.059732 2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b32269df-1268-4a51-94af-655b26f959b0-dshm\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:46:04.059935 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:04.059749 2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b32269df-1268-4a51-94af-655b26f959b0-tls-certs\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:46:04.441884 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:04.441846 2555 generic.go:356] "Generic (PLEG): container finished" podID="b32269df-1268-4a51-94af-655b26f959b0" containerID="539d0543d40a3ed00af142512a58e166326e636721c6aff69b13389c41b8e160" exitCode=0 Jul 06 22:46:04.442099 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:04.441891 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-944f9f/auth-test-model-944f9f-kserve-7bc56b8fd7-wcfkx" Jul 06 22:46:04.442099 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:04.441931 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-944f9f/auth-test-model-944f9f-kserve-7bc56b8fd7-wcfkx" event={"ID":"b32269df-1268-4a51-94af-655b26f959b0","Type":"ContainerDied","Data":"539d0543d40a3ed00af142512a58e166326e636721c6aff69b13389c41b8e160"} Jul 06 22:46:04.442099 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:04.441978 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-944f9f/auth-test-model-944f9f-kserve-7bc56b8fd7-wcfkx" event={"ID":"b32269df-1268-4a51-94af-655b26f959b0","Type":"ContainerDied","Data":"acd7ec8666293d3639553970dd51bd5531ffe6ee5f14dc3e2f8c0ecaffac629a"} Jul 06 22:46:04.442099 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:04.441998 2555 scope.go:117] "RemoveContainer" containerID="539d0543d40a3ed00af142512a58e166326e636721c6aff69b13389c41b8e160" Jul 06 22:46:04.451219 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:04.451199 2555 scope.go:117] "RemoveContainer" containerID="539d0543d40a3ed00af142512a58e166326e636721c6aff69b13389c41b8e160" Jul 06 22:46:04.451514 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:46:04.451494 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"539d0543d40a3ed00af142512a58e166326e636721c6aff69b13389c41b8e160\": container with ID starting with 539d0543d40a3ed00af142512a58e166326e636721c6aff69b13389c41b8e160 not found: ID does not exist" containerID="539d0543d40a3ed00af142512a58e166326e636721c6aff69b13389c41b8e160" Jul 06 22:46:04.451566 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:04.451523 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"539d0543d40a3ed00af142512a58e166326e636721c6aff69b13389c41b8e160"} err="failed to get container status \"539d0543d40a3ed00af142512a58e166326e636721c6aff69b13389c41b8e160\": rpc error: code = NotFound desc = could not find container \"539d0543d40a3ed00af142512a58e166326e636721c6aff69b13389c41b8e160\": container with ID starting with 539d0543d40a3ed00af142512a58e166326e636721c6aff69b13389c41b8e160 not found: ID does not exist" Jul 06 22:46:04.463824 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:04.463795 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-944f9f/auth-test-model-944f9f-kserve-7bc56b8fd7-wcfkx"] Jul 06 22:46:04.470546 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:04.470512 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-944f9f/auth-test-model-944f9f-kserve-7bc56b8fd7-wcfkx"] Jul 06 22:46:05.340931 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:05.340896 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b32269df-1268-4a51-94af-655b26f959b0" path="/var/lib/kubelet/pods/b32269df-1268-4a51-94af-655b26f959b0/volumes" Jul 06 22:46:05.341331 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:05.341323 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd96e2e4-b951-4367-b57c-81d1fc57b00e" path="/var/lib/kubelet/pods/cd96e2e4-b951-4367-b57c-81d1fc57b00e/volumes" Jul 06 22:46:22.094268 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:22.094217 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-944f9f/sub-test-model-944f9f-kserve-64ff6b5975-lj4v6"] Jul 06 22:46:22.095496 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:22.095470 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b32269df-1268-4a51-94af-655b26f959b0" containerName="main" Jul 06 22:46:22.095655 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:22.095641 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b32269df-1268-4a51-94af-655b26f959b0" containerName="main" Jul 06 22:46:22.095772 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:22.095761 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd96e2e4-b951-4367-b57c-81d1fc57b00e" containerName="main" Jul 06 22:46:22.095853 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:22.095843 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd96e2e4-b951-4367-b57c-81d1fc57b00e" containerName="main" Jul 06 22:46:22.096078 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:22.096062 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b32269df-1268-4a51-94af-655b26f959b0" containerName="main" Jul 06 22:46:22.096207 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:22.096195 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd96e2e4-b951-4367-b57c-81d1fc57b00e" containerName="main" Jul 06 22:46:22.106444 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:22.105962 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-944f9f/sub-test-model-944f9f-kserve-64ff6b5975-lj4v6" Jul 06 22:46:22.109053 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:22.109024 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-944f9f\"/\"default-dockercfg-8tnxt\"" Jul 06 22:46:22.109270 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:22.109029 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-944f9f\"/\"kube-root-ca.crt\"" Jul 06 22:46:22.109416 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:22.109090 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-944f9f\"/\"sub-test-model-944f9f-kserve-self-signed-certs\"" Jul 06 22:46:22.109535 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:22.109207 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-944f9f\"/\"openshift-service-ca.crt\"" Jul 06 22:46:22.110025 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:22.109981 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-944f9f/sub-test-model-944f9f-kserve-64ff6b5975-lj4v6"] Jul 06 22:46:22.132621 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:22.132576 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71a3ad33-6719-46fb-bcc4-9fa79d1ae76f-tmp-dir\") pod \"sub-test-model-944f9f-kserve-64ff6b5975-lj4v6\" (UID: \"71a3ad33-6719-46fb-bcc4-9fa79d1ae76f\") " pod="ai-tenant-e2e-shared-b-944f9f/sub-test-model-944f9f-kserve-64ff6b5975-lj4v6" Jul 06 22:46:22.132812 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:22.132712 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71a3ad33-6719-46fb-bcc4-9fa79d1ae76f-dshm\") pod \"sub-test-model-944f9f-kserve-64ff6b5975-lj4v6\" (UID: \"71a3ad33-6719-46fb-bcc4-9fa79d1ae76f\") " pod="ai-tenant-e2e-shared-b-944f9f/sub-test-model-944f9f-kserve-64ff6b5975-lj4v6" Jul 06 22:46:22.132812 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:22.132749 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71a3ad33-6719-46fb-bcc4-9fa79d1ae76f-home\") pod \"sub-test-model-944f9f-kserve-64ff6b5975-lj4v6\" (UID: \"71a3ad33-6719-46fb-bcc4-9fa79d1ae76f\") " pod="ai-tenant-e2e-shared-b-944f9f/sub-test-model-944f9f-kserve-64ff6b5975-lj4v6" Jul 06 22:46:22.132920 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:22.132820 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4stdk\" (UniqueName: \"kubernetes.io/projected/71a3ad33-6719-46fb-bcc4-9fa79d1ae76f-kube-api-access-4stdk\") pod \"sub-test-model-944f9f-kserve-64ff6b5975-lj4v6\" (UID: \"71a3ad33-6719-46fb-bcc4-9fa79d1ae76f\") " pod="ai-tenant-e2e-shared-b-944f9f/sub-test-model-944f9f-kserve-64ff6b5975-lj4v6" Jul 06 22:46:22.132920 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:22.132857 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71a3ad33-6719-46fb-bcc4-9fa79d1ae76f-tls-certs\") pod \"sub-test-model-944f9f-kserve-64ff6b5975-lj4v6\" (UID: \"71a3ad33-6719-46fb-bcc4-9fa79d1ae76f\") " pod="ai-tenant-e2e-shared-b-944f9f/sub-test-model-944f9f-kserve-64ff6b5975-lj4v6" Jul 06 22:46:22.132920 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:22.132889 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71a3ad33-6719-46fb-bcc4-9fa79d1ae76f-model-cache\") pod \"sub-test-model-944f9f-kserve-64ff6b5975-lj4v6\" (UID: \"71a3ad33-6719-46fb-bcc4-9fa79d1ae76f\") " pod="ai-tenant-e2e-shared-b-944f9f/sub-test-model-944f9f-kserve-64ff6b5975-lj4v6" Jul 06 22:46:22.233683 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:22.233639 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71a3ad33-6719-46fb-bcc4-9fa79d1ae76f-dshm\") pod \"sub-test-model-944f9f-kserve-64ff6b5975-lj4v6\" (UID: \"71a3ad33-6719-46fb-bcc4-9fa79d1ae76f\") " pod="ai-tenant-e2e-shared-b-944f9f/sub-test-model-944f9f-kserve-64ff6b5975-lj4v6" Jul 06 22:46:22.233873 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:22.233699 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71a3ad33-6719-46fb-bcc4-9fa79d1ae76f-home\") pod \"sub-test-model-944f9f-kserve-64ff6b5975-lj4v6\" (UID: \"71a3ad33-6719-46fb-bcc4-9fa79d1ae76f\") " pod="ai-tenant-e2e-shared-b-944f9f/sub-test-model-944f9f-kserve-64ff6b5975-lj4v6" Jul 06 22:46:22.233873 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:22.233772 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4stdk\" (UniqueName: \"kubernetes.io/projected/71a3ad33-6719-46fb-bcc4-9fa79d1ae76f-kube-api-access-4stdk\") pod \"sub-test-model-944f9f-kserve-64ff6b5975-lj4v6\" (UID: \"71a3ad33-6719-46fb-bcc4-9fa79d1ae76f\") " pod="ai-tenant-e2e-shared-b-944f9f/sub-test-model-944f9f-kserve-64ff6b5975-lj4v6" Jul 06 22:46:22.233873 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:22.233811 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71a3ad33-6719-46fb-bcc4-9fa79d1ae76f-tls-certs\") pod \"sub-test-model-944f9f-kserve-64ff6b5975-lj4v6\" (UID: \"71a3ad33-6719-46fb-bcc4-9fa79d1ae76f\") " pod="ai-tenant-e2e-shared-b-944f9f/sub-test-model-944f9f-kserve-64ff6b5975-lj4v6" Jul 06 22:46:22.233873 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:22.233844 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71a3ad33-6719-46fb-bcc4-9fa79d1ae76f-model-cache\") pod \"sub-test-model-944f9f-kserve-64ff6b5975-lj4v6\" (UID: \"71a3ad33-6719-46fb-bcc4-9fa79d1ae76f\") " pod="ai-tenant-e2e-shared-b-944f9f/sub-test-model-944f9f-kserve-64ff6b5975-lj4v6" Jul 06 22:46:22.234112 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:22.233887 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71a3ad33-6719-46fb-bcc4-9fa79d1ae76f-tmp-dir\") pod \"sub-test-model-944f9f-kserve-64ff6b5975-lj4v6\" (UID: \"71a3ad33-6719-46fb-bcc4-9fa79d1ae76f\") " pod="ai-tenant-e2e-shared-b-944f9f/sub-test-model-944f9f-kserve-64ff6b5975-lj4v6" Jul 06 22:46:22.234169 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:22.234128 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71a3ad33-6719-46fb-bcc4-9fa79d1ae76f-home\") pod \"sub-test-model-944f9f-kserve-64ff6b5975-lj4v6\" (UID: \"71a3ad33-6719-46fb-bcc4-9fa79d1ae76f\") " pod="ai-tenant-e2e-shared-b-944f9f/sub-test-model-944f9f-kserve-64ff6b5975-lj4v6" Jul 06 22:46:22.234235 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:22.234200 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71a3ad33-6719-46fb-bcc4-9fa79d1ae76f-tmp-dir\") pod \"sub-test-model-944f9f-kserve-64ff6b5975-lj4v6\" (UID: \"71a3ad33-6719-46fb-bcc4-9fa79d1ae76f\") " pod="ai-tenant-e2e-shared-b-944f9f/sub-test-model-944f9f-kserve-64ff6b5975-lj4v6" Jul 06 22:46:22.234474 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:22.234437 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71a3ad33-6719-46fb-bcc4-9fa79d1ae76f-model-cache\") pod \"sub-test-model-944f9f-kserve-64ff6b5975-lj4v6\" (UID: \"71a3ad33-6719-46fb-bcc4-9fa79d1ae76f\") " pod="ai-tenant-e2e-shared-b-944f9f/sub-test-model-944f9f-kserve-64ff6b5975-lj4v6" Jul 06 22:46:22.236109 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:22.236079 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71a3ad33-6719-46fb-bcc4-9fa79d1ae76f-dshm\") pod \"sub-test-model-944f9f-kserve-64ff6b5975-lj4v6\" (UID: \"71a3ad33-6719-46fb-bcc4-9fa79d1ae76f\") " pod="ai-tenant-e2e-shared-b-944f9f/sub-test-model-944f9f-kserve-64ff6b5975-lj4v6" Jul 06 22:46:22.236410 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:22.236392 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71a3ad33-6719-46fb-bcc4-9fa79d1ae76f-tls-certs\") pod \"sub-test-model-944f9f-kserve-64ff6b5975-lj4v6\" (UID: \"71a3ad33-6719-46fb-bcc4-9fa79d1ae76f\") " pod="ai-tenant-e2e-shared-b-944f9f/sub-test-model-944f9f-kserve-64ff6b5975-lj4v6" Jul 06 22:46:22.245156 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:22.245127 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4stdk\" (UniqueName: \"kubernetes.io/projected/71a3ad33-6719-46fb-bcc4-9fa79d1ae76f-kube-api-access-4stdk\") pod \"sub-test-model-944f9f-kserve-64ff6b5975-lj4v6\" (UID: \"71a3ad33-6719-46fb-bcc4-9fa79d1ae76f\") " pod="ai-tenant-e2e-shared-b-944f9f/sub-test-model-944f9f-kserve-64ff6b5975-lj4v6" Jul 06 22:46:22.418645 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:22.418609 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-944f9f/sub-test-model-944f9f-kserve-64ff6b5975-lj4v6" Jul 06 22:46:22.555386 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:22.555360 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-944f9f/sub-test-model-944f9f-kserve-64ff6b5975-lj4v6"] Jul 06 22:46:22.557653 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:46:22.557613 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod71a3ad33_6719_46fb_bcc4_9fa79d1ae76f.slice/crio-24e1d20706e2779d7c577aa7ca8397a806f4b965cf0c5ec846bc3a4eb51a417a WatchSource:0}: Error finding container 24e1d20706e2779d7c577aa7ca8397a806f4b965cf0c5ec846bc3a4eb51a417a: Status 404 returned error can't find the container with id 24e1d20706e2779d7c577aa7ca8397a806f4b965cf0c5ec846bc3a4eb51a417a Jul 06 22:46:23.543664 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:23.543625 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-944f9f/sub-test-model-944f9f-kserve-64ff6b5975-lj4v6" event={"ID":"71a3ad33-6719-46fb-bcc4-9fa79d1ae76f","Type":"ContainerStarted","Data":"d1beb62bc4928f6838b3cabcc4308b48018660bc70fb51f70c87be0fe9d06b9c"} Jul 06 22:46:23.544169 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:23.543676 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-944f9f/sub-test-model-944f9f-kserve-64ff6b5975-lj4v6" event={"ID":"71a3ad33-6719-46fb-bcc4-9fa79d1ae76f","Type":"ContainerStarted","Data":"24e1d20706e2779d7c577aa7ca8397a806f4b965cf0c5ec846bc3a4eb51a417a"} Jul 06 22:46:23.564069 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:23.563999 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-944f9f/sub-test-model-944f9f-kserve-64ff6b5975-lj4v6" podStartSLOduration=1.563983034 podStartE2EDuration="1.563983034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:46:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:46:23.562867632 +0000 UTC m=+2234.862579480" watchObservedRunningTime="2026-07-06 22:46:23.563983034 +0000 UTC m=+2234.863694882" Jul 06 22:46:32.419599 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:32.419505 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-944f9f/sub-test-model-944f9f-kserve-64ff6b5975-lj4v6" Jul 06 22:46:32.419599 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:32.419575 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-944f9f/sub-test-model-944f9f-kserve-64ff6b5975-lj4v6" Jul 06 22:46:32.431826 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:32.431786 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-944f9f/sub-test-model-944f9f-kserve-64ff6b5975-lj4v6" Jul 06 22:46:32.599154 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:46:32.599126 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-944f9f/sub-test-model-944f9f-kserve-64ff6b5975-lj4v6" Jul 06 22:47:21.873744 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:47:21.873699 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-944f9f/sub-test-model-944f9f-kserve-64ff6b5975-lj4v6"] Jul 06 22:47:29.302135 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:47:29.302097 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-65387d/rate-test-model-65387d-kserve-55bbb4bfdc-kzfdh"] Jul 06 22:47:29.305686 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:47:29.305667 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-65387d/rate-test-model-65387d-kserve-55bbb4bfdc-kzfdh" Jul 06 22:47:29.309627 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:47:29.309598 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-65387d\"/\"openshift-service-ca.crt\"" Jul 06 22:47:29.309627 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:47:29.309616 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-65387d\"/\"rate-test-model-65387d-kserve-self-signed-certs\"" Jul 06 22:47:29.309627 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:47:29.309628 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-65387d\"/\"default-dockercfg-6qls2\"" Jul 06 22:47:29.309848 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:47:29.309599 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-65387d\"/\"kube-root-ca.crt\"" Jul 06 22:47:29.313865 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:47:29.313840 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-65387d/rate-test-model-65387d-kserve-55bbb4bfdc-kzfdh"] Jul 06 22:47:29.499664 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:47:29.499620 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a1c0651-b003-4dff-b83a-b98e5fb15721-tmp-dir\") pod \"rate-test-model-65387d-kserve-55bbb4bfdc-kzfdh\" (UID: \"4a1c0651-b003-4dff-b83a-b98e5fb15721\") " pod="ai-tenant-e2e-shared-a-65387d/rate-test-model-65387d-kserve-55bbb4bfdc-kzfdh" Jul 06 22:47:29.499996 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:47:29.499746 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a1c0651-b003-4dff-b83a-b98e5fb15721-tls-certs\") pod \"rate-test-model-65387d-kserve-55bbb4bfdc-kzfdh\" (UID: \"4a1c0651-b003-4dff-b83a-b98e5fb15721\") " pod="ai-tenant-e2e-shared-a-65387d/rate-test-model-65387d-kserve-55bbb4bfdc-kzfdh" Jul 06 22:47:29.499996 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:47:29.499820 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a1c0651-b003-4dff-b83a-b98e5fb15721-model-cache\") pod \"rate-test-model-65387d-kserve-55bbb4bfdc-kzfdh\" (UID: \"4a1c0651-b003-4dff-b83a-b98e5fb15721\") " pod="ai-tenant-e2e-shared-a-65387d/rate-test-model-65387d-kserve-55bbb4bfdc-kzfdh" Jul 06 22:47:29.499996 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:47:29.499906 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a1c0651-b003-4dff-b83a-b98e5fb15721-home\") pod \"rate-test-model-65387d-kserve-55bbb4bfdc-kzfdh\" (UID: \"4a1c0651-b003-4dff-b83a-b98e5fb15721\") " pod="ai-tenant-e2e-shared-a-65387d/rate-test-model-65387d-kserve-55bbb4bfdc-kzfdh" Jul 06 22:47:29.500179 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:47:29.500096 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a1c0651-b003-4dff-b83a-b98e5fb15721-dshm\") pod \"rate-test-model-65387d-kserve-55bbb4bfdc-kzfdh\" (UID: \"4a1c0651-b003-4dff-b83a-b98e5fb15721\") " pod="ai-tenant-e2e-shared-a-65387d/rate-test-model-65387d-kserve-55bbb4bfdc-kzfdh" Jul 06 22:47:29.500224 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:47:29.500181 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cjpg\" (UniqueName: \"kubernetes.io/projected/4a1c0651-b003-4dff-b83a-b98e5fb15721-kube-api-access-8cjpg\") pod \"rate-test-model-65387d-kserve-55bbb4bfdc-kzfdh\" (UID: \"4a1c0651-b003-4dff-b83a-b98e5fb15721\") " pod="ai-tenant-e2e-shared-a-65387d/rate-test-model-65387d-kserve-55bbb4bfdc-kzfdh" Jul 06 22:47:29.601835 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:47:29.601737 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a1c0651-b003-4dff-b83a-b98e5fb15721-dshm\") pod \"rate-test-model-65387d-kserve-55bbb4bfdc-kzfdh\" (UID: \"4a1c0651-b003-4dff-b83a-b98e5fb15721\") " pod="ai-tenant-e2e-shared-a-65387d/rate-test-model-65387d-kserve-55bbb4bfdc-kzfdh" Jul 06 22:47:29.601835 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:47:29.601788 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cjpg\" (UniqueName: \"kubernetes.io/projected/4a1c0651-b003-4dff-b83a-b98e5fb15721-kube-api-access-8cjpg\") pod \"rate-test-model-65387d-kserve-55bbb4bfdc-kzfdh\" (UID: \"4a1c0651-b003-4dff-b83a-b98e5fb15721\") " pod="ai-tenant-e2e-shared-a-65387d/rate-test-model-65387d-kserve-55bbb4bfdc-kzfdh" Jul 06 22:47:29.602124 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:47:29.601845 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a1c0651-b003-4dff-b83a-b98e5fb15721-tmp-dir\") pod \"rate-test-model-65387d-kserve-55bbb4bfdc-kzfdh\" (UID: \"4a1c0651-b003-4dff-b83a-b98e5fb15721\") " pod="ai-tenant-e2e-shared-a-65387d/rate-test-model-65387d-kserve-55bbb4bfdc-kzfdh" Jul 06 22:47:29.602124 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:47:29.601928 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a1c0651-b003-4dff-b83a-b98e5fb15721-tls-certs\") pod \"rate-test-model-65387d-kserve-55bbb4bfdc-kzfdh\" (UID: \"4a1c0651-b003-4dff-b83a-b98e5fb15721\") " pod="ai-tenant-e2e-shared-a-65387d/rate-test-model-65387d-kserve-55bbb4bfdc-kzfdh" Jul 06 22:47:29.602124 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:47:29.601972 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a1c0651-b003-4dff-b83a-b98e5fb15721-model-cache\") pod \"rate-test-model-65387d-kserve-55bbb4bfdc-kzfdh\" (UID: \"4a1c0651-b003-4dff-b83a-b98e5fb15721\") " pod="ai-tenant-e2e-shared-a-65387d/rate-test-model-65387d-kserve-55bbb4bfdc-kzfdh" Jul 06 22:47:29.602124 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:47:29.602043 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a1c0651-b003-4dff-b83a-b98e5fb15721-home\") pod \"rate-test-model-65387d-kserve-55bbb4bfdc-kzfdh\" (UID: \"4a1c0651-b003-4dff-b83a-b98e5fb15721\") " pod="ai-tenant-e2e-shared-a-65387d/rate-test-model-65387d-kserve-55bbb4bfdc-kzfdh" Jul 06 22:47:29.602382 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:47:29.602347 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a1c0651-b003-4dff-b83a-b98e5fb15721-tmp-dir\") pod \"rate-test-model-65387d-kserve-55bbb4bfdc-kzfdh\" (UID: \"4a1c0651-b003-4dff-b83a-b98e5fb15721\") " pod="ai-tenant-e2e-shared-a-65387d/rate-test-model-65387d-kserve-55bbb4bfdc-kzfdh" Jul 06 22:47:29.602382 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:47:29.602364 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a1c0651-b003-4dff-b83a-b98e5fb15721-home\") pod \"rate-test-model-65387d-kserve-55bbb4bfdc-kzfdh\" (UID: \"4a1c0651-b003-4dff-b83a-b98e5fb15721\") " pod="ai-tenant-e2e-shared-a-65387d/rate-test-model-65387d-kserve-55bbb4bfdc-kzfdh" Jul 06 22:47:29.602502 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:47:29.602401 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a1c0651-b003-4dff-b83a-b98e5fb15721-model-cache\") pod \"rate-test-model-65387d-kserve-55bbb4bfdc-kzfdh\" (UID: \"4a1c0651-b003-4dff-b83a-b98e5fb15721\") " pod="ai-tenant-e2e-shared-a-65387d/rate-test-model-65387d-kserve-55bbb4bfdc-kzfdh" Jul 06 22:47:29.603978 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:47:29.603957 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a1c0651-b003-4dff-b83a-b98e5fb15721-dshm\") pod \"rate-test-model-65387d-kserve-55bbb4bfdc-kzfdh\" (UID: \"4a1c0651-b003-4dff-b83a-b98e5fb15721\") " pod="ai-tenant-e2e-shared-a-65387d/rate-test-model-65387d-kserve-55bbb4bfdc-kzfdh" Jul 06 22:47:29.604349 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:47:29.604332 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a1c0651-b003-4dff-b83a-b98e5fb15721-tls-certs\") pod \"rate-test-model-65387d-kserve-55bbb4bfdc-kzfdh\" (UID: \"4a1c0651-b003-4dff-b83a-b98e5fb15721\") " pod="ai-tenant-e2e-shared-a-65387d/rate-test-model-65387d-kserve-55bbb4bfdc-kzfdh" Jul 06 22:47:29.610259 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:47:29.610232 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cjpg\" (UniqueName: \"kubernetes.io/projected/4a1c0651-b003-4dff-b83a-b98e5fb15721-kube-api-access-8cjpg\") pod \"rate-test-model-65387d-kserve-55bbb4bfdc-kzfdh\" (UID: \"4a1c0651-b003-4dff-b83a-b98e5fb15721\") " pod="ai-tenant-e2e-shared-a-65387d/rate-test-model-65387d-kserve-55bbb4bfdc-kzfdh" Jul 06 22:47:29.616108 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:47:29.616081 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-65387d/rate-test-model-65387d-kserve-55bbb4bfdc-kzfdh" Jul 06 22:47:29.752247 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:47:29.752216 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-65387d/rate-test-model-65387d-kserve-55bbb4bfdc-kzfdh"] Jul 06 22:47:29.754161 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:47:29.754137 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a1c0651_b003_4dff_b83a_b98e5fb15721.slice/crio-b1477953edc590353728231c4d1e7d06a3eb01b35aaacd62c4db5e0ff9deaddc WatchSource:0}: Error finding container b1477953edc590353728231c4d1e7d06a3eb01b35aaacd62c4db5e0ff9deaddc: Status 404 returned error can't find the container with id b1477953edc590353728231c4d1e7d06a3eb01b35aaacd62c4db5e0ff9deaddc Jul 06 22:47:29.877224 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:47:29.877130 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-65387d/rate-test-model-65387d-kserve-55bbb4bfdc-kzfdh" event={"ID":"4a1c0651-b003-4dff-b83a-b98e5fb15721","Type":"ContainerStarted","Data":"e2c30044188e0615c855d24184388401e0b93f1f15f1c79f3346f0331885b26c"} Jul 06 22:47:29.877224 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:47:29.877180 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-65387d/rate-test-model-65387d-kserve-55bbb4bfdc-kzfdh" event={"ID":"4a1c0651-b003-4dff-b83a-b98e5fb15721","Type":"ContainerStarted","Data":"b1477953edc590353728231c4d1e7d06a3eb01b35aaacd62c4db5e0ff9deaddc"} Jul 06 22:47:29.895453 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:47:29.895402 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-65387d/rate-test-model-65387d-kserve-55bbb4bfdc-kzfdh" podStartSLOduration=0.895387058 podStartE2EDuration="895.387058ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:47:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:47:29.892857269 +0000 UTC m=+2301.192569115" watchObservedRunningTime="2026-07-06 22:47:29.895387058 +0000 UTC m=+2301.195098907" Jul 06 22:47:36.895977 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:47:36.895932 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-944f9f/sub-test-model-944f9f-kserve-64ff6b5975-lj4v6" podUID="71a3ad33-6719-46fb-bcc4-9fa79d1ae76f" containerName="main" containerID="cri-o://d1beb62bc4928f6838b3cabcc4308b48018660bc70fb51f70c87be0fe9d06b9c" gracePeriod=45 Jul 06 22:47:37.158160 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:47:37.158084 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-944f9f/sub-test-model-944f9f-kserve-64ff6b5975-lj4v6" Jul 06 22:47:37.271557 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:47:37.271515 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71a3ad33-6719-46fb-bcc4-9fa79d1ae76f-model-cache\") pod \"71a3ad33-6719-46fb-bcc4-9fa79d1ae76f\" (UID: \"71a3ad33-6719-46fb-bcc4-9fa79d1ae76f\") " Jul 06 22:47:37.271557 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:47:37.271566 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4stdk\" (UniqueName: \"kubernetes.io/projected/71a3ad33-6719-46fb-bcc4-9fa79d1ae76f-kube-api-access-4stdk\") pod \"71a3ad33-6719-46fb-bcc4-9fa79d1ae76f\" (UID: \"71a3ad33-6719-46fb-bcc4-9fa79d1ae76f\") " Jul 06 22:47:37.271836 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:47:37.271611 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71a3ad33-6719-46fb-bcc4-9fa79d1ae76f-tmp-dir\") pod \"71a3ad33-6719-46fb-bcc4-9fa79d1ae76f\" (UID: \"71a3ad33-6719-46fb-bcc4-9fa79d1ae76f\") " Jul 06 22:47:37.271836 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:47:37.271627 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71a3ad33-6719-46fb-bcc4-9fa79d1ae76f-dshm\") pod \"71a3ad33-6719-46fb-bcc4-9fa79d1ae76f\" (UID: \"71a3ad33-6719-46fb-bcc4-9fa79d1ae76f\") " Jul 06 22:47:37.271836 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:47:37.271651 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71a3ad33-6719-46fb-bcc4-9fa79d1ae76f-tls-certs\") pod \"71a3ad33-6719-46fb-bcc4-9fa79d1ae76f\" (UID: \"71a3ad33-6719-46fb-bcc4-9fa79d1ae76f\") " Jul 06 22:47:37.271836 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:47:37.271714 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71a3ad33-6719-46fb-bcc4-9fa79d1ae76f-home\") pod \"71a3ad33-6719-46fb-bcc4-9fa79d1ae76f\" (UID: \"71a3ad33-6719-46fb-bcc4-9fa79d1ae76f\") " Jul 06 22:47:37.272107 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:47:37.271861 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a3ad33-6719-46fb-bcc4-9fa79d1ae76f-model-cache" (OuterVolumeSpecName: "model-cache") pod "71a3ad33-6719-46fb-bcc4-9fa79d1ae76f" (UID: "71a3ad33-6719-46fb-bcc4-9fa79d1ae76f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:47:37.272107 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:47:37.271953 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a3ad33-6719-46fb-bcc4-9fa79d1ae76f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "71a3ad33-6719-46fb-bcc4-9fa79d1ae76f" (UID: "71a3ad33-6719-46fb-bcc4-9fa79d1ae76f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:47:37.272107 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:47:37.272041 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a3ad33-6719-46fb-bcc4-9fa79d1ae76f-home" (OuterVolumeSpecName: "home") pod "71a3ad33-6719-46fb-bcc4-9fa79d1ae76f" (UID: "71a3ad33-6719-46fb-bcc4-9fa79d1ae76f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:47:37.272273 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:47:37.272202 2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71a3ad33-6719-46fb-bcc4-9fa79d1ae76f-model-cache\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:47:37.272273 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:47:37.272217 2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71a3ad33-6719-46fb-bcc4-9fa79d1ae76f-tmp-dir\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:47:37.272273 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:47:37.272229 2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71a3ad33-6719-46fb-bcc4-9fa79d1ae76f-home\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:47:37.273872 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:47:37.273841 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a3ad33-6719-46fb-bcc4-9fa79d1ae76f-dshm" (OuterVolumeSpecName: "dshm") pod "71a3ad33-6719-46fb-bcc4-9fa79d1ae76f" (UID: "71a3ad33-6719-46fb-bcc4-9fa79d1ae76f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:47:37.274000 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:47:37.273891 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71a3ad33-6719-46fb-bcc4-9fa79d1ae76f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "71a3ad33-6719-46fb-bcc4-9fa79d1ae76f" (UID: "71a3ad33-6719-46fb-bcc4-9fa79d1ae76f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:47:37.274000 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:47:37.273923 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71a3ad33-6719-46fb-bcc4-9fa79d1ae76f-kube-api-access-4stdk" (OuterVolumeSpecName: "kube-api-access-4stdk") pod "71a3ad33-6719-46fb-bcc4-9fa79d1ae76f" (UID: "71a3ad33-6719-46fb-bcc4-9fa79d1ae76f"). InnerVolumeSpecName "kube-api-access-4stdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:47:37.372966 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:47:37.372925 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4stdk\" (UniqueName: \"kubernetes.io/projected/71a3ad33-6719-46fb-bcc4-9fa79d1ae76f-kube-api-access-4stdk\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:47:37.372966 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:47:37.372958 2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71a3ad33-6719-46fb-bcc4-9fa79d1ae76f-dshm\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:47:37.372966 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:47:37.372968 2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71a3ad33-6719-46fb-bcc4-9fa79d1ae76f-tls-certs\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:47:37.924104 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:47:37.924063 2555 generic.go:356] "Generic (PLEG): container finished" podID="71a3ad33-6719-46fb-bcc4-9fa79d1ae76f" containerID="d1beb62bc4928f6838b3cabcc4308b48018660bc70fb51f70c87be0fe9d06b9c" exitCode=0 Jul 06 22:47:37.924104 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:47:37.924109 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-944f9f/sub-test-model-944f9f-kserve-64ff6b5975-lj4v6" Jul 06 22:47:37.924618 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:47:37.924153 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-944f9f/sub-test-model-944f9f-kserve-64ff6b5975-lj4v6" event={"ID":"71a3ad33-6719-46fb-bcc4-9fa79d1ae76f","Type":"ContainerDied","Data":"d1beb62bc4928f6838b3cabcc4308b48018660bc70fb51f70c87be0fe9d06b9c"} Jul 06 22:47:37.924618 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:47:37.924204 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-944f9f/sub-test-model-944f9f-kserve-64ff6b5975-lj4v6" event={"ID":"71a3ad33-6719-46fb-bcc4-9fa79d1ae76f","Type":"ContainerDied","Data":"24e1d20706e2779d7c577aa7ca8397a806f4b965cf0c5ec846bc3a4eb51a417a"} Jul 06 22:47:37.924618 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:47:37.924222 2555 scope.go:117] "RemoveContainer" containerID="d1beb62bc4928f6838b3cabcc4308b48018660bc70fb51f70c87be0fe9d06b9c" Jul 06 22:47:37.932839 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:47:37.932825 2555 scope.go:117] "RemoveContainer" containerID="d1beb62bc4928f6838b3cabcc4308b48018660bc70fb51f70c87be0fe9d06b9c" Jul 06 22:47:37.933111 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:47:37.933089 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1beb62bc4928f6838b3cabcc4308b48018660bc70fb51f70c87be0fe9d06b9c\": container with ID starting with d1beb62bc4928f6838b3cabcc4308b48018660bc70fb51f70c87be0fe9d06b9c not found: ID does not exist" containerID="d1beb62bc4928f6838b3cabcc4308b48018660bc70fb51f70c87be0fe9d06b9c" Jul 06 22:47:37.933176 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:47:37.933120 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1beb62bc4928f6838b3cabcc4308b48018660bc70fb51f70c87be0fe9d06b9c"} err="failed to get container status \"d1beb62bc4928f6838b3cabcc4308b48018660bc70fb51f70c87be0fe9d06b9c\": rpc error: code = NotFound desc = could not find container \"d1beb62bc4928f6838b3cabcc4308b48018660bc70fb51f70c87be0fe9d06b9c\": container with ID starting with d1beb62bc4928f6838b3cabcc4308b48018660bc70fb51f70c87be0fe9d06b9c not found: ID does not exist" Jul 06 22:47:37.941144 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:47:37.941114 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-944f9f/sub-test-model-944f9f-kserve-64ff6b5975-lj4v6"] Jul 06 22:47:37.942899 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:47:37.942878 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-944f9f/sub-test-model-944f9f-kserve-64ff6b5975-lj4v6"] Jul 06 22:47:39.341477 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:47:39.341440 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71a3ad33-6719-46fb-bcc4-9fa79d1ae76f" path="/var/lib/kubelet/pods/71a3ad33-6719-46fb-bcc4-9fa79d1ae76f/volumes" Jul 06 22:47:39.617192 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:47:39.617091 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-65387d/rate-test-model-65387d-kserve-55bbb4bfdc-kzfdh" Jul 06 22:47:39.617385 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:47:39.617365 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-65387d/rate-test-model-65387d-kserve-55bbb4bfdc-kzfdh" Jul 06 22:47:39.630068 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:47:39.630038 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-65387d/rate-test-model-65387d-kserve-55bbb4bfdc-kzfdh" Jul 06 22:47:39.947563 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:47:39.947537 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-65387d/rate-test-model-65387d-kserve-55bbb4bfdc-kzfdh" Jul 06 22:48:34.925901 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:48:34.925863 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-65387d/rate-test-model-65387d-kserve-55bbb4bfdc-kzfdh"] Jul 06 22:48:37.554427 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:48:37.554383 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-65387d-b7947677b-5cblp"] Jul 06 22:48:37.554844 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:48:37.554620 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-a-65387d-b7947677b-5cblp" podUID="a8bc2d37-3211-464c-8393-abc51bddbdbd" containerName="maas-api" containerID="cri-o://4281163b9ebda52f890d6941b02f700f13b9ef97a1e228045e5f210c5529e13c" gracePeriod=30 Jul 06 22:48:37.805791 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:48:37.805731 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-65387d-b7947677b-5cblp" Jul 06 22:48:37.983342 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:48:37.983309 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-68686\" (UniqueName: \"kubernetes.io/projected/a8bc2d37-3211-464c-8393-abc51bddbdbd-kube-api-access-68686\") pod \"a8bc2d37-3211-464c-8393-abc51bddbdbd\" (UID: \"a8bc2d37-3211-464c-8393-abc51bddbdbd\") " Jul 06 22:48:37.983342 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:48:37.983353 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8bc2d37-3211-464c-8393-abc51bddbdbd-maas-api-tls\") pod \"a8bc2d37-3211-464c-8393-abc51bddbdbd\" (UID: \"a8bc2d37-3211-464c-8393-abc51bddbdbd\") " Jul 06 22:48:37.983573 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:48:37.983408 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8bc2d37-3211-464c-8393-abc51bddbdbd-openshift-service-ca\") pod \"a8bc2d37-3211-464c-8393-abc51bddbdbd\" (UID: \"a8bc2d37-3211-464c-8393-abc51bddbdbd\") " Jul 06 22:48:37.983867 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:48:37.983840 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8bc2d37-3211-464c-8393-abc51bddbdbd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a8bc2d37-3211-464c-8393-abc51bddbdbd" (UID: "a8bc2d37-3211-464c-8393-abc51bddbdbd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:48:37.985456 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:48:37.985422 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8bc2d37-3211-464c-8393-abc51bddbdbd-kube-api-access-68686" (OuterVolumeSpecName: "kube-api-access-68686") pod "a8bc2d37-3211-464c-8393-abc51bddbdbd" (UID: "a8bc2d37-3211-464c-8393-abc51bddbdbd"). InnerVolumeSpecName "kube-api-access-68686". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:48:37.985618 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:48:37.985454 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8bc2d37-3211-464c-8393-abc51bddbdbd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a8bc2d37-3211-464c-8393-abc51bddbdbd" (UID: "a8bc2d37-3211-464c-8393-abc51bddbdbd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:48:38.084554 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:48:38.084458 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8bc2d37-3211-464c-8393-abc51bddbdbd-openshift-service-ca\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:48:38.084554 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:48:38.084490 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68686\" (UniqueName: \"kubernetes.io/projected/a8bc2d37-3211-464c-8393-abc51bddbdbd-kube-api-access-68686\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:48:38.084554 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:48:38.084505 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8bc2d37-3211-464c-8393-abc51bddbdbd-maas-api-tls\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:48:38.231558 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:48:38.231524 2555 generic.go:356] "Generic (PLEG): container finished" podID="a8bc2d37-3211-464c-8393-abc51bddbdbd" containerID="4281163b9ebda52f890d6941b02f700f13b9ef97a1e228045e5f210c5529e13c" exitCode=0 Jul 06 22:48:38.231740 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:48:38.231580 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-65387d-b7947677b-5cblp" Jul 06 22:48:38.231740 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:48:38.231614 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-65387d-b7947677b-5cblp" event={"ID":"a8bc2d37-3211-464c-8393-abc51bddbdbd","Type":"ContainerDied","Data":"4281163b9ebda52f890d6941b02f700f13b9ef97a1e228045e5f210c5529e13c"} Jul 06 22:48:38.231740 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:48:38.231671 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-65387d-b7947677b-5cblp" event={"ID":"a8bc2d37-3211-464c-8393-abc51bddbdbd","Type":"ContainerDied","Data":"85129520950ba99bce1fc6eb7bb44f979292e2afa145fbdea0b8e1f09a81cfa7"} Jul 06 22:48:38.231740 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:48:38.231703 2555 scope.go:117] "RemoveContainer" containerID="4281163b9ebda52f890d6941b02f700f13b9ef97a1e228045e5f210c5529e13c" Jul 06 22:48:38.241308 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:48:38.241290 2555 scope.go:117] "RemoveContainer" containerID="4281163b9ebda52f890d6941b02f700f13b9ef97a1e228045e5f210c5529e13c" Jul 06 22:48:38.241598 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:48:38.241578 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4281163b9ebda52f890d6941b02f700f13b9ef97a1e228045e5f210c5529e13c\": container with ID starting with 4281163b9ebda52f890d6941b02f700f13b9ef97a1e228045e5f210c5529e13c not found: ID does not exist" containerID="4281163b9ebda52f890d6941b02f700f13b9ef97a1e228045e5f210c5529e13c" Jul 06 22:48:38.241647 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:48:38.241606 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4281163b9ebda52f890d6941b02f700f13b9ef97a1e228045e5f210c5529e13c"} err="failed to get container status \"4281163b9ebda52f890d6941b02f700f13b9ef97a1e228045e5f210c5529e13c\": rpc error: code = NotFound desc = could not find container \"4281163b9ebda52f890d6941b02f700f13b9ef97a1e228045e5f210c5529e13c\": container with ID starting with 4281163b9ebda52f890d6941b02f700f13b9ef97a1e228045e5f210c5529e13c not found: ID does not exist" Jul 06 22:48:38.253813 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:48:38.253786 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-65387d-b7947677b-5cblp"] Jul 06 22:48:38.257245 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:48:38.257220 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-65387d-b7947677b-5cblp"] Jul 06 22:48:39.340255 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:48:39.340222 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8bc2d37-3211-464c-8393-abc51bddbdbd" path="/var/lib/kubelet/pods/a8bc2d37-3211-464c-8393-abc51bddbdbd/volumes" Jul 06 22:48:49.946055 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:48:49.945967 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-65387d/rate-test-model-65387d-kserve-55bbb4bfdc-kzfdh" podUID="4a1c0651-b003-4dff-b83a-b98e5fb15721" containerName="main" containerID="cri-o://e2c30044188e0615c855d24184388401e0b93f1f15f1c79f3346f0331885b26c" gracePeriod=45 Jul 06 22:48:50.201751 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:48:50.201678 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-65387d/rate-test-model-65387d-kserve-55bbb4bfdc-kzfdh" Jul 06 22:48:50.298618 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:48:50.298588 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a1c0651-b003-4dff-b83a-b98e5fb15721-model-cache\") pod \"4a1c0651-b003-4dff-b83a-b98e5fb15721\" (UID: \"4a1c0651-b003-4dff-b83a-b98e5fb15721\") " Jul 06 22:48:50.298837 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:48:50.298654 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a1c0651-b003-4dff-b83a-b98e5fb15721-dshm\") pod \"4a1c0651-b003-4dff-b83a-b98e5fb15721\" (UID: \"4a1c0651-b003-4dff-b83a-b98e5fb15721\") " Jul 06 22:48:50.298837 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:48:50.298680 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a1c0651-b003-4dff-b83a-b98e5fb15721-home\") pod \"4a1c0651-b003-4dff-b83a-b98e5fb15721\" (UID: \"4a1c0651-b003-4dff-b83a-b98e5fb15721\") " Jul 06 22:48:50.298837 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:48:50.298709 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a1c0651-b003-4dff-b83a-b98e5fb15721-tmp-dir\") pod \"4a1c0651-b003-4dff-b83a-b98e5fb15721\" (UID: \"4a1c0651-b003-4dff-b83a-b98e5fb15721\") " Jul 06 22:48:50.298837 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:48:50.298780 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8cjpg\" (UniqueName: \"kubernetes.io/projected/4a1c0651-b003-4dff-b83a-b98e5fb15721-kube-api-access-8cjpg\") pod \"4a1c0651-b003-4dff-b83a-b98e5fb15721\" (UID: \"4a1c0651-b003-4dff-b83a-b98e5fb15721\") " Jul 06 22:48:50.299097 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:48:50.298858 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a1c0651-b003-4dff-b83a-b98e5fb15721-tls-certs\") pod \"4a1c0651-b003-4dff-b83a-b98e5fb15721\" (UID: \"4a1c0651-b003-4dff-b83a-b98e5fb15721\") " Jul 06 22:48:50.299097 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:48:50.298984 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a1c0651-b003-4dff-b83a-b98e5fb15721-home" (OuterVolumeSpecName: "home") pod "4a1c0651-b003-4dff-b83a-b98e5fb15721" (UID: "4a1c0651-b003-4dff-b83a-b98e5fb15721"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:48:50.299097 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:48:50.299001 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a1c0651-b003-4dff-b83a-b98e5fb15721-model-cache" (OuterVolumeSpecName: "model-cache") pod "4a1c0651-b003-4dff-b83a-b98e5fb15721" (UID: "4a1c0651-b003-4dff-b83a-b98e5fb15721"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:48:50.299239 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:48:50.299219 2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a1c0651-b003-4dff-b83a-b98e5fb15721-home\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:48:50.299297 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:48:50.299247 2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a1c0651-b003-4dff-b83a-b98e5fb15721-model-cache\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:48:50.299355 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:48:50.299306 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a1c0651-b003-4dff-b83a-b98e5fb15721-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4a1c0651-b003-4dff-b83a-b98e5fb15721" (UID: "4a1c0651-b003-4dff-b83a-b98e5fb15721"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:48:50.301111 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:48:50.301089 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a1c0651-b003-4dff-b83a-b98e5fb15721-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4a1c0651-b003-4dff-b83a-b98e5fb15721" (UID: "4a1c0651-b003-4dff-b83a-b98e5fb15721"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:48:50.301220 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:48:50.301202 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a1c0651-b003-4dff-b83a-b98e5fb15721-dshm" (OuterVolumeSpecName: "dshm") pod "4a1c0651-b003-4dff-b83a-b98e5fb15721" (UID: "4a1c0651-b003-4dff-b83a-b98e5fb15721"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:48:50.301357 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:48:50.301335 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a1c0651-b003-4dff-b83a-b98e5fb15721-kube-api-access-8cjpg" (OuterVolumeSpecName: "kube-api-access-8cjpg") pod "4a1c0651-b003-4dff-b83a-b98e5fb15721" (UID: "4a1c0651-b003-4dff-b83a-b98e5fb15721"). InnerVolumeSpecName "kube-api-access-8cjpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:48:50.303141 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:48:50.303105 2555 generic.go:356] "Generic (PLEG): container finished" podID="4a1c0651-b003-4dff-b83a-b98e5fb15721" containerID="e2c30044188e0615c855d24184388401e0b93f1f15f1c79f3346f0331885b26c" exitCode=0 Jul 06 22:48:50.303248 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:48:50.303162 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-65387d/rate-test-model-65387d-kserve-55bbb4bfdc-kzfdh" Jul 06 22:48:50.303248 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:48:50.303181 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-65387d/rate-test-model-65387d-kserve-55bbb4bfdc-kzfdh" event={"ID":"4a1c0651-b003-4dff-b83a-b98e5fb15721","Type":"ContainerDied","Data":"e2c30044188e0615c855d24184388401e0b93f1f15f1c79f3346f0331885b26c"} Jul 06 22:48:50.303248 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:48:50.303232 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-65387d/rate-test-model-65387d-kserve-55bbb4bfdc-kzfdh" event={"ID":"4a1c0651-b003-4dff-b83a-b98e5fb15721","Type":"ContainerDied","Data":"b1477953edc590353728231c4d1e7d06a3eb01b35aaacd62c4db5e0ff9deaddc"} Jul 06 22:48:50.303361 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:48:50.303255 2555 scope.go:117] "RemoveContainer" containerID="e2c30044188e0615c855d24184388401e0b93f1f15f1c79f3346f0331885b26c" Jul 06 22:48:50.321641 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:48:50.321616 2555 scope.go:117] "RemoveContainer" containerID="e2c30044188e0615c855d24184388401e0b93f1f15f1c79f3346f0331885b26c" Jul 06 22:48:50.321938 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:48:50.321918 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2c30044188e0615c855d24184388401e0b93f1f15f1c79f3346f0331885b26c\": container with ID starting with e2c30044188e0615c855d24184388401e0b93f1f15f1c79f3346f0331885b26c not found: ID does not exist" containerID="e2c30044188e0615c855d24184388401e0b93f1f15f1c79f3346f0331885b26c" Jul 06 22:48:50.321994 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:48:50.321954 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2c30044188e0615c855d24184388401e0b93f1f15f1c79f3346f0331885b26c"} err="failed to get container status \"e2c30044188e0615c855d24184388401e0b93f1f15f1c79f3346f0331885b26c\": rpc error: code = NotFound desc = could not find container \"e2c30044188e0615c855d24184388401e0b93f1f15f1c79f3346f0331885b26c\": container with ID starting with e2c30044188e0615c855d24184388401e0b93f1f15f1c79f3346f0331885b26c not found: ID does not exist" Jul 06 22:48:50.332339 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:48:50.332312 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-65387d/rate-test-model-65387d-kserve-55bbb4bfdc-kzfdh"] Jul 06 22:48:50.336069 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:48:50.336045 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-65387d/rate-test-model-65387d-kserve-55bbb4bfdc-kzfdh"] Jul 06 22:48:50.400923 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:48:50.400871 2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a1c0651-b003-4dff-b83a-b98e5fb15721-dshm\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:48:50.401122 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:48:50.400991 2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a1c0651-b003-4dff-b83a-b98e5fb15721-tmp-dir\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:48:50.401122 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:48:50.401048 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cjpg\" (UniqueName: \"kubernetes.io/projected/4a1c0651-b003-4dff-b83a-b98e5fb15721-kube-api-access-8cjpg\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:48:50.401122 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:48:50.401066 2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a1c0651-b003-4dff-b83a-b98e5fb15721-tls-certs\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:48:51.341450 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:48:51.341417 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a1c0651-b003-4dff-b83a-b98e5fb15721" path="/var/lib/kubelet/pods/4a1c0651-b003-4dff-b83a-b98e5fb15721/volumes" Jul 06 22:49:09.377606 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:09.377469 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7mq8_e50d3b5f-832e-481f-8807-7774ff8d5521/ovn-acl-logging/0.log" Jul 06 22:49:09.385320 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:09.385300 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7mq8_e50d3b5f-832e-481f-8807-7774ff8d5521/ovn-acl-logging/0.log" Jul 06 22:49:20.369470 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:20.369429 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-944f9f-58fdff6c46-6nfjh"] Jul 06 22:49:20.369914 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:20.369702 2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-b-944f9f-58fdff6c46-6nfjh" podUID="67a6cf3b-4068-4cac-a2ec-6a2bb92dc1b4" containerName="maas-api" containerID="cri-o://cec6efba97648e70c6ca19bc78f3218a8f050fada5f68f0985a0a1b5dadbeee4" gracePeriod=30 Jul 06 22:49:20.620497 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:20.620426 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-944f9f-58fdff6c46-6nfjh" Jul 06 22:49:20.701031 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:20.700972 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67a6cf3b-4068-4cac-a2ec-6a2bb92dc1b4-maas-api-tls\") pod \"67a6cf3b-4068-4cac-a2ec-6a2bb92dc1b4\" (UID: \"67a6cf3b-4068-4cac-a2ec-6a2bb92dc1b4\") " Jul 06 22:49:20.701243 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:20.701093 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zj4xk\" (UniqueName: \"kubernetes.io/projected/67a6cf3b-4068-4cac-a2ec-6a2bb92dc1b4-kube-api-access-zj4xk\") pod \"67a6cf3b-4068-4cac-a2ec-6a2bb92dc1b4\" (UID: \"67a6cf3b-4068-4cac-a2ec-6a2bb92dc1b4\") " Jul 06 22:49:20.701243 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:20.701176 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67a6cf3b-4068-4cac-a2ec-6a2bb92dc1b4-openshift-service-ca\") pod \"67a6cf3b-4068-4cac-a2ec-6a2bb92dc1b4\" (UID: \"67a6cf3b-4068-4cac-a2ec-6a2bb92dc1b4\") " Jul 06 22:49:20.701519 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:20.701495 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67a6cf3b-4068-4cac-a2ec-6a2bb92dc1b4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "67a6cf3b-4068-4cac-a2ec-6a2bb92dc1b4" (UID: "67a6cf3b-4068-4cac-a2ec-6a2bb92dc1b4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:49:20.703285 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:20.703258 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67a6cf3b-4068-4cac-a2ec-6a2bb92dc1b4-kube-api-access-zj4xk" (OuterVolumeSpecName: "kube-api-access-zj4xk") pod "67a6cf3b-4068-4cac-a2ec-6a2bb92dc1b4" (UID: "67a6cf3b-4068-4cac-a2ec-6a2bb92dc1b4"). InnerVolumeSpecName "kube-api-access-zj4xk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:49:20.703389 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:20.703364 2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67a6cf3b-4068-4cac-a2ec-6a2bb92dc1b4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "67a6cf3b-4068-4cac-a2ec-6a2bb92dc1b4" (UID: "67a6cf3b-4068-4cac-a2ec-6a2bb92dc1b4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:49:20.802538 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:20.802490 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67a6cf3b-4068-4cac-a2ec-6a2bb92dc1b4-maas-api-tls\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:49:20.802538 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:20.802529 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zj4xk\" (UniqueName: \"kubernetes.io/projected/67a6cf3b-4068-4cac-a2ec-6a2bb92dc1b4-kube-api-access-zj4xk\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:49:20.802538 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:20.802542 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67a6cf3b-4068-4cac-a2ec-6a2bb92dc1b4-openshift-service-ca\") on node \"ip-10-0-128-71.ec2.internal\" DevicePath \"\"" Jul 06 22:49:21.459932 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:21.459898 2555 generic.go:356] "Generic (PLEG): container finished" podID="67a6cf3b-4068-4cac-a2ec-6a2bb92dc1b4" containerID="cec6efba97648e70c6ca19bc78f3218a8f050fada5f68f0985a0a1b5dadbeee4" exitCode=0 Jul 06 22:49:21.460445 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:21.459943 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-944f9f-58fdff6c46-6nfjh" Jul 06 22:49:21.460445 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:21.459980 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-944f9f-58fdff6c46-6nfjh" event={"ID":"67a6cf3b-4068-4cac-a2ec-6a2bb92dc1b4","Type":"ContainerDied","Data":"cec6efba97648e70c6ca19bc78f3218a8f050fada5f68f0985a0a1b5dadbeee4"} Jul 06 22:49:21.460445 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:21.460058 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-944f9f-58fdff6c46-6nfjh" event={"ID":"67a6cf3b-4068-4cac-a2ec-6a2bb92dc1b4","Type":"ContainerDied","Data":"3ef3caed0d4934594494c48b2801fcf136ddefcc04fb2a155c4c27ef459b011a"} Jul 06 22:49:21.460445 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:21.460083 2555 scope.go:117] "RemoveContainer" containerID="cec6efba97648e70c6ca19bc78f3218a8f050fada5f68f0985a0a1b5dadbeee4" Jul 06 22:49:21.469384 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:21.469363 2555 scope.go:117] "RemoveContainer" containerID="cec6efba97648e70c6ca19bc78f3218a8f050fada5f68f0985a0a1b5dadbeee4" Jul 06 22:49:21.469674 ip-10-0-128-71 kubenswrapper[2555]: E0706 22:49:21.469651 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cec6efba97648e70c6ca19bc78f3218a8f050fada5f68f0985a0a1b5dadbeee4\": container with ID starting with cec6efba97648e70c6ca19bc78f3218a8f050fada5f68f0985a0a1b5dadbeee4 not found: ID does not exist" containerID="cec6efba97648e70c6ca19bc78f3218a8f050fada5f68f0985a0a1b5dadbeee4" Jul 06 22:49:21.469760 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:21.469685 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cec6efba97648e70c6ca19bc78f3218a8f050fada5f68f0985a0a1b5dadbeee4"} err="failed to get container status \"cec6efba97648e70c6ca19bc78f3218a8f050fada5f68f0985a0a1b5dadbeee4\": rpc error: code = NotFound desc = could not find container \"cec6efba97648e70c6ca19bc78f3218a8f050fada5f68f0985a0a1b5dadbeee4\": container with ID starting with cec6efba97648e70c6ca19bc78f3218a8f050fada5f68f0985a0a1b5dadbeee4 not found: ID does not exist" Jul 06 22:49:21.477041 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:21.476999 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-944f9f-58fdff6c46-6nfjh"] Jul 06 22:49:21.481031 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:21.480992 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-944f9f-58fdff6c46-6nfjh"] Jul 06 22:49:23.339575 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:23.339540 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67a6cf3b-4068-4cac-a2ec-6a2bb92dc1b4" path="/var/lib/kubelet/pods/67a6cf3b-4068-4cac-a2ec-6a2bb92dc1b4/volumes" Jul 06 22:49:27.308480 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:27.308446 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-79b4c9c9b9-7kzcr_787cb0c4-927b-4829-8813-b17cbbc6933d/authorino/0.log" Jul 06 22:49:32.027447 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:32.027361 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-6c74cd8d45-rdzxd_13ecb710-e391-4cd3-85a9-e508b11d6dc8/maas-api/0.log" Jul 06 22:49:32.842745 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:32.842716 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c695b4674-75zsx_bc5a156b-600c-418e-98ca-4cb925001ce5/copy-manifests/0.log" Jul 06 22:49:32.849326 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:32.849283 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c695b4674-75zsx_bc5a156b-600c-418e-98ca-4cb925001ce5/manager/0.log" Jul 06 22:49:34.326326 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:34.326290 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-79b4c9c9b9-7kzcr_787cb0c4-927b-4829-8813-b17cbbc6933d/authorino/0.log" Jul 06 22:49:34.653501 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:34.653468 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-rgzxd_8e03dcbd-aef7-452e-a850-6331b00df90b/kuadrant-console-plugin/0.log" Jul 06 22:49:35.130682 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:35.130595 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-jwqrx_4df8a97d-bc32-4790-b551-e97c65523756/manager/0.log" Jul 06 22:49:35.815915 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:35.815884 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-658774566b-l959z_2ca2aaf5-c87c-498e-aec9-da080ea88381/kube-auth-proxy/0.log" Jul 06 22:49:36.874677 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:36.874649 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-jxsc9_522a53e6-aca2-4f21-8756-961e2a4d3c09/main/0.log" Jul 06 22:49:43.781893 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:43.781848 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-zt94q_d681bb3e-9bb1-4705-98cb-fd2cd5dc7063/global-pull-secret-syncer/0.log" Jul 06 22:49:43.900761 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:43.900729 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-grmjd_f7d6b6f8-91a4-4341-b018-4e768abdd9a0/konnectivity-agent/0.log" Jul 06 22:49:43.925856 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:43.925820 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-71.ec2.internal_33df906486200f7aaa67d6b5660b85fa/haproxy/0.log" Jul 06 22:49:49.689604 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:49.689504 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-79b4c9c9b9-7kzcr_787cb0c4-927b-4829-8813-b17cbbc6933d/authorino/0.log" Jul 06 22:49:49.776063 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:49.776031 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-rgzxd_8e03dcbd-aef7-452e-a850-6331b00df90b/kuadrant-console-plugin/0.log" Jul 06 22:49:50.092576 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:50.092533 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-jwqrx_4df8a97d-bc32-4790-b551-e97c65523756/manager/0.log" Jul 06 22:49:51.704979 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:51.704945 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_38ffd73f-16ff-4301-9084-a4dc29c0cdef/alertmanager/0.log" Jul 06 22:49:51.726365 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:51.726328 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_38ffd73f-16ff-4301-9084-a4dc29c0cdef/config-reloader/0.log" Jul 06 22:49:51.747059 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:51.747028 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_38ffd73f-16ff-4301-9084-a4dc29c0cdef/kube-rbac-proxy-web/0.log" Jul 06 22:49:51.767340 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:51.767317 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_38ffd73f-16ff-4301-9084-a4dc29c0cdef/kube-rbac-proxy/0.log" Jul 06 22:49:51.792092 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:51.792061 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_38ffd73f-16ff-4301-9084-a4dc29c0cdef/kube-rbac-proxy-metric/0.log" Jul 06 22:49:51.811982 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:51.811950 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_38ffd73f-16ff-4301-9084-a4dc29c0cdef/prom-label-proxy/0.log" Jul 06 22:49:51.835642 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:51.835612 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_38ffd73f-16ff-4301-9084-a4dc29c0cdef/init-config-reloader/0.log" Jul 06 22:49:51.903788 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:51.903766 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-xfb9f_2c5f15ee-59ba-4a1a-9e8b-5826acf7a438/kube-state-metrics/0.log" Jul 06 22:49:51.922225 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:51.922187 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-xfb9f_2c5f15ee-59ba-4a1a-9e8b-5826acf7a438/kube-rbac-proxy-main/0.log" Jul 06 22:49:51.942112 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:51.942068 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-xfb9f_2c5f15ee-59ba-4a1a-9e8b-5826acf7a438/kube-rbac-proxy-self/0.log" Jul 06 22:49:51.968599 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:51.968572 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5678f86565-bhz5r_a0bd2d8d-b4a6-456b-9f88-fb699e4ed523/metrics-server/0.log" Jul 06 22:49:51.995220 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:51.995189 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-swjnf_74152ba8-c71c-42ad-8109-60c71839ae56/monitoring-plugin/0.log" Jul 06 22:49:52.179579 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:52.179489 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p5xwh_e1a16d17-2367-4bd9-834a-c2da55613ab4/node-exporter/0.log" Jul 06 22:49:52.199218 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:52.199186 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p5xwh_e1a16d17-2367-4bd9-834a-c2da55613ab4/kube-rbac-proxy/0.log" Jul 06 22:49:52.219315 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:52.219290 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p5xwh_e1a16d17-2367-4bd9-834a-c2da55613ab4/init-textfile/0.log" Jul 06 22:49:52.246178 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:52.246150 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-st8th_ea64e052-1c86-42b5-a8fd-b3f7e2f042cb/kube-rbac-proxy-main/0.log" Jul 06 22:49:52.265717 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:52.265683 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-st8th_ea64e052-1c86-42b5-a8fd-b3f7e2f042cb/kube-rbac-proxy-self/0.log" Jul 06 22:49:52.285978 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:52.285946 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-st8th_ea64e052-1c86-42b5-a8fd-b3f7e2f042cb/openshift-state-metrics/0.log" Jul 06 22:49:52.324136 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:52.324107 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_85868a97-948b-4c4b-b5ab-30c7fbbfef4f/prometheus/0.log" Jul 06 22:49:52.345142 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:52.345117 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_85868a97-948b-4c4b-b5ab-30c7fbbfef4f/config-reloader/0.log" Jul 06 22:49:52.368872 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:52.368836 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_85868a97-948b-4c4b-b5ab-30c7fbbfef4f/thanos-sidecar/0.log" Jul 06 22:49:52.390390 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:52.390365 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_85868a97-948b-4c4b-b5ab-30c7fbbfef4f/kube-rbac-proxy-web/0.log" Jul 06 22:49:52.410758 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:52.410727 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_85868a97-948b-4c4b-b5ab-30c7fbbfef4f/kube-rbac-proxy/0.log" Jul 06 22:49:52.431723 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:52.431651 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_85868a97-948b-4c4b-b5ab-30c7fbbfef4f/kube-rbac-proxy-thanos/0.log" Jul 06 22:49:52.451983 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:52.451958 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_85868a97-948b-4c4b-b5ab-30c7fbbfef4f/init-config-reloader/0.log" Jul 06 22:49:52.481468 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:52.481437 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-kcp9n_acbf1ef4-43b4-413a-bbf9-49485cfa7cba/prometheus-operator/0.log" Jul 06 22:49:52.501312 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:52.501281 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-kcp9n_acbf1ef4-43b4-413a-bbf9-49485cfa7cba/kube-rbac-proxy/0.log" Jul 06 22:49:52.523970 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:52.523940 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-d559d66d5-dn48d_a1e09957-d459-471b-a220-d04d9c960228/prometheus-operator-admission-webhook/0.log" Jul 06 22:49:52.551711 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:52.551678 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-67d899d7c8-vlbqf_491f519b-6854-4770-876e-df3f7fe5bb84/telemeter-client/0.log" Jul 06 22:49:52.571616 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:52.571586 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-67d899d7c8-vlbqf_491f519b-6854-4770-876e-df3f7fe5bb84/reload/0.log" Jul 06 22:49:52.597838 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:52.597805 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-67d899d7c8-vlbqf_491f519b-6854-4770-876e-df3f7fe5bb84/kube-rbac-proxy/0.log" Jul 06 22:49:52.628565 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:52.628537 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75c6b47584-6bdk2_d1a7e001-36db-4a34-ae02-b8c05ef42d77/thanos-query/0.log" Jul 06 22:49:52.649711 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:52.649682 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75c6b47584-6bdk2_d1a7e001-36db-4a34-ae02-b8c05ef42d77/kube-rbac-proxy-web/0.log" Jul 06 22:49:52.686343 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:52.686272 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75c6b47584-6bdk2_d1a7e001-36db-4a34-ae02-b8c05ef42d77/kube-rbac-proxy/0.log" Jul 06 22:49:52.708966 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:52.708943 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75c6b47584-6bdk2_d1a7e001-36db-4a34-ae02-b8c05ef42d77/prom-label-proxy/0.log" Jul 06 22:49:52.729124 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:52.729084 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75c6b47584-6bdk2_d1a7e001-36db-4a34-ae02-b8c05ef42d77/kube-rbac-proxy-rules/0.log" Jul 06 22:49:52.748901 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:52.748874 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75c6b47584-6bdk2_d1a7e001-36db-4a34-ae02-b8c05ef42d77/kube-rbac-proxy-metrics/0.log" Jul 06 22:49:53.617874 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:53.617843 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fc7g8/perf-node-gather-daemonset-7jgs9"] Jul 06 22:49:53.618270 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:53.618257 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71a3ad33-6719-46fb-bcc4-9fa79d1ae76f" containerName="main" Jul 06 22:49:53.618335 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:53.618272 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="71a3ad33-6719-46fb-bcc4-9fa79d1ae76f" containerName="main" Jul 06 22:49:53.618335 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:53.618286 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a1c0651-b003-4dff-b83a-b98e5fb15721" containerName="main" Jul 06 22:49:53.618335 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:53.618292 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a1c0651-b003-4dff-b83a-b98e5fb15721" containerName="main" Jul 06 22:49:53.618335 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:53.618301 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8bc2d37-3211-464c-8393-abc51bddbdbd" containerName="maas-api" Jul 06 22:49:53.618335 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:53.618307 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8bc2d37-3211-464c-8393-abc51bddbdbd" containerName="maas-api" Jul 06 22:49:53.618335 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:53.618313 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67a6cf3b-4068-4cac-a2ec-6a2bb92dc1b4" containerName="maas-api" Jul 06 22:49:53.618335 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:53.618318 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="67a6cf3b-4068-4cac-a2ec-6a2bb92dc1b4" containerName="maas-api" Jul 06 22:49:53.618652 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:53.618388 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8bc2d37-3211-464c-8393-abc51bddbdbd" containerName="maas-api" Jul 06 22:49:53.618652 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:53.618396 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="71a3ad33-6719-46fb-bcc4-9fa79d1ae76f" containerName="main" Jul 06 22:49:53.618652 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:53.618405 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a1c0651-b003-4dff-b83a-b98e5fb15721" containerName="main" Jul 06 22:49:53.618652 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:53.618411 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="67a6cf3b-4068-4cac-a2ec-6a2bb92dc1b4" containerName="maas-api" Jul 06 22:49:53.622138 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:53.622115 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fc7g8/perf-node-gather-daemonset-7jgs9" Jul 06 22:49:53.624841 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:53.624821 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fc7g8\"/\"openshift-service-ca.crt\"" Jul 06 22:49:53.624967 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:53.624822 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fc7g8\"/\"kube-root-ca.crt\"" Jul 06 22:49:53.625139 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:53.625127 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fc7g8\"/\"default-dockercfg-5gjkx\"" Jul 06 22:49:53.631788 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:53.631768 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fc7g8/perf-node-gather-daemonset-7jgs9"] Jul 06 22:49:53.718928 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:53.718888 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6cfaa7be-4b9b-402f-8521-258eb47e07ff-lib-modules\") pod \"perf-node-gather-daemonset-7jgs9\" (UID: \"6cfaa7be-4b9b-402f-8521-258eb47e07ff\") " pod="openshift-must-gather-fc7g8/perf-node-gather-daemonset-7jgs9" Jul 06 22:49:53.719356 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:53.718946 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6cfaa7be-4b9b-402f-8521-258eb47e07ff-podres\") pod \"perf-node-gather-daemonset-7jgs9\" (UID: \"6cfaa7be-4b9b-402f-8521-258eb47e07ff\") " pod="openshift-must-gather-fc7g8/perf-node-gather-daemonset-7jgs9" Jul 06 22:49:53.719356 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:53.718998 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b488s\" (UniqueName: \"kubernetes.io/projected/6cfaa7be-4b9b-402f-8521-258eb47e07ff-kube-api-access-b488s\") pod \"perf-node-gather-daemonset-7jgs9\" (UID: \"6cfaa7be-4b9b-402f-8521-258eb47e07ff\") " pod="openshift-must-gather-fc7g8/perf-node-gather-daemonset-7jgs9" Jul 06 22:49:53.719356 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:53.719130 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6cfaa7be-4b9b-402f-8521-258eb47e07ff-sys\") pod \"perf-node-gather-daemonset-7jgs9\" (UID: \"6cfaa7be-4b9b-402f-8521-258eb47e07ff\") " pod="openshift-must-gather-fc7g8/perf-node-gather-daemonset-7jgs9" Jul 06 22:49:53.719356 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:53.719160 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6cfaa7be-4b9b-402f-8521-258eb47e07ff-proc\") pod \"perf-node-gather-daemonset-7jgs9\" (UID: \"6cfaa7be-4b9b-402f-8521-258eb47e07ff\") " pod="openshift-must-gather-fc7g8/perf-node-gather-daemonset-7jgs9" Jul 06 22:49:53.819953 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:53.819914 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b488s\" (UniqueName: \"kubernetes.io/projected/6cfaa7be-4b9b-402f-8521-258eb47e07ff-kube-api-access-b488s\") pod \"perf-node-gather-daemonset-7jgs9\" (UID: \"6cfaa7be-4b9b-402f-8521-258eb47e07ff\") " pod="openshift-must-gather-fc7g8/perf-node-gather-daemonset-7jgs9" Jul 06 22:49:53.820181 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:53.819984 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6cfaa7be-4b9b-402f-8521-258eb47e07ff-sys\") pod \"perf-node-gather-daemonset-7jgs9\" (UID: \"6cfaa7be-4b9b-402f-8521-258eb47e07ff\") " pod="openshift-must-gather-fc7g8/perf-node-gather-daemonset-7jgs9" Jul 06 22:49:53.820181 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:53.820001 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6cfaa7be-4b9b-402f-8521-258eb47e07ff-proc\") pod \"perf-node-gather-daemonset-7jgs9\" (UID: \"6cfaa7be-4b9b-402f-8521-258eb47e07ff\") " pod="openshift-must-gather-fc7g8/perf-node-gather-daemonset-7jgs9" Jul 06 22:49:53.820181 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:53.820065 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6cfaa7be-4b9b-402f-8521-258eb47e07ff-lib-modules\") pod \"perf-node-gather-daemonset-7jgs9\" (UID: \"6cfaa7be-4b9b-402f-8521-258eb47e07ff\") " pod="openshift-must-gather-fc7g8/perf-node-gather-daemonset-7jgs9" Jul 06 22:49:53.820181 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:53.820097 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6cfaa7be-4b9b-402f-8521-258eb47e07ff-podres\") pod \"perf-node-gather-daemonset-7jgs9\" (UID: \"6cfaa7be-4b9b-402f-8521-258eb47e07ff\") " pod="openshift-must-gather-fc7g8/perf-node-gather-daemonset-7jgs9" Jul 06 22:49:53.820181 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:53.820130 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6cfaa7be-4b9b-402f-8521-258eb47e07ff-sys\") pod \"perf-node-gather-daemonset-7jgs9\" (UID: \"6cfaa7be-4b9b-402f-8521-258eb47e07ff\") " pod="openshift-must-gather-fc7g8/perf-node-gather-daemonset-7jgs9" Jul 06 22:49:53.820181 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:53.820130 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6cfaa7be-4b9b-402f-8521-258eb47e07ff-proc\") pod \"perf-node-gather-daemonset-7jgs9\" (UID: \"6cfaa7be-4b9b-402f-8521-258eb47e07ff\") " pod="openshift-must-gather-fc7g8/perf-node-gather-daemonset-7jgs9" Jul 06 22:49:53.820433 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:53.820236 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6cfaa7be-4b9b-402f-8521-258eb47e07ff-lib-modules\") pod \"perf-node-gather-daemonset-7jgs9\" (UID: \"6cfaa7be-4b9b-402f-8521-258eb47e07ff\") " pod="openshift-must-gather-fc7g8/perf-node-gather-daemonset-7jgs9" Jul 06 22:49:53.820433 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:53.820238 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6cfaa7be-4b9b-402f-8521-258eb47e07ff-podres\") pod \"perf-node-gather-daemonset-7jgs9\" (UID: \"6cfaa7be-4b9b-402f-8521-258eb47e07ff\") " pod="openshift-must-gather-fc7g8/perf-node-gather-daemonset-7jgs9" Jul 06 22:49:53.828181 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:53.828155 2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b488s\" (UniqueName: \"kubernetes.io/projected/6cfaa7be-4b9b-402f-8521-258eb47e07ff-kube-api-access-b488s\") pod \"perf-node-gather-daemonset-7jgs9\" (UID: \"6cfaa7be-4b9b-402f-8521-258eb47e07ff\") " pod="openshift-must-gather-fc7g8/perf-node-gather-daemonset-7jgs9" Jul 06 22:49:53.933454 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:53.933413 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fc7g8/perf-node-gather-daemonset-7jgs9" Jul 06 22:49:54.062136 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:54.062110 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fc7g8/perf-node-gather-daemonset-7jgs9"] Jul 06 22:49:54.064049 ip-10-0-128-71 kubenswrapper[2555]: W0706 22:49:54.063990 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6cfaa7be_4b9b_402f_8521_258eb47e07ff.slice/crio-fd9585131b68a270df7abeca6b9e9c770a4ee000cf63f9912f496f4a228bdc34 WatchSource:0}: Error finding container fd9585131b68a270df7abeca6b9e9c770a4ee000cf63f9912f496f4a228bdc34: Status 404 returned error can't find the container with id fd9585131b68a270df7abeca6b9e9c770a4ee000cf63f9912f496f4a228bdc34 Jul 06 22:49:54.623219 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:54.623185 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fc7g8/perf-node-gather-daemonset-7jgs9" event={"ID":"6cfaa7be-4b9b-402f-8521-258eb47e07ff","Type":"ContainerStarted","Data":"b938f521d21ccf2e1d7e8fc1713c70b80137b11192857c88f1efbdee91e69942"} Jul 06 22:49:54.623434 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:54.623240 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fc7g8/perf-node-gather-daemonset-7jgs9" event={"ID":"6cfaa7be-4b9b-402f-8521-258eb47e07ff","Type":"ContainerStarted","Data":"fd9585131b68a270df7abeca6b9e9c770a4ee000cf63f9912f496f4a228bdc34"} Jul 06 22:49:54.623434 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:54.623261 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-fc7g8/perf-node-gather-daemonset-7jgs9" Jul 06 22:49:54.639467 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:54.639417 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-fc7g8/perf-node-gather-daemonset-7jgs9" podStartSLOduration=1.639402708 podStartE2EDuration="1.639402708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:49:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:49:54.637853306 +0000 UTC m=+2445.937565157" watchObservedRunningTime="2026-07-06 22:49:54.639402708 +0000 UTC m=+2445.939114556" Jul 06 22:49:55.035880 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:55.035850 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-576896d655-4w5c6_27b7b97e-de46-420f-bfb2-edfe27d7301e/console/0.log" Jul 06 22:49:56.748864 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:56.748828 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xj686_d0dd52ea-c96a-480a-8c24-629502b393a3/dns/0.log" Jul 06 22:49:56.770100 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:56.770070 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xj686_d0dd52ea-c96a-480a-8c24-629502b393a3/kube-rbac-proxy/0.log" Jul 06 22:49:56.818934 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:56.818901 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dfd8v_7139257f-7bcb-4486-8818-83a0c465eea4/dns-node-resolver/0.log" Jul 06 22:49:57.405760 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:57.405726 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9tlpj_7c3b8e45-4537-4010-97c9-6081a2e3edcf/node-ca/0.log" Jul 06 22:49:58.560351 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:58.560319 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-658774566b-l959z_2ca2aaf5-c87c-498e-aec9-da080ea88381/kube-auth-proxy/0.log" Jul 06 22:49:59.462668 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:49:59.462634 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9xdn4_1dbc58da-974a-43a5-8106-008bccb0da53/serve-healthcheck-canary/0.log" Jul 06 22:50:00.140641 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:50:00.140608 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-m4k2h_8eb86aa0-0a0b-4de6-8302-1e0f557df2ba/kube-rbac-proxy/0.log" Jul 06 22:50:00.160986 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:50:00.160961 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-m4k2h_8eb86aa0-0a0b-4de6-8302-1e0f557df2ba/exporter/0.log" Jul 06 22:50:00.183932 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:50:00.183904 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-m4k2h_8eb86aa0-0a0b-4de6-8302-1e0f557df2ba/extractor/0.log" Jul 06 22:50:00.638713 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:50:00.638637 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-fc7g8/perf-node-gather-daemonset-7jgs9" Jul 06 22:50:02.817409 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:50:02.817373 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-6c74cd8d45-rdzxd_13ecb710-e391-4cd3-85a9-e508b11d6dc8/maas-api/0.log" Jul 06 22:50:03.238246 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:50:03.238214 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c695b4674-75zsx_bc5a156b-600c-418e-98ca-4cb925001ce5/manager/0.log" Jul 06 22:50:03.263989 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:50:03.263961 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c695b4674-75zsx_bc5a156b-600c-418e-98ca-4cb925001ce5/copy-manifests/0.log" Jul 06 22:50:04.748342 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:50:04.748313 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7bcf4db656-jlzqb_08a08704-1234-4ea2-9c3c-af2d3dad8fda/manager/0.log" Jul 06 22:50:04.776878 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:50:04.776851 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-658774566b-l959z_2ca2aaf5-c87c-498e-aec9-da080ea88381/kube-auth-proxy/0.log" Jul 06 22:50:06.288772 ip-10-0-128-71 kubenswrapper[2555]: I0706 22:50:06.288740 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-658774566b-l959z_2ca2aaf5-c87c-498e-aec9-da080ea88381/kube-auth-proxy/0.log"