Jul 07 22:03:40.056752 ip-10-0-134-156 systemd[1]: Starting Kubernetes Kubelet... Jul 07 22:03:40.447378 ip-10-0-134-156 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 22:03:40.447378 ip-10-0-134-156 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 07 22:03:40.447378 ip-10-0-134-156 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 22:03:40.447378 ip-10-0-134-156 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 07 22:03:40.447378 ip-10-0-134-156 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 22:03:40.448053 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.447395 2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 07 22:03:40.451979 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.451962 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 22:03:40.451979 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.451979 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 22:03:40.452053 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.451983 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 22:03:40.452053 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.451987 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 22:03:40.452053 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.451990 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 22:03:40.452053 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.451993 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 22:03:40.452053 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.451996 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 22:03:40.452053 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.451999 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 22:03:40.452053 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452001 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 22:03:40.452053 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452004 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 22:03:40.452053 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452006 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 22:03:40.452053 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452009 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 22:03:40.452053 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452011 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 22:03:40.452053 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452014 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 22:03:40.452053 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452016 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 22:03:40.452053 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452019 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 22:03:40.452053 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452029 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 22:03:40.452053 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452032 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 22:03:40.452053 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452035 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 22:03:40.452053 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452037 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 22:03:40.452053 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452039 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 22:03:40.452053 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452042 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 22:03:40.452629 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452044 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 22:03:40.452629 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452047 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 22:03:40.452629 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452049 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 22:03:40.452629 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452052 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 22:03:40.452629 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452054 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 22:03:40.452629 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452057 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 22:03:40.452629 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452059 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 22:03:40.452629 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452062 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 22:03:40.452629 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452065 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 22:03:40.452629 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452068 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 22:03:40.452629 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452070 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 22:03:40.452629 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452073 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 22:03:40.452629 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452076 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 22:03:40.452629 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452078 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 22:03:40.452629 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452081 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 22:03:40.452629 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452083 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 22:03:40.452629 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452085 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 22:03:40.452629 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452088 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 22:03:40.452629 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452090 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 22:03:40.452629 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452093 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 22:03:40.453425 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452096 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 22:03:40.453425 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452099 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 22:03:40.453425 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452102 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 22:03:40.453425 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452105 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 22:03:40.453425 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452107 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 22:03:40.453425 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452111 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 22:03:40.453425 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452115 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 22:03:40.453425 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452118 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 22:03:40.453425 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452121 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 22:03:40.453425 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452123 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 22:03:40.453425 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452125 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 22:03:40.453425 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452128 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 07 22:03:40.453425 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452130 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 22:03:40.453425 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452133 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 22:03:40.453425 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452136 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 22:03:40.453425 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452139 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 22:03:40.453425 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452141 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 22:03:40.453425 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452144 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 22:03:40.453425 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452146 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 22:03:40.453425 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452149 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 22:03:40.454080 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452151 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 22:03:40.454080 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452154 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 22:03:40.454080 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452156 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 22:03:40.454080 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452158 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 22:03:40.454080 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452161 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 22:03:40.454080 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452163 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 22:03:40.454080 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452166 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 22:03:40.454080 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452168 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 22:03:40.454080 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452170 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 22:03:40.454080 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452173 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 22:03:40.454080 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452175 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 22:03:40.454080 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452178 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 22:03:40.454080 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452180 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 22:03:40.454080 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452182 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 22:03:40.454080 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452185 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 22:03:40.454080 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452187 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 22:03:40.454080 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452190 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 22:03:40.454080 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452193 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 22:03:40.454080 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452195 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 22:03:40.454080 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452198 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 22:03:40.454590 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452201 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 22:03:40.454590 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452204 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 22:03:40.454590 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452206 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 22:03:40.454590 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452208 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 22:03:40.454590 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452211 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 22:03:40.454590 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452217 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 22:03:40.454590 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452221 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 22:03:40.454590 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452223 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 22:03:40.454590 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452226 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 22:03:40.454590 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452230 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 22:03:40.454590 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452233 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 22:03:40.454590 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452236 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 22:03:40.454590 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452238 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 22:03:40.454590 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452241 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 22:03:40.454590 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452244 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 22:03:40.454590 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452246 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 22:03:40.454590 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.452249 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 22:03:40.455787 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455771 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 22:03:40.455787 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455787 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 07 22:03:40.455917 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455791 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 22:03:40.455917 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455794 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 22:03:40.455917 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455798 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 22:03:40.455917 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455801 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 22:03:40.455917 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455803 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 22:03:40.455917 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455806 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 22:03:40.455917 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455809 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 22:03:40.455917 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455814 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 22:03:40.455917 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455818 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 22:03:40.455917 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455822 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 22:03:40.455917 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455825 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 22:03:40.455917 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455828 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 22:03:40.455917 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455831 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 22:03:40.455917 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455833 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 22:03:40.455917 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455836 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 22:03:40.455917 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455839 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 22:03:40.455917 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455841 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 22:03:40.455917 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455845 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 22:03:40.455917 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455848 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 22:03:40.455917 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455850 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 22:03:40.456465 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455853 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 22:03:40.456465 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455855 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 22:03:40.456465 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455858 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 22:03:40.456465 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455860 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 22:03:40.456465 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455863 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 22:03:40.456465 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455865 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 22:03:40.456465 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455868 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 22:03:40.456465 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455870 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 22:03:40.456465 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455873 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 22:03:40.456465 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455876 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 22:03:40.456465 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455878 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 22:03:40.456465 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455880 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 22:03:40.456465 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455883 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 22:03:40.456465 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455886 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 22:03:40.456465 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455889 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 22:03:40.456465 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455892 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 22:03:40.456465 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455894 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 22:03:40.456465 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455896 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 22:03:40.456465 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455899 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 22:03:40.456465 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455901 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 22:03:40.456965 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455904 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 22:03:40.456965 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455907 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 22:03:40.456965 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455910 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 22:03:40.456965 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455912 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 22:03:40.456965 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455914 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 22:03:40.456965 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455917 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 22:03:40.456965 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455919 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 22:03:40.456965 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455922 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 22:03:40.456965 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455924 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 22:03:40.456965 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455927 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 22:03:40.456965 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455931 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 22:03:40.456965 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455935 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 22:03:40.456965 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455938 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 22:03:40.456965 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455941 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 22:03:40.456965 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455944 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 22:03:40.456965 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455947 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 22:03:40.456965 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455949 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 22:03:40.456965 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455952 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 22:03:40.456965 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455954 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 22:03:40.457430 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455957 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 22:03:40.457430 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455959 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 22:03:40.457430 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455962 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 22:03:40.457430 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455964 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 22:03:40.457430 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455967 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 22:03:40.457430 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455969 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 22:03:40.457430 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455973 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 22:03:40.457430 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455977 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 22:03:40.457430 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455980 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 22:03:40.457430 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455983 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 22:03:40.457430 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455985 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 22:03:40.457430 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455988 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 22:03:40.457430 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455990 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 22:03:40.457430 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455993 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 22:03:40.457430 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455996 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 22:03:40.457430 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.455998 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 22:03:40.457430 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456001 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 22:03:40.457430 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456003 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 22:03:40.457430 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456006 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 22:03:40.457430 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456009 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 22:03:40.457933 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456011 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 22:03:40.457933 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456013 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 22:03:40.457933 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456017 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 22:03:40.457933 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456019 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 22:03:40.457933 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456022 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 22:03:40.457933 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456024 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 22:03:40.457933 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456027 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 22:03:40.457933 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456029 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 22:03:40.457933 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456031 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 22:03:40.457933 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456034 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 22:03:40.457933 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456037 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 22:03:40.457933 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456041 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 22:03:40.457933 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456044 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 22:03:40.457933 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456046 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 22:03:40.457933 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456049 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 22:03:40.457933 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456051 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 22:03:40.457933 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456054 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 22:03:40.457933 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456057 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 22:03:40.457933 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456484 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 22:03:40.457933 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456490 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 22:03:40.458416 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456494 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 22:03:40.458416 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456497 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 22:03:40.458416 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456500 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 22:03:40.458416 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456503 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 22:03:40.458416 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456506 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 22:03:40.458416 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456508 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 22:03:40.458416 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456511 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 22:03:40.458416 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456513 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 22:03:40.458416 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456516 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 22:03:40.458416 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456519 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 22:03:40.458416 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456521 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 22:03:40.458416 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456523 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 22:03:40.458416 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456526 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 22:03:40.458416 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456529 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 22:03:40.458416 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456532 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 22:03:40.458416 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456534 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 22:03:40.458416 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456537 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 22:03:40.458416 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456540 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 22:03:40.458416 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456543 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 22:03:40.458416 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456545 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 22:03:40.458912 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456547 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 22:03:40.458912 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456550 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 22:03:40.458912 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456552 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 22:03:40.458912 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456556 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 22:03:40.458912 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456559 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 22:03:40.458912 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456562 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 22:03:40.458912 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456565 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 22:03:40.458912 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456567 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 22:03:40.458912 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456570 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 22:03:40.458912 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456572 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 22:03:40.458912 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456575 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 22:03:40.458912 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456578 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 22:03:40.458912 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456581 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 22:03:40.458912 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456583 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 22:03:40.458912 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456586 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 22:03:40.458912 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456588 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 22:03:40.458912 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456591 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 22:03:40.458912 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456593 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 22:03:40.458912 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456595 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 22:03:40.458912 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456598 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 22:03:40.459395 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456600 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 22:03:40.459395 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456603 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 22:03:40.459395 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456605 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 22:03:40.459395 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456608 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 22:03:40.459395 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456611 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 22:03:40.459395 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456613 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 22:03:40.459395 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456616 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 22:03:40.459395 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456619 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 22:03:40.459395 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456621 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 22:03:40.459395 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456623 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 22:03:40.459395 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456626 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 22:03:40.459395 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456629 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 22:03:40.459395 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456631 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 22:03:40.459395 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456633 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 22:03:40.459395 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456636 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 22:03:40.459395 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456638 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 22:03:40.459395 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456641 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 22:03:40.459395 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456643 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 22:03:40.459395 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456645 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 22:03:40.459395 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456648 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 22:03:40.459905 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456650 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 22:03:40.459905 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456653 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 22:03:40.459905 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456656 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 22:03:40.459905 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456658 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 22:03:40.459905 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456661 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 07 22:03:40.459905 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456663 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 22:03:40.459905 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456666 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 22:03:40.459905 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456668 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 22:03:40.459905 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456671 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 22:03:40.459905 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456674 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 22:03:40.459905 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456678 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 22:03:40.459905 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456681 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 22:03:40.459905 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456684 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 22:03:40.459905 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456687 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 22:03:40.459905 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456690 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 22:03:40.459905 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456693 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 22:03:40.459905 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456696 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 22:03:40.459905 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456699 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 22:03:40.459905 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456702 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 22:03:40.460359 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456704 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 22:03:40.460359 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456707 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 22:03:40.460359 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456710 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 22:03:40.460359 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456713 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 22:03:40.460359 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456716 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 22:03:40.460359 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456719 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 22:03:40.460359 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456722 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 22:03:40.460359 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456725 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 22:03:40.460359 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456727 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 22:03:40.460359 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456730 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 22:03:40.460359 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456732 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 22:03:40.460359 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456735 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 22:03:40.460359 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456737 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 22:03:40.460359 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456740 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 22:03:40.460359 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456742 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 22:03:40.460359 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456745 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 22:03:40.460359 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456748 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 22:03:40.460359 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.456751 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 22:03:40.460359 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.456834 2564 flags.go:64] FLAG: --address="0.0.0.0" Jul 07 22:03:40.460359 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.456842 2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 07 22:03:40.460865 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.456849 2564 flags.go:64] FLAG: --anonymous-auth="true" Jul 07 22:03:40.460865 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.456853 2564 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 07 22:03:40.460865 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.456857 2564 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 07 22:03:40.460865 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.456861 2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 07 22:03:40.460865 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.456865 2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 07 22:03:40.460865 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.456869 2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 07 22:03:40.460865 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.456872 2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 07 22:03:40.460865 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.456875 2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 07 22:03:40.460865 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.456878 2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 07 22:03:40.460865 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.456881 2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 07 22:03:40.460865 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.456884 2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 07 22:03:40.460865 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.456887 2564 flags.go:64] FLAG: --cgroup-root="" Jul 07 22:03:40.460865 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.456890 2564 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 07 22:03:40.460865 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.456893 2564 flags.go:64] FLAG: --client-ca-file="" Jul 07 22:03:40.460865 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.456896 2564 flags.go:64] FLAG: --cloud-provider="external" Jul 07 22:03:40.460865 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.456899 2564 flags.go:64] FLAG: --cluster-dns="[]" Jul 07 22:03:40.460865 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.456903 2564 flags.go:64] FLAG: --cluster-domain="" Jul 07 22:03:40.460865 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.456906 2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 07 22:03:40.460865 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.456909 2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 07 22:03:40.460865 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.456912 2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 07 22:03:40.460865 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.456915 2564 flags.go:64] FLAG: --container-log-max-files="5" Jul 07 22:03:40.460865 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.456918 2564 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 07 22:03:40.460865 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.456921 2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 07 22:03:40.461396 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.456924 2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 07 22:03:40.461396 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.456928 2564 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 07 22:03:40.461396 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.456930 2564 flags.go:64] FLAG: --contention-profiling="false" Jul 07 22:03:40.461396 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.456934 2564 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 07 22:03:40.461396 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.456939 2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 07 22:03:40.461396 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.456943 2564 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 07 22:03:40.461396 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.456946 2564 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 07 22:03:40.461396 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.456950 2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 07 22:03:40.461396 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.456959 2564 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 07 22:03:40.461396 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.456963 2564 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 07 22:03:40.461396 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.456966 2564 flags.go:64] FLAG: --enable-load-reader="false" Jul 07 22:03:40.461396 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.456970 2564 flags.go:64] FLAG: --enable-server="true" Jul 07 22:03:40.461396 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.456974 2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 07 22:03:40.461396 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.456978 2564 flags.go:64] FLAG: --event-burst="100" Jul 07 22:03:40.461396 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.456981 2564 flags.go:64] FLAG: --event-qps="50" Jul 07 22:03:40.461396 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.456984 2564 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 07 22:03:40.461396 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.456987 2564 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 07 22:03:40.461396 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.456990 2564 flags.go:64] FLAG: --eviction-hard="" Jul 07 22:03:40.461396 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.456994 2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 07 22:03:40.461396 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.456997 2564 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 07 22:03:40.461396 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457000 2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 07 22:03:40.461396 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457003 2564 flags.go:64] FLAG: --eviction-soft="" Jul 07 22:03:40.461396 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457005 2564 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 07 22:03:40.461396 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457008 2564 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 07 22:03:40.461396 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457011 2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 07 22:03:40.462098 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457014 2564 flags.go:64] FLAG: --experimental-mounter-path="" Jul 07 22:03:40.462098 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457017 2564 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 07 22:03:40.462098 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457020 2564 flags.go:64] FLAG: --fail-swap-on="true" Jul 07 22:03:40.462098 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457022 2564 flags.go:64] FLAG: --feature-gates="" Jul 07 22:03:40.462098 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457026 2564 flags.go:64] FLAG: --file-check-frequency="20s" Jul 07 22:03:40.462098 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457029 2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 07 22:03:40.462098 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457031 2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 07 22:03:40.462098 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457034 2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 07 22:03:40.462098 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457037 2564 flags.go:64] FLAG: --healthz-port="10248" Jul 07 22:03:40.462098 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457040 2564 flags.go:64] FLAG: --help="false" Jul 07 22:03:40.462098 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457043 2564 flags.go:64] FLAG: --hostname-override="ip-10-0-134-156.ec2.internal" Jul 07 22:03:40.462098 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457048 2564 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 07 22:03:40.462098 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457051 2564 flags.go:64] FLAG: --http-check-frequency="20s" Jul 07 22:03:40.462098 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457054 2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 07 22:03:40.462098 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457057 2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 07 22:03:40.462098 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457060 2564 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 07 22:03:40.462098 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457063 2564 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 07 22:03:40.462098 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457065 2564 flags.go:64] FLAG: --image-service-endpoint="" Jul 07 22:03:40.462098 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457068 2564 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 07 22:03:40.462098 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457071 2564 flags.go:64] FLAG: --kube-api-burst="100" Jul 07 22:03:40.462098 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457074 2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 07 22:03:40.462098 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457076 2564 flags.go:64] FLAG: --kube-api-qps="50" Jul 07 22:03:40.462098 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457080 2564 flags.go:64] FLAG: --kube-reserved="" Jul 07 22:03:40.462098 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457082 2564 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 07 22:03:40.462704 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457085 2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 07 22:03:40.462704 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457088 2564 flags.go:64] FLAG: --kubelet-cgroups="" Jul 07 22:03:40.462704 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457090 2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 07 22:03:40.462704 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457093 2564 flags.go:64] FLAG: --lock-file="" Jul 07 22:03:40.462704 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457096 2564 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 07 22:03:40.462704 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457098 2564 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 07 22:03:40.462704 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457101 2564 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 07 22:03:40.462704 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457108 2564 flags.go:64] FLAG: --log-json-split-stream="false" Jul 07 22:03:40.462704 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457111 2564 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 07 22:03:40.462704 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457114 2564 flags.go:64] FLAG: --log-text-split-stream="false" Jul 07 22:03:40.462704 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457117 2564 flags.go:64] FLAG: --logging-format="text" Jul 07 22:03:40.462704 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457119 2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 07 22:03:40.462704 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457123 2564 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 07 22:03:40.462704 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457125 2564 flags.go:64] FLAG: --manifest-url="" Jul 07 22:03:40.462704 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457128 2564 flags.go:64] FLAG: --manifest-url-header="" Jul 07 22:03:40.462704 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457132 2564 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 07 22:03:40.462704 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457135 2564 flags.go:64] FLAG: --max-open-files="1000000" Jul 07 22:03:40.462704 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457139 2564 flags.go:64] FLAG: --max-pods="110" Jul 07 22:03:40.462704 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457142 2564 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 07 22:03:40.462704 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457146 2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 07 22:03:40.462704 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457149 2564 flags.go:64] FLAG: --memory-manager-policy="None" Jul 07 22:03:40.462704 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457152 2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 07 22:03:40.462704 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457155 2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 07 22:03:40.462704 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457158 2564 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 07 22:03:40.462704 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457160 2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 07 22:03:40.463307 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457168 2564 flags.go:64] FLAG: --node-status-max-images="50" Jul 07 22:03:40.463307 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457171 2564 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 07 22:03:40.463307 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457174 2564 flags.go:64] FLAG: --oom-score-adj="-999" Jul 07 22:03:40.463307 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457177 2564 flags.go:64] FLAG: --pod-cidr="" Jul 07 22:03:40.463307 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457180 2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 07 22:03:40.463307 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457186 2564 flags.go:64] FLAG: --pod-manifest-path="" Jul 07 22:03:40.463307 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457189 2564 flags.go:64] FLAG: --pod-max-pids="-1" Jul 07 22:03:40.463307 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457192 2564 flags.go:64] FLAG: --pods-per-core="0" Jul 07 22:03:40.463307 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457194 2564 flags.go:64] FLAG: --port="10250" Jul 07 22:03:40.463307 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457197 2564 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 07 22:03:40.463307 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457200 2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01f3b8df8884dab34" Jul 07 22:03:40.463307 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457203 2564 flags.go:64] FLAG: --qos-reserved="" Jul 07 22:03:40.463307 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457206 2564 flags.go:64] FLAG: --read-only-port="10255" Jul 07 22:03:40.463307 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457209 2564 flags.go:64] FLAG: --register-node="true" Jul 07 22:03:40.463307 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457211 2564 flags.go:64] FLAG: --register-with-taints="" Jul 07 22:03:40.463307 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457216 2564 flags.go:64] FLAG: --registry-burst="10" Jul 07 22:03:40.463307 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457220 2564 flags.go:64] FLAG: --registry-qps="5" Jul 07 22:03:40.463307 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457223 2564 flags.go:64] FLAG: --reserved-cpus="" Jul 07 22:03:40.463307 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457226 2564 flags.go:64] FLAG: --reserved-memory="" Jul 07 22:03:40.463307 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457229 2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 07 22:03:40.463307 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457232 2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 07 22:03:40.463307 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457235 2564 flags.go:64] FLAG: --rotate-certificates="false" Jul 07 22:03:40.463307 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457237 2564 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 07 22:03:40.463307 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457240 2564 flags.go:64] FLAG: --runonce="false" Jul 07 22:03:40.463307 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457243 2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 07 22:03:40.463940 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457246 2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 07 22:03:40.463940 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457250 2564 flags.go:64] FLAG: --seccomp-default="false" Jul 07 22:03:40.463940 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457253 2564 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 07 22:03:40.463940 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457255 2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 07 22:03:40.463940 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457258 2564 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 07 22:03:40.463940 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457261 2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 07 22:03:40.463940 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457264 2564 flags.go:64] FLAG: --storage-driver-password="root" Jul 07 22:03:40.463940 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457266 2564 flags.go:64] FLAG: --storage-driver-secure="false" Jul 07 22:03:40.463940 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457269 2564 flags.go:64] FLAG: --storage-driver-table="stats" Jul 07 22:03:40.463940 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457271 2564 flags.go:64] FLAG: --storage-driver-user="root" Jul 07 22:03:40.463940 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457274 2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 07 22:03:40.463940 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457277 2564 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 07 22:03:40.463940 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457280 2564 flags.go:64] FLAG: --system-cgroups="" Jul 07 22:03:40.463940 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457283 2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 07 22:03:40.463940 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457288 2564 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 07 22:03:40.463940 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457290 2564 flags.go:64] FLAG: --tls-cert-file="" Jul 07 22:03:40.463940 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457293 2564 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 07 22:03:40.463940 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457297 2564 flags.go:64] FLAG: --tls-min-version="" Jul 07 22:03:40.463940 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457299 2564 flags.go:64] FLAG: --tls-private-key-file="" Jul 07 22:03:40.463940 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457302 2564 flags.go:64] FLAG: --topology-manager-policy="none" Jul 07 22:03:40.463940 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457304 2564 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 07 22:03:40.463940 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457307 2564 flags.go:64] FLAG: --topology-manager-scope="container" Jul 07 22:03:40.463940 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457311 2564 flags.go:64] FLAG: --v="2" Jul 07 22:03:40.463940 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457315 2564 flags.go:64] FLAG: --version="false" Jul 07 22:03:40.463940 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457319 2564 flags.go:64] FLAG: --vmodule="" Jul 07 22:03:40.464620 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457323 2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 07 22:03:40.464620 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.457326 2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 07 22:03:40.464620 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457462 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 22:03:40.464620 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457468 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 22:03:40.464620 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457471 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 22:03:40.464620 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457474 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 22:03:40.464620 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457476 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 22:03:40.464620 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457479 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 22:03:40.464620 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457485 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 22:03:40.464620 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457487 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 22:03:40.464620 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457490 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 22:03:40.464620 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457492 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 22:03:40.464620 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457495 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 22:03:40.464620 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457497 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 22:03:40.464620 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457500 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 22:03:40.464620 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457503 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 22:03:40.464620 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457505 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 07 22:03:40.464620 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457508 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 22:03:40.464620 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457510 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 22:03:40.464620 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457512 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 22:03:40.465109 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457515 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 22:03:40.465109 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457517 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 22:03:40.465109 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457520 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 22:03:40.465109 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457522 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 22:03:40.465109 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457525 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 22:03:40.465109 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457527 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 22:03:40.465109 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457529 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 22:03:40.465109 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457532 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 22:03:40.465109 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457535 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 22:03:40.465109 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457538 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 22:03:40.465109 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457541 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 22:03:40.465109 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457544 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 22:03:40.465109 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457547 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 22:03:40.465109 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457549 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 22:03:40.465109 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457552 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 22:03:40.465109 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457554 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 22:03:40.465109 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457557 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 22:03:40.465109 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457559 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 22:03:40.465109 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457562 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 22:03:40.465109 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457564 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 22:03:40.465649 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457568 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 22:03:40.465649 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457572 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 22:03:40.465649 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457575 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 22:03:40.465649 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457578 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 22:03:40.465649 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457581 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 22:03:40.465649 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457584 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 22:03:40.465649 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457587 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 22:03:40.465649 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457589 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 22:03:40.465649 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457592 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 22:03:40.465649 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457594 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 22:03:40.465649 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457597 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 22:03:40.465649 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457599 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 22:03:40.465649 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457601 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 22:03:40.465649 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457605 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 22:03:40.465649 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457609 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 22:03:40.465649 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457611 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 22:03:40.465649 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457614 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 22:03:40.465649 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457616 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 22:03:40.465649 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457619 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 22:03:40.466120 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457621 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 22:03:40.466120 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457624 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 22:03:40.466120 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457629 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 22:03:40.466120 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457633 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 22:03:40.466120 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457636 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 22:03:40.466120 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457639 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 22:03:40.466120 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457642 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 22:03:40.466120 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457645 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 22:03:40.466120 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457648 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 22:03:40.466120 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457651 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 22:03:40.466120 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457653 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 22:03:40.466120 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457656 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 22:03:40.466120 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457659 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 22:03:40.466120 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457663 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 22:03:40.466120 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457666 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 22:03:40.466120 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457669 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 22:03:40.466120 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457671 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 22:03:40.466120 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457674 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 22:03:40.466120 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457677 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 22:03:40.466120 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457679 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 22:03:40.466629 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457682 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 22:03:40.466629 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457684 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 22:03:40.466629 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457687 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 22:03:40.466629 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457689 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 22:03:40.466629 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457691 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 22:03:40.466629 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457694 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 22:03:40.466629 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457696 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 22:03:40.466629 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457699 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 22:03:40.466629 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457701 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 22:03:40.466629 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457704 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 22:03:40.466629 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457706 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 22:03:40.466629 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457709 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 22:03:40.466629 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457711 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 22:03:40.466629 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457713 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 22:03:40.466629 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457717 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 22:03:40.466629 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457720 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 22:03:40.466629 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457722 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 22:03:40.466629 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457724 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 22:03:40.466629 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457727 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 22:03:40.466629 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457733 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 22:03:40.467097 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457735 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 22:03:40.467097 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.457738 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 22:03:40.467097 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.458461 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 22:03:40.467097 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.465802 2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 07 22:03:40.467097 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.465817 2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 07 22:03:40.467097 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.465831 2564 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 07 22:03:40.467097 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.465837 2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 07 22:03:40.467097 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.465907 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 22:03:40.467097 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.465912 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 22:03:40.467097 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.465915 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 22:03:40.467097 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.465918 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 22:03:40.467097 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.465921 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 22:03:40.467097 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.465924 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 22:03:40.467097 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.465927 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 22:03:40.467540 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.465929 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 22:03:40.467540 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.465932 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 22:03:40.467540 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.465935 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 22:03:40.467540 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.465937 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 22:03:40.467540 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.465940 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 22:03:40.467540 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.465942 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 22:03:40.467540 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.465945 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 22:03:40.467540 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.465947 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 22:03:40.467540 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.465950 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 22:03:40.467540 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.465952 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 22:03:40.467540 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.465955 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 22:03:40.467540 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.465958 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 22:03:40.467540 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.465960 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 22:03:40.467540 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.465963 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 22:03:40.467540 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.465966 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 22:03:40.467540 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.465968 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 22:03:40.467540 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.465971 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 22:03:40.467540 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.465973 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 22:03:40.467540 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.465976 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 22:03:40.467540 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.465979 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 22:03:40.467540 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.465981 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 22:03:40.468045 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.465983 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 22:03:40.468045 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.465987 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 22:03:40.468045 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.465990 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 22:03:40.468045 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.465992 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 22:03:40.468045 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.465995 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 22:03:40.468045 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.465997 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 22:03:40.468045 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466000 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 22:03:40.468045 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466002 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 22:03:40.468045 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466005 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 22:03:40.468045 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466007 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 22:03:40.468045 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466009 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 22:03:40.468045 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466012 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 22:03:40.468045 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466016 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 22:03:40.468045 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466020 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 22:03:40.468045 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466022 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 22:03:40.468045 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466025 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 22:03:40.468045 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466036 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 22:03:40.468045 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466039 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 22:03:40.468045 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466041 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 07 22:03:40.468521 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466044 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 22:03:40.468521 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466047 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 22:03:40.468521 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466049 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 22:03:40.468521 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466052 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 22:03:40.468521 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466054 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 22:03:40.468521 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466057 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 22:03:40.468521 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466059 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 22:03:40.468521 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466062 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 22:03:40.468521 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466064 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 22:03:40.468521 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466066 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 22:03:40.468521 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466069 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 22:03:40.468521 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466071 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 22:03:40.468521 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466074 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 22:03:40.468521 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466076 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 22:03:40.468521 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466079 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 22:03:40.468521 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466083 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 22:03:40.468521 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466085 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 22:03:40.468521 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466088 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 22:03:40.468521 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466091 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 22:03:40.468521 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466093 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 22:03:40.469002 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466096 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 22:03:40.469002 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466098 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 22:03:40.469002 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466100 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 22:03:40.469002 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466104 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 22:03:40.469002 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466107 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 22:03:40.469002 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466110 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 22:03:40.469002 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466113 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 22:03:40.469002 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466115 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 22:03:40.469002 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466118 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 22:03:40.469002 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466120 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 22:03:40.469002 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466122 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 22:03:40.469002 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466125 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 22:03:40.469002 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466128 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 22:03:40.469002 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466130 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 22:03:40.469002 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466133 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 22:03:40.469002 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466136 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 22:03:40.469002 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466139 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 22:03:40.469002 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466142 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 22:03:40.469002 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466144 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 22:03:40.469498 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466147 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 22:03:40.469498 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466149 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 22:03:40.469498 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466152 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 22:03:40.469498 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466154 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 22:03:40.469498 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466156 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 22:03:40.469498 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466160 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 22:03:40.469498 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466164 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 22:03:40.469498 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466167 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 22:03:40.469498 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466169 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 22:03:40.469498 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466177 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 22:03:40.469498 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466180 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 22:03:40.469498 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466182 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 22:03:40.469498 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466185 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 22:03:40.469498 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.466190 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 22:03:40.469844 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466310 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 22:03:40.469844 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466315 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 22:03:40.469844 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466318 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 22:03:40.469844 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466320 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 22:03:40.469844 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466324 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 22:03:40.469844 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466326 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 22:03:40.469844 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466329 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 22:03:40.469844 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466331 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 22:03:40.469844 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466334 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 22:03:40.469844 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466336 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 22:03:40.469844 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466339 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 22:03:40.469844 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466341 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 22:03:40.469844 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466344 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 22:03:40.469844 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466346 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 22:03:40.469844 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466349 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 22:03:40.469844 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466351 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 22:03:40.469844 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466354 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 22:03:40.469844 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466362 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 22:03:40.469844 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466365 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 22:03:40.469844 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466367 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 22:03:40.470341 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466370 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 22:03:40.470341 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466372 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 22:03:40.470341 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466375 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 22:03:40.470341 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466378 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 22:03:40.470341 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466381 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 22:03:40.470341 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466383 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 22:03:40.470341 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466386 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 22:03:40.470341 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466388 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 22:03:40.470341 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466391 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 22:03:40.470341 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466394 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 22:03:40.470341 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466397 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 22:03:40.470341 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466399 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 22:03:40.470341 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466401 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 22:03:40.470341 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466403 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 22:03:40.470341 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466406 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 22:03:40.470341 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466408 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 22:03:40.470341 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466411 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 22:03:40.470341 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466413 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 22:03:40.470341 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466416 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 22:03:40.470911 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466418 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 22:03:40.470911 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466420 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 22:03:40.470911 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466423 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 22:03:40.470911 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466425 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 22:03:40.470911 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466427 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 22:03:40.470911 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466431 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 22:03:40.470911 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466434 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 22:03:40.470911 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466437 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 22:03:40.470911 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466457 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 22:03:40.470911 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466462 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 22:03:40.470911 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466466 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 22:03:40.470911 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466469 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 22:03:40.470911 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466472 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 22:03:40.470911 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466474 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 22:03:40.470911 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466477 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 22:03:40.470911 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466479 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 22:03:40.470911 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466481 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 22:03:40.470911 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466484 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 22:03:40.470911 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466486 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 22:03:40.471361 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466489 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 22:03:40.471361 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466491 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 22:03:40.471361 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466494 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 22:03:40.471361 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466497 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 22:03:40.471361 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466499 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 22:03:40.471361 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466501 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 22:03:40.471361 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466504 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 22:03:40.471361 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466506 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 22:03:40.471361 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466509 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 22:03:40.471361 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466511 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 22:03:40.471361 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466514 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 22:03:40.471361 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466516 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 22:03:40.471361 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466519 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 22:03:40.471361 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466521 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 22:03:40.471361 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466523 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 22:03:40.471361 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466526 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 22:03:40.471361 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466528 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 22:03:40.471361 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466531 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 22:03:40.471361 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466533 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 22:03:40.471361 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466535 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 22:03:40.471882 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466538 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 22:03:40.471882 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466540 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 22:03:40.471882 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466543 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 22:03:40.471882 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466545 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 07 22:03:40.471882 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466548 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 22:03:40.471882 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466550 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 22:03:40.471882 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466552 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 22:03:40.471882 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466555 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 22:03:40.471882 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466557 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 22:03:40.471882 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466559 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 22:03:40.471882 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466562 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 22:03:40.471882 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466564 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 22:03:40.471882 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466567 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 22:03:40.471882 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466570 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 22:03:40.471882 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466572 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 22:03:40.471882 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466574 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 22:03:40.471882 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466577 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 22:03:40.471882 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466580 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 22:03:40.471882 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466582 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 22:03:40.471882 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466585 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 22:03:40.472393 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:40.466587 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 22:03:40.472393 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.466592 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 22:03:40.472393 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.466691 2564 server.go:962] "Client rotation is on, will bootstrap in background" Jul 07 22:03:40.472393 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.469094 2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 07 22:03:40.472393 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.470612 2564 server.go:1019] "Starting client certificate rotation" Jul 07 22:03:40.472393 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.470717 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 07 22:03:40.472393 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.470770 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 07 22:03:40.473181 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.473164 2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 07 22:03:40.474325 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.474304 2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 07 22:03:40.483306 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.483289 2564 log.go:25] "Validated CRI v1 runtime API" Jul 07 22:03:40.486627 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.486608 2564 log.go:25] "Validated CRI v1 image API" Jul 07 22:03:40.487410 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.487396 2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 07 22:03:40.489513 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.489495 2564 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 a71c335a-a020-46f0-b148-fc6c88a0a62a:/dev/nvme0n1p4 db932df7-8652-40cb-a3d8-0516390448b9:/dev/nvme0n1p3] Jul 07 22:03:40.489566 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.489513 2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 07 22:03:40.496453 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.496311 2564 manager.go:216] Machine: {Timestamp:2026-07-07 22:03:40.494885123 +0000 UTC m=+0.323014572 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3089099 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22cbc6b21ff63b49b8f1343cf98faa SystemUUID:ec22cbc6-b21f-f63b-49b8-f1343cf98faa BootID:ed29e1f7-5828-4176-be16-b20f7080bea0 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:f3:69:f4:98:2b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:f3:69:f4:98:2b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6e:b1:50:e3:5a:11 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 07 22:03:40.496524 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.496454 2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 07 22:03:40.496559 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.496533 2564 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 07 22:03:40.498495 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.498477 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 07 22:03:40.499969 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.499941 2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 07 22:03:40.500095 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.499972 2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-156.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 07 22:03:40.500794 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.500782 2564 topology_manager.go:138] "Creating topology manager with none policy" Jul 07 22:03:40.500835 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.500798 2564 container_manager_linux.go:309] "Creating device plugin manager" Jul 07 22:03:40.500835 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.500805 2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 07 22:03:40.501583 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.501564 2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 07 22:03:40.503389 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.503379 2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 07 22:03:40.507790 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.507778 2564 state_mem.go:36] "Initialized new in-memory state store" Jul 07 22:03:40.508065 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.508054 2564 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 07 22:03:40.508105 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.508086 2564 kubelet.go:486] "Attempting to sync node with API server" Jul 07 22:03:40.508105 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.508097 2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 07 22:03:40.508180 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.508110 2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 07 22:03:40.508180 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.508119 2564 kubelet.go:398] "Adding apiserver pod source" Jul 07 22:03:40.508180 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.508127 2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 07 22:03:40.508467 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.508438 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 07 22:03:40.508511 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.508476 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 07 22:03:40.510526 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.510489 2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 07 22:03:40.512803 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.512790 2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 07 22:03:40.512852 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.512820 2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 07 22:03:40.513876 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.513859 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 07 22:03:40.513876 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.513879 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 07 22:03:40.513998 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.513888 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 07 22:03:40.513998 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.513894 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 07 22:03:40.513998 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.513903 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 07 22:03:40.513998 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.513911 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 07 22:03:40.513998 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.513924 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 07 22:03:40.513998 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.513930 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 07 22:03:40.513998 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.513937 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 07 22:03:40.513998 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.513942 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 07 22:03:40.513998 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.513951 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 07 22:03:40.513998 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.513956 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 07 22:03:40.513998 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.513981 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 07 22:03:40.513998 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.513991 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 07 22:03:40.517467 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.517438 2564 server.go:1268] "Started kubelet" Jul 07 22:03:40.517591 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.517564 2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 07 22:03:40.517821 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.517804 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9s5tt" Jul 07 22:03:40.517956 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.517910 2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 07 22:03:40.518037 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.517975 2564 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 07 22:03:40.519329 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.519297 2564 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-156.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 07 22:03:40.519414 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:40.519369 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-156.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 07 22:03:40.519493 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:40.519477 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 07 22:03:40.519814 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.519796 2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 07 22:03:40.520121 ip-10-0-134-156 systemd[1]: Started Kubernetes Kubelet. Jul 07 22:03:40.521254 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.521238 2564 server.go:310] "Adding debug handlers to kubelet server" Jul 07 22:03:40.524335 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.524314 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9s5tt" Jul 07 22:03:40.524852 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:40.524828 2564 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 07 22:03:40.525943 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:40.523116 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-156.ec2.internal.18c021139a695e6b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-156.ec2.internal,UID:ip-10-0-134-156.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-156.ec2.internal,},FirstTimestamp:2026-07-07 22:03:40.517408363 +0000 UTC m=+0.345537812,LastTimestamp:2026-07-07 22:03:40.517408363 +0000 UTC m=+0.345537812,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-156.ec2.internal,}" Jul 07 22:03:40.529127 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.529114 2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 07 22:03:40.529230 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.529116 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 07 22:03:40.529287 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:40.529244 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-156.ec2.internal\" not found" Jul 07 22:03:40.529927 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.529907 2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 07 22:03:40.529927 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.529908 2564 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 07 22:03:40.530082 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.529936 2564 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 07 22:03:40.530082 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.529985 2564 factory.go:55] Registering systemd factory Jul 07 22:03:40.530082 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.529998 2564 factory.go:223] Registration of the systemd container factory successfully Jul 07 22:03:40.530082 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.530037 2564 reconstruct.go:97] "Volume reconstruction finished" Jul 07 22:03:40.530082 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.530047 2564 reconciler.go:29] "Reconciler: start to sync state" Jul 07 22:03:40.530269 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.530240 2564 factory.go:153] Registering CRI-O factory Jul 07 22:03:40.530269 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.530254 2564 factory.go:223] Registration of the crio container factory successfully Jul 07 22:03:40.530353 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.530306 2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 07 22:03:40.530353 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.530329 2564 factory.go:103] Registering Raw factory Jul 07 22:03:40.530353 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.530344 2564 manager.go:1196] Started watching for new ooms in manager Jul 07 22:03:40.530957 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.530795 2564 manager.go:349] Starting recovery of all containers Jul 07 22:03:40.539096 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.538904 2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 22:03:40.539169 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.538930 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 07 22:03:40.541678 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:40.541638 2564 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-134-156.ec2.internal\" not found" node="ip-10-0-134-156.ec2.internal" Jul 07 22:03:40.541824 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.541737 2564 manager.go:354] Recovery completed Jul 07 22:03:40.545899 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.545886 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 22:03:40.548828 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.548809 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasSufficientMemory" Jul 07 22:03:40.548891 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.548848 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 22:03:40.548891 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.548862 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasSufficientPID" Jul 07 22:03:40.549755 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.549738 2564 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 07 22:03:40.549755 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.549749 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 22:03:40.549755 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.549755 2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 07 22:03:40.549910 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.549776 2564 state_mem.go:36] "Initialized new in-memory state store" Jul 07 22:03:40.550459 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.550431 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasSufficientMemory" Jul 07 22:03:40.550506 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.550471 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 22:03:40.550506 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.550484 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasSufficientPID" Jul 07 22:03:40.551265 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.551252 2564 policy_none.go:49] "None policy: Start" Jul 07 22:03:40.551324 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.551272 2564 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 07 22:03:40.551324 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.551285 2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 07 22:03:40.552364 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.552352 2564 policy_none.go:47] "Start" Jul 07 22:03:40.586053 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.586020 2564 manager.go:337] "Starting Device Plugin manager" Jul 07 22:03:40.586053 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:40.586053 2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 07 22:03:40.586227 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.586062 2564 server.go:87] "Starting device plugin registration server" Jul 07 22:03:40.586286 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.586275 2564 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 07 22:03:40.586318 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.586287 2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 07 22:03:40.587156 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.587136 2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 07 22:03:40.587254 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.587224 2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 07 22:03:40.587254 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.587234 2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 07 22:03:40.588128 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:40.588102 2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 07 22:03:40.588205 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:40.588173 2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-156.ec2.internal\" not found" Jul 07 22:03:40.686947 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.686916 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 22:03:40.688264 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.687967 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasSufficientMemory" Jul 07 22:03:40.688264 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.688002 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 22:03:40.688264 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.688017 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasSufficientPID" Jul 07 22:03:40.688264 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.688051 2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-156.ec2.internal" Jul 07 22:03:40.695958 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.695935 2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-156.ec2.internal" Jul 07 22:03:40.696051 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:40.695960 2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-156.ec2.internal\": node \"ip-10-0-134-156.ec2.internal\" not found" Jul 07 22:03:40.701101 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.701042 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 07 22:03:40.701101 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.701087 2564 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 07 22:03:40.701248 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.701107 2564 kubelet.go:2448] "Starting kubelet main sync loop" Jul 07 22:03:40.701248 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:40.701137 2564 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 07 22:03:40.703204 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.703183 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 22:03:40.709007 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:40.708983 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-156.ec2.internal\" not found" Jul 07 22:03:40.801438 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.801402 2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-156.ec2.internal"] Jul 07 22:03:40.801600 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.801516 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 22:03:40.803499 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.803483 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasSufficientMemory" Jul 07 22:03:40.803499 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.803511 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 22:03:40.803632 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.803520 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasSufficientPID" Jul 07 22:03:40.804954 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.804941 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 22:03:40.805081 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.805067 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal" Jul 07 22:03:40.805141 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.805104 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 22:03:40.805707 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.805690 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasSufficientMemory" Jul 07 22:03:40.805795 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.805692 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasSufficientMemory" Jul 07 22:03:40.805795 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.805746 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 22:03:40.805795 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.805758 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasSufficientPID" Jul 07 22:03:40.805795 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.805724 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 22:03:40.805795 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.805790 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasSufficientPID" Jul 07 22:03:40.806992 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.806979 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-156.ec2.internal" Jul 07 22:03:40.807044 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.807007 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 22:03:40.807747 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.807718 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasSufficientMemory" Jul 07 22:03:40.807747 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.807747 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 22:03:40.807895 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.807760 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeHasSufficientPID" Jul 07 22:03:40.809063 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:40.809050 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-156.ec2.internal\" not found" Jul 07 22:03:40.831930 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.831903 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7a6e89d5fb8cfce240a45d8bfd07e986-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal\" (UID: \"7a6e89d5fb8cfce240a45d8bfd07e986\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal" Jul 07 22:03:40.831930 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.831928 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a6e89d5fb8cfce240a45d8bfd07e986-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal\" (UID: \"7a6e89d5fb8cfce240a45d8bfd07e986\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal" Jul 07 22:03:40.832064 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:40.831945 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-156.ec2.internal\" not found" node="ip-10-0-134-156.ec2.internal" Jul 07 22:03:40.832064 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.831946 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7e758ef0bacc8b47cfb5ed8977696fa1-config\") pod \"kube-apiserver-proxy-ip-10-0-134-156.ec2.internal\" (UID: \"7e758ef0bacc8b47cfb5ed8977696fa1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-156.ec2.internal" Jul 07 22:03:40.836373 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:40.836355 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-156.ec2.internal\" not found" node="ip-10-0-134-156.ec2.internal" Jul 07 22:03:40.910018 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:40.909994 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-156.ec2.internal\" not found" Jul 07 22:03:40.932345 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.932325 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7a6e89d5fb8cfce240a45d8bfd07e986-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal\" (UID: \"7a6e89d5fb8cfce240a45d8bfd07e986\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal" Jul 07 22:03:40.932411 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.932341 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7a6e89d5fb8cfce240a45d8bfd07e986-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal\" (UID: \"7a6e89d5fb8cfce240a45d8bfd07e986\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal" Jul 07 22:03:40.932411 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.932362 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a6e89d5fb8cfce240a45d8bfd07e986-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal\" (UID: \"7a6e89d5fb8cfce240a45d8bfd07e986\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal" Jul 07 22:03:40.932411 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.932382 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7e758ef0bacc8b47cfb5ed8977696fa1-config\") pod \"kube-apiserver-proxy-ip-10-0-134-156.ec2.internal\" (UID: \"7e758ef0bacc8b47cfb5ed8977696fa1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-156.ec2.internal" Jul 07 22:03:40.932526 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.932412 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a6e89d5fb8cfce240a45d8bfd07e986-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal\" (UID: \"7a6e89d5fb8cfce240a45d8bfd07e986\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal" Jul 07 22:03:40.932526 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:40.932421 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7e758ef0bacc8b47cfb5ed8977696fa1-config\") pod \"kube-apiserver-proxy-ip-10-0-134-156.ec2.internal\" (UID: \"7e758ef0bacc8b47cfb5ed8977696fa1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-156.ec2.internal" Jul 07 22:03:41.010698 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:41.010639 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-156.ec2.internal\" not found" Jul 07 22:03:41.111191 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:41.111159 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-156.ec2.internal\" not found" Jul 07 22:03:41.136622 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:41.136595 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal" Jul 07 22:03:41.141144 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:41.140972 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-156.ec2.internal" Jul 07 22:03:41.212237 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:41.212208 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-156.ec2.internal\" not found" Jul 07 22:03:41.318663 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:41.318589 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-156.ec2.internal\" not found" Jul 07 22:03:41.424913 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:41.424885 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-156.ec2.internal\" not found" Jul 07 22:03:41.472702 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:41.472681 2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 07 22:03:41.473207 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:41.472831 2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 07 22:03:41.473207 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:41.472851 2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 07 22:03:41.525483 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:41.525457 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-156.ec2.internal\" not found" Jul 07 22:03:41.529130 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:41.529099 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-06 21:58:40 +0000 UTC" deadline="2027-12-22 23:53:44.745927785 +0000 UTC" Jul 07 22:03:41.529130 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:41.529129 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12793h50m3.216802175s" Jul 07 22:03:41.626643 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:41.626574 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-156.ec2.internal\" not found" Jul 07 22:03:41.663147 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:41.663113 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a6e89d5fb8cfce240a45d8bfd07e986.slice/crio-c30805aaea53c8b7fca1ecb5d6196f14fa667d77affb5543b899c49c690f9360 WatchSource:0}: Error finding container c30805aaea53c8b7fca1ecb5d6196f14fa667d77affb5543b899c49c690f9360: Status 404 returned error can't find the container with id c30805aaea53c8b7fca1ecb5d6196f14fa667d77affb5543b899c49c690f9360 Jul 07 22:03:41.663425 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:41.663400 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e758ef0bacc8b47cfb5ed8977696fa1.slice/crio-3c10539603ee380e76b411be6f8c507c23405841661552680e7b12a35db01d51 WatchSource:0}: Error finding container 3c10539603ee380e76b411be6f8c507c23405841661552680e7b12a35db01d51: Status 404 returned error can't find the container with id 3c10539603ee380e76b411be6f8c507c23405841661552680e7b12a35db01d51 Jul 07 22:03:41.669390 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:41.669319 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 22:03:41.704594 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:41.704539 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal" event={"ID":"7a6e89d5fb8cfce240a45d8bfd07e986","Type":"ContainerStarted","Data":"c30805aaea53c8b7fca1ecb5d6196f14fa667d77affb5543b899c49c690f9360"} Jul 07 22:03:41.705604 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:41.705581 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-156.ec2.internal" event={"ID":"7e758ef0bacc8b47cfb5ed8977696fa1","Type":"ContainerStarted","Data":"3c10539603ee380e76b411be6f8c507c23405841661552680e7b12a35db01d51"} Jul 07 22:03:41.728434 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:41.728414 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-156.ec2.internal\" not found" Jul 07 22:03:41.835621 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:41.835586 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-156.ec2.internal\" not found" Jul 07 22:03:41.936632 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:41.936560 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-156.ec2.internal\" not found" Jul 07 22:03:42.004976 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.004946 2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 22:03:42.037537 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:42.037508 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-156.ec2.internal\" not found" Jul 07 22:03:42.057153 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.057129 2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 22:03:42.129961 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.129933 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal" Jul 07 22:03:42.141421 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.141400 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 07 22:03:42.142529 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.142481 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-156.ec2.internal" Jul 07 22:03:42.150245 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.150220 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 07 22:03:42.442404 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.442366 2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 22:03:42.511215 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.511186 2564 apiserver.go:52] "Watching apiserver" Jul 07 22:03:42.518570 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.518357 2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 07 22:03:42.518797 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.518773 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-134-156.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsfv2","openshift-cluster-node-tuning-operator/tuned-wpg82","openshift-multus/multus-bnx62","openshift-ovn-kubernetes/ovnkube-node-4h8zm","kube-system/global-pull-secret-syncer-mzzl4","openshift-dns/node-resolver-2gbtg","openshift-image-registry/node-ca-c9npj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal","openshift-multus/multus-additional-cni-plugins-9z8v8","openshift-multus/network-metrics-daemon-m7lhr","openshift-network-diagnostics/network-check-target-zlr8s","openshift-network-operator/iptables-alerter-llpjl","kube-system/konnectivity-agent-lc54x"] Jul 07 22:03:42.520773 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.520747 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2gbtg" Jul 07 22:03:42.521928 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.521898 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-llpjl" Jul 07 22:03:42.523568 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.523546 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wpg82" Jul 07 22:03:42.524981 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.524964 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kspkn\"" Jul 07 22:03:42.527540 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.525345 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 07 22:03:42.527540 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.525618 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 07 22:03:42.527540 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.525949 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 07 22:03:42.528822 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.528804 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-c9wvd\"" Jul 07 22:03:42.529109 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.529074 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-x7r65\"" Jul 07 22:03:42.529173 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.529148 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 07 22:03:42.529226 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.529019 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 07 22:03:42.529279 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.528850 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 07 22:03:42.529359 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.528850 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 07 22:03:42.529895 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.529876 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bnx62" Jul 07 22:03:42.530438 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.530423 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 07 22:03:42.532554 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.531908 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:03:42.532554 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.532446 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 07 22:03:42.532723 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.532595 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-nvcvm\"" Jul 07 22:03:42.532723 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.532685 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 07 22:03:42.532723 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.532714 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 07 22:03:42.532893 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.532875 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 07 22:03:42.534004 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.533982 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mzzl4" Jul 07 22:03:42.534106 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:42.534069 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mzzl4" podUID="acb69b34-f020-4c76-89c2-0ab5b6e6f3d5" Jul 07 22:03:42.534226 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.534200 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 07 22:03:42.534658 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.534517 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mzcd6\"" Jul 07 22:03:42.534658 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.534542 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 07 22:03:42.535610 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.534931 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 07 22:03:42.535610 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.535245 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 07 22:03:42.535610 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.534938 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 07 22:03:42.535610 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.535431 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 07 22:03:42.537797 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.537777 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lc54x" Jul 07 22:03:42.539646 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.539625 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-c9npj" Jul 07 22:03:42.540209 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.540187 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 07 22:03:42.540301 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.540212 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-647l9\"" Jul 07 22:03:42.540486 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.540469 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 07 22:03:42.541328 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.541307 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1d2e51fd-592d-4fef-a8e0-5b70ae836780-etc-sysctl-d\") pod \"tuned-wpg82\" (UID: \"1d2e51fd-592d-4fef-a8e0-5b70ae836780\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg82" Jul 07 22:03:42.541423 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.541336 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1d2e51fd-592d-4fef-a8e0-5b70ae836780-etc-tuned\") pod \"tuned-wpg82\" (UID: \"1d2e51fd-592d-4fef-a8e0-5b70ae836780\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg82" Jul 07 22:03:42.541423 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.541357 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6f9d7564-2f1c-4d78-863f-b2da1db479cf-multus-cni-dir\") pod \"multus-bnx62\" (UID: \"6f9d7564-2f1c-4d78-863f-b2da1db479cf\") " pod="openshift-multus/multus-bnx62" Jul 07 22:03:42.541423 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.541382 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6f9d7564-2f1c-4d78-863f-b2da1db479cf-multus-socket-dir-parent\") pod \"multus-bnx62\" (UID: \"6f9d7564-2f1c-4d78-863f-b2da1db479cf\") " pod="openshift-multus/multus-bnx62" Jul 07 22:03:42.541423 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.541405 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6f9d7564-2f1c-4d78-863f-b2da1db479cf-host-var-lib-cni-bin\") pod \"multus-bnx62\" (UID: \"6f9d7564-2f1c-4d78-863f-b2da1db479cf\") " pod="openshift-multus/multus-bnx62" Jul 07 22:03:42.541423 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.541421 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6f9d7564-2f1c-4d78-863f-b2da1db479cf-cnibin\") pod \"multus-bnx62\" (UID: \"6f9d7564-2f1c-4d78-863f-b2da1db479cf\") " pod="openshift-multus/multus-bnx62" Jul 07 22:03:42.541423 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.541435 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6f9d7564-2f1c-4d78-863f-b2da1db479cf-os-release\") pod \"multus-bnx62\" (UID: \"6f9d7564-2f1c-4d78-863f-b2da1db479cf\") " pod="openshift-multus/multus-bnx62" Jul 07 22:03:42.541839 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.541449 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6f9d7564-2f1c-4d78-863f-b2da1db479cf-cni-binary-copy\") pod \"multus-bnx62\" (UID: \"6f9d7564-2f1c-4d78-863f-b2da1db479cf\") " pod="openshift-multus/multus-bnx62" Jul 07 22:03:42.541839 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.541444 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsfv2" Jul 07 22:03:42.541839 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.541467 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6f9d7564-2f1c-4d78-863f-b2da1db479cf-host-var-lib-cni-multus\") pod \"multus-bnx62\" (UID: \"6f9d7564-2f1c-4d78-863f-b2da1db479cf\") " pod="openshift-multus/multus-bnx62" Jul 07 22:03:42.541839 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.541705 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4b4234f5-6cda-45ea-b559-f09bcad77232-run-openvswitch\") pod \"ovnkube-node-4h8zm\" (UID: \"4b4234f5-6cda-45ea-b559-f09bcad77232\") " pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:03:42.541839 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.541720 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4b4234f5-6cda-45ea-b559-f09bcad77232-node-log\") pod \"ovnkube-node-4h8zm\" (UID: \"4b4234f5-6cda-45ea-b559-f09bcad77232\") " pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:03:42.541839 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.541735 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t24rv\" (UniqueName: \"kubernetes.io/projected/4b4234f5-6cda-45ea-b559-f09bcad77232-kube-api-access-t24rv\") pod \"ovnkube-node-4h8zm\" (UID: \"4b4234f5-6cda-45ea-b559-f09bcad77232\") " pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:03:42.541839 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.541750 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mpn8\" (UniqueName: \"kubernetes.io/projected/b22ba61c-8e69-4913-acd7-b58c7a92c8d4-kube-api-access-7mpn8\") pod \"node-resolver-2gbtg\" (UID: \"b22ba61c-8e69-4913-acd7-b58c7a92c8d4\") " pod="openshift-dns/node-resolver-2gbtg" Jul 07 22:03:42.541839 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.541764 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1d2e51fd-592d-4fef-a8e0-5b70ae836780-etc-sysconfig\") pod \"tuned-wpg82\" (UID: \"1d2e51fd-592d-4fef-a8e0-5b70ae836780\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg82" Jul 07 22:03:42.541839 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.541780 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d2e51fd-592d-4fef-a8e0-5b70ae836780-etc-kubernetes\") pod \"tuned-wpg82\" (UID: \"1d2e51fd-592d-4fef-a8e0-5b70ae836780\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg82" Jul 07 22:03:42.541839 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.541793 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1d2e51fd-592d-4fef-a8e0-5b70ae836780-run\") pod \"tuned-wpg82\" (UID: \"1d2e51fd-592d-4fef-a8e0-5b70ae836780\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg82" Jul 07 22:03:42.541839 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.541834 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4b4234f5-6cda-45ea-b559-f09bcad77232-var-lib-openvswitch\") pod \"ovnkube-node-4h8zm\" (UID: \"4b4234f5-6cda-45ea-b559-f09bcad77232\") " pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:03:42.541839 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.541856 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d2e51fd-592d-4fef-a8e0-5b70ae836780-tmp\") pod \"tuned-wpg82\" (UID: \"1d2e51fd-592d-4fef-a8e0-5b70ae836780\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg82" Jul 07 22:03:42.542556 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.541879 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4b4234f5-6cda-45ea-b559-f09bcad77232-host-cni-netd\") pod \"ovnkube-node-4h8zm\" (UID: \"4b4234f5-6cda-45ea-b559-f09bcad77232\") " pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:03:42.542556 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.541920 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4b4234f5-6cda-45ea-b559-f09bcad77232-ovnkube-config\") pod \"ovnkube-node-4h8zm\" (UID: \"4b4234f5-6cda-45ea-b559-f09bcad77232\") " pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:03:42.542556 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.541947 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1a20bbf4-728c-4d13-b2a1-846fdf8c5a2e-host-slash\") pod \"iptables-alerter-llpjl\" (UID: \"1a20bbf4-728c-4d13-b2a1-846fdf8c5a2e\") " pod="openshift-network-operator/iptables-alerter-llpjl" Jul 07 22:03:42.542556 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.541971 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6f9d7564-2f1c-4d78-863f-b2da1db479cf-host-run-k8s-cni-cncf-io\") pod \"multus-bnx62\" (UID: \"6f9d7564-2f1c-4d78-863f-b2da1db479cf\") " pod="openshift-multus/multus-bnx62" Jul 07 22:03:42.542556 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.541989 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f9d7564-2f1c-4d78-863f-b2da1db479cf-host-var-lib-kubelet\") pod \"multus-bnx62\" (UID: \"6f9d7564-2f1c-4d78-863f-b2da1db479cf\") " pod="openshift-multus/multus-bnx62" Jul 07 22:03:42.542556 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.542030 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f9d7564-2f1c-4d78-863f-b2da1db479cf-etc-kubernetes\") pod \"multus-bnx62\" (UID: \"6f9d7564-2f1c-4d78-863f-b2da1db479cf\") " pod="openshift-multus/multus-bnx62" Jul 07 22:03:42.542556 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.542057 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4b4234f5-6cda-45ea-b559-f09bcad77232-ovn-node-metrics-cert\") pod \"ovnkube-node-4h8zm\" (UID: \"4b4234f5-6cda-45ea-b559-f09bcad77232\") " pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:03:42.542556 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.542073 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b22ba61c-8e69-4913-acd7-b58c7a92c8d4-hosts-file\") pod \"node-resolver-2gbtg\" (UID: \"b22ba61c-8e69-4913-acd7-b58c7a92c8d4\") " pod="openshift-dns/node-resolver-2gbtg" Jul 07 22:03:42.542556 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.542092 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6f9d7564-2f1c-4d78-863f-b2da1db479cf-host-run-multus-certs\") pod \"multus-bnx62\" (UID: \"6f9d7564-2f1c-4d78-863f-b2da1db479cf\") " pod="openshift-multus/multus-bnx62" Jul 07 22:03:42.542556 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.542117 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4b4234f5-6cda-45ea-b559-f09bcad77232-etc-openvswitch\") pod \"ovnkube-node-4h8zm\" (UID: \"4b4234f5-6cda-45ea-b559-f09bcad77232\") " pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:03:42.542556 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.542141 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96m5j\" (UniqueName: \"kubernetes.io/projected/1a20bbf4-728c-4d13-b2a1-846fdf8c5a2e-kube-api-access-96m5j\") pod \"iptables-alerter-llpjl\" (UID: \"1a20bbf4-728c-4d13-b2a1-846fdf8c5a2e\") " pod="openshift-network-operator/iptables-alerter-llpjl" Jul 07 22:03:42.542556 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.542183 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1d2e51fd-592d-4fef-a8e0-5b70ae836780-etc-systemd\") pod \"tuned-wpg82\" (UID: \"1d2e51fd-592d-4fef-a8e0-5b70ae836780\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg82" Jul 07 22:03:42.542556 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.542207 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d2e51fd-592d-4fef-a8e0-5b70ae836780-var-lib-kubelet\") pod \"tuned-wpg82\" (UID: \"1d2e51fd-592d-4fef-a8e0-5b70ae836780\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg82" Jul 07 22:03:42.542556 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.542231 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b4234f5-6cda-45ea-b559-f09bcad77232-host-kubelet\") pod \"ovnkube-node-4h8zm\" (UID: \"4b4234f5-6cda-45ea-b559-f09bcad77232\") " pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:03:42.542556 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.542253 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4b4234f5-6cda-45ea-b559-f09bcad77232-log-socket\") pod \"ovnkube-node-4h8zm\" (UID: \"4b4234f5-6cda-45ea-b559-f09bcad77232\") " pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:03:42.542556 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.542275 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4b4234f5-6cda-45ea-b559-f09bcad77232-host-cni-bin\") pod \"ovnkube-node-4h8zm\" (UID: \"4b4234f5-6cda-45ea-b559-f09bcad77232\") " pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:03:42.542556 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.542280 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 07 22:03:42.543604 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.542298 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6f9d7564-2f1c-4d78-863f-b2da1db479cf-multus-conf-dir\") pod \"multus-bnx62\" (UID: \"6f9d7564-2f1c-4d78-863f-b2da1db479cf\") " pod="openshift-multus/multus-bnx62" Jul 07 22:03:42.543604 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.542310 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-r4vcl\"" Jul 07 22:03:42.543604 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.542322 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4b4234f5-6cda-45ea-b559-f09bcad77232-ovnkube-script-lib\") pod \"ovnkube-node-4h8zm\" (UID: \"4b4234f5-6cda-45ea-b559-f09bcad77232\") " pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:03:42.543604 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.542334 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 07 22:03:42.543604 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.542348 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1d2e51fd-592d-4fef-a8e0-5b70ae836780-lib-modules\") pod \"tuned-wpg82\" (UID: \"1d2e51fd-592d-4fef-a8e0-5b70ae836780\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg82" Jul 07 22:03:42.543604 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.542372 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6f9d7564-2f1c-4d78-863f-b2da1db479cf-multus-daemon-config\") pod \"multus-bnx62\" (UID: \"6f9d7564-2f1c-4d78-863f-b2da1db479cf\") " pod="openshift-multus/multus-bnx62" Jul 07 22:03:42.543604 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.542397 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqsg9\" (UniqueName: \"kubernetes.io/projected/6f9d7564-2f1c-4d78-863f-b2da1db479cf-kube-api-access-hqsg9\") pod \"multus-bnx62\" (UID: \"6f9d7564-2f1c-4d78-863f-b2da1db479cf\") " pod="openshift-multus/multus-bnx62" Jul 07 22:03:42.543604 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.542448 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4b4234f5-6cda-45ea-b559-f09bcad77232-run-systemd\") pod \"ovnkube-node-4h8zm\" (UID: \"4b4234f5-6cda-45ea-b559-f09bcad77232\") " pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:03:42.543604 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.542492 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4b4234f5-6cda-45ea-b559-f09bcad77232-env-overrides\") pod \"ovnkube-node-4h8zm\" (UID: \"4b4234f5-6cda-45ea-b559-f09bcad77232\") " pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:03:42.543604 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.542553 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 07 22:03:42.543604 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.542542 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4b4234f5-6cda-45ea-b559-f09bcad77232-host-slash\") pod \"ovnkube-node-4h8zm\" (UID: \"4b4234f5-6cda-45ea-b559-f09bcad77232\") " pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:03:42.543604 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.542604 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4b4234f5-6cda-45ea-b559-f09bcad77232-host-run-netns\") pod \"ovnkube-node-4h8zm\" (UID: \"4b4234f5-6cda-45ea-b559-f09bcad77232\") " pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:03:42.543604 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.542636 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b22ba61c-8e69-4913-acd7-b58c7a92c8d4-tmp-dir\") pod \"node-resolver-2gbtg\" (UID: \"b22ba61c-8e69-4913-acd7-b58c7a92c8d4\") " pod="openshift-dns/node-resolver-2gbtg" Jul 07 22:03:42.543604 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.542677 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b4234f5-6cda-45ea-b559-f09bcad77232-host-run-ovn-kubernetes\") pod \"ovnkube-node-4h8zm\" (UID: \"4b4234f5-6cda-45ea-b559-f09bcad77232\") " pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:03:42.543604 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.542712 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b4234f5-6cda-45ea-b559-f09bcad77232-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4h8zm\" (UID: \"4b4234f5-6cda-45ea-b559-f09bcad77232\") " pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:03:42.543604 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.542736 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1d2e51fd-592d-4fef-a8e0-5b70ae836780-sys\") pod \"tuned-wpg82\" (UID: \"1d2e51fd-592d-4fef-a8e0-5b70ae836780\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg82" Jul 07 22:03:42.543604 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.542778 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6f9d7564-2f1c-4d78-863f-b2da1db479cf-system-cni-dir\") pod \"multus-bnx62\" (UID: \"6f9d7564-2f1c-4d78-863f-b2da1db479cf\") " pod="openshift-multus/multus-bnx62" Jul 07 22:03:42.543604 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.542821 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6f9d7564-2f1c-4d78-863f-b2da1db479cf-host-run-netns\") pod \"multus-bnx62\" (UID: \"6f9d7564-2f1c-4d78-863f-b2da1db479cf\") " pod="openshift-multus/multus-bnx62" Jul 07 22:03:42.544632 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.542848 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4b4234f5-6cda-45ea-b559-f09bcad77232-run-ovn\") pod \"ovnkube-node-4h8zm\" (UID: \"4b4234f5-6cda-45ea-b559-f09bcad77232\") " pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:03:42.544632 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.542892 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1d2e51fd-592d-4fef-a8e0-5b70ae836780-etc-modprobe-d\") pod \"tuned-wpg82\" (UID: \"1d2e51fd-592d-4fef-a8e0-5b70ae836780\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg82" Jul 07 22:03:42.544632 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.542926 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1d2e51fd-592d-4fef-a8e0-5b70ae836780-host\") pod \"tuned-wpg82\" (UID: \"1d2e51fd-592d-4fef-a8e0-5b70ae836780\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg82" Jul 07 22:03:42.544632 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.542958 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4b4234f5-6cda-45ea-b559-f09bcad77232-systemd-units\") pod \"ovnkube-node-4h8zm\" (UID: \"4b4234f5-6cda-45ea-b559-f09bcad77232\") " pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:03:42.544632 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.542977 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1a20bbf4-728c-4d13-b2a1-846fdf8c5a2e-iptables-alerter-script\") pod \"iptables-alerter-llpjl\" (UID: \"1a20bbf4-728c-4d13-b2a1-846fdf8c5a2e\") " pod="openshift-network-operator/iptables-alerter-llpjl" Jul 07 22:03:42.544632 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.543008 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1d2e51fd-592d-4fef-a8e0-5b70ae836780-etc-sysctl-conf\") pod \"tuned-wpg82\" (UID: \"1d2e51fd-592d-4fef-a8e0-5b70ae836780\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg82" Jul 07 22:03:42.544632 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.543025 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wqsp\" (UniqueName: \"kubernetes.io/projected/1d2e51fd-592d-4fef-a8e0-5b70ae836780-kube-api-access-9wqsp\") pod \"tuned-wpg82\" (UID: \"1d2e51fd-592d-4fef-a8e0-5b70ae836780\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg82" Jul 07 22:03:42.544632 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.543061 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6f9d7564-2f1c-4d78-863f-b2da1db479cf-hostroot\") pod \"multus-bnx62\" (UID: \"6f9d7564-2f1c-4d78-863f-b2da1db479cf\") " pod="openshift-multus/multus-bnx62" Jul 07 22:03:42.544632 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.543530 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-ncx4t\"" Jul 07 22:03:42.544632 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.543538 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 07 22:03:42.544632 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.543813 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 07 22:03:42.544632 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.543905 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 07 22:03:42.544632 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.544195 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9z8v8" Jul 07 22:03:42.545422 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.545400 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 07 22:03:42.546503 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.546413 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 07 22:03:42.546503 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.546430 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m7lhr" Jul 07 22:03:42.546686 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:42.546511 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m7lhr" podUID="6aef29a2-20f0-4804-951f-4bd876b4b8b5" Jul 07 22:03:42.546798 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.546749 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-w5wtv\"" Jul 07 22:03:42.547134 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.547115 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 07 22:03:42.548169 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.548148 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zlr8s" Jul 07 22:03:42.548267 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:42.548210 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zlr8s" podUID="95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9" Jul 07 22:03:42.571628 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.571599 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-5z25m" Jul 07 22:03:42.580810 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.580790 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5z25m" Jul 07 22:03:42.589344 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.589314 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 22:03:42.633471 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.633434 2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 07 22:03:42.643865 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.643839 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1d2e51fd-592d-4fef-a8e0-5b70ae836780-etc-modprobe-d\") pod \"tuned-wpg82\" (UID: \"1d2e51fd-592d-4fef-a8e0-5b70ae836780\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg82" Jul 07 22:03:42.644000 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.643875 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1d2e51fd-592d-4fef-a8e0-5b70ae836780-host\") pod \"tuned-wpg82\" (UID: \"1d2e51fd-592d-4fef-a8e0-5b70ae836780\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg82" Jul 07 22:03:42.644000 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.643905 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4b4234f5-6cda-45ea-b559-f09bcad77232-systemd-units\") pod \"ovnkube-node-4h8zm\" (UID: \"4b4234f5-6cda-45ea-b559-f09bcad77232\") " pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:03:42.644000 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.643948 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94ccg\" (UniqueName: \"kubernetes.io/projected/6aef29a2-20f0-4804-951f-4bd876b4b8b5-kube-api-access-94ccg\") pod \"network-metrics-daemon-m7lhr\" (UID: \"6aef29a2-20f0-4804-951f-4bd876b4b8b5\") " pod="openshift-multus/network-metrics-daemon-m7lhr" Jul 07 22:03:42.644000 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.643988 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1a20bbf4-728c-4d13-b2a1-846fdf8c5a2e-iptables-alerter-script\") pod \"iptables-alerter-llpjl\" (UID: \"1a20bbf4-728c-4d13-b2a1-846fdf8c5a2e\") " pod="openshift-network-operator/iptables-alerter-llpjl" Jul 07 22:03:42.644211 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.644028 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1d2e51fd-592d-4fef-a8e0-5b70ae836780-etc-sysctl-conf\") pod \"tuned-wpg82\" (UID: \"1d2e51fd-592d-4fef-a8e0-5b70ae836780\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg82" Jul 07 22:03:42.644211 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.644050 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1d2e51fd-592d-4fef-a8e0-5b70ae836780-host\") pod \"tuned-wpg82\" (UID: \"1d2e51fd-592d-4fef-a8e0-5b70ae836780\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg82" Jul 07 22:03:42.644211 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.644056 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wqsp\" (UniqueName: \"kubernetes.io/projected/1d2e51fd-592d-4fef-a8e0-5b70ae836780-kube-api-access-9wqsp\") pod \"tuned-wpg82\" (UID: \"1d2e51fd-592d-4fef-a8e0-5b70ae836780\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg82" Jul 07 22:03:42.644211 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.644135 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6f9d7564-2f1c-4d78-863f-b2da1db479cf-hostroot\") pod \"multus-bnx62\" (UID: \"6f9d7564-2f1c-4d78-863f-b2da1db479cf\") " pod="openshift-multus/multus-bnx62" Jul 07 22:03:42.644211 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.644170 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/acb69b34-f020-4c76-89c2-0ab5b6e6f3d5-kubelet-config\") pod \"global-pull-secret-syncer-mzzl4\" (UID: \"acb69b34-f020-4c76-89c2-0ab5b6e6f3d5\") " pod="kube-system/global-pull-secret-syncer-mzzl4" Jul 07 22:03:42.644211 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.644196 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1d2e51fd-592d-4fef-a8e0-5b70ae836780-etc-sysctl-d\") pod \"tuned-wpg82\" (UID: \"1d2e51fd-592d-4fef-a8e0-5b70ae836780\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg82" Jul 07 22:03:42.644560 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.644224 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1d2e51fd-592d-4fef-a8e0-5b70ae836780-etc-tuned\") pod \"tuned-wpg82\" (UID: \"1d2e51fd-592d-4fef-a8e0-5b70ae836780\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg82" Jul 07 22:03:42.644560 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.644243 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6f9d7564-2f1c-4d78-863f-b2da1db479cf-multus-cni-dir\") pod \"multus-bnx62\" (UID: \"6f9d7564-2f1c-4d78-863f-b2da1db479cf\") " pod="openshift-multus/multus-bnx62" Jul 07 22:03:42.644560 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.644265 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/97032a35-9ecb-4f59-b494-505ec4c051f4-socket-dir\") pod \"aws-ebs-csi-driver-node-bsfv2\" (UID: \"97032a35-9ecb-4f59-b494-505ec4c051f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsfv2" Jul 07 22:03:42.644560 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.644290 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/94b5ccd3-cdf2-4d52-af68-30f21b855887-cnibin\") pod \"multus-additional-cni-plugins-9z8v8\" (UID: \"94b5ccd3-cdf2-4d52-af68-30f21b855887\") " pod="openshift-multus/multus-additional-cni-plugins-9z8v8" Jul 07 22:03:42.644560 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.644308 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6f9d7564-2f1c-4d78-863f-b2da1db479cf-multus-socket-dir-parent\") pod \"multus-bnx62\" (UID: \"6f9d7564-2f1c-4d78-863f-b2da1db479cf\") " pod="openshift-multus/multus-bnx62" Jul 07 22:03:42.644560 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.644324 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6f9d7564-2f1c-4d78-863f-b2da1db479cf-host-var-lib-cni-bin\") pod \"multus-bnx62\" (UID: \"6f9d7564-2f1c-4d78-863f-b2da1db479cf\") " pod="openshift-multus/multus-bnx62" Jul 07 22:03:42.644560 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.644371 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4b4234f5-6cda-45ea-b559-f09bcad77232-systemd-units\") pod \"ovnkube-node-4h8zm\" (UID: \"4b4234f5-6cda-45ea-b559-f09bcad77232\") " pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:03:42.644560 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.644391 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6f9d7564-2f1c-4d78-863f-b2da1db479cf-host-var-lib-cni-bin\") pod \"multus-bnx62\" (UID: \"6f9d7564-2f1c-4d78-863f-b2da1db479cf\") " pod="openshift-multus/multus-bnx62" Jul 07 22:03:42.644560 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.644412 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55b4g\" (UniqueName: \"kubernetes.io/projected/97032a35-9ecb-4f59-b494-505ec4c051f4-kube-api-access-55b4g\") pod \"aws-ebs-csi-driver-node-bsfv2\" (UID: \"97032a35-9ecb-4f59-b494-505ec4c051f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsfv2" Jul 07 22:03:42.644560 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.644442 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/94b5ccd3-cdf2-4d52-af68-30f21b855887-cni-binary-copy\") pod \"multus-additional-cni-plugins-9z8v8\" (UID: \"94b5ccd3-cdf2-4d52-af68-30f21b855887\") " pod="openshift-multus/multus-additional-cni-plugins-9z8v8" Jul 07 22:03:42.644560 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.644469 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj8mx\" (UniqueName: \"kubernetes.io/projected/95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9-kube-api-access-bj8mx\") pod \"network-check-target-zlr8s\" (UID: \"95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9\") " pod="openshift-network-diagnostics/network-check-target-zlr8s" Jul 07 22:03:42.644560 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.644523 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6f9d7564-2f1c-4d78-863f-b2da1db479cf-cnibin\") pod \"multus-bnx62\" (UID: \"6f9d7564-2f1c-4d78-863f-b2da1db479cf\") " pod="openshift-multus/multus-bnx62" Jul 07 22:03:42.644560 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.644546 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1d2e51fd-592d-4fef-a8e0-5b70ae836780-etc-modprobe-d\") pod \"tuned-wpg82\" (UID: \"1d2e51fd-592d-4fef-a8e0-5b70ae836780\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg82" Jul 07 22:03:42.644560 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.644549 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6f9d7564-2f1c-4d78-863f-b2da1db479cf-os-release\") pod \"multus-bnx62\" (UID: \"6f9d7564-2f1c-4d78-863f-b2da1db479cf\") " pod="openshift-multus/multus-bnx62" Jul 07 22:03:42.645253 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.644580 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6f9d7564-2f1c-4d78-863f-b2da1db479cf-hostroot\") pod \"multus-bnx62\" (UID: \"6f9d7564-2f1c-4d78-863f-b2da1db479cf\") " pod="openshift-multus/multus-bnx62" Jul 07 22:03:42.645253 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.644588 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6f9d7564-2f1c-4d78-863f-b2da1db479cf-cni-binary-copy\") pod \"multus-bnx62\" (UID: \"6f9d7564-2f1c-4d78-863f-b2da1db479cf\") " pod="openshift-multus/multus-bnx62" Jul 07 22:03:42.645253 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.644620 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6f9d7564-2f1c-4d78-863f-b2da1db479cf-host-var-lib-cni-multus\") pod \"multus-bnx62\" (UID: \"6f9d7564-2f1c-4d78-863f-b2da1db479cf\") " pod="openshift-multus/multus-bnx62" Jul 07 22:03:42.645253 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.644625 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1a20bbf4-728c-4d13-b2a1-846fdf8c5a2e-iptables-alerter-script\") pod \"iptables-alerter-llpjl\" (UID: \"1a20bbf4-728c-4d13-b2a1-846fdf8c5a2e\") " pod="openshift-network-operator/iptables-alerter-llpjl" Jul 07 22:03:42.645253 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.644650 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4b4234f5-6cda-45ea-b559-f09bcad77232-run-openvswitch\") pod \"ovnkube-node-4h8zm\" (UID: \"4b4234f5-6cda-45ea-b559-f09bcad77232\") " pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:03:42.645253 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.644675 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4b4234f5-6cda-45ea-b559-f09bcad77232-node-log\") pod \"ovnkube-node-4h8zm\" (UID: \"4b4234f5-6cda-45ea-b559-f09bcad77232\") " pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:03:42.645253 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.644698 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t24rv\" (UniqueName: \"kubernetes.io/projected/4b4234f5-6cda-45ea-b559-f09bcad77232-kube-api-access-t24rv\") pod \"ovnkube-node-4h8zm\" (UID: \"4b4234f5-6cda-45ea-b559-f09bcad77232\") " pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:03:42.645253 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.644763 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4b4234f5-6cda-45ea-b559-f09bcad77232-node-log\") pod \"ovnkube-node-4h8zm\" (UID: \"4b4234f5-6cda-45ea-b559-f09bcad77232\") " pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:03:42.645253 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.644852 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1d2e51fd-592d-4fef-a8e0-5b70ae836780-etc-sysctl-d\") pod \"tuned-wpg82\" (UID: \"1d2e51fd-592d-4fef-a8e0-5b70ae836780\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg82" Jul 07 22:03:42.645253 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.645062 2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 07 22:03:42.645253 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.645097 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6f9d7564-2f1c-4d78-863f-b2da1db479cf-multus-cni-dir\") pod \"multus-bnx62\" (UID: \"6f9d7564-2f1c-4d78-863f-b2da1db479cf\") " pod="openshift-multus/multus-bnx62" Jul 07 22:03:42.645253 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.645123 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6f9d7564-2f1c-4d78-863f-b2da1db479cf-cni-binary-copy\") pod \"multus-bnx62\" (UID: \"6f9d7564-2f1c-4d78-863f-b2da1db479cf\") " pod="openshift-multus/multus-bnx62" Jul 07 22:03:42.645253 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.645185 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mpn8\" (UniqueName: \"kubernetes.io/projected/b22ba61c-8e69-4913-acd7-b58c7a92c8d4-kube-api-access-7mpn8\") pod \"node-resolver-2gbtg\" (UID: \"b22ba61c-8e69-4913-acd7-b58c7a92c8d4\") " pod="openshift-dns/node-resolver-2gbtg" Jul 07 22:03:42.645253 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.645215 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1d2e51fd-592d-4fef-a8e0-5b70ae836780-etc-sysconfig\") pod \"tuned-wpg82\" (UID: \"1d2e51fd-592d-4fef-a8e0-5b70ae836780\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg82" Jul 07 22:03:42.645253 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.645243 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d2e51fd-592d-4fef-a8e0-5b70ae836780-etc-kubernetes\") pod \"tuned-wpg82\" (UID: \"1d2e51fd-592d-4fef-a8e0-5b70ae836780\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg82" Jul 07 22:03:42.646139 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.645267 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1d2e51fd-592d-4fef-a8e0-5b70ae836780-run\") pod \"tuned-wpg82\" (UID: \"1d2e51fd-592d-4fef-a8e0-5b70ae836780\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg82" Jul 07 22:03:42.646139 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.645292 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4b4234f5-6cda-45ea-b559-f09bcad77232-var-lib-openvswitch\") pod \"ovnkube-node-4h8zm\" (UID: \"4b4234f5-6cda-45ea-b559-f09bcad77232\") " pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:03:42.646139 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.645321 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/acb69b34-f020-4c76-89c2-0ab5b6e6f3d5-dbus\") pod \"global-pull-secret-syncer-mzzl4\" (UID: \"acb69b34-f020-4c76-89c2-0ab5b6e6f3d5\") " pod="kube-system/global-pull-secret-syncer-mzzl4" Jul 07 22:03:42.646139 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.645219 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1d2e51fd-592d-4fef-a8e0-5b70ae836780-etc-sysctl-conf\") pod \"tuned-wpg82\" (UID: \"1d2e51fd-592d-4fef-a8e0-5b70ae836780\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg82" Jul 07 22:03:42.646139 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.645246 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6f9d7564-2f1c-4d78-863f-b2da1db479cf-cnibin\") pod \"multus-bnx62\" (UID: \"6f9d7564-2f1c-4d78-863f-b2da1db479cf\") " pod="openshift-multus/multus-bnx62" Jul 07 22:03:42.646139 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.645388 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/acb69b34-f020-4c76-89c2-0ab5b6e6f3d5-original-pull-secret\") pod \"global-pull-secret-syncer-mzzl4\" (UID: \"acb69b34-f020-4c76-89c2-0ab5b6e6f3d5\") " pod="kube-system/global-pull-secret-syncer-mzzl4" Jul 07 22:03:42.646139 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.645413 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/68c8466f-1a04-4d64-a571-35aa35b5c8b9-serviceca\") pod \"node-ca-c9npj\" (UID: \"68c8466f-1a04-4d64-a571-35aa35b5c8b9\") " pod="openshift-image-registry/node-ca-c9npj" Jul 07 22:03:42.646139 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.645438 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d2e51fd-592d-4fef-a8e0-5b70ae836780-tmp\") pod \"tuned-wpg82\" (UID: \"1d2e51fd-592d-4fef-a8e0-5b70ae836780\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg82" Jul 07 22:03:42.646139 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.645467 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4b4234f5-6cda-45ea-b559-f09bcad77232-host-cni-netd\") pod \"ovnkube-node-4h8zm\" (UID: \"4b4234f5-6cda-45ea-b559-f09bcad77232\") " pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:03:42.646139 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.645494 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4b4234f5-6cda-45ea-b559-f09bcad77232-ovnkube-config\") pod \"ovnkube-node-4h8zm\" (UID: \"4b4234f5-6cda-45ea-b559-f09bcad77232\") " pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:03:42.646139 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.645504 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6f9d7564-2f1c-4d78-863f-b2da1db479cf-host-var-lib-cni-multus\") pod \"multus-bnx62\" (UID: \"6f9d7564-2f1c-4d78-863f-b2da1db479cf\") " pod="openshift-multus/multus-bnx62" Jul 07 22:03:42.646139 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.645544 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a38ac499-afcf-4298-9918-a825227cab29-konnectivity-ca\") pod \"konnectivity-agent-lc54x\" (UID: \"a38ac499-afcf-4298-9918-a825227cab29\") " pod="kube-system/konnectivity-agent-lc54x" Jul 07 22:03:42.646139 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.645554 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4b4234f5-6cda-45ea-b559-f09bcad77232-var-lib-openvswitch\") pod \"ovnkube-node-4h8zm\" (UID: \"4b4234f5-6cda-45ea-b559-f09bcad77232\") " pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:03:42.646139 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.645597 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6f9d7564-2f1c-4d78-863f-b2da1db479cf-multus-socket-dir-parent\") pod \"multus-bnx62\" (UID: \"6f9d7564-2f1c-4d78-863f-b2da1db479cf\") " pod="openshift-multus/multus-bnx62" Jul 07 22:03:42.646139 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.645600 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4b4234f5-6cda-45ea-b559-f09bcad77232-run-openvswitch\") pod \"ovnkube-node-4h8zm\" (UID: \"4b4234f5-6cda-45ea-b559-f09bcad77232\") " pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:03:42.646139 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.645614 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4b4234f5-6cda-45ea-b559-f09bcad77232-host-cni-netd\") pod \"ovnkube-node-4h8zm\" (UID: \"4b4234f5-6cda-45ea-b559-f09bcad77232\") " pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:03:42.646139 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.645664 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1d2e51fd-592d-4fef-a8e0-5b70ae836780-etc-sysconfig\") pod \"tuned-wpg82\" (UID: \"1d2e51fd-592d-4fef-a8e0-5b70ae836780\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg82" Jul 07 22:03:42.646139 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.645692 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6f9d7564-2f1c-4d78-863f-b2da1db479cf-os-release\") pod \"multus-bnx62\" (UID: \"6f9d7564-2f1c-4d78-863f-b2da1db479cf\") " pod="openshift-multus/multus-bnx62" Jul 07 22:03:42.647278 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.645708 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d2e51fd-592d-4fef-a8e0-5b70ae836780-etc-kubernetes\") pod \"tuned-wpg82\" (UID: \"1d2e51fd-592d-4fef-a8e0-5b70ae836780\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg82" Jul 07 22:03:42.647278 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.645767 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1d2e51fd-592d-4fef-a8e0-5b70ae836780-run\") pod \"tuned-wpg82\" (UID: \"1d2e51fd-592d-4fef-a8e0-5b70ae836780\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg82" Jul 07 22:03:42.647278 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.645876 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1a20bbf4-728c-4d13-b2a1-846fdf8c5a2e-host-slash\") pod \"iptables-alerter-llpjl\" (UID: \"1a20bbf4-728c-4d13-b2a1-846fdf8c5a2e\") " pod="openshift-network-operator/iptables-alerter-llpjl" Jul 07 22:03:42.647278 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.645908 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6f9d7564-2f1c-4d78-863f-b2da1db479cf-host-run-k8s-cni-cncf-io\") pod \"multus-bnx62\" (UID: \"6f9d7564-2f1c-4d78-863f-b2da1db479cf\") " pod="openshift-multus/multus-bnx62" Jul 07 22:03:42.647278 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.645955 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f9d7564-2f1c-4d78-863f-b2da1db479cf-host-var-lib-kubelet\") pod \"multus-bnx62\" (UID: \"6f9d7564-2f1c-4d78-863f-b2da1db479cf\") " pod="openshift-multus/multus-bnx62" Jul 07 22:03:42.647278 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.645967 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6f9d7564-2f1c-4d78-863f-b2da1db479cf-host-run-k8s-cni-cncf-io\") pod \"multus-bnx62\" (UID: \"6f9d7564-2f1c-4d78-863f-b2da1db479cf\") " pod="openshift-multus/multus-bnx62" Jul 07 22:03:42.647278 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.646010 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f9d7564-2f1c-4d78-863f-b2da1db479cf-host-var-lib-kubelet\") pod \"multus-bnx62\" (UID: \"6f9d7564-2f1c-4d78-863f-b2da1db479cf\") " pod="openshift-multus/multus-bnx62" Jul 07 22:03:42.647278 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.646008 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f9d7564-2f1c-4d78-863f-b2da1db479cf-etc-kubernetes\") pod \"multus-bnx62\" (UID: \"6f9d7564-2f1c-4d78-863f-b2da1db479cf\") " pod="openshift-multus/multus-bnx62" Jul 07 22:03:42.647278 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.646045 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4b4234f5-6cda-45ea-b559-f09bcad77232-ovn-node-metrics-cert\") pod \"ovnkube-node-4h8zm\" (UID: \"4b4234f5-6cda-45ea-b559-f09bcad77232\") " pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:03:42.647278 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.646094 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1a20bbf4-728c-4d13-b2a1-846fdf8c5a2e-host-slash\") pod \"iptables-alerter-llpjl\" (UID: \"1a20bbf4-728c-4d13-b2a1-846fdf8c5a2e\") " pod="openshift-network-operator/iptables-alerter-llpjl" Jul 07 22:03:42.647278 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.646127 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b22ba61c-8e69-4913-acd7-b58c7a92c8d4-hosts-file\") pod \"node-resolver-2gbtg\" (UID: \"b22ba61c-8e69-4913-acd7-b58c7a92c8d4\") " pod="openshift-dns/node-resolver-2gbtg" Jul 07 22:03:42.647278 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.646159 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/97032a35-9ecb-4f59-b494-505ec4c051f4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bsfv2\" (UID: \"97032a35-9ecb-4f59-b494-505ec4c051f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsfv2" Jul 07 22:03:42.647278 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.646200 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/97032a35-9ecb-4f59-b494-505ec4c051f4-registration-dir\") pod \"aws-ebs-csi-driver-node-bsfv2\" (UID: \"97032a35-9ecb-4f59-b494-505ec4c051f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsfv2" Jul 07 22:03:42.647278 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.646232 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6f9d7564-2f1c-4d78-863f-b2da1db479cf-host-run-multus-certs\") pod \"multus-bnx62\" (UID: \"6f9d7564-2f1c-4d78-863f-b2da1db479cf\") " pod="openshift-multus/multus-bnx62" Jul 07 22:03:42.647278 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.646257 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4b4234f5-6cda-45ea-b559-f09bcad77232-etc-openvswitch\") pod \"ovnkube-node-4h8zm\" (UID: \"4b4234f5-6cda-45ea-b559-f09bcad77232\") " pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:03:42.647278 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.646269 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4b4234f5-6cda-45ea-b559-f09bcad77232-ovnkube-config\") pod \"ovnkube-node-4h8zm\" (UID: \"4b4234f5-6cda-45ea-b559-f09bcad77232\") " pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:03:42.647278 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.646288 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/97032a35-9ecb-4f59-b494-505ec4c051f4-etc-selinux\") pod \"aws-ebs-csi-driver-node-bsfv2\" (UID: \"97032a35-9ecb-4f59-b494-505ec4c051f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsfv2" Jul 07 22:03:42.648247 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.646304 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/94b5ccd3-cdf2-4d52-af68-30f21b855887-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9z8v8\" (UID: \"94b5ccd3-cdf2-4d52-af68-30f21b855887\") " pod="openshift-multus/multus-additional-cni-plugins-9z8v8" Jul 07 22:03:42.648247 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.646322 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96m5j\" (UniqueName: \"kubernetes.io/projected/1a20bbf4-728c-4d13-b2a1-846fdf8c5a2e-kube-api-access-96m5j\") pod \"iptables-alerter-llpjl\" (UID: \"1a20bbf4-728c-4d13-b2a1-846fdf8c5a2e\") " pod="openshift-network-operator/iptables-alerter-llpjl" Jul 07 22:03:42.648247 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.646331 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f9d7564-2f1c-4d78-863f-b2da1db479cf-etc-kubernetes\") pod \"multus-bnx62\" (UID: \"6f9d7564-2f1c-4d78-863f-b2da1db479cf\") " pod="openshift-multus/multus-bnx62" Jul 07 22:03:42.648247 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.646338 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1d2e51fd-592d-4fef-a8e0-5b70ae836780-etc-systemd\") pod \"tuned-wpg82\" (UID: \"1d2e51fd-592d-4fef-a8e0-5b70ae836780\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg82" Jul 07 22:03:42.648247 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.646375 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d2e51fd-592d-4fef-a8e0-5b70ae836780-var-lib-kubelet\") pod \"tuned-wpg82\" (UID: \"1d2e51fd-592d-4fef-a8e0-5b70ae836780\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg82" Jul 07 22:03:42.648247 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.646373 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4b4234f5-6cda-45ea-b559-f09bcad77232-etc-openvswitch\") pod \"ovnkube-node-4h8zm\" (UID: \"4b4234f5-6cda-45ea-b559-f09bcad77232\") " pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:03:42.648247 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.646415 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b4234f5-6cda-45ea-b559-f09bcad77232-host-kubelet\") pod \"ovnkube-node-4h8zm\" (UID: \"4b4234f5-6cda-45ea-b559-f09bcad77232\") " pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:03:42.648247 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.646421 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b22ba61c-8e69-4913-acd7-b58c7a92c8d4-hosts-file\") pod \"node-resolver-2gbtg\" (UID: \"b22ba61c-8e69-4913-acd7-b58c7a92c8d4\") " pod="openshift-dns/node-resolver-2gbtg" Jul 07 22:03:42.648247 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.646445 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4b4234f5-6cda-45ea-b559-f09bcad77232-log-socket\") pod \"ovnkube-node-4h8zm\" (UID: \"4b4234f5-6cda-45ea-b559-f09bcad77232\") " pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:03:42.648247 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.646459 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6f9d7564-2f1c-4d78-863f-b2da1db479cf-host-run-multus-certs\") pod \"multus-bnx62\" (UID: \"6f9d7564-2f1c-4d78-863f-b2da1db479cf\") " pod="openshift-multus/multus-bnx62" Jul 07 22:03:42.648247 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.646509 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4b4234f5-6cda-45ea-b559-f09bcad77232-host-cni-bin\") pod \"ovnkube-node-4h8zm\" (UID: \"4b4234f5-6cda-45ea-b559-f09bcad77232\") " pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:03:42.648247 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.646534 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6f9d7564-2f1c-4d78-863f-b2da1db479cf-multus-conf-dir\") pod \"multus-bnx62\" (UID: \"6f9d7564-2f1c-4d78-863f-b2da1db479cf\") " pod="openshift-multus/multus-bnx62" Jul 07 22:03:42.648247 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.646547 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d2e51fd-592d-4fef-a8e0-5b70ae836780-var-lib-kubelet\") pod \"tuned-wpg82\" (UID: \"1d2e51fd-592d-4fef-a8e0-5b70ae836780\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg82" Jul 07 22:03:42.648247 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.646553 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4b4234f5-6cda-45ea-b559-f09bcad77232-ovnkube-script-lib\") pod \"ovnkube-node-4h8zm\" (UID: \"4b4234f5-6cda-45ea-b559-f09bcad77232\") " pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:03:42.648247 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.646586 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/97032a35-9ecb-4f59-b494-505ec4c051f4-sys-fs\") pod \"aws-ebs-csi-driver-node-bsfv2\" (UID: \"97032a35-9ecb-4f59-b494-505ec4c051f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsfv2" Jul 07 22:03:42.648247 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.646618 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1d2e51fd-592d-4fef-a8e0-5b70ae836780-lib-modules\") pod \"tuned-wpg82\" (UID: \"1d2e51fd-592d-4fef-a8e0-5b70ae836780\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg82" Jul 07 22:03:42.648247 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.646656 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6f9d7564-2f1c-4d78-863f-b2da1db479cf-multus-daemon-config\") pod \"multus-bnx62\" (UID: \"6f9d7564-2f1c-4d78-863f-b2da1db479cf\") " pod="openshift-multus/multus-bnx62" Jul 07 22:03:42.649101 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.646684 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqsg9\" (UniqueName: \"kubernetes.io/projected/6f9d7564-2f1c-4d78-863f-b2da1db479cf-kube-api-access-hqsg9\") pod \"multus-bnx62\" (UID: \"6f9d7564-2f1c-4d78-863f-b2da1db479cf\") " pod="openshift-multus/multus-bnx62" Jul 07 22:03:42.649101 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.646714 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4b4234f5-6cda-45ea-b559-f09bcad77232-run-systemd\") pod \"ovnkube-node-4h8zm\" (UID: \"4b4234f5-6cda-45ea-b559-f09bcad77232\") " pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:03:42.649101 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.646743 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4b4234f5-6cda-45ea-b559-f09bcad77232-env-overrides\") pod \"ovnkube-node-4h8zm\" (UID: \"4b4234f5-6cda-45ea-b559-f09bcad77232\") " pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:03:42.649101 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.646770 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/94b5ccd3-cdf2-4d52-af68-30f21b855887-system-cni-dir\") pod \"multus-additional-cni-plugins-9z8v8\" (UID: \"94b5ccd3-cdf2-4d52-af68-30f21b855887\") " pod="openshift-multus/multus-additional-cni-plugins-9z8v8" Jul 07 22:03:42.649101 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.646798 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/94b5ccd3-cdf2-4d52-af68-30f21b855887-os-release\") pod \"multus-additional-cni-plugins-9z8v8\" (UID: \"94b5ccd3-cdf2-4d52-af68-30f21b855887\") " pod="openshift-multus/multus-additional-cni-plugins-9z8v8" Jul 07 22:03:42.649101 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.646839 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4b4234f5-6cda-45ea-b559-f09bcad77232-host-slash\") pod \"ovnkube-node-4h8zm\" (UID: \"4b4234f5-6cda-45ea-b559-f09bcad77232\") " pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:03:42.649101 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.646868 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4b4234f5-6cda-45ea-b559-f09bcad77232-host-run-netns\") pod \"ovnkube-node-4h8zm\" (UID: \"4b4234f5-6cda-45ea-b559-f09bcad77232\") " pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:03:42.649101 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.647019 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4b4234f5-6cda-45ea-b559-f09bcad77232-ovnkube-script-lib\") pod \"ovnkube-node-4h8zm\" (UID: \"4b4234f5-6cda-45ea-b559-f09bcad77232\") " pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:03:42.649101 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.647047 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zxdv\" (UniqueName: \"kubernetes.io/projected/68c8466f-1a04-4d64-a571-35aa35b5c8b9-kube-api-access-5zxdv\") pod \"node-ca-c9npj\" (UID: \"68c8466f-1a04-4d64-a571-35aa35b5c8b9\") " pod="openshift-image-registry/node-ca-c9npj" Jul 07 22:03:42.649101 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.647080 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b4234f5-6cda-45ea-b559-f09bcad77232-host-kubelet\") pod \"ovnkube-node-4h8zm\" (UID: \"4b4234f5-6cda-45ea-b559-f09bcad77232\") " pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:03:42.649101 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.647090 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1d2e51fd-592d-4fef-a8e0-5b70ae836780-etc-systemd\") pod \"tuned-wpg82\" (UID: \"1d2e51fd-592d-4fef-a8e0-5b70ae836780\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg82" Jul 07 22:03:42.649101 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.647123 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4b4234f5-6cda-45ea-b559-f09bcad77232-run-systemd\") pod \"ovnkube-node-4h8zm\" (UID: \"4b4234f5-6cda-45ea-b559-f09bcad77232\") " pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:03:42.649101 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.647210 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1d2e51fd-592d-4fef-a8e0-5b70ae836780-lib-modules\") pod \"tuned-wpg82\" (UID: \"1d2e51fd-592d-4fef-a8e0-5b70ae836780\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg82" Jul 07 22:03:42.649101 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.647258 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6f9d7564-2f1c-4d78-863f-b2da1db479cf-multus-conf-dir\") pod \"multus-bnx62\" (UID: \"6f9d7564-2f1c-4d78-863f-b2da1db479cf\") " pod="openshift-multus/multus-bnx62" Jul 07 22:03:42.649101 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.647292 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4b4234f5-6cda-45ea-b559-f09bcad77232-host-cni-bin\") pod \"ovnkube-node-4h8zm\" (UID: \"4b4234f5-6cda-45ea-b559-f09bcad77232\") " pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:03:42.649101 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.647356 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4b4234f5-6cda-45ea-b559-f09bcad77232-host-slash\") pod \"ovnkube-node-4h8zm\" (UID: \"4b4234f5-6cda-45ea-b559-f09bcad77232\") " pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:03:42.649101 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.647083 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/94b5ccd3-cdf2-4d52-af68-30f21b855887-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9z8v8\" (UID: \"94b5ccd3-cdf2-4d52-af68-30f21b855887\") " pod="openshift-multus/multus-additional-cni-plugins-9z8v8" Jul 07 22:03:42.649733 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.647396 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4b4234f5-6cda-45ea-b559-f09bcad77232-log-socket\") pod \"ovnkube-node-4h8zm\" (UID: \"4b4234f5-6cda-45ea-b559-f09bcad77232\") " pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:03:42.649733 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.647435 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b22ba61c-8e69-4913-acd7-b58c7a92c8d4-tmp-dir\") pod \"node-resolver-2gbtg\" (UID: \"b22ba61c-8e69-4913-acd7-b58c7a92c8d4\") " pod="openshift-dns/node-resolver-2gbtg" Jul 07 22:03:42.649733 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.647634 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a38ac499-afcf-4298-9918-a825227cab29-agent-certs\") pod \"konnectivity-agent-lc54x\" (UID: \"a38ac499-afcf-4298-9918-a825227cab29\") " pod="kube-system/konnectivity-agent-lc54x" Jul 07 22:03:42.649733 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.647664 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scwjg\" (UniqueName: \"kubernetes.io/projected/94b5ccd3-cdf2-4d52-af68-30f21b855887-kube-api-access-scwjg\") pod \"multus-additional-cni-plugins-9z8v8\" (UID: \"94b5ccd3-cdf2-4d52-af68-30f21b855887\") " pod="openshift-multus/multus-additional-cni-plugins-9z8v8" Jul 07 22:03:42.649733 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.647690 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6aef29a2-20f0-4804-951f-4bd876b4b8b5-metrics-certs\") pod \"network-metrics-daemon-m7lhr\" (UID: \"6aef29a2-20f0-4804-951f-4bd876b4b8b5\") " pod="openshift-multus/network-metrics-daemon-m7lhr" Jul 07 22:03:42.649733 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.647718 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b4234f5-6cda-45ea-b559-f09bcad77232-host-run-ovn-kubernetes\") pod \"ovnkube-node-4h8zm\" (UID: \"4b4234f5-6cda-45ea-b559-f09bcad77232\") " pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:03:42.649733 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.647721 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4b4234f5-6cda-45ea-b559-f09bcad77232-env-overrides\") pod \"ovnkube-node-4h8zm\" (UID: \"4b4234f5-6cda-45ea-b559-f09bcad77232\") " pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:03:42.649733 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.647745 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b4234f5-6cda-45ea-b559-f09bcad77232-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4h8zm\" (UID: \"4b4234f5-6cda-45ea-b559-f09bcad77232\") " pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:03:42.649733 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.647771 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1d2e51fd-592d-4fef-a8e0-5b70ae836780-sys\") pod \"tuned-wpg82\" (UID: \"1d2e51fd-592d-4fef-a8e0-5b70ae836780\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg82" Jul 07 22:03:42.649733 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.647790 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b22ba61c-8e69-4913-acd7-b58c7a92c8d4-tmp-dir\") pod \"node-resolver-2gbtg\" (UID: \"b22ba61c-8e69-4913-acd7-b58c7a92c8d4\") " pod="openshift-dns/node-resolver-2gbtg" Jul 07 22:03:42.649733 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.647812 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6f9d7564-2f1c-4d78-863f-b2da1db479cf-multus-daemon-config\") pod \"multus-bnx62\" (UID: \"6f9d7564-2f1c-4d78-863f-b2da1db479cf\") " pod="openshift-multus/multus-bnx62" Jul 07 22:03:42.649733 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.647792 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b4234f5-6cda-45ea-b559-f09bcad77232-host-run-ovn-kubernetes\") pod \"ovnkube-node-4h8zm\" (UID: \"4b4234f5-6cda-45ea-b559-f09bcad77232\") " pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:03:42.649733 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.647827 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b4234f5-6cda-45ea-b559-f09bcad77232-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4h8zm\" (UID: \"4b4234f5-6cda-45ea-b559-f09bcad77232\") " pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:03:42.649733 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.647841 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1d2e51fd-592d-4fef-a8e0-5b70ae836780-sys\") pod \"tuned-wpg82\" (UID: \"1d2e51fd-592d-4fef-a8e0-5b70ae836780\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg82" Jul 07 22:03:42.649733 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.647795 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6f9d7564-2f1c-4d78-863f-b2da1db479cf-system-cni-dir\") pod \"multus-bnx62\" (UID: \"6f9d7564-2f1c-4d78-863f-b2da1db479cf\") " pod="openshift-multus/multus-bnx62" Jul 07 22:03:42.649733 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.647867 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4b4234f5-6cda-45ea-b559-f09bcad77232-host-run-netns\") pod \"ovnkube-node-4h8zm\" (UID: \"4b4234f5-6cda-45ea-b559-f09bcad77232\") " pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:03:42.649733 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.647895 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6f9d7564-2f1c-4d78-863f-b2da1db479cf-host-run-netns\") pod \"multus-bnx62\" (UID: \"6f9d7564-2f1c-4d78-863f-b2da1db479cf\") " pod="openshift-multus/multus-bnx62" Jul 07 22:03:42.650305 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.647918 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4b4234f5-6cda-45ea-b559-f09bcad77232-run-ovn\") pod \"ovnkube-node-4h8zm\" (UID: \"4b4234f5-6cda-45ea-b559-f09bcad77232\") " pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:03:42.650305 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.647894 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6f9d7564-2f1c-4d78-863f-b2da1db479cf-system-cni-dir\") pod \"multus-bnx62\" (UID: \"6f9d7564-2f1c-4d78-863f-b2da1db479cf\") " pod="openshift-multus/multus-bnx62" Jul 07 22:03:42.650305 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.647939 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/68c8466f-1a04-4d64-a571-35aa35b5c8b9-host\") pod \"node-ca-c9npj\" (UID: \"68c8466f-1a04-4d64-a571-35aa35b5c8b9\") " pod="openshift-image-registry/node-ca-c9npj" Jul 07 22:03:42.650305 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.648006 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/97032a35-9ecb-4f59-b494-505ec4c051f4-device-dir\") pod \"aws-ebs-csi-driver-node-bsfv2\" (UID: \"97032a35-9ecb-4f59-b494-505ec4c051f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsfv2" Jul 07 22:03:42.650305 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.647997 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6f9d7564-2f1c-4d78-863f-b2da1db479cf-host-run-netns\") pod \"multus-bnx62\" (UID: \"6f9d7564-2f1c-4d78-863f-b2da1db479cf\") " pod="openshift-multus/multus-bnx62" Jul 07 22:03:42.650305 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.648046 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/94b5ccd3-cdf2-4d52-af68-30f21b855887-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9z8v8\" (UID: \"94b5ccd3-cdf2-4d52-af68-30f21b855887\") " pod="openshift-multus/multus-additional-cni-plugins-9z8v8" Jul 07 22:03:42.650305 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.648054 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4b4234f5-6cda-45ea-b559-f09bcad77232-run-ovn\") pod \"ovnkube-node-4h8zm\" (UID: \"4b4234f5-6cda-45ea-b559-f09bcad77232\") " pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:03:42.650305 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.648708 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1d2e51fd-592d-4fef-a8e0-5b70ae836780-etc-tuned\") pod \"tuned-wpg82\" (UID: \"1d2e51fd-592d-4fef-a8e0-5b70ae836780\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg82" Jul 07 22:03:42.650305 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.649048 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4b4234f5-6cda-45ea-b559-f09bcad77232-ovn-node-metrics-cert\") pod \"ovnkube-node-4h8zm\" (UID: \"4b4234f5-6cda-45ea-b559-f09bcad77232\") " pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:03:42.650305 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.649265 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d2e51fd-592d-4fef-a8e0-5b70ae836780-tmp\") pod \"tuned-wpg82\" (UID: \"1d2e51fd-592d-4fef-a8e0-5b70ae836780\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg82" Jul 07 22:03:42.653082 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.653052 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wqsp\" (UniqueName: \"kubernetes.io/projected/1d2e51fd-592d-4fef-a8e0-5b70ae836780-kube-api-access-9wqsp\") pod \"tuned-wpg82\" (UID: \"1d2e51fd-592d-4fef-a8e0-5b70ae836780\") " pod="openshift-cluster-node-tuning-operator/tuned-wpg82" Jul 07 22:03:42.655199 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.655178 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t24rv\" (UniqueName: \"kubernetes.io/projected/4b4234f5-6cda-45ea-b559-f09bcad77232-kube-api-access-t24rv\") pod \"ovnkube-node-4h8zm\" (UID: \"4b4234f5-6cda-45ea-b559-f09bcad77232\") " pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:03:42.658027 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.657946 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mpn8\" (UniqueName: \"kubernetes.io/projected/b22ba61c-8e69-4913-acd7-b58c7a92c8d4-kube-api-access-7mpn8\") pod \"node-resolver-2gbtg\" (UID: \"b22ba61c-8e69-4913-acd7-b58c7a92c8d4\") " pod="openshift-dns/node-resolver-2gbtg" Jul 07 22:03:42.658419 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.658385 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqsg9\" (UniqueName: \"kubernetes.io/projected/6f9d7564-2f1c-4d78-863f-b2da1db479cf-kube-api-access-hqsg9\") pod \"multus-bnx62\" (UID: \"6f9d7564-2f1c-4d78-863f-b2da1db479cf\") " pod="openshift-multus/multus-bnx62" Jul 07 22:03:42.659909 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.659890 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96m5j\" (UniqueName: \"kubernetes.io/projected/1a20bbf4-728c-4d13-b2a1-846fdf8c5a2e-kube-api-access-96m5j\") pod \"iptables-alerter-llpjl\" (UID: \"1a20bbf4-728c-4d13-b2a1-846fdf8c5a2e\") " pod="openshift-network-operator/iptables-alerter-llpjl" Jul 07 22:03:42.748776 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.748690 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a38ac499-afcf-4298-9918-a825227cab29-konnectivity-ca\") pod \"konnectivity-agent-lc54x\" (UID: \"a38ac499-afcf-4298-9918-a825227cab29\") " pod="kube-system/konnectivity-agent-lc54x" Jul 07 22:03:42.748776 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.748724 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/97032a35-9ecb-4f59-b494-505ec4c051f4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bsfv2\" (UID: \"97032a35-9ecb-4f59-b494-505ec4c051f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsfv2" Jul 07 22:03:42.748776 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.748751 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/97032a35-9ecb-4f59-b494-505ec4c051f4-registration-dir\") pod \"aws-ebs-csi-driver-node-bsfv2\" (UID: \"97032a35-9ecb-4f59-b494-505ec4c051f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsfv2" Jul 07 22:03:42.748776 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.748778 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/97032a35-9ecb-4f59-b494-505ec4c051f4-etc-selinux\") pod \"aws-ebs-csi-driver-node-bsfv2\" (UID: \"97032a35-9ecb-4f59-b494-505ec4c051f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsfv2" Jul 07 22:03:42.749087 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.748804 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/94b5ccd3-cdf2-4d52-af68-30f21b855887-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9z8v8\" (UID: \"94b5ccd3-cdf2-4d52-af68-30f21b855887\") " pod="openshift-multus/multus-additional-cni-plugins-9z8v8" Jul 07 22:03:42.749087 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.748834 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/97032a35-9ecb-4f59-b494-505ec4c051f4-sys-fs\") pod \"aws-ebs-csi-driver-node-bsfv2\" (UID: \"97032a35-9ecb-4f59-b494-505ec4c051f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsfv2" Jul 07 22:03:42.749087 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.748871 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/94b5ccd3-cdf2-4d52-af68-30f21b855887-system-cni-dir\") pod \"multus-additional-cni-plugins-9z8v8\" (UID: \"94b5ccd3-cdf2-4d52-af68-30f21b855887\") " pod="openshift-multus/multus-additional-cni-plugins-9z8v8" Jul 07 22:03:42.749087 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.748890 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/94b5ccd3-cdf2-4d52-af68-30f21b855887-os-release\") pod \"multus-additional-cni-plugins-9z8v8\" (UID: \"94b5ccd3-cdf2-4d52-af68-30f21b855887\") " pod="openshift-multus/multus-additional-cni-plugins-9z8v8" Jul 07 22:03:42.749087 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.748918 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zxdv\" (UniqueName: \"kubernetes.io/projected/68c8466f-1a04-4d64-a571-35aa35b5c8b9-kube-api-access-5zxdv\") pod \"node-ca-c9npj\" (UID: \"68c8466f-1a04-4d64-a571-35aa35b5c8b9\") " pod="openshift-image-registry/node-ca-c9npj" Jul 07 22:03:42.749087 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.748938 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/94b5ccd3-cdf2-4d52-af68-30f21b855887-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9z8v8\" (UID: \"94b5ccd3-cdf2-4d52-af68-30f21b855887\") " pod="openshift-multus/multus-additional-cni-plugins-9z8v8" Jul 07 22:03:42.749087 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.748957 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a38ac499-afcf-4298-9918-a825227cab29-agent-certs\") pod \"konnectivity-agent-lc54x\" (UID: \"a38ac499-afcf-4298-9918-a825227cab29\") " pod="kube-system/konnectivity-agent-lc54x" Jul 07 22:03:42.749087 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.748983 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scwjg\" (UniqueName: \"kubernetes.io/projected/94b5ccd3-cdf2-4d52-af68-30f21b855887-kube-api-access-scwjg\") pod \"multus-additional-cni-plugins-9z8v8\" (UID: \"94b5ccd3-cdf2-4d52-af68-30f21b855887\") " pod="openshift-multus/multus-additional-cni-plugins-9z8v8" Jul 07 22:03:42.749087 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.749010 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6aef29a2-20f0-4804-951f-4bd876b4b8b5-metrics-certs\") pod \"network-metrics-daemon-m7lhr\" (UID: \"6aef29a2-20f0-4804-951f-4bd876b4b8b5\") " pod="openshift-multus/network-metrics-daemon-m7lhr" Jul 07 22:03:42.749087 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.749035 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/68c8466f-1a04-4d64-a571-35aa35b5c8b9-host\") pod \"node-ca-c9npj\" (UID: \"68c8466f-1a04-4d64-a571-35aa35b5c8b9\") " pod="openshift-image-registry/node-ca-c9npj" Jul 07 22:03:42.749087 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.749049 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/97032a35-9ecb-4f59-b494-505ec4c051f4-device-dir\") pod \"aws-ebs-csi-driver-node-bsfv2\" (UID: \"97032a35-9ecb-4f59-b494-505ec4c051f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsfv2" Jul 07 22:03:42.749087 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.749066 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/94b5ccd3-cdf2-4d52-af68-30f21b855887-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9z8v8\" (UID: \"94b5ccd3-cdf2-4d52-af68-30f21b855887\") " pod="openshift-multus/multus-additional-cni-plugins-9z8v8" Jul 07 22:03:42.749619 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.749093 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94ccg\" (UniqueName: \"kubernetes.io/projected/6aef29a2-20f0-4804-951f-4bd876b4b8b5-kube-api-access-94ccg\") pod \"network-metrics-daemon-m7lhr\" (UID: \"6aef29a2-20f0-4804-951f-4bd876b4b8b5\") " pod="openshift-multus/network-metrics-daemon-m7lhr" Jul 07 22:03:42.749619 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.749130 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/acb69b34-f020-4c76-89c2-0ab5b6e6f3d5-kubelet-config\") pod \"global-pull-secret-syncer-mzzl4\" (UID: \"acb69b34-f020-4c76-89c2-0ab5b6e6f3d5\") " pod="kube-system/global-pull-secret-syncer-mzzl4" Jul 07 22:03:42.749619 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.749205 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/97032a35-9ecb-4f59-b494-505ec4c051f4-socket-dir\") pod \"aws-ebs-csi-driver-node-bsfv2\" (UID: \"97032a35-9ecb-4f59-b494-505ec4c051f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsfv2" Jul 07 22:03:42.749619 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.749229 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/94b5ccd3-cdf2-4d52-af68-30f21b855887-cnibin\") pod \"multus-additional-cni-plugins-9z8v8\" (UID: \"94b5ccd3-cdf2-4d52-af68-30f21b855887\") " pod="openshift-multus/multus-additional-cni-plugins-9z8v8" Jul 07 22:03:42.749619 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.749254 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55b4g\" (UniqueName: \"kubernetes.io/projected/97032a35-9ecb-4f59-b494-505ec4c051f4-kube-api-access-55b4g\") pod \"aws-ebs-csi-driver-node-bsfv2\" (UID: \"97032a35-9ecb-4f59-b494-505ec4c051f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsfv2" Jul 07 22:03:42.749619 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.749281 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/94b5ccd3-cdf2-4d52-af68-30f21b855887-cni-binary-copy\") pod \"multus-additional-cni-plugins-9z8v8\" (UID: \"94b5ccd3-cdf2-4d52-af68-30f21b855887\") " pod="openshift-multus/multus-additional-cni-plugins-9z8v8" Jul 07 22:03:42.749619 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.749311 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj8mx\" (UniqueName: \"kubernetes.io/projected/95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9-kube-api-access-bj8mx\") pod \"network-check-target-zlr8s\" (UID: \"95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9\") " pod="openshift-network-diagnostics/network-check-target-zlr8s" Jul 07 22:03:42.749619 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.749337 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/acb69b34-f020-4c76-89c2-0ab5b6e6f3d5-dbus\") pod \"global-pull-secret-syncer-mzzl4\" (UID: \"acb69b34-f020-4c76-89c2-0ab5b6e6f3d5\") " pod="kube-system/global-pull-secret-syncer-mzzl4" Jul 07 22:03:42.749619 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.749374 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/acb69b34-f020-4c76-89c2-0ab5b6e6f3d5-original-pull-secret\") pod \"global-pull-secret-syncer-mzzl4\" (UID: \"acb69b34-f020-4c76-89c2-0ab5b6e6f3d5\") " pod="kube-system/global-pull-secret-syncer-mzzl4" Jul 07 22:03:42.749619 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.749384 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a38ac499-afcf-4298-9918-a825227cab29-konnectivity-ca\") pod \"konnectivity-agent-lc54x\" (UID: \"a38ac499-afcf-4298-9918-a825227cab29\") " pod="kube-system/konnectivity-agent-lc54x" Jul 07 22:03:42.749619 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.749391 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/68c8466f-1a04-4d64-a571-35aa35b5c8b9-serviceca\") pod \"node-ca-c9npj\" (UID: \"68c8466f-1a04-4d64-a571-35aa35b5c8b9\") " pod="openshift-image-registry/node-ca-c9npj" Jul 07 22:03:42.749619 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.749432 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/97032a35-9ecb-4f59-b494-505ec4c051f4-registration-dir\") pod \"aws-ebs-csi-driver-node-bsfv2\" (UID: \"97032a35-9ecb-4f59-b494-505ec4c051f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsfv2" Jul 07 22:03:42.749619 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.749391 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/97032a35-9ecb-4f59-b494-505ec4c051f4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bsfv2\" (UID: \"97032a35-9ecb-4f59-b494-505ec4c051f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsfv2" Jul 07 22:03:42.749619 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.749539 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/97032a35-9ecb-4f59-b494-505ec4c051f4-sys-fs\") pod \"aws-ebs-csi-driver-node-bsfv2\" (UID: \"97032a35-9ecb-4f59-b494-505ec4c051f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsfv2" Jul 07 22:03:42.750331 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.749704 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/94b5ccd3-cdf2-4d52-af68-30f21b855887-os-release\") pod \"multus-additional-cni-plugins-9z8v8\" (UID: \"94b5ccd3-cdf2-4d52-af68-30f21b855887\") " pod="openshift-multus/multus-additional-cni-plugins-9z8v8" Jul 07 22:03:42.750331 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.749732 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/94b5ccd3-cdf2-4d52-af68-30f21b855887-system-cni-dir\") pod \"multus-additional-cni-plugins-9z8v8\" (UID: \"94b5ccd3-cdf2-4d52-af68-30f21b855887\") " pod="openshift-multus/multus-additional-cni-plugins-9z8v8" Jul 07 22:03:42.750331 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.749844 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/68c8466f-1a04-4d64-a571-35aa35b5c8b9-serviceca\") pod \"node-ca-c9npj\" (UID: \"68c8466f-1a04-4d64-a571-35aa35b5c8b9\") " pod="openshift-image-registry/node-ca-c9npj" Jul 07 22:03:42.750331 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.749858 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/94b5ccd3-cdf2-4d52-af68-30f21b855887-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9z8v8\" (UID: \"94b5ccd3-cdf2-4d52-af68-30f21b855887\") " pod="openshift-multus/multus-additional-cni-plugins-9z8v8" Jul 07 22:03:42.750331 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.749945 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/68c8466f-1a04-4d64-a571-35aa35b5c8b9-host\") pod \"node-ca-c9npj\" (UID: \"68c8466f-1a04-4d64-a571-35aa35b5c8b9\") " pod="openshift-image-registry/node-ca-c9npj" Jul 07 22:03:42.750331 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.749946 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/acb69b34-f020-4c76-89c2-0ab5b6e6f3d5-kubelet-config\") pod \"global-pull-secret-syncer-mzzl4\" (UID: \"acb69b34-f020-4c76-89c2-0ab5b6e6f3d5\") " pod="kube-system/global-pull-secret-syncer-mzzl4" Jul 07 22:03:42.750331 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.749965 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/97032a35-9ecb-4f59-b494-505ec4c051f4-etc-selinux\") pod \"aws-ebs-csi-driver-node-bsfv2\" (UID: \"97032a35-9ecb-4f59-b494-505ec4c051f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsfv2" Jul 07 22:03:42.750331 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.750001 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/97032a35-9ecb-4f59-b494-505ec4c051f4-device-dir\") pod \"aws-ebs-csi-driver-node-bsfv2\" (UID: \"97032a35-9ecb-4f59-b494-505ec4c051f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsfv2" Jul 07 22:03:42.750331 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.750055 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/acb69b34-f020-4c76-89c2-0ab5b6e6f3d5-dbus\") pod \"global-pull-secret-syncer-mzzl4\" (UID: \"acb69b34-f020-4c76-89c2-0ab5b6e6f3d5\") " pod="kube-system/global-pull-secret-syncer-mzzl4" Jul 07 22:03:42.750331 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.750110 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/97032a35-9ecb-4f59-b494-505ec4c051f4-socket-dir\") pod \"aws-ebs-csi-driver-node-bsfv2\" (UID: \"97032a35-9ecb-4f59-b494-505ec4c051f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsfv2" Jul 07 22:03:42.750331 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:42.750129 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 22:03:42.750331 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.750151 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/94b5ccd3-cdf2-4d52-af68-30f21b855887-cnibin\") pod \"multus-additional-cni-plugins-9z8v8\" (UID: \"94b5ccd3-cdf2-4d52-af68-30f21b855887\") " pod="openshift-multus/multus-additional-cni-plugins-9z8v8" Jul 07 22:03:42.750331 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:42.750230 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acb69b34-f020-4c76-89c2-0ab5b6e6f3d5-original-pull-secret podName:acb69b34-f020-4c76-89c2-0ab5b6e6f3d5 nodeName:}" failed. No retries permitted until 2026-07-07 22:03:43.250185137 +0000 UTC m=+3.078314574 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/acb69b34-f020-4c76-89c2-0ab5b6e6f3d5-original-pull-secret") pod "global-pull-secret-syncer-mzzl4" (UID: "acb69b34-f020-4c76-89c2-0ab5b6e6f3d5") : object "kube-system"/"original-pull-secret" not registered Jul 07 22:03:42.750946 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:42.750470 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 22:03:42.750946 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:42.750547 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6aef29a2-20f0-4804-951f-4bd876b4b8b5-metrics-certs podName:6aef29a2-20f0-4804-951f-4bd876b4b8b5 nodeName:}" failed. No retries permitted until 2026-07-07 22:03:43.250504683 +0000 UTC m=+3.078634121 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6aef29a2-20f0-4804-951f-4bd876b4b8b5-metrics-certs") pod "network-metrics-daemon-m7lhr" (UID: "6aef29a2-20f0-4804-951f-4bd876b4b8b5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 22:03:42.750946 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.750630 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/94b5ccd3-cdf2-4d52-af68-30f21b855887-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9z8v8\" (UID: \"94b5ccd3-cdf2-4d52-af68-30f21b855887\") " pod="openshift-multus/multus-additional-cni-plugins-9z8v8" Jul 07 22:03:42.750946 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.750768 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/94b5ccd3-cdf2-4d52-af68-30f21b855887-cni-binary-copy\") pod \"multus-additional-cni-plugins-9z8v8\" (UID: \"94b5ccd3-cdf2-4d52-af68-30f21b855887\") " pod="openshift-multus/multus-additional-cni-plugins-9z8v8" Jul 07 22:03:42.751099 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.750972 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/94b5ccd3-cdf2-4d52-af68-30f21b855887-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9z8v8\" (UID: \"94b5ccd3-cdf2-4d52-af68-30f21b855887\") " pod="openshift-multus/multus-additional-cni-plugins-9z8v8" Jul 07 22:03:42.753161 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.753139 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a38ac499-afcf-4298-9918-a825227cab29-agent-certs\") pod \"konnectivity-agent-lc54x\" (UID: \"a38ac499-afcf-4298-9918-a825227cab29\") " pod="kube-system/konnectivity-agent-lc54x" Jul 07 22:03:42.757088 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:42.757068 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 22:03:42.757224 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:42.757093 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 22:03:42.757224 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:42.757106 2564 projected.go:196] Error preparing data for projected volume kube-api-access-bj8mx for pod openshift-network-diagnostics/network-check-target-zlr8s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 22:03:42.757224 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:42.757166 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9-kube-api-access-bj8mx podName:95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9 nodeName:}" failed. No retries permitted until 2026-07-07 22:03:43.257149471 +0000 UTC m=+3.085278910 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-bj8mx" (UniqueName: "kubernetes.io/projected/95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9-kube-api-access-bj8mx") pod "network-check-target-zlr8s" (UID: "95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 22:03:42.759340 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.759310 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94ccg\" (UniqueName: \"kubernetes.io/projected/6aef29a2-20f0-4804-951f-4bd876b4b8b5-kube-api-access-94ccg\") pod \"network-metrics-daemon-m7lhr\" (UID: \"6aef29a2-20f0-4804-951f-4bd876b4b8b5\") " pod="openshift-multus/network-metrics-daemon-m7lhr" Jul 07 22:03:42.759794 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.759773 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55b4g\" (UniqueName: \"kubernetes.io/projected/97032a35-9ecb-4f59-b494-505ec4c051f4-kube-api-access-55b4g\") pod \"aws-ebs-csi-driver-node-bsfv2\" (UID: \"97032a35-9ecb-4f59-b494-505ec4c051f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsfv2" Jul 07 22:03:42.759794 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.759788 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scwjg\" (UniqueName: \"kubernetes.io/projected/94b5ccd3-cdf2-4d52-af68-30f21b855887-kube-api-access-scwjg\") pod \"multus-additional-cni-plugins-9z8v8\" (UID: \"94b5ccd3-cdf2-4d52-af68-30f21b855887\") " pod="openshift-multus/multus-additional-cni-plugins-9z8v8" Jul 07 22:03:42.760824 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.760801 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zxdv\" (UniqueName: \"kubernetes.io/projected/68c8466f-1a04-4d64-a571-35aa35b5c8b9-kube-api-access-5zxdv\") pod \"node-ca-c9npj\" (UID: \"68c8466f-1a04-4d64-a571-35aa35b5c8b9\") " pod="openshift-image-registry/node-ca-c9npj" Jul 07 22:03:42.836938 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.836894 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2gbtg" Jul 07 22:03:42.854660 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.854634 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-llpjl" Jul 07 22:03:42.864952 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.864928 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wpg82" Jul 07 22:03:42.871212 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.871191 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bnx62" Jul 07 22:03:42.879305 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.879289 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:03:42.888595 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.888577 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lc54x" Jul 07 22:03:42.896622 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.896591 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-c9npj" Jul 07 22:03:42.905622 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.905603 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsfv2" Jul 07 22:03:42.912587 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:42.912570 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9z8v8" Jul 07 22:03:43.254462 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:43.254428 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/acb69b34-f020-4c76-89c2-0ab5b6e6f3d5-original-pull-secret\") pod \"global-pull-secret-syncer-mzzl4\" (UID: \"acb69b34-f020-4c76-89c2-0ab5b6e6f3d5\") " pod="kube-system/global-pull-secret-syncer-mzzl4" Jul 07 22:03:43.254702 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:43.254498 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6aef29a2-20f0-4804-951f-4bd876b4b8b5-metrics-certs\") pod \"network-metrics-daemon-m7lhr\" (UID: \"6aef29a2-20f0-4804-951f-4bd876b4b8b5\") " pod="openshift-multus/network-metrics-daemon-m7lhr" Jul 07 22:03:43.254702 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:43.254624 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 22:03:43.254702 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:43.254676 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6aef29a2-20f0-4804-951f-4bd876b4b8b5-metrics-certs podName:6aef29a2-20f0-4804-951f-4bd876b4b8b5 nodeName:}" failed. No retries permitted until 2026-07-07 22:03:44.254662081 +0000 UTC m=+4.082791518 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6aef29a2-20f0-4804-951f-4bd876b4b8b5-metrics-certs") pod "network-metrics-daemon-m7lhr" (UID: "6aef29a2-20f0-4804-951f-4bd876b4b8b5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 22:03:43.254907 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:43.254712 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 22:03:43.254907 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:43.254801 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acb69b34-f020-4c76-89c2-0ab5b6e6f3d5-original-pull-secret podName:acb69b34-f020-4c76-89c2-0ab5b6e6f3d5 nodeName:}" failed. No retries permitted until 2026-07-07 22:03:44.254773768 +0000 UTC m=+4.082903220 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/acb69b34-f020-4c76-89c2-0ab5b6e6f3d5-original-pull-secret") pod "global-pull-secret-syncer-mzzl4" (UID: "acb69b34-f020-4c76-89c2-0ab5b6e6f3d5") : object "kube-system"/"original-pull-secret" not registered Jul 07 22:03:43.355388 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:43.355353 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj8mx\" (UniqueName: \"kubernetes.io/projected/95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9-kube-api-access-bj8mx\") pod \"network-check-target-zlr8s\" (UID: \"95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9\") " pod="openshift-network-diagnostics/network-check-target-zlr8s" Jul 07 22:03:43.355578 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:43.355555 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 22:03:43.355638 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:43.355587 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 22:03:43.355638 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:43.355601 2564 projected.go:196] Error preparing data for projected volume kube-api-access-bj8mx for pod openshift-network-diagnostics/network-check-target-zlr8s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 22:03:43.355718 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:43.355677 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9-kube-api-access-bj8mx podName:95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9 nodeName:}" failed. No retries permitted until 2026-07-07 22:03:44.355656819 +0000 UTC m=+4.183786278 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-bj8mx" (UniqueName: "kubernetes.io/projected/95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9-kube-api-access-bj8mx") pod "network-check-target-zlr8s" (UID: "95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 22:03:43.584161 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:43.583527 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 21:58:42 +0000 UTC" deadline="2028-02-06 02:28:05.603215956 +0000 UTC" Jul 07 22:03:43.584161 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:43.583588 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13876h24m22.019631871s" Jul 07 22:03:43.701944 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:43.701907 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m7lhr" Jul 07 22:03:43.702142 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:43.702068 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m7lhr" podUID="6aef29a2-20f0-4804-951f-4bd876b4b8b5" Jul 07 22:03:43.746255 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:43.746219 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda38ac499_afcf_4298_9918_a825227cab29.slice/crio-1dac7e69fe264af3644597a78f857018f5d5c41967d4b9d5f77947d83a15c122 WatchSource:0}: Error finding container 1dac7e69fe264af3644597a78f857018f5d5c41967d4b9d5f77947d83a15c122: Status 404 returned error can't find the container with id 1dac7e69fe264af3644597a78f857018f5d5c41967d4b9d5f77947d83a15c122 Jul 07 22:03:43.748148 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:43.748120 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f9d7564_2f1c_4d78_863f_b2da1db479cf.slice/crio-9c1646b097544729de83ae3c3c50bf57248b9a8a8be51a224b36fa4bb026c04e WatchSource:0}: Error finding container 9c1646b097544729de83ae3c3c50bf57248b9a8a8be51a224b36fa4bb026c04e: Status 404 returned error can't find the container with id 9c1646b097544729de83ae3c3c50bf57248b9a8a8be51a224b36fa4bb026c04e Jul 07 22:03:43.756047 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:43.756023 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b4234f5_6cda_45ea_b559_f09bcad77232.slice/crio-39af92a44b329b9fe9d03d7018527a7b4bd7bc94c18a51cc19312cf0e40d6e35 WatchSource:0}: Error finding container 39af92a44b329b9fe9d03d7018527a7b4bd7bc94c18a51cc19312cf0e40d6e35: Status 404 returned error can't find the container with id 39af92a44b329b9fe9d03d7018527a7b4bd7bc94c18a51cc19312cf0e40d6e35 Jul 07 22:03:43.756819 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:43.756801 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a20bbf4_728c_4d13_b2a1_846fdf8c5a2e.slice/crio-826c73dc3663f1a95ee2dacc09979ecb6e44e9bd49742bf6f8841a81c7f87447 WatchSource:0}: Error finding container 826c73dc3663f1a95ee2dacc09979ecb6e44e9bd49742bf6f8841a81c7f87447: Status 404 returned error can't find the container with id 826c73dc3663f1a95ee2dacc09979ecb6e44e9bd49742bf6f8841a81c7f87447 Jul 07 22:03:43.758829 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:43.758532 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d2e51fd_592d_4fef_a8e0_5b70ae836780.slice/crio-0975c27d10cc22c196bd82e8201002378b2ff5a6bc34539c52119cd13c6efa09 WatchSource:0}: Error finding container 0975c27d10cc22c196bd82e8201002378b2ff5a6bc34539c52119cd13c6efa09: Status 404 returned error can't find the container with id 0975c27d10cc22c196bd82e8201002378b2ff5a6bc34539c52119cd13c6efa09 Jul 07 22:03:43.759882 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:43.759821 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb22ba61c_8e69_4913_acd7_b58c7a92c8d4.slice/crio-86e36c34eb5635b14f1fc0e55990659353708b13417ce5512727400d46edb96b WatchSource:0}: Error finding container 86e36c34eb5635b14f1fc0e55990659353708b13417ce5512727400d46edb96b: Status 404 returned error can't find the container with id 86e36c34eb5635b14f1fc0e55990659353708b13417ce5512727400d46edb96b Jul 07 22:03:43.761506 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:43.761407 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68c8466f_1a04_4d64_a571_35aa35b5c8b9.slice/crio-8c73a7e359a240f48aa350b8748bc0020d9ffac8af950608e334476978c3e1d9 WatchSource:0}: Error finding container 8c73a7e359a240f48aa350b8748bc0020d9ffac8af950608e334476978c3e1d9: Status 404 returned error can't find the container with id 8c73a7e359a240f48aa350b8748bc0020d9ffac8af950608e334476978c3e1d9 Jul 07 22:03:43.762779 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:03:43.762759 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94b5ccd3_cdf2_4d52_af68_30f21b855887.slice/crio-87c11c0ec0afc584109ce483484a12af16765e56e5e1b57915132aefc3e8dd46 WatchSource:0}: Error finding container 87c11c0ec0afc584109ce483484a12af16765e56e5e1b57915132aefc3e8dd46: Status 404 returned error can't find the container with id 87c11c0ec0afc584109ce483484a12af16765e56e5e1b57915132aefc3e8dd46 Jul 07 22:03:44.263225 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:44.262896 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/acb69b34-f020-4c76-89c2-0ab5b6e6f3d5-original-pull-secret\") pod \"global-pull-secret-syncer-mzzl4\" (UID: \"acb69b34-f020-4c76-89c2-0ab5b6e6f3d5\") " pod="kube-system/global-pull-secret-syncer-mzzl4" Jul 07 22:03:44.263391 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:44.263224 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6aef29a2-20f0-4804-951f-4bd876b4b8b5-metrics-certs\") pod \"network-metrics-daemon-m7lhr\" (UID: \"6aef29a2-20f0-4804-951f-4bd876b4b8b5\") " pod="openshift-multus/network-metrics-daemon-m7lhr" Jul 07 22:03:44.263391 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:44.263370 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 22:03:44.263557 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:44.263437 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6aef29a2-20f0-4804-951f-4bd876b4b8b5-metrics-certs podName:6aef29a2-20f0-4804-951f-4bd876b4b8b5 nodeName:}" failed. No retries permitted until 2026-07-07 22:03:46.263416709 +0000 UTC m=+6.091546153 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6aef29a2-20f0-4804-951f-4bd876b4b8b5-metrics-certs") pod "network-metrics-daemon-m7lhr" (UID: "6aef29a2-20f0-4804-951f-4bd876b4b8b5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 22:03:44.264050 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:44.263930 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 22:03:44.264050 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:44.263987 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acb69b34-f020-4c76-89c2-0ab5b6e6f3d5-original-pull-secret podName:acb69b34-f020-4c76-89c2-0ab5b6e6f3d5 nodeName:}" failed. No retries permitted until 2026-07-07 22:03:46.263970869 +0000 UTC m=+6.092100313 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/acb69b34-f020-4c76-89c2-0ab5b6e6f3d5-original-pull-secret") pod "global-pull-secret-syncer-mzzl4" (UID: "acb69b34-f020-4c76-89c2-0ab5b6e6f3d5") : object "kube-system"/"original-pull-secret" not registered Jul 07 22:03:44.365194 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:44.364390 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj8mx\" (UniqueName: \"kubernetes.io/projected/95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9-kube-api-access-bj8mx\") pod \"network-check-target-zlr8s\" (UID: \"95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9\") " pod="openshift-network-diagnostics/network-check-target-zlr8s" Jul 07 22:03:44.365194 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:44.364672 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 22:03:44.365194 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:44.364722 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 22:03:44.365194 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:44.364740 2564 projected.go:196] Error preparing data for projected volume kube-api-access-bj8mx for pod openshift-network-diagnostics/network-check-target-zlr8s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 22:03:44.365194 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:44.364805 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9-kube-api-access-bj8mx podName:95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9 nodeName:}" failed. No retries permitted until 2026-07-07 22:03:46.364783405 +0000 UTC m=+6.192912852 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-bj8mx" (UniqueName: "kubernetes.io/projected/95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9-kube-api-access-bj8mx") pod "network-check-target-zlr8s" (UID: "95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 22:03:44.584340 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:44.584203 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 21:58:42 +0000 UTC" deadline="2028-01-26 07:30:30.010919811 +0000 UTC" Jul 07 22:03:44.584340 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:44.584248 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13617h26m45.426676543s" Jul 07 22:03:44.704585 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:44.704472 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zlr8s" Jul 07 22:03:44.704768 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:44.704646 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mzzl4" Jul 07 22:03:44.704837 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:44.704768 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mzzl4" podUID="acb69b34-f020-4c76-89c2-0ab5b6e6f3d5" Jul 07 22:03:44.705147 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:44.705082 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zlr8s" podUID="95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9" Jul 07 22:03:44.724696 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:44.724660 2564 generic.go:356] "Generic (PLEG): container finished" podID="7a6e89d5fb8cfce240a45d8bfd07e986" containerID="b5b4ee2ba5217e50b6890b136f4c7fe513859ee7a7e62d01d07753fbe1d97725" exitCode=0 Jul 07 22:03:44.724844 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:44.724758 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal" event={"ID":"7a6e89d5fb8cfce240a45d8bfd07e986","Type":"ContainerDied","Data":"b5b4ee2ba5217e50b6890b136f4c7fe513859ee7a7e62d01d07753fbe1d97725"} Jul 07 22:03:44.744354 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:44.744051 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9z8v8" event={"ID":"94b5ccd3-cdf2-4d52-af68-30f21b855887","Type":"ContainerStarted","Data":"87c11c0ec0afc584109ce483484a12af16765e56e5e1b57915132aefc3e8dd46"} Jul 07 22:03:44.748287 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:44.748256 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2gbtg" event={"ID":"b22ba61c-8e69-4913-acd7-b58c7a92c8d4","Type":"ContainerStarted","Data":"86e36c34eb5635b14f1fc0e55990659353708b13417ce5512727400d46edb96b"} Jul 07 22:03:44.755248 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:44.755216 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-llpjl" event={"ID":"1a20bbf4-728c-4d13-b2a1-846fdf8c5a2e","Type":"ContainerStarted","Data":"826c73dc3663f1a95ee2dacc09979ecb6e44e9bd49742bf6f8841a81c7f87447"} Jul 07 22:03:44.762612 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:44.762579 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bnx62" event={"ID":"6f9d7564-2f1c-4d78-863f-b2da1db479cf","Type":"ContainerStarted","Data":"9c1646b097544729de83ae3c3c50bf57248b9a8a8be51a224b36fa4bb026c04e"} Jul 07 22:03:44.767172 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:44.767144 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-156.ec2.internal" event={"ID":"7e758ef0bacc8b47cfb5ed8977696fa1","Type":"ContainerStarted","Data":"3fd13f3c70fcfb1085ff12a315176fcbc76423d5e892794e8391750b69af1421"} Jul 07 22:03:44.774923 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:44.774894 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-c9npj" event={"ID":"68c8466f-1a04-4d64-a571-35aa35b5c8b9","Type":"ContainerStarted","Data":"8c73a7e359a240f48aa350b8748bc0020d9ffac8af950608e334476978c3e1d9"} Jul 07 22:03:44.779115 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:44.779084 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wpg82" event={"ID":"1d2e51fd-592d-4fef-a8e0-5b70ae836780","Type":"ContainerStarted","Data":"0975c27d10cc22c196bd82e8201002378b2ff5a6bc34539c52119cd13c6efa09"} Jul 07 22:03:44.789368 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:44.789245 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" event={"ID":"4b4234f5-6cda-45ea-b559-f09bcad77232","Type":"ContainerStarted","Data":"39af92a44b329b9fe9d03d7018527a7b4bd7bc94c18a51cc19312cf0e40d6e35"} Jul 07 22:03:44.794172 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:44.794145 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lc54x" event={"ID":"a38ac499-afcf-4298-9918-a825227cab29","Type":"ContainerStarted","Data":"1dac7e69fe264af3644597a78f857018f5d5c41967d4b9d5f77947d83a15c122"} Jul 07 22:03:44.798711 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:44.798684 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsfv2" event={"ID":"97032a35-9ecb-4f59-b494-505ec4c051f4","Type":"ContainerStarted","Data":"7c922d0ee258f4870f12012cb5f6654884f8dbc2c58e8df3c1e562c6635169d8"} Jul 07 22:03:45.702423 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:45.702350 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m7lhr" Jul 07 22:03:45.702945 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:45.702630 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m7lhr" podUID="6aef29a2-20f0-4804-951f-4bd876b4b8b5" Jul 07 22:03:45.822029 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:45.821452 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal" event={"ID":"7a6e89d5fb8cfce240a45d8bfd07e986","Type":"ContainerStarted","Data":"d0841cb034a1d765e1359011500d4669a4a8884f1cc24751d26dd49083e0992d"} Jul 07 22:03:45.836510 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:45.836434 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-156.ec2.internal" podStartSLOduration=3.8364144209999997 podStartE2EDuration="3.836414421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 22:03:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:03:44.783049921 +0000 UTC m=+4.611179385" watchObservedRunningTime="2026-07-07 22:03:45.836414421 +0000 UTC m=+5.664543885" Jul 07 22:03:45.837261 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:45.837218 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-156.ec2.internal" podStartSLOduration=3.8372032000000003 podStartE2EDuration="3.8372032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 22:03:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:03:45.835992643 +0000 UTC m=+5.664122106" watchObservedRunningTime="2026-07-07 22:03:45.8372032 +0000 UTC m=+5.665332662" Jul 07 22:03:46.280790 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:46.280747 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6aef29a2-20f0-4804-951f-4bd876b4b8b5-metrics-certs\") pod \"network-metrics-daemon-m7lhr\" (UID: \"6aef29a2-20f0-4804-951f-4bd876b4b8b5\") " pod="openshift-multus/network-metrics-daemon-m7lhr" Jul 07 22:03:46.280991 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:46.280838 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/acb69b34-f020-4c76-89c2-0ab5b6e6f3d5-original-pull-secret\") pod \"global-pull-secret-syncer-mzzl4\" (UID: \"acb69b34-f020-4c76-89c2-0ab5b6e6f3d5\") " pod="kube-system/global-pull-secret-syncer-mzzl4" Jul 07 22:03:46.280991 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:46.280975 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 22:03:46.281109 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:46.281042 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acb69b34-f020-4c76-89c2-0ab5b6e6f3d5-original-pull-secret podName:acb69b34-f020-4c76-89c2-0ab5b6e6f3d5 nodeName:}" failed. No retries permitted until 2026-07-07 22:03:50.281020416 +0000 UTC m=+10.109149872 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/acb69b34-f020-4c76-89c2-0ab5b6e6f3d5-original-pull-secret") pod "global-pull-secret-syncer-mzzl4" (UID: "acb69b34-f020-4c76-89c2-0ab5b6e6f3d5") : object "kube-system"/"original-pull-secret" not registered Jul 07 22:03:46.281636 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:46.281507 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 22:03:46.281636 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:46.281587 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6aef29a2-20f0-4804-951f-4bd876b4b8b5-metrics-certs podName:6aef29a2-20f0-4804-951f-4bd876b4b8b5 nodeName:}" failed. No retries permitted until 2026-07-07 22:03:50.281567565 +0000 UTC m=+10.109697020 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6aef29a2-20f0-4804-951f-4bd876b4b8b5-metrics-certs") pod "network-metrics-daemon-m7lhr" (UID: "6aef29a2-20f0-4804-951f-4bd876b4b8b5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 22:03:46.382032 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:46.381234 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj8mx\" (UniqueName: \"kubernetes.io/projected/95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9-kube-api-access-bj8mx\") pod \"network-check-target-zlr8s\" (UID: \"95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9\") " pod="openshift-network-diagnostics/network-check-target-zlr8s" Jul 07 22:03:46.382032 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:46.381470 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 22:03:46.382032 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:46.381508 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 22:03:46.382032 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:46.381522 2564 projected.go:196] Error preparing data for projected volume kube-api-access-bj8mx for pod openshift-network-diagnostics/network-check-target-zlr8s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 22:03:46.382032 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:46.381587 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9-kube-api-access-bj8mx podName:95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9 nodeName:}" failed. No retries permitted until 2026-07-07 22:03:50.381566618 +0000 UTC m=+10.209696059 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-bj8mx" (UniqueName: "kubernetes.io/projected/95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9-kube-api-access-bj8mx") pod "network-check-target-zlr8s" (UID: "95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 22:03:46.701473 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:46.701363 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mzzl4" Jul 07 22:03:46.701646 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:46.701526 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mzzl4" podUID="acb69b34-f020-4c76-89c2-0ab5b6e6f3d5" Jul 07 22:03:46.701646 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:46.701610 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zlr8s" Jul 07 22:03:46.701775 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:46.701689 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zlr8s" podUID="95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9" Jul 07 22:03:47.702077 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:47.702046 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m7lhr" Jul 07 22:03:47.702585 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:47.702195 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m7lhr" podUID="6aef29a2-20f0-4804-951f-4bd876b4b8b5" Jul 07 22:03:48.701597 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:48.701524 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zlr8s" Jul 07 22:03:48.701776 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:48.701658 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zlr8s" podUID="95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9" Jul 07 22:03:48.702072 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:48.702051 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mzzl4" Jul 07 22:03:48.702220 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:48.702195 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mzzl4" podUID="acb69b34-f020-4c76-89c2-0ab5b6e6f3d5" Jul 07 22:03:49.702051 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:49.701993 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m7lhr" Jul 07 22:03:49.702209 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:49.702122 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m7lhr" podUID="6aef29a2-20f0-4804-951f-4bd876b4b8b5" Jul 07 22:03:50.313983 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:50.313940 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6aef29a2-20f0-4804-951f-4bd876b4b8b5-metrics-certs\") pod \"network-metrics-daemon-m7lhr\" (UID: \"6aef29a2-20f0-4804-951f-4bd876b4b8b5\") " pod="openshift-multus/network-metrics-daemon-m7lhr" Jul 07 22:03:50.314336 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:50.314031 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/acb69b34-f020-4c76-89c2-0ab5b6e6f3d5-original-pull-secret\") pod \"global-pull-secret-syncer-mzzl4\" (UID: \"acb69b34-f020-4c76-89c2-0ab5b6e6f3d5\") " pod="kube-system/global-pull-secret-syncer-mzzl4" Jul 07 22:03:50.314336 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:50.314169 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 22:03:50.314336 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:50.314229 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acb69b34-f020-4c76-89c2-0ab5b6e6f3d5-original-pull-secret podName:acb69b34-f020-4c76-89c2-0ab5b6e6f3d5 nodeName:}" failed. No retries permitted until 2026-07-07 22:03:58.314210941 +0000 UTC m=+18.142340392 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/acb69b34-f020-4c76-89c2-0ab5b6e6f3d5-original-pull-secret") pod "global-pull-secret-syncer-mzzl4" (UID: "acb69b34-f020-4c76-89c2-0ab5b6e6f3d5") : object "kube-system"/"original-pull-secret" not registered Jul 07 22:03:50.314680 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:50.314664 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 22:03:50.314734 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:50.314723 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6aef29a2-20f0-4804-951f-4bd876b4b8b5-metrics-certs podName:6aef29a2-20f0-4804-951f-4bd876b4b8b5 nodeName:}" failed. No retries permitted until 2026-07-07 22:03:58.314707068 +0000 UTC m=+18.142836524 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6aef29a2-20f0-4804-951f-4bd876b4b8b5-metrics-certs") pod "network-metrics-daemon-m7lhr" (UID: "6aef29a2-20f0-4804-951f-4bd876b4b8b5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 22:03:50.415287 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:50.415220 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj8mx\" (UniqueName: \"kubernetes.io/projected/95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9-kube-api-access-bj8mx\") pod \"network-check-target-zlr8s\" (UID: \"95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9\") " pod="openshift-network-diagnostics/network-check-target-zlr8s" Jul 07 22:03:50.415485 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:50.415431 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 22:03:50.415485 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:50.415469 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 22:03:50.415485 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:50.415484 2564 projected.go:196] Error preparing data for projected volume kube-api-access-bj8mx for pod openshift-network-diagnostics/network-check-target-zlr8s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 22:03:50.415688 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:50.415542 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9-kube-api-access-bj8mx podName:95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9 nodeName:}" failed. No retries permitted until 2026-07-07 22:03:58.415523842 +0000 UTC m=+18.243653278 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-bj8mx" (UniqueName: "kubernetes.io/projected/95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9-kube-api-access-bj8mx") pod "network-check-target-zlr8s" (UID: "95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 22:03:50.705070 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:50.704399 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zlr8s" Jul 07 22:03:50.705070 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:50.704513 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zlr8s" podUID="95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9" Jul 07 22:03:50.705070 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:50.704865 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mzzl4" Jul 07 22:03:50.705070 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:50.704961 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mzzl4" podUID="acb69b34-f020-4c76-89c2-0ab5b6e6f3d5" Jul 07 22:03:51.701644 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:51.701609 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m7lhr" Jul 07 22:03:51.702007 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:51.701798 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m7lhr" podUID="6aef29a2-20f0-4804-951f-4bd876b4b8b5" Jul 07 22:03:52.705679 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:52.705610 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mzzl4" Jul 07 22:03:52.706038 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:52.705714 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mzzl4" podUID="acb69b34-f020-4c76-89c2-0ab5b6e6f3d5" Jul 07 22:03:52.706038 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:52.706001 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zlr8s" Jul 07 22:03:52.706118 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:52.706066 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zlr8s" podUID="95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9" Jul 07 22:03:53.702362 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:53.702324 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m7lhr" Jul 07 22:03:53.702548 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:53.702499 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m7lhr" podUID="6aef29a2-20f0-4804-951f-4bd876b4b8b5" Jul 07 22:03:54.701590 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:54.701555 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mzzl4" Jul 07 22:03:54.701945 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:54.701681 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mzzl4" podUID="acb69b34-f020-4c76-89c2-0ab5b6e6f3d5" Jul 07 22:03:54.701945 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:54.701727 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zlr8s" Jul 07 22:03:54.701945 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:54.701815 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zlr8s" podUID="95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9" Jul 07 22:03:55.701575 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:55.701540 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m7lhr" Jul 07 22:03:55.701740 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:55.701679 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m7lhr" podUID="6aef29a2-20f0-4804-951f-4bd876b4b8b5" Jul 07 22:03:56.702029 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:56.701990 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zlr8s" Jul 07 22:03:56.702492 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:56.702040 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mzzl4" Jul 07 22:03:56.702492 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:56.702130 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zlr8s" podUID="95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9" Jul 07 22:03:56.702492 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:56.702269 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mzzl4" podUID="acb69b34-f020-4c76-89c2-0ab5b6e6f3d5" Jul 07 22:03:57.702046 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:57.702009 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m7lhr" Jul 07 22:03:57.702571 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:57.702151 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m7lhr" podUID="6aef29a2-20f0-4804-951f-4bd876b4b8b5" Jul 07 22:03:58.372845 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:58.372802 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/acb69b34-f020-4c76-89c2-0ab5b6e6f3d5-original-pull-secret\") pod \"global-pull-secret-syncer-mzzl4\" (UID: \"acb69b34-f020-4c76-89c2-0ab5b6e6f3d5\") " pod="kube-system/global-pull-secret-syncer-mzzl4" Jul 07 22:03:58.373097 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:58.372861 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6aef29a2-20f0-4804-951f-4bd876b4b8b5-metrics-certs\") pod \"network-metrics-daemon-m7lhr\" (UID: \"6aef29a2-20f0-4804-951f-4bd876b4b8b5\") " pod="openshift-multus/network-metrics-daemon-m7lhr" Jul 07 22:03:58.373097 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:58.372962 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 22:03:58.373097 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:58.372967 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 22:03:58.373097 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:58.373025 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6aef29a2-20f0-4804-951f-4bd876b4b8b5-metrics-certs podName:6aef29a2-20f0-4804-951f-4bd876b4b8b5 nodeName:}" failed. No retries permitted until 2026-07-07 22:04:14.373004401 +0000 UTC m=+34.201133839 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6aef29a2-20f0-4804-951f-4bd876b4b8b5-metrics-certs") pod "network-metrics-daemon-m7lhr" (UID: "6aef29a2-20f0-4804-951f-4bd876b4b8b5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 22:03:58.373097 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:58.373045 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acb69b34-f020-4c76-89c2-0ab5b6e6f3d5-original-pull-secret podName:acb69b34-f020-4c76-89c2-0ab5b6e6f3d5 nodeName:}" failed. No retries permitted until 2026-07-07 22:04:14.373034254 +0000 UTC m=+34.201163693 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/acb69b34-f020-4c76-89c2-0ab5b6e6f3d5-original-pull-secret") pod "global-pull-secret-syncer-mzzl4" (UID: "acb69b34-f020-4c76-89c2-0ab5b6e6f3d5") : object "kube-system"/"original-pull-secret" not registered Jul 07 22:03:58.473539 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:58.473507 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj8mx\" (UniqueName: \"kubernetes.io/projected/95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9-kube-api-access-bj8mx\") pod \"network-check-target-zlr8s\" (UID: \"95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9\") " pod="openshift-network-diagnostics/network-check-target-zlr8s" Jul 07 22:03:58.473722 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:58.473704 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 22:03:58.473799 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:58.473730 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 22:03:58.473799 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:58.473742 2564 projected.go:196] Error preparing data for projected volume kube-api-access-bj8mx for pod openshift-network-diagnostics/network-check-target-zlr8s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 22:03:58.473896 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:58.473802 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9-kube-api-access-bj8mx podName:95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9 nodeName:}" failed. No retries permitted until 2026-07-07 22:04:14.473783768 +0000 UTC m=+34.301913208 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-bj8mx" (UniqueName: "kubernetes.io/projected/95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9-kube-api-access-bj8mx") pod "network-check-target-zlr8s" (UID: "95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 22:03:58.701614 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:58.701535 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mzzl4" Jul 07 22:03:58.701785 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:58.701535 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zlr8s" Jul 07 22:03:58.701785 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:58.701693 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mzzl4" podUID="acb69b34-f020-4c76-89c2-0ab5b6e6f3d5" Jul 07 22:03:58.701785 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:58.701714 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zlr8s" podUID="95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9" Jul 07 22:03:59.701692 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:03:59.701611 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m7lhr" Jul 07 22:03:59.702077 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:03:59.701740 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m7lhr" podUID="6aef29a2-20f0-4804-951f-4bd876b4b8b5" Jul 07 22:04:00.705551 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:00.705522 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zlr8s" Jul 07 22:04:00.706008 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:00.705522 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mzzl4" Jul 07 22:04:00.706008 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:04:00.705636 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zlr8s" podUID="95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9" Jul 07 22:04:00.706008 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:04:00.705728 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mzzl4" podUID="acb69b34-f020-4c76-89c2-0ab5b6e6f3d5" Jul 07 22:04:01.701589 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:01.701555 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m7lhr" Jul 07 22:04:01.701755 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:04:01.701682 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m7lhr" podUID="6aef29a2-20f0-4804-951f-4bd876b4b8b5" Jul 07 22:04:02.702197 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:02.702161 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zlr8s" Jul 07 22:04:02.702786 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:04:02.702286 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zlr8s" podUID="95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9" Jul 07 22:04:02.702786 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:02.702604 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mzzl4" Jul 07 22:04:02.702786 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:04:02.702687 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mzzl4" podUID="acb69b34-f020-4c76-89c2-0ab5b6e6f3d5" Jul 07 22:04:03.701983 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:03.701817 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m7lhr" Jul 07 22:04:03.702135 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:04:03.702108 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m7lhr" podUID="6aef29a2-20f0-4804-951f-4bd876b4b8b5" Jul 07 22:04:03.861417 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:03.861378 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bnx62" event={"ID":"6f9d7564-2f1c-4d78-863f-b2da1db479cf","Type":"ContainerStarted","Data":"c581546ffef000f71bacffa6ea44caf41f468804598d9cd622454d952b0485c4"} Jul 07 22:04:03.863477 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:03.863311 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-c9npj" event={"ID":"68c8466f-1a04-4d64-a571-35aa35b5c8b9","Type":"ContainerStarted","Data":"e1d732db82663b65a4475b6576cf4339dbc7dc40490a216441a10d0f12c79a09"} Jul 07 22:04:03.865113 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:03.865085 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wpg82" event={"ID":"1d2e51fd-592d-4fef-a8e0-5b70ae836780","Type":"ContainerStarted","Data":"b8986ea9ce5a2e32d40f24b171a7083bec93483d2e92f4d455b569bdb981dfe2"} Jul 07 22:04:03.868625 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:03.868604 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4h8zm_4b4234f5-6cda-45ea-b559-f09bcad77232/ovn-acl-logging/0.log" Jul 07 22:04:03.869014 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:03.868984 2564 generic.go:356] "Generic (PLEG): container finished" podID="4b4234f5-6cda-45ea-b559-f09bcad77232" containerID="70ebf17818025cd68926e62b6495e3ed02d65d9fece5b639d08a907533160283" exitCode=1 Jul 07 22:04:03.869093 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:03.869054 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" event={"ID":"4b4234f5-6cda-45ea-b559-f09bcad77232","Type":"ContainerStarted","Data":"e316b983f34be0b1addd50dcce036e5bad9f0ab1a27aa2d7a5a00a4e112c6d82"} Jul 07 22:04:03.869149 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:03.869131 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" event={"ID":"4b4234f5-6cda-45ea-b559-f09bcad77232","Type":"ContainerStarted","Data":"ffd2333908d9d989a97b92f4285dd35ff0186d9aa34938257ea2cd72824da1b7"} Jul 07 22:04:03.869197 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:03.869179 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" event={"ID":"4b4234f5-6cda-45ea-b559-f09bcad77232","Type":"ContainerStarted","Data":"05caacc8efc2916887b5735d39c8d1668973becad4561017fd33ecb5f16c8362"} Jul 07 22:04:03.869251 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:03.869226 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" event={"ID":"4b4234f5-6cda-45ea-b559-f09bcad77232","Type":"ContainerStarted","Data":"5ef7f41b6abd1417bd4c4d6b945b548bfd1fd1d1db52498a4cb25a6741c29a4d"} Jul 07 22:04:03.869298 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:03.869271 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" event={"ID":"4b4234f5-6cda-45ea-b559-f09bcad77232","Type":"ContainerDied","Data":"70ebf17818025cd68926e62b6495e3ed02d65d9fece5b639d08a907533160283"} Jul 07 22:04:03.869347 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:03.869335 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" event={"ID":"4b4234f5-6cda-45ea-b559-f09bcad77232","Type":"ContainerStarted","Data":"bafd2334a960db248fd232a87faef8cff60fb577fb19d2106d7c699f5daaa64a"} Jul 07 22:04:03.870554 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:03.870531 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lc54x" event={"ID":"a38ac499-afcf-4298-9918-a825227cab29","Type":"ContainerStarted","Data":"697212b515ebafba4096ec8466c77e939d295e447127592a10e6b5dad2a01ba8"} Jul 07 22:04:03.872326 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:03.872303 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsfv2" event={"ID":"97032a35-9ecb-4f59-b494-505ec4c051f4","Type":"ContainerStarted","Data":"75024bd27695d65ee559398fcdda86e4f3d1f4f7bed6e4ee39fa522488d4003d"} Jul 07 22:04:03.873970 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:03.873950 2564 generic.go:356] "Generic (PLEG): container finished" podID="94b5ccd3-cdf2-4d52-af68-30f21b855887" containerID="1f212ca43b9ae4eefbaebb3adc83cb8d1e38ba3cb406165d51ea826aed1c7e43" exitCode=0 Jul 07 22:04:03.874065 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:03.874014 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9z8v8" event={"ID":"94b5ccd3-cdf2-4d52-af68-30f21b855887","Type":"ContainerDied","Data":"1f212ca43b9ae4eefbaebb3adc83cb8d1e38ba3cb406165d51ea826aed1c7e43"} Jul 07 22:04:03.876413 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:03.876389 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2gbtg" event={"ID":"b22ba61c-8e69-4913-acd7-b58c7a92c8d4","Type":"ContainerStarted","Data":"f3a7ab554a57ba301bf3bd1bc25f145b19a29ceff1b13e19ac4943801d64199b"} Jul 07 22:04:03.878989 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:03.878620 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-bnx62" podStartSLOduration=3.511478996 podStartE2EDuration="22.878605776s" totalImagesPullingTime="19.36712678s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 22:03:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:04:03.878098958 +0000 UTC m=+23.706228418" watchObservedRunningTime="2026-07-07 22:04:03.878605776 +0000 UTC m=+23.706735233" Jul 07 22:04:03.913361 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:03.913310 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-wpg82" podStartSLOduration=3.593992917 podStartE2EDuration="22.913291153s" totalImagesPullingTime="19.319298236s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 22:03:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:04:03.913046103 +0000 UTC m=+23.741175562" watchObservedRunningTime="2026-07-07 22:04:03.913291153 +0000 UTC m=+23.741420612" Jul 07 22:04:03.925993 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:03.925939 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-lc54x" podStartSLOduration=7.322158928 podStartE2EDuration="22.925922179s" totalImagesPullingTime="15.603763251s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 22:03:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:04:03.925651452 +0000 UTC m=+23.753780910" watchObservedRunningTime="2026-07-07 22:04:03.925922179 +0000 UTC m=+23.754051637" Jul 07 22:04:03.937987 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:03.937940 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-c9npj" podStartSLOduration=3.637555288 podStartE2EDuration="22.937923486s" totalImagesPullingTime="19.300368198s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 22:03:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:04:03.937882122 +0000 UTC m=+23.766011581" watchObservedRunningTime="2026-07-07 22:04:03.937923486 +0000 UTC m=+23.766052945" Jul 07 22:04:03.950874 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:03.950829 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-2gbtg" podStartSLOduration=3.640259648 podStartE2EDuration="22.950816163s" totalImagesPullingTime="19.310556515s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 22:03:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:04:03.950602539 +0000 UTC m=+23.778731999" watchObservedRunningTime="2026-07-07 22:04:03.950816163 +0000 UTC m=+23.778945621" Jul 07 22:04:04.359861 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:04.359837 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-lc54x" Jul 07 22:04:04.384072 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:04.384049 2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 07 22:04:04.603718 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:04.603560 2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-07T22:04:04.384065921Z","UUID":"2ea738a7-000a-4479-bd2a-08c3ec116cfd","Handler":null,"Name":"","Endpoint":""} Jul 07 22:04:04.605532 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:04.605513 2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 07 22:04:04.605686 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:04.605554 2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 07 22:04:04.701912 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:04.701879 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zlr8s" Jul 07 22:04:04.702194 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:04.701890 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mzzl4" Jul 07 22:04:04.702194 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:04:04.701997 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zlr8s" podUID="95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9" Jul 07 22:04:04.702194 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:04:04.702100 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mzzl4" podUID="acb69b34-f020-4c76-89c2-0ab5b6e6f3d5" Jul 07 22:04:04.885319 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:04.885239 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsfv2" event={"ID":"97032a35-9ecb-4f59-b494-505ec4c051f4","Type":"ContainerStarted","Data":"24382cd7b5fcfefd38bb32a1209e57e7646484e05da13fb5aa97528b044f0557"} Jul 07 22:04:04.888402 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:04.887288 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-llpjl" event={"ID":"1a20bbf4-728c-4d13-b2a1-846fdf8c5a2e","Type":"ContainerStarted","Data":"1071b9597dd587b361f37ff5fa2c35fb79a456e66f996c7ff95b8128781430e7"} Jul 07 22:04:05.702129 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:05.702059 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m7lhr" Jul 07 22:04:05.702270 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:04:05.702164 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m7lhr" podUID="6aef29a2-20f0-4804-951f-4bd876b4b8b5" Jul 07 22:04:05.891891 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:05.891683 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsfv2" event={"ID":"97032a35-9ecb-4f59-b494-505ec4c051f4","Type":"ContainerStarted","Data":"7a8f3c4360179ecfa9a2957bd0ff02544f59f6fee991d0bc88581e6b85c4f977"} Jul 07 22:04:05.939877 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:05.939834 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsfv2" podStartSLOduration=3.644228804 podStartE2EDuration="24.939817777s" totalImagesPullingTime="21.295588973s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 22:03:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:04:05.939727457 +0000 UTC m=+25.767856916" watchObservedRunningTime="2026-07-07 22:04:05.939817777 +0000 UTC m=+25.767947235" Jul 07 22:04:05.940700 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:05.940659 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-llpjl" podStartSLOduration=5.636062636 podStartE2EDuration="24.940647973s" totalImagesPullingTime="19.304585337s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 22:03:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:04:04.901468329 +0000 UTC m=+24.729597788" watchObservedRunningTime="2026-07-07 22:04:05.940647973 +0000 UTC m=+25.768777432" Jul 07 22:04:06.701873 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:06.701838 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mzzl4" Jul 07 22:04:06.702096 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:06.701837 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zlr8s" Jul 07 22:04:06.702096 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:04:06.701980 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mzzl4" podUID="acb69b34-f020-4c76-89c2-0ab5b6e6f3d5" Jul 07 22:04:06.702096 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:04:06.702045 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zlr8s" podUID="95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9" Jul 07 22:04:06.900226 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:06.900196 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4h8zm_4b4234f5-6cda-45ea-b559-f09bcad77232/ovn-acl-logging/0.log" Jul 07 22:04:06.900802 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:06.900619 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" event={"ID":"4b4234f5-6cda-45ea-b559-f09bcad77232","Type":"ContainerStarted","Data":"12eb7c97fa79eeb051fd33f75dc4323b2d56949896abf868d325441d45766484"} Jul 07 22:04:07.701894 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:07.701861 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m7lhr" Jul 07 22:04:07.702064 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:04:07.701968 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m7lhr" podUID="6aef29a2-20f0-4804-951f-4bd876b4b8b5" Jul 07 22:04:08.005939 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:08.005860 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-lc54x" Jul 07 22:04:08.006487 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:08.006409 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-lc54x" Jul 07 22:04:08.702255 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:08.702072 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mzzl4" Jul 07 22:04:08.702435 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:08.702136 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zlr8s" Jul 07 22:04:08.702435 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:04:08.702325 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mzzl4" podUID="acb69b34-f020-4c76-89c2-0ab5b6e6f3d5" Jul 07 22:04:08.702435 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:04:08.702420 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zlr8s" podUID="95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9" Jul 07 22:04:08.908177 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:08.908143 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4h8zm_4b4234f5-6cda-45ea-b559-f09bcad77232/ovn-acl-logging/0.log" Jul 07 22:04:08.908571 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:08.908538 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" event={"ID":"4b4234f5-6cda-45ea-b559-f09bcad77232","Type":"ContainerStarted","Data":"df822ba1c256067e639ba0eed4cccf3e4160f4376172ffe84456f168467ef6b9"} Jul 07 22:04:08.908826 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:08.908798 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:04:08.908931 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:08.908873 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:04:08.908980 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:08.908946 2564 scope.go:117] "RemoveContainer" containerID="70ebf17818025cd68926e62b6495e3ed02d65d9fece5b639d08a907533160283" Jul 07 22:04:08.910375 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:08.910349 2564 generic.go:356] "Generic (PLEG): container finished" podID="94b5ccd3-cdf2-4d52-af68-30f21b855887" containerID="0201acaf6636a04a6732b2150e0e248f95e6e5b5425938ae4484df8adbf9f4d3" exitCode=0 Jul 07 22:04:08.910502 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:08.910456 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9z8v8" event={"ID":"94b5ccd3-cdf2-4d52-af68-30f21b855887","Type":"ContainerDied","Data":"0201acaf6636a04a6732b2150e0e248f95e6e5b5425938ae4484df8adbf9f4d3"} Jul 07 22:04:08.911121 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:08.911106 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-lc54x" Jul 07 22:04:08.927068 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:08.927050 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:04:09.701874 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:09.701840 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m7lhr" Jul 07 22:04:09.702526 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:04:09.701997 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m7lhr" podUID="6aef29a2-20f0-4804-951f-4bd876b4b8b5" Jul 07 22:04:09.921062 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:09.921036 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4h8zm_4b4234f5-6cda-45ea-b559-f09bcad77232/ovn-acl-logging/0.log" Jul 07 22:04:09.921510 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:09.921403 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" event={"ID":"4b4234f5-6cda-45ea-b559-f09bcad77232","Type":"ContainerStarted","Data":"a67201a0d7090b0b287ef5133c24ea14304ee8d13c284a6d2961483e0fc1eb28"} Jul 07 22:04:09.922000 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:09.921630 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:04:09.939640 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:09.939614 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:04:09.952693 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:09.952604 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" podStartSLOduration=9.582152236 podStartE2EDuration="28.952586223s" totalImagesPullingTime="19.370433987s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 22:03:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:04:09.951593954 +0000 UTC m=+29.779723413" watchObservedRunningTime="2026-07-07 22:04:09.952586223 +0000 UTC m=+29.780715813" Jul 07 22:04:10.182168 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:10.181939 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mzzl4"] Jul 07 22:04:10.182287 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:10.182275 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mzzl4" Jul 07 22:04:10.182404 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:04:10.182380 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mzzl4" podUID="acb69b34-f020-4c76-89c2-0ab5b6e6f3d5" Jul 07 22:04:10.184710 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:10.184689 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zlr8s"] Jul 07 22:04:10.184828 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:10.184785 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zlr8s" Jul 07 22:04:10.184904 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:04:10.184880 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zlr8s" podUID="95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9" Jul 07 22:04:10.185426 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:10.185397 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-m7lhr"] Jul 07 22:04:10.185527 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:10.185514 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m7lhr" Jul 07 22:04:10.185633 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:04:10.185617 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m7lhr" podUID="6aef29a2-20f0-4804-951f-4bd876b4b8b5" Jul 07 22:04:10.925215 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:10.925182 2564 generic.go:356] "Generic (PLEG): container finished" podID="94b5ccd3-cdf2-4d52-af68-30f21b855887" containerID="1b3a3e1c6a492833bf3b1b5519fccce6f57224b55d80f2bae3581cb657b7d7c4" exitCode=0 Jul 07 22:04:10.925862 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:10.925274 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9z8v8" event={"ID":"94b5ccd3-cdf2-4d52-af68-30f21b855887","Type":"ContainerDied","Data":"1b3a3e1c6a492833bf3b1b5519fccce6f57224b55d80f2bae3581cb657b7d7c4"} Jul 07 22:04:11.701392 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:11.701357 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mzzl4" Jul 07 22:04:11.701392 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:11.701376 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m7lhr" Jul 07 22:04:11.701606 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:11.701376 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zlr8s" Jul 07 22:04:11.701606 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:04:11.701477 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mzzl4" podUID="acb69b34-f020-4c76-89c2-0ab5b6e6f3d5" Jul 07 22:04:11.701606 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:04:11.701539 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m7lhr" podUID="6aef29a2-20f0-4804-951f-4bd876b4b8b5" Jul 07 22:04:11.701606 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:04:11.701591 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zlr8s" podUID="95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9" Jul 07 22:04:12.931623 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:12.931586 2564 generic.go:356] "Generic (PLEG): container finished" podID="94b5ccd3-cdf2-4d52-af68-30f21b855887" containerID="a89c1e0708db8352a67c0e78b0487cf84a9dc44639650400cebd2818fa2879e2" exitCode=0 Jul 07 22:04:12.932164 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:12.931640 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9z8v8" event={"ID":"94b5ccd3-cdf2-4d52-af68-30f21b855887","Type":"ContainerDied","Data":"a89c1e0708db8352a67c0e78b0487cf84a9dc44639650400cebd2818fa2879e2"} Jul 07 22:04:13.701727 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:13.701546 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m7lhr" Jul 07 22:04:13.701885 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:13.701574 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mzzl4" Jul 07 22:04:13.701885 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:04:13.701822 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m7lhr" podUID="6aef29a2-20f0-4804-951f-4bd876b4b8b5" Jul 07 22:04:13.701885 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:04:13.701873 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mzzl4" podUID="acb69b34-f020-4c76-89c2-0ab5b6e6f3d5" Jul 07 22:04:13.701986 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:13.701594 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zlr8s" Jul 07 22:04:13.701986 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:04:13.701954 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zlr8s" podUID="95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9" Jul 07 22:04:14.387394 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:14.387353 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6aef29a2-20f0-4804-951f-4bd876b4b8b5-metrics-certs\") pod \"network-metrics-daemon-m7lhr\" (UID: \"6aef29a2-20f0-4804-951f-4bd876b4b8b5\") " pod="openshift-multus/network-metrics-daemon-m7lhr" Jul 07 22:04:14.387840 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:14.387457 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/acb69b34-f020-4c76-89c2-0ab5b6e6f3d5-original-pull-secret\") pod \"global-pull-secret-syncer-mzzl4\" (UID: \"acb69b34-f020-4c76-89c2-0ab5b6e6f3d5\") " pod="kube-system/global-pull-secret-syncer-mzzl4" Jul 07 22:04:14.388217 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:04:14.388190 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 22:04:14.388217 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:04:14.388209 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 22:04:14.388478 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:04:14.388289 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6aef29a2-20f0-4804-951f-4bd876b4b8b5-metrics-certs podName:6aef29a2-20f0-4804-951f-4bd876b4b8b5 nodeName:}" failed. No retries permitted until 2026-07-07 22:04:46.388255115 +0000 UTC m=+66.216384553 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6aef29a2-20f0-4804-951f-4bd876b4b8b5-metrics-certs") pod "network-metrics-daemon-m7lhr" (UID: "6aef29a2-20f0-4804-951f-4bd876b4b8b5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 22:04:14.388478 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:04:14.388310 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acb69b34-f020-4c76-89c2-0ab5b6e6f3d5-original-pull-secret podName:acb69b34-f020-4c76-89c2-0ab5b6e6f3d5 nodeName:}" failed. No retries permitted until 2026-07-07 22:04:46.388299717 +0000 UTC m=+66.216429158 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/acb69b34-f020-4c76-89c2-0ab5b6e6f3d5-original-pull-secret") pod "global-pull-secret-syncer-mzzl4" (UID: "acb69b34-f020-4c76-89c2-0ab5b6e6f3d5") : object "kube-system"/"original-pull-secret" not registered Jul 07 22:04:14.488463 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:14.488410 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj8mx\" (UniqueName: \"kubernetes.io/projected/95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9-kube-api-access-bj8mx\") pod \"network-check-target-zlr8s\" (UID: \"95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9\") " pod="openshift-network-diagnostics/network-check-target-zlr8s" Jul 07 22:04:14.488618 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:04:14.488589 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 22:04:14.488668 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:04:14.488618 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 22:04:14.488668 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:04:14.488632 2564 projected.go:196] Error preparing data for projected volume kube-api-access-bj8mx for pod openshift-network-diagnostics/network-check-target-zlr8s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 22:04:14.488760 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:04:14.488700 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9-kube-api-access-bj8mx podName:95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9 nodeName:}" failed. No retries permitted until 2026-07-07 22:04:46.488680503 +0000 UTC m=+66.316809954 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-bj8mx" (UniqueName: "kubernetes.io/projected/95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9-kube-api-access-bj8mx") pod "network-check-target-zlr8s" (UID: "95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 22:04:15.702274 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:15.702238 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zlr8s" Jul 07 22:04:15.702803 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:15.702284 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m7lhr" Jul 07 22:04:15.702803 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:15.702284 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mzzl4" Jul 07 22:04:15.702803 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:04:15.702364 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zlr8s" podUID="95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9" Jul 07 22:04:15.702803 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:04:15.702468 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m7lhr" podUID="6aef29a2-20f0-4804-951f-4bd876b4b8b5" Jul 07 22:04:15.702803 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:04:15.702537 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mzzl4" podUID="acb69b34-f020-4c76-89c2-0ab5b6e6f3d5" Jul 07 22:04:15.926723 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:15.926695 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-156.ec2.internal" event="NodeReady" Jul 07 22:04:15.926891 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:15.926855 2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 07 22:04:15.982567 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:15.982478 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-2bkmt"] Jul 07 22:04:15.985128 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:15.985100 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-hbqkg"] Jul 07 22:04:15.985252 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:15.985209 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2bkmt" Jul 07 22:04:15.987354 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:15.987325 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hbqkg" Jul 07 22:04:15.988021 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:15.987626 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 07 22:04:15.988021 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:15.987697 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 07 22:04:15.988021 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:15.987705 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-8gs5j\"" Jul 07 22:04:15.988021 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:15.987981 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 07 22:04:15.989574 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:15.989478 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 07 22:04:15.989574 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:15.989526 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 07 22:04:15.989574 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:15.989532 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kp4p7\"" Jul 07 22:04:15.999573 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:15.999552 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2bkmt"] Jul 07 22:04:16.000347 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:16.000328 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hbqkg"] Jul 07 22:04:16.102647 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:16.102607 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9becf7d9-4d28-4ae3-a0df-c7b1a87ddf8c-cert\") pod \"ingress-canary-2bkmt\" (UID: \"9becf7d9-4d28-4ae3-a0df-c7b1a87ddf8c\") " pod="openshift-ingress-canary/ingress-canary-2bkmt" Jul 07 22:04:16.102831 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:16.102663 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/20b4d981-ad57-4ff9-b293-48c8f4a6f43a-metrics-tls\") pod \"dns-default-hbqkg\" (UID: \"20b4d981-ad57-4ff9-b293-48c8f4a6f43a\") " pod="openshift-dns/dns-default-hbqkg" Jul 07 22:04:16.102831 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:16.102725 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20b4d981-ad57-4ff9-b293-48c8f4a6f43a-tmp-dir\") pod \"dns-default-hbqkg\" (UID: \"20b4d981-ad57-4ff9-b293-48c8f4a6f43a\") " pod="openshift-dns/dns-default-hbqkg" Jul 07 22:04:16.102831 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:16.102763 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn7gw\" (UniqueName: \"kubernetes.io/projected/20b4d981-ad57-4ff9-b293-48c8f4a6f43a-kube-api-access-kn7gw\") pod \"dns-default-hbqkg\" (UID: \"20b4d981-ad57-4ff9-b293-48c8f4a6f43a\") " pod="openshift-dns/dns-default-hbqkg" Jul 07 22:04:16.102831 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:16.102810 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhrm2\" (UniqueName: \"kubernetes.io/projected/9becf7d9-4d28-4ae3-a0df-c7b1a87ddf8c-kube-api-access-vhrm2\") pod \"ingress-canary-2bkmt\" (UID: \"9becf7d9-4d28-4ae3-a0df-c7b1a87ddf8c\") " pod="openshift-ingress-canary/ingress-canary-2bkmt" Jul 07 22:04:16.102986 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:16.102860 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/20b4d981-ad57-4ff9-b293-48c8f4a6f43a-config-volume\") pod \"dns-default-hbqkg\" (UID: \"20b4d981-ad57-4ff9-b293-48c8f4a6f43a\") " pod="openshift-dns/dns-default-hbqkg" Jul 07 22:04:16.203928 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:16.203894 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/20b4d981-ad57-4ff9-b293-48c8f4a6f43a-config-volume\") pod \"dns-default-hbqkg\" (UID: \"20b4d981-ad57-4ff9-b293-48c8f4a6f43a\") " pod="openshift-dns/dns-default-hbqkg" Jul 07 22:04:16.204111 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:16.203975 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9becf7d9-4d28-4ae3-a0df-c7b1a87ddf8c-cert\") pod \"ingress-canary-2bkmt\" (UID: \"9becf7d9-4d28-4ae3-a0df-c7b1a87ddf8c\") " pod="openshift-ingress-canary/ingress-canary-2bkmt" Jul 07 22:04:16.204111 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:16.204018 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/20b4d981-ad57-4ff9-b293-48c8f4a6f43a-metrics-tls\") pod \"dns-default-hbqkg\" (UID: \"20b4d981-ad57-4ff9-b293-48c8f4a6f43a\") " pod="openshift-dns/dns-default-hbqkg" Jul 07 22:04:16.204111 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:16.204047 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20b4d981-ad57-4ff9-b293-48c8f4a6f43a-tmp-dir\") pod \"dns-default-hbqkg\" (UID: \"20b4d981-ad57-4ff9-b293-48c8f4a6f43a\") " pod="openshift-dns/dns-default-hbqkg" Jul 07 22:04:16.204111 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:16.204068 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn7gw\" (UniqueName: \"kubernetes.io/projected/20b4d981-ad57-4ff9-b293-48c8f4a6f43a-kube-api-access-kn7gw\") pod \"dns-default-hbqkg\" (UID: \"20b4d981-ad57-4ff9-b293-48c8f4a6f43a\") " pod="openshift-dns/dns-default-hbqkg" Jul 07 22:04:16.204111 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:16.204108 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhrm2\" (UniqueName: \"kubernetes.io/projected/9becf7d9-4d28-4ae3-a0df-c7b1a87ddf8c-kube-api-access-vhrm2\") pod \"ingress-canary-2bkmt\" (UID: \"9becf7d9-4d28-4ae3-a0df-c7b1a87ddf8c\") " pod="openshift-ingress-canary/ingress-canary-2bkmt" Jul 07 22:04:16.204533 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:04:16.204508 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 22:04:16.204646 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:04:16.204587 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20b4d981-ad57-4ff9-b293-48c8f4a6f43a-metrics-tls podName:20b4d981-ad57-4ff9-b293-48c8f4a6f43a nodeName:}" failed. No retries permitted until 2026-07-07 22:04:16.704564589 +0000 UTC m=+36.532694041 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/20b4d981-ad57-4ff9-b293-48c8f4a6f43a-metrics-tls") pod "dns-default-hbqkg" (UID: "20b4d981-ad57-4ff9-b293-48c8f4a6f43a") : secret "dns-default-metrics-tls" not found Jul 07 22:04:16.204780 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:16.204747 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20b4d981-ad57-4ff9-b293-48c8f4a6f43a-tmp-dir\") pod \"dns-default-hbqkg\" (UID: \"20b4d981-ad57-4ff9-b293-48c8f4a6f43a\") " pod="openshift-dns/dns-default-hbqkg" Jul 07 22:04:16.204946 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:16.204929 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/20b4d981-ad57-4ff9-b293-48c8f4a6f43a-config-volume\") pod \"dns-default-hbqkg\" (UID: \"20b4d981-ad57-4ff9-b293-48c8f4a6f43a\") " pod="openshift-dns/dns-default-hbqkg" Jul 07 22:04:16.209550 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:16.209528 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9becf7d9-4d28-4ae3-a0df-c7b1a87ddf8c-cert\") pod \"ingress-canary-2bkmt\" (UID: \"9becf7d9-4d28-4ae3-a0df-c7b1a87ddf8c\") " pod="openshift-ingress-canary/ingress-canary-2bkmt" Jul 07 22:04:16.212679 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:16.212658 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn7gw\" (UniqueName: \"kubernetes.io/projected/20b4d981-ad57-4ff9-b293-48c8f4a6f43a-kube-api-access-kn7gw\") pod \"dns-default-hbqkg\" (UID: \"20b4d981-ad57-4ff9-b293-48c8f4a6f43a\") " pod="openshift-dns/dns-default-hbqkg" Jul 07 22:04:16.212786 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:16.212723 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhrm2\" (UniqueName: \"kubernetes.io/projected/9becf7d9-4d28-4ae3-a0df-c7b1a87ddf8c-kube-api-access-vhrm2\") pod \"ingress-canary-2bkmt\" (UID: \"9becf7d9-4d28-4ae3-a0df-c7b1a87ddf8c\") " pod="openshift-ingress-canary/ingress-canary-2bkmt" Jul 07 22:04:16.299768 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:16.299733 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2bkmt" Jul 07 22:04:16.708373 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:16.708302 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/20b4d981-ad57-4ff9-b293-48c8f4a6f43a-metrics-tls\") pod \"dns-default-hbqkg\" (UID: \"20b4d981-ad57-4ff9-b293-48c8f4a6f43a\") " pod="openshift-dns/dns-default-hbqkg" Jul 07 22:04:16.708776 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:04:16.708431 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 22:04:16.708776 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:04:16.708509 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20b4d981-ad57-4ff9-b293-48c8f4a6f43a-metrics-tls podName:20b4d981-ad57-4ff9-b293-48c8f4a6f43a nodeName:}" failed. No retries permitted until 2026-07-07 22:04:17.708494123 +0000 UTC m=+37.536623558 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/20b4d981-ad57-4ff9-b293-48c8f4a6f43a-metrics-tls") pod "dns-default-hbqkg" (UID: "20b4d981-ad57-4ff9-b293-48c8f4a6f43a") : secret "dns-default-metrics-tls" not found Jul 07 22:04:17.160482 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:17.160306 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2bkmt"] Jul 07 22:04:17.166341 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:04:17.166317 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9becf7d9_4d28_4ae3_a0df_c7b1a87ddf8c.slice/crio-98c349c35418cb5c0245587dd96c9a333966a54fa8be21107e66081ad4f2b17b WatchSource:0}: Error finding container 98c349c35418cb5c0245587dd96c9a333966a54fa8be21107e66081ad4f2b17b: Status 404 returned error can't find the container with id 98c349c35418cb5c0245587dd96c9a333966a54fa8be21107e66081ad4f2b17b Jul 07 22:04:17.701579 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:17.701543 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m7lhr" Jul 07 22:04:17.701774 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:17.701672 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mzzl4" Jul 07 22:04:17.701774 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:17.701707 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zlr8s" Jul 07 22:04:17.705182 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:17.705141 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hdlmc\"" Jul 07 22:04:17.705182 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:17.705161 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 07 22:04:17.705182 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:17.705154 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-p2w6w\"" Jul 07 22:04:17.705384 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:17.705198 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 07 22:04:17.705384 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:17.705159 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 07 22:04:17.705384 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:17.705145 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 07 22:04:17.715010 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:17.714988 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/20b4d981-ad57-4ff9-b293-48c8f4a6f43a-metrics-tls\") pod \"dns-default-hbqkg\" (UID: \"20b4d981-ad57-4ff9-b293-48c8f4a6f43a\") " pod="openshift-dns/dns-default-hbqkg" Jul 07 22:04:17.715273 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:04:17.715113 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 22:04:17.715273 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:04:17.715173 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20b4d981-ad57-4ff9-b293-48c8f4a6f43a-metrics-tls podName:20b4d981-ad57-4ff9-b293-48c8f4a6f43a nodeName:}" failed. No retries permitted until 2026-07-07 22:04:19.715156711 +0000 UTC m=+39.543286147 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/20b4d981-ad57-4ff9-b293-48c8f4a6f43a-metrics-tls") pod "dns-default-hbqkg" (UID: "20b4d981-ad57-4ff9-b293-48c8f4a6f43a") : secret "dns-default-metrics-tls" not found Jul 07 22:04:17.947815 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:17.947778 2564 generic.go:356] "Generic (PLEG): container finished" podID="94b5ccd3-cdf2-4d52-af68-30f21b855887" containerID="763ed6028c1489ccb848bf2b20dd97174fc28e2817e748fe8f9eb5e05e61ef5b" exitCode=0 Jul 07 22:04:17.947958 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:17.947819 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9z8v8" event={"ID":"94b5ccd3-cdf2-4d52-af68-30f21b855887","Type":"ContainerDied","Data":"763ed6028c1489ccb848bf2b20dd97174fc28e2817e748fe8f9eb5e05e61ef5b"} Jul 07 22:04:17.949724 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:17.949697 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2bkmt" event={"ID":"9becf7d9-4d28-4ae3-a0df-c7b1a87ddf8c","Type":"ContainerStarted","Data":"98c349c35418cb5c0245587dd96c9a333966a54fa8be21107e66081ad4f2b17b"} Jul 07 22:04:18.955707 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:18.955668 2564 generic.go:356] "Generic (PLEG): container finished" podID="94b5ccd3-cdf2-4d52-af68-30f21b855887" containerID="30bca1330c1854674d6d8f5a3e2039b164731a92bf9933900a9c439cf46c7f89" exitCode=0 Jul 07 22:04:18.956354 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:18.955723 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9z8v8" event={"ID":"94b5ccd3-cdf2-4d52-af68-30f21b855887","Type":"ContainerDied","Data":"30bca1330c1854674d6d8f5a3e2039b164731a92bf9933900a9c439cf46c7f89"} Jul 07 22:04:19.730481 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:19.730435 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/20b4d981-ad57-4ff9-b293-48c8f4a6f43a-metrics-tls\") pod \"dns-default-hbqkg\" (UID: \"20b4d981-ad57-4ff9-b293-48c8f4a6f43a\") " pod="openshift-dns/dns-default-hbqkg" Jul 07 22:04:19.730604 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:04:19.730587 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 22:04:19.730654 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:04:19.730646 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20b4d981-ad57-4ff9-b293-48c8f4a6f43a-metrics-tls podName:20b4d981-ad57-4ff9-b293-48c8f4a6f43a nodeName:}" failed. No retries permitted until 2026-07-07 22:04:23.73063122 +0000 UTC m=+43.558760656 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/20b4d981-ad57-4ff9-b293-48c8f4a6f43a-metrics-tls") pod "dns-default-hbqkg" (UID: "20b4d981-ad57-4ff9-b293-48c8f4a6f43a") : secret "dns-default-metrics-tls" not found Jul 07 22:04:19.962536 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:19.962504 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9z8v8" event={"ID":"94b5ccd3-cdf2-4d52-af68-30f21b855887","Type":"ContainerStarted","Data":"5d8026e746100434032c303cde355d1d936bfcf81b27bc9b5b6f9097a3b4ca0c"} Jul 07 22:04:19.984994 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:19.984938 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9z8v8" podStartSLOduration=9.206410418 podStartE2EDuration="38.984919383s" totalImagesPullingTime="29.778508965s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 22:03:41 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:04:19.983594477 +0000 UTC m=+39.811723935" watchObservedRunningTime="2026-07-07 22:04:19.984919383 +0000 UTC m=+39.813048842" Jul 07 22:04:20.966848 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:20.966804 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2bkmt" event={"ID":"9becf7d9-4d28-4ae3-a0df-c7b1a87ddf8c","Type":"ContainerStarted","Data":"68aa2a6f84d49065d6f2249e5d7d5353df29cb8ed58713dcbb425248ced5e6a1"} Jul 07 22:04:20.981995 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:20.981946 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-2bkmt" podStartSLOduration=3.227044002 podStartE2EDuration="5.9819332s" totalImagesPullingTime="2.754889198s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 22:04:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:04:20.981300431 +0000 UTC m=+40.809429890" watchObservedRunningTime="2026-07-07 22:04:20.9819332 +0000 UTC m=+40.810062657" Jul 07 22:04:23.146924 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:23.146891 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-88k64"] Jul 07 22:04:23.181291 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:23.181265 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-88k64"] Jul 07 22:04:23.181417 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:23.181397 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-88k64" Jul 07 22:04:23.183437 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:23.183408 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 07 22:04:23.183660 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:23.183642 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 07 22:04:23.183660 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:23.183657 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 07 22:04:23.184457 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:23.184430 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bwh5s\"" Jul 07 22:04:23.184563 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:23.184436 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 07 22:04:23.255335 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:23.255304 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3fecd654-8a55-4c3d-91f6-dca0be8172e2-data-volume\") pod \"insights-runtime-extractor-88k64\" (UID: \"3fecd654-8a55-4c3d-91f6-dca0be8172e2\") " pod="openshift-insights/insights-runtime-extractor-88k64" Jul 07 22:04:23.255335 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:23.255336 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whq55\" (UniqueName: \"kubernetes.io/projected/3fecd654-8a55-4c3d-91f6-dca0be8172e2-kube-api-access-whq55\") pod \"insights-runtime-extractor-88k64\" (UID: \"3fecd654-8a55-4c3d-91f6-dca0be8172e2\") " pod="openshift-insights/insights-runtime-extractor-88k64" Jul 07 22:04:23.255542 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:23.255359 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3fecd654-8a55-4c3d-91f6-dca0be8172e2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-88k64\" (UID: \"3fecd654-8a55-4c3d-91f6-dca0be8172e2\") " pod="openshift-insights/insights-runtime-extractor-88k64" Jul 07 22:04:23.255542 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:23.255528 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3fecd654-8a55-4c3d-91f6-dca0be8172e2-crio-socket\") pod \"insights-runtime-extractor-88k64\" (UID: \"3fecd654-8a55-4c3d-91f6-dca0be8172e2\") " pod="openshift-insights/insights-runtime-extractor-88k64" Jul 07 22:04:23.255616 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:23.255556 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3fecd654-8a55-4c3d-91f6-dca0be8172e2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-88k64\" (UID: \"3fecd654-8a55-4c3d-91f6-dca0be8172e2\") " pod="openshift-insights/insights-runtime-extractor-88k64" Jul 07 22:04:23.356614 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:23.356586 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3fecd654-8a55-4c3d-91f6-dca0be8172e2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-88k64\" (UID: \"3fecd654-8a55-4c3d-91f6-dca0be8172e2\") " pod="openshift-insights/insights-runtime-extractor-88k64" Jul 07 22:04:23.356723 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:23.356651 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3fecd654-8a55-4c3d-91f6-dca0be8172e2-crio-socket\") pod \"insights-runtime-extractor-88k64\" (UID: \"3fecd654-8a55-4c3d-91f6-dca0be8172e2\") " pod="openshift-insights/insights-runtime-extractor-88k64" Jul 07 22:04:23.356723 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:23.356678 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3fecd654-8a55-4c3d-91f6-dca0be8172e2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-88k64\" (UID: \"3fecd654-8a55-4c3d-91f6-dca0be8172e2\") " pod="openshift-insights/insights-runtime-extractor-88k64" Jul 07 22:04:23.356723 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:23.356717 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3fecd654-8a55-4c3d-91f6-dca0be8172e2-data-volume\") pod \"insights-runtime-extractor-88k64\" (UID: \"3fecd654-8a55-4c3d-91f6-dca0be8172e2\") " pod="openshift-insights/insights-runtime-extractor-88k64" Jul 07 22:04:23.356857 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:04:23.356736 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 22:04:23.356857 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:04:23.356795 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 22:04:23.356857 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:04:23.356815 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fecd654-8a55-4c3d-91f6-dca0be8172e2-insights-runtime-extractor-tls podName:3fecd654-8a55-4c3d-91f6-dca0be8172e2 nodeName:}" failed. No retries permitted until 2026-07-07 22:04:23.856788519 +0000 UTC m=+43.684917961 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3fecd654-8a55-4c3d-91f6-dca0be8172e2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-88k64" (UID: "3fecd654-8a55-4c3d-91f6-dca0be8172e2") : secret "insights-runtime-extractor-tls" not found Jul 07 22:04:23.356857 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:23.356743 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whq55\" (UniqueName: \"kubernetes.io/projected/3fecd654-8a55-4c3d-91f6-dca0be8172e2-kube-api-access-whq55\") pod \"insights-runtime-extractor-88k64\" (UID: \"3fecd654-8a55-4c3d-91f6-dca0be8172e2\") " pod="openshift-insights/insights-runtime-extractor-88k64" Jul 07 22:04:23.357060 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:04:23.356864 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3fecd654-8a55-4c3d-91f6-dca0be8172e2-kube-rbac-proxy-cm podName:3fecd654-8a55-4c3d-91f6-dca0be8172e2 nodeName:}" failed. No retries permitted until 2026-07-07 22:04:23.856847669 +0000 UTC m=+43.684977130 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3fecd654-8a55-4c3d-91f6-dca0be8172e2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-88k64" (UID: "3fecd654-8a55-4c3d-91f6-dca0be8172e2") : configmap "kube-rbac-proxy" not found Jul 07 22:04:23.357060 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:23.356916 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3fecd654-8a55-4c3d-91f6-dca0be8172e2-crio-socket\") pod \"insights-runtime-extractor-88k64\" (UID: \"3fecd654-8a55-4c3d-91f6-dca0be8172e2\") " pod="openshift-insights/insights-runtime-extractor-88k64" Jul 07 22:04:23.357139 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:23.357075 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3fecd654-8a55-4c3d-91f6-dca0be8172e2-data-volume\") pod \"insights-runtime-extractor-88k64\" (UID: \"3fecd654-8a55-4c3d-91f6-dca0be8172e2\") " pod="openshift-insights/insights-runtime-extractor-88k64" Jul 07 22:04:23.368753 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:23.368726 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whq55\" (UniqueName: \"kubernetes.io/projected/3fecd654-8a55-4c3d-91f6-dca0be8172e2-kube-api-access-whq55\") pod \"insights-runtime-extractor-88k64\" (UID: \"3fecd654-8a55-4c3d-91f6-dca0be8172e2\") " pod="openshift-insights/insights-runtime-extractor-88k64" Jul 07 22:04:23.608035 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:23.608001 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-dnztd"] Jul 07 22:04:23.625614 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:23.625591 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-dnztd"] Jul 07 22:04:23.625742 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:23.625705 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-dnztd" Jul 07 22:04:23.627931 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:23.627912 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 07 22:04:23.628016 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:23.627914 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 07 22:04:23.628192 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:23.628175 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-92lgs\"" Jul 07 22:04:23.759665 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:23.759632 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/20b4d981-ad57-4ff9-b293-48c8f4a6f43a-metrics-tls\") pod \"dns-default-hbqkg\" (UID: \"20b4d981-ad57-4ff9-b293-48c8f4a6f43a\") " pod="openshift-dns/dns-default-hbqkg" Jul 07 22:04:23.759848 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:23.759698 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9vh2\" (UniqueName: \"kubernetes.io/projected/5cd8d4d5-75db-49f2-b8e3-4291e58ec967-kube-api-access-v9vh2\") pod \"migrator-d57d58cc9-dnztd\" (UID: \"5cd8d4d5-75db-49f2-b8e3-4291e58ec967\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-dnztd" Jul 07 22:04:23.759848 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:04:23.759811 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 22:04:23.759971 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:04:23.759894 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20b4d981-ad57-4ff9-b293-48c8f4a6f43a-metrics-tls podName:20b4d981-ad57-4ff9-b293-48c8f4a6f43a nodeName:}" failed. No retries permitted until 2026-07-07 22:04:31.759873795 +0000 UTC m=+51.588003231 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/20b4d981-ad57-4ff9-b293-48c8f4a6f43a-metrics-tls") pod "dns-default-hbqkg" (UID: "20b4d981-ad57-4ff9-b293-48c8f4a6f43a") : secret "dns-default-metrics-tls" not found Jul 07 22:04:23.860799 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:23.860714 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9vh2\" (UniqueName: \"kubernetes.io/projected/5cd8d4d5-75db-49f2-b8e3-4291e58ec967-kube-api-access-v9vh2\") pod \"migrator-d57d58cc9-dnztd\" (UID: \"5cd8d4d5-75db-49f2-b8e3-4291e58ec967\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-dnztd" Jul 07 22:04:23.860799 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:23.860759 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3fecd654-8a55-4c3d-91f6-dca0be8172e2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-88k64\" (UID: \"3fecd654-8a55-4c3d-91f6-dca0be8172e2\") " pod="openshift-insights/insights-runtime-extractor-88k64" Jul 07 22:04:23.861015 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:23.860818 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3fecd654-8a55-4c3d-91f6-dca0be8172e2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-88k64\" (UID: \"3fecd654-8a55-4c3d-91f6-dca0be8172e2\") " pod="openshift-insights/insights-runtime-extractor-88k64" Jul 07 22:04:23.861015 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:04:23.860908 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 22:04:23.861015 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:04:23.860931 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 22:04:23.861015 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:04:23.860969 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3fecd654-8a55-4c3d-91f6-dca0be8172e2-kube-rbac-proxy-cm podName:3fecd654-8a55-4c3d-91f6-dca0be8172e2 nodeName:}" failed. No retries permitted until 2026-07-07 22:04:24.860955767 +0000 UTC m=+44.689085203 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3fecd654-8a55-4c3d-91f6-dca0be8172e2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-88k64" (UID: "3fecd654-8a55-4c3d-91f6-dca0be8172e2") : configmap "kube-rbac-proxy" not found Jul 07 22:04:23.861015 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:04:23.860987 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fecd654-8a55-4c3d-91f6-dca0be8172e2-insights-runtime-extractor-tls podName:3fecd654-8a55-4c3d-91f6-dca0be8172e2 nodeName:}" failed. No retries permitted until 2026-07-07 22:04:24.860974271 +0000 UTC m=+44.689103710 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3fecd654-8a55-4c3d-91f6-dca0be8172e2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-88k64" (UID: "3fecd654-8a55-4c3d-91f6-dca0be8172e2") : secret "insights-runtime-extractor-tls" not found Jul 07 22:04:23.868193 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:23.868164 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9vh2\" (UniqueName: \"kubernetes.io/projected/5cd8d4d5-75db-49f2-b8e3-4291e58ec967-kube-api-access-v9vh2\") pod \"migrator-d57d58cc9-dnztd\" (UID: \"5cd8d4d5-75db-49f2-b8e3-4291e58ec967\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-dnztd" Jul 07 22:04:23.934275 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:23.934250 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-dnztd" Jul 07 22:04:24.073975 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:24.073936 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-dnztd"] Jul 07 22:04:24.077697 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:04:24.077670 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cd8d4d5_75db_49f2_b8e3_4291e58ec967.slice/crio-99e16eff8f12489d03caeb96e9551c7974bf989c9e35906ef1bc7c49ffebc744 WatchSource:0}: Error finding container 99e16eff8f12489d03caeb96e9551c7974bf989c9e35906ef1bc7c49ffebc744: Status 404 returned error can't find the container with id 99e16eff8f12489d03caeb96e9551c7974bf989c9e35906ef1bc7c49ffebc744 Jul 07 22:04:24.868337 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:24.868297 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3fecd654-8a55-4c3d-91f6-dca0be8172e2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-88k64\" (UID: \"3fecd654-8a55-4c3d-91f6-dca0be8172e2\") " pod="openshift-insights/insights-runtime-extractor-88k64" Jul 07 22:04:24.868784 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:24.868374 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3fecd654-8a55-4c3d-91f6-dca0be8172e2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-88k64\" (UID: \"3fecd654-8a55-4c3d-91f6-dca0be8172e2\") " pod="openshift-insights/insights-runtime-extractor-88k64" Jul 07 22:04:24.868784 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:04:24.868435 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 22:04:24.868784 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:04:24.868527 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3fecd654-8a55-4c3d-91f6-dca0be8172e2-kube-rbac-proxy-cm podName:3fecd654-8a55-4c3d-91f6-dca0be8172e2 nodeName:}" failed. No retries permitted until 2026-07-07 22:04:26.86850431 +0000 UTC m=+46.696633767 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3fecd654-8a55-4c3d-91f6-dca0be8172e2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-88k64" (UID: "3fecd654-8a55-4c3d-91f6-dca0be8172e2") : configmap "kube-rbac-proxy" not found Jul 07 22:04:24.868784 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:04:24.868526 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 22:04:24.868784 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:04:24.868583 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fecd654-8a55-4c3d-91f6-dca0be8172e2-insights-runtime-extractor-tls podName:3fecd654-8a55-4c3d-91f6-dca0be8172e2 nodeName:}" failed. No retries permitted until 2026-07-07 22:04:26.868567859 +0000 UTC m=+46.696697299 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3fecd654-8a55-4c3d-91f6-dca0be8172e2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-88k64" (UID: "3fecd654-8a55-4c3d-91f6-dca0be8172e2") : secret "insights-runtime-extractor-tls" not found Jul 07 22:04:24.929405 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:24.929377 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2gbtg_b22ba61c-8e69-4913-acd7-b58c7a92c8d4/dns-node-resolver/0.log" Jul 07 22:04:24.977553 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:24.977529 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-dnztd" event={"ID":"5cd8d4d5-75db-49f2-b8e3-4291e58ec967","Type":"ContainerStarted","Data":"99e16eff8f12489d03caeb96e9551c7974bf989c9e35906ef1bc7c49ffebc744"} Jul 07 22:04:25.386891 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:25.386844 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-6499p"] Jul 07 22:04:25.417724 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:25.417690 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-6499p"] Jul 07 22:04:25.417860 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:25.417791 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-6499p" Jul 07 22:04:25.420706 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:25.420680 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 07 22:04:25.420865 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:25.420744 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 07 22:04:25.420865 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:25.420745 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 07 22:04:25.421407 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:25.421387 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-92lps\"" Jul 07 22:04:25.421550 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:25.421478 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 07 22:04:25.573786 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:25.573756 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/61b82340-8090-479c-b3c0-e69658f30bfb-signing-cabundle\") pod \"service-ca-758f599c5-6499p\" (UID: \"61b82340-8090-479c-b3c0-e69658f30bfb\") " pod="openshift-service-ca/service-ca-758f599c5-6499p" Jul 07 22:04:25.573929 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:25.573802 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h5m6\" (UniqueName: \"kubernetes.io/projected/61b82340-8090-479c-b3c0-e69658f30bfb-kube-api-access-6h5m6\") pod \"service-ca-758f599c5-6499p\" (UID: \"61b82340-8090-479c-b3c0-e69658f30bfb\") " pod="openshift-service-ca/service-ca-758f599c5-6499p" Jul 07 22:04:25.573929 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:25.573914 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/61b82340-8090-479c-b3c0-e69658f30bfb-signing-key\") pod \"service-ca-758f599c5-6499p\" (UID: \"61b82340-8090-479c-b3c0-e69658f30bfb\") " pod="openshift-service-ca/service-ca-758f599c5-6499p" Jul 07 22:04:25.675180 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:25.675113 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/61b82340-8090-479c-b3c0-e69658f30bfb-signing-cabundle\") pod \"service-ca-758f599c5-6499p\" (UID: \"61b82340-8090-479c-b3c0-e69658f30bfb\") " pod="openshift-service-ca/service-ca-758f599c5-6499p" Jul 07 22:04:25.675180 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:25.675154 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h5m6\" (UniqueName: \"kubernetes.io/projected/61b82340-8090-479c-b3c0-e69658f30bfb-kube-api-access-6h5m6\") pod \"service-ca-758f599c5-6499p\" (UID: \"61b82340-8090-479c-b3c0-e69658f30bfb\") " pod="openshift-service-ca/service-ca-758f599c5-6499p" Jul 07 22:04:25.675347 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:25.675223 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/61b82340-8090-479c-b3c0-e69658f30bfb-signing-key\") pod \"service-ca-758f599c5-6499p\" (UID: \"61b82340-8090-479c-b3c0-e69658f30bfb\") " pod="openshift-service-ca/service-ca-758f599c5-6499p" Jul 07 22:04:25.675769 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:25.675749 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/61b82340-8090-479c-b3c0-e69658f30bfb-signing-cabundle\") pod \"service-ca-758f599c5-6499p\" (UID: \"61b82340-8090-479c-b3c0-e69658f30bfb\") " pod="openshift-service-ca/service-ca-758f599c5-6499p" Jul 07 22:04:25.677839 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:25.677822 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/61b82340-8090-479c-b3c0-e69658f30bfb-signing-key\") pod \"service-ca-758f599c5-6499p\" (UID: \"61b82340-8090-479c-b3c0-e69658f30bfb\") " pod="openshift-service-ca/service-ca-758f599c5-6499p" Jul 07 22:04:25.683803 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:25.683774 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h5m6\" (UniqueName: \"kubernetes.io/projected/61b82340-8090-479c-b3c0-e69658f30bfb-kube-api-access-6h5m6\") pod \"service-ca-758f599c5-6499p\" (UID: \"61b82340-8090-479c-b3c0-e69658f30bfb\") " pod="openshift-service-ca/service-ca-758f599c5-6499p" Jul 07 22:04:25.727825 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:25.727798 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-6499p" Jul 07 22:04:25.932262 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:25.932235 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-c9npj_68c8466f-1a04-4d64-a571-35aa35b5c8b9/node-ca/0.log" Jul 07 22:04:26.032017 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:26.031988 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-6499p"] Jul 07 22:04:26.035788 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:04:26.035761 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61b82340_8090_479c_b3c0_e69658f30bfb.slice/crio-2ec0b53b2303e96b57a7f68e8e0f5439df0822e09650eae299769d3921dd27ed WatchSource:0}: Error finding container 2ec0b53b2303e96b57a7f68e8e0f5439df0822e09650eae299769d3921dd27ed: Status 404 returned error can't find the container with id 2ec0b53b2303e96b57a7f68e8e0f5439df0822e09650eae299769d3921dd27ed Jul 07 22:04:26.729703 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:26.729674 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2bkmt_9becf7d9-4d28-4ae3-a0df-c7b1a87ddf8c/serve-healthcheck-canary/0.log" Jul 07 22:04:26.886591 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:26.886558 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3fecd654-8a55-4c3d-91f6-dca0be8172e2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-88k64\" (UID: \"3fecd654-8a55-4c3d-91f6-dca0be8172e2\") " pod="openshift-insights/insights-runtime-extractor-88k64" Jul 07 22:04:26.886750 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:26.886624 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3fecd654-8a55-4c3d-91f6-dca0be8172e2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-88k64\" (UID: \"3fecd654-8a55-4c3d-91f6-dca0be8172e2\") " pod="openshift-insights/insights-runtime-extractor-88k64" Jul 07 22:04:26.886750 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:04:26.886700 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 22:04:26.886750 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:04:26.886733 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 22:04:26.886848 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:04:26.886755 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3fecd654-8a55-4c3d-91f6-dca0be8172e2-kube-rbac-proxy-cm podName:3fecd654-8a55-4c3d-91f6-dca0be8172e2 nodeName:}" failed. No retries permitted until 2026-07-07 22:04:30.886740971 +0000 UTC m=+50.714870407 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3fecd654-8a55-4c3d-91f6-dca0be8172e2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-88k64" (UID: "3fecd654-8a55-4c3d-91f6-dca0be8172e2") : configmap "kube-rbac-proxy" not found Jul 07 22:04:26.886848 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:04:26.886788 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fecd654-8a55-4c3d-91f6-dca0be8172e2-insights-runtime-extractor-tls podName:3fecd654-8a55-4c3d-91f6-dca0be8172e2 nodeName:}" failed. No retries permitted until 2026-07-07 22:04:30.886768782 +0000 UTC m=+50.714898218 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3fecd654-8a55-4c3d-91f6-dca0be8172e2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-88k64" (UID: "3fecd654-8a55-4c3d-91f6-dca0be8172e2") : secret "insights-runtime-extractor-tls" not found Jul 07 22:04:26.984371 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:26.984285 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-dnztd" event={"ID":"5cd8d4d5-75db-49f2-b8e3-4291e58ec967","Type":"ContainerStarted","Data":"38c217695fb1fa4e0a4d86c57da91e6d6420dd7759d45f6444ce3cf48bdb411c"} Jul 07 22:04:26.984769 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:26.984373 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-dnztd" event={"ID":"5cd8d4d5-75db-49f2-b8e3-4291e58ec967","Type":"ContainerStarted","Data":"faa69358ec0056bfec81ae0084aad884912c62a6dd99e096bba5e4bb9d2a0d1a"} Jul 07 22:04:26.985550 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:26.985520 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-6499p" event={"ID":"61b82340-8090-479c-b3c0-e69658f30bfb","Type":"ContainerStarted","Data":"2ec0b53b2303e96b57a7f68e8e0f5439df0822e09650eae299769d3921dd27ed"} Jul 07 22:04:27.000787 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:27.000739 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-dnztd" podStartSLOduration=2.184557483 podStartE2EDuration="4.000723165s" totalImagesPullingTime="1.816165682s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 22:04:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:04:26.999822355 +0000 UTC m=+46.827951813" watchObservedRunningTime="2026-07-07 22:04:27.000723165 +0000 UTC m=+46.828852624" Jul 07 22:04:29.995338 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:29.995298 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-6499p" event={"ID":"61b82340-8090-479c-b3c0-e69658f30bfb","Type":"ContainerStarted","Data":"e0eba562c228c973070cc924da46596b751e4ee851275d8c2edf66f002642f3f"} Jul 07 22:04:30.010415 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:30.010346 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-6499p" podStartSLOduration=2.09989296 podStartE2EDuration="5.010326201s" totalImagesPullingTime="2.910433241s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 22:04:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:04:30.010235331 +0000 UTC m=+49.838364790" watchObservedRunningTime="2026-07-07 22:04:30.010326201 +0000 UTC m=+49.838455639" Jul 07 22:04:30.923929 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:30.923892 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3fecd654-8a55-4c3d-91f6-dca0be8172e2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-88k64\" (UID: \"3fecd654-8a55-4c3d-91f6-dca0be8172e2\") " pod="openshift-insights/insights-runtime-extractor-88k64" Jul 07 22:04:30.924132 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:30.923955 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3fecd654-8a55-4c3d-91f6-dca0be8172e2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-88k64\" (UID: \"3fecd654-8a55-4c3d-91f6-dca0be8172e2\") " pod="openshift-insights/insights-runtime-extractor-88k64" Jul 07 22:04:30.924132 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:04:30.924106 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 22:04:30.924247 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:04:30.924164 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fecd654-8a55-4c3d-91f6-dca0be8172e2-insights-runtime-extractor-tls podName:3fecd654-8a55-4c3d-91f6-dca0be8172e2 nodeName:}" failed. No retries permitted until 2026-07-07 22:04:38.924146881 +0000 UTC m=+58.752276317 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3fecd654-8a55-4c3d-91f6-dca0be8172e2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-88k64" (UID: "3fecd654-8a55-4c3d-91f6-dca0be8172e2") : secret "insights-runtime-extractor-tls" not found Jul 07 22:04:30.924478 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:30.924437 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3fecd654-8a55-4c3d-91f6-dca0be8172e2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-88k64\" (UID: \"3fecd654-8a55-4c3d-91f6-dca0be8172e2\") " pod="openshift-insights/insights-runtime-extractor-88k64" Jul 07 22:04:31.831461 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:31.831415 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/20b4d981-ad57-4ff9-b293-48c8f4a6f43a-metrics-tls\") pod \"dns-default-hbqkg\" (UID: \"20b4d981-ad57-4ff9-b293-48c8f4a6f43a\") " pod="openshift-dns/dns-default-hbqkg" Jul 07 22:04:31.831828 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:04:31.831557 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 22:04:31.831828 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:04:31.831617 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20b4d981-ad57-4ff9-b293-48c8f4a6f43a-metrics-tls podName:20b4d981-ad57-4ff9-b293-48c8f4a6f43a nodeName:}" failed. No retries permitted until 2026-07-07 22:04:47.831602196 +0000 UTC m=+67.659731632 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/20b4d981-ad57-4ff9-b293-48c8f4a6f43a-metrics-tls") pod "dns-default-hbqkg" (UID: "20b4d981-ad57-4ff9-b293-48c8f4a6f43a") : secret "dns-default-metrics-tls" not found Jul 07 22:04:38.986479 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:38.986426 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3fecd654-8a55-4c3d-91f6-dca0be8172e2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-88k64\" (UID: \"3fecd654-8a55-4c3d-91f6-dca0be8172e2\") " pod="openshift-insights/insights-runtime-extractor-88k64" Jul 07 22:04:38.988979 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:38.988942 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3fecd654-8a55-4c3d-91f6-dca0be8172e2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-88k64\" (UID: \"3fecd654-8a55-4c3d-91f6-dca0be8172e2\") " pod="openshift-insights/insights-runtime-extractor-88k64" Jul 07 22:04:39.089977 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:39.089922 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-88k64" Jul 07 22:04:39.218679 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:39.218648 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-88k64"] Jul 07 22:04:39.222023 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:04:39.221990 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fecd654_8a55_4c3d_91f6_dca0be8172e2.slice/crio-c8d3141ab6625c2c2aa0a7900e31922306719fb41f060a12159bb66e994b66ac WatchSource:0}: Error finding container c8d3141ab6625c2c2aa0a7900e31922306719fb41f060a12159bb66e994b66ac: Status 404 returned error can't find the container with id c8d3141ab6625c2c2aa0a7900e31922306719fb41f060a12159bb66e994b66ac Jul 07 22:04:40.023163 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:40.023118 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-88k64" event={"ID":"3fecd654-8a55-4c3d-91f6-dca0be8172e2","Type":"ContainerStarted","Data":"c8d3141ab6625c2c2aa0a7900e31922306719fb41f060a12159bb66e994b66ac"} Jul 07 22:04:41.947957 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:41.947928 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4h8zm" Jul 07 22:04:44.036723 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:44.036693 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-88k64" event={"ID":"3fecd654-8a55-4c3d-91f6-dca0be8172e2","Type":"ContainerStarted","Data":"9f1a4949ff299cb88df5fe1b8fc8802c2c2c89a4006b6e92ac0a8abcd2261d18"} Jul 07 22:04:44.037084 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:44.036746 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-88k64" event={"ID":"3fecd654-8a55-4c3d-91f6-dca0be8172e2","Type":"ContainerStarted","Data":"dc2d377bcf9310784672b7b7b722cdd9997573f6ac8dc04151b07e3d017cf94d"} Jul 07 22:04:46.445916 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.445866 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6aef29a2-20f0-4804-951f-4bd876b4b8b5-metrics-certs\") pod \"network-metrics-daemon-m7lhr\" (UID: \"6aef29a2-20f0-4804-951f-4bd876b4b8b5\") " pod="openshift-multus/network-metrics-daemon-m7lhr" Jul 07 22:04:46.446305 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.445961 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/acb69b34-f020-4c76-89c2-0ab5b6e6f3d5-original-pull-secret\") pod \"global-pull-secret-syncer-mzzl4\" (UID: \"acb69b34-f020-4c76-89c2-0ab5b6e6f3d5\") " pod="kube-system/global-pull-secret-syncer-mzzl4" Jul 07 22:04:46.447830 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.447809 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 07 22:04:46.447924 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.447811 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 07 22:04:46.458848 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.458816 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/acb69b34-f020-4c76-89c2-0ab5b6e6f3d5-original-pull-secret\") pod \"global-pull-secret-syncer-mzzl4\" (UID: \"acb69b34-f020-4c76-89c2-0ab5b6e6f3d5\") " pod="kube-system/global-pull-secret-syncer-mzzl4" Jul 07 22:04:46.458848 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.458842 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6aef29a2-20f0-4804-951f-4bd876b4b8b5-metrics-certs\") pod \"network-metrics-daemon-m7lhr\" (UID: \"6aef29a2-20f0-4804-951f-4bd876b4b8b5\") " pod="openshift-multus/network-metrics-daemon-m7lhr" Jul 07 22:04:46.513393 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.513362 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hdlmc\"" Jul 07 22:04:46.517025 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.517012 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mzzl4" Jul 07 22:04:46.521870 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.521854 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m7lhr" Jul 07 22:04:46.546692 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.546660 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj8mx\" (UniqueName: \"kubernetes.io/projected/95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9-kube-api-access-bj8mx\") pod \"network-check-target-zlr8s\" (UID: \"95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9\") " pod="openshift-network-diagnostics/network-check-target-zlr8s" Jul 07 22:04:46.549019 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.548991 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 07 22:04:46.560770 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.560743 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 07 22:04:46.574108 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.574006 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj8mx\" (UniqueName: \"kubernetes.io/projected/95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9-kube-api-access-bj8mx\") pod \"network-check-target-zlr8s\" (UID: \"95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9\") " pod="openshift-network-diagnostics/network-check-target-zlr8s" Jul 07 22:04:46.587184 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.587146 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86f8c6bc9b-nzsj5"] Jul 07 22:04:46.590350 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.590324 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86f8c6bc9b-nzsj5" Jul 07 22:04:46.595986 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.595960 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 07 22:04:46.596118 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.595960 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 07 22:04:46.596326 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.596307 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-82cns\"" Jul 07 22:04:46.596589 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.596515 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 07 22:04:46.596821 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.596803 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 07 22:04:46.598982 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.598674 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-584f5b89bf-x9djv"] Jul 07 22:04:46.600641 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.600626 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-584f5b89bf-x9djv" Jul 07 22:04:46.602606 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.602592 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 07 22:04:46.607996 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.607935 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86f8c6bc9b-nzsj5"] Jul 07 22:04:46.616392 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.616323 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-584f5b89bf-x9djv"] Jul 07 22:04:46.647897 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.647870 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/277a0060-4e2d-4275-89ae-bf98a7ef7bff-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-86f8c6bc9b-nzsj5\" (UID: \"277a0060-4e2d-4275-89ae-bf98a7ef7bff\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86f8c6bc9b-nzsj5" Jul 07 22:04:46.648058 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.647908 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkhn8\" (UniqueName: \"kubernetes.io/projected/277a0060-4e2d-4275-89ae-bf98a7ef7bff-kube-api-access-rkhn8\") pod \"managed-serviceaccount-addon-agent-86f8c6bc9b-nzsj5\" (UID: \"277a0060-4e2d-4275-89ae-bf98a7ef7bff\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86f8c6bc9b-nzsj5" Jul 07 22:04:46.648058 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.647927 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3552d7fb-1996-4346-862a-d88b1ab5742a-tmp\") pod \"klusterlet-addon-workmgr-584f5b89bf-x9djv\" (UID: \"3552d7fb-1996-4346-862a-d88b1ab5742a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-584f5b89bf-x9djv" Jul 07 22:04:46.648058 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.647941 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/3552d7fb-1996-4346-862a-d88b1ab5742a-klusterlet-config\") pod \"klusterlet-addon-workmgr-584f5b89bf-x9djv\" (UID: \"3552d7fb-1996-4346-862a-d88b1ab5742a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-584f5b89bf-x9djv" Jul 07 22:04:46.648238 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.648055 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zchbw\" (UniqueName: \"kubernetes.io/projected/3552d7fb-1996-4346-862a-d88b1ab5742a-kube-api-access-zchbw\") pod \"klusterlet-addon-workmgr-584f5b89bf-x9djv\" (UID: \"3552d7fb-1996-4346-862a-d88b1ab5742a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-584f5b89bf-x9djv" Jul 07 22:04:46.664666 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.664637 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mzzl4"] Jul 07 22:04:46.668305 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:04:46.668282 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacb69b34_f020_4c76_89c2_0ab5b6e6f3d5.slice/crio-cc30b431c5aaf5e458b70681c9a32db9b005a2d30a59d909f3ad9e99985f39d7 WatchSource:0}: Error finding container cc30b431c5aaf5e458b70681c9a32db9b005a2d30a59d909f3ad9e99985f39d7: Status 404 returned error can't find the container with id cc30b431c5aaf5e458b70681c9a32db9b005a2d30a59d909f3ad9e99985f39d7 Jul 07 22:04:46.674925 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.674901 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-m7lhr"] Jul 07 22:04:46.678099 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:04:46.678077 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aef29a2_20f0_4804_951f_4bd876b4b8b5.slice/crio-c86dde7454f75fb4b78d42bc95f15382762d5a53abdbfced385712ecbf96534e WatchSource:0}: Error finding container c86dde7454f75fb4b78d42bc95f15382762d5a53abdbfced385712ecbf96534e: Status 404 returned error can't find the container with id c86dde7454f75fb4b78d42bc95f15382762d5a53abdbfced385712ecbf96534e Jul 07 22:04:46.749097 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.749069 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkhn8\" (UniqueName: \"kubernetes.io/projected/277a0060-4e2d-4275-89ae-bf98a7ef7bff-kube-api-access-rkhn8\") pod \"managed-serviceaccount-addon-agent-86f8c6bc9b-nzsj5\" (UID: \"277a0060-4e2d-4275-89ae-bf98a7ef7bff\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86f8c6bc9b-nzsj5" Jul 07 22:04:46.749097 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.749097 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3552d7fb-1996-4346-862a-d88b1ab5742a-tmp\") pod \"klusterlet-addon-workmgr-584f5b89bf-x9djv\" (UID: \"3552d7fb-1996-4346-862a-d88b1ab5742a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-584f5b89bf-x9djv" Jul 07 22:04:46.749337 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.749115 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/3552d7fb-1996-4346-862a-d88b1ab5742a-klusterlet-config\") pod \"klusterlet-addon-workmgr-584f5b89bf-x9djv\" (UID: \"3552d7fb-1996-4346-862a-d88b1ab5742a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-584f5b89bf-x9djv" Jul 07 22:04:46.749337 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.749158 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zchbw\" (UniqueName: \"kubernetes.io/projected/3552d7fb-1996-4346-862a-d88b1ab5742a-kube-api-access-zchbw\") pod \"klusterlet-addon-workmgr-584f5b89bf-x9djv\" (UID: \"3552d7fb-1996-4346-862a-d88b1ab5742a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-584f5b89bf-x9djv" Jul 07 22:04:46.749337 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.749222 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/277a0060-4e2d-4275-89ae-bf98a7ef7bff-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-86f8c6bc9b-nzsj5\" (UID: \"277a0060-4e2d-4275-89ae-bf98a7ef7bff\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86f8c6bc9b-nzsj5" Jul 07 22:04:46.749575 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.749554 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3552d7fb-1996-4346-862a-d88b1ab5742a-tmp\") pod \"klusterlet-addon-workmgr-584f5b89bf-x9djv\" (UID: \"3552d7fb-1996-4346-862a-d88b1ab5742a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-584f5b89bf-x9djv" Jul 07 22:04:46.751722 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.751706 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/3552d7fb-1996-4346-862a-d88b1ab5742a-klusterlet-config\") pod \"klusterlet-addon-workmgr-584f5b89bf-x9djv\" (UID: \"3552d7fb-1996-4346-862a-d88b1ab5742a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-584f5b89bf-x9djv" Jul 07 22:04:46.751779 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.751765 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/277a0060-4e2d-4275-89ae-bf98a7ef7bff-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-86f8c6bc9b-nzsj5\" (UID: \"277a0060-4e2d-4275-89ae-bf98a7ef7bff\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86f8c6bc9b-nzsj5" Jul 07 22:04:46.760704 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.760673 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zchbw\" (UniqueName: \"kubernetes.io/projected/3552d7fb-1996-4346-862a-d88b1ab5742a-kube-api-access-zchbw\") pod \"klusterlet-addon-workmgr-584f5b89bf-x9djv\" (UID: \"3552d7fb-1996-4346-862a-d88b1ab5742a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-584f5b89bf-x9djv" Jul 07 22:04:46.760882 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.760851 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkhn8\" (UniqueName: \"kubernetes.io/projected/277a0060-4e2d-4275-89ae-bf98a7ef7bff-kube-api-access-rkhn8\") pod \"managed-serviceaccount-addon-agent-86f8c6bc9b-nzsj5\" (UID: \"277a0060-4e2d-4275-89ae-bf98a7ef7bff\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86f8c6bc9b-nzsj5" Jul 07 22:04:46.777204 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.777174 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5554df4897-v5rn8"] Jul 07 22:04:46.780087 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.780068 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5554df4897-v5rn8" Jul 07 22:04:46.782256 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.782238 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 07 22:04:46.782547 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.782529 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-c748h\"" Jul 07 22:04:46.782602 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.782563 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 07 22:04:46.782891 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.782876 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 07 22:04:46.788579 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.788560 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 07 22:04:46.796890 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.796865 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5554df4897-v5rn8"] Jul 07 22:04:46.823883 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.823857 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-p2w6w\"" Jul 07 22:04:46.832869 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.832800 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zlr8s" Jul 07 22:04:46.850123 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.850096 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d2380ae8-40b7-40a7-a87e-900314d58511-registry-certificates\") pod \"image-registry-5554df4897-v5rn8\" (UID: \"d2380ae8-40b7-40a7-a87e-900314d58511\") " pod="openshift-image-registry/image-registry-5554df4897-v5rn8" Jul 07 22:04:46.850217 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.850136 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d2380ae8-40b7-40a7-a87e-900314d58511-installation-pull-secrets\") pod \"image-registry-5554df4897-v5rn8\" (UID: \"d2380ae8-40b7-40a7-a87e-900314d58511\") " pod="openshift-image-registry/image-registry-5554df4897-v5rn8" Jul 07 22:04:46.850217 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.850164 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d2380ae8-40b7-40a7-a87e-900314d58511-image-registry-private-configuration\") pod \"image-registry-5554df4897-v5rn8\" (UID: \"d2380ae8-40b7-40a7-a87e-900314d58511\") " pod="openshift-image-registry/image-registry-5554df4897-v5rn8" Jul 07 22:04:46.850217 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.850204 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d2380ae8-40b7-40a7-a87e-900314d58511-trusted-ca\") pod \"image-registry-5554df4897-v5rn8\" (UID: \"d2380ae8-40b7-40a7-a87e-900314d58511\") " pod="openshift-image-registry/image-registry-5554df4897-v5rn8" Jul 07 22:04:46.850357 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.850252 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d2380ae8-40b7-40a7-a87e-900314d58511-registry-tls\") pod \"image-registry-5554df4897-v5rn8\" (UID: \"d2380ae8-40b7-40a7-a87e-900314d58511\") " pod="openshift-image-registry/image-registry-5554df4897-v5rn8" Jul 07 22:04:46.850357 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.850313 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjhfm\" (UniqueName: \"kubernetes.io/projected/d2380ae8-40b7-40a7-a87e-900314d58511-kube-api-access-hjhfm\") pod \"image-registry-5554df4897-v5rn8\" (UID: \"d2380ae8-40b7-40a7-a87e-900314d58511\") " pod="openshift-image-registry/image-registry-5554df4897-v5rn8" Jul 07 22:04:46.850357 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.850338 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d2380ae8-40b7-40a7-a87e-900314d58511-bound-sa-token\") pod \"image-registry-5554df4897-v5rn8\" (UID: \"d2380ae8-40b7-40a7-a87e-900314d58511\") " pod="openshift-image-registry/image-registry-5554df4897-v5rn8" Jul 07 22:04:46.850526 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.850397 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d2380ae8-40b7-40a7-a87e-900314d58511-ca-trust-extracted\") pod \"image-registry-5554df4897-v5rn8\" (UID: \"d2380ae8-40b7-40a7-a87e-900314d58511\") " pod="openshift-image-registry/image-registry-5554df4897-v5rn8" Jul 07 22:04:46.909716 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.909659 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86f8c6bc9b-nzsj5" Jul 07 22:04:46.919384 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.919361 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-584f5b89bf-x9djv" Jul 07 22:04:46.951297 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.951259 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d2380ae8-40b7-40a7-a87e-900314d58511-installation-pull-secrets\") pod \"image-registry-5554df4897-v5rn8\" (UID: \"d2380ae8-40b7-40a7-a87e-900314d58511\") " pod="openshift-image-registry/image-registry-5554df4897-v5rn8" Jul 07 22:04:46.951477 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.951320 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d2380ae8-40b7-40a7-a87e-900314d58511-image-registry-private-configuration\") pod \"image-registry-5554df4897-v5rn8\" (UID: \"d2380ae8-40b7-40a7-a87e-900314d58511\") " pod="openshift-image-registry/image-registry-5554df4897-v5rn8" Jul 07 22:04:46.951477 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.951341 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d2380ae8-40b7-40a7-a87e-900314d58511-trusted-ca\") pod \"image-registry-5554df4897-v5rn8\" (UID: \"d2380ae8-40b7-40a7-a87e-900314d58511\") " pod="openshift-image-registry/image-registry-5554df4897-v5rn8" Jul 07 22:04:46.951477 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.951372 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d2380ae8-40b7-40a7-a87e-900314d58511-registry-tls\") pod \"image-registry-5554df4897-v5rn8\" (UID: \"d2380ae8-40b7-40a7-a87e-900314d58511\") " pod="openshift-image-registry/image-registry-5554df4897-v5rn8" Jul 07 22:04:46.951477 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.951406 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjhfm\" (UniqueName: \"kubernetes.io/projected/d2380ae8-40b7-40a7-a87e-900314d58511-kube-api-access-hjhfm\") pod \"image-registry-5554df4897-v5rn8\" (UID: \"d2380ae8-40b7-40a7-a87e-900314d58511\") " pod="openshift-image-registry/image-registry-5554df4897-v5rn8" Jul 07 22:04:46.951477 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.951433 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d2380ae8-40b7-40a7-a87e-900314d58511-bound-sa-token\") pod \"image-registry-5554df4897-v5rn8\" (UID: \"d2380ae8-40b7-40a7-a87e-900314d58511\") " pod="openshift-image-registry/image-registry-5554df4897-v5rn8" Jul 07 22:04:46.951477 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.951471 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d2380ae8-40b7-40a7-a87e-900314d58511-ca-trust-extracted\") pod \"image-registry-5554df4897-v5rn8\" (UID: \"d2380ae8-40b7-40a7-a87e-900314d58511\") " pod="openshift-image-registry/image-registry-5554df4897-v5rn8" Jul 07 22:04:46.951797 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.951490 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d2380ae8-40b7-40a7-a87e-900314d58511-registry-certificates\") pod \"image-registry-5554df4897-v5rn8\" (UID: \"d2380ae8-40b7-40a7-a87e-900314d58511\") " pod="openshift-image-registry/image-registry-5554df4897-v5rn8" Jul 07 22:04:46.952415 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.952351 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d2380ae8-40b7-40a7-a87e-900314d58511-ca-trust-extracted\") pod \"image-registry-5554df4897-v5rn8\" (UID: \"d2380ae8-40b7-40a7-a87e-900314d58511\") " pod="openshift-image-registry/image-registry-5554df4897-v5rn8" Jul 07 22:04:46.952552 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.952416 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d2380ae8-40b7-40a7-a87e-900314d58511-registry-certificates\") pod \"image-registry-5554df4897-v5rn8\" (UID: \"d2380ae8-40b7-40a7-a87e-900314d58511\") " pod="openshift-image-registry/image-registry-5554df4897-v5rn8" Jul 07 22:04:46.952625 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.952575 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d2380ae8-40b7-40a7-a87e-900314d58511-trusted-ca\") pod \"image-registry-5554df4897-v5rn8\" (UID: \"d2380ae8-40b7-40a7-a87e-900314d58511\") " pod="openshift-image-registry/image-registry-5554df4897-v5rn8" Jul 07 22:04:46.954692 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.954669 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d2380ae8-40b7-40a7-a87e-900314d58511-registry-tls\") pod \"image-registry-5554df4897-v5rn8\" (UID: \"d2380ae8-40b7-40a7-a87e-900314d58511\") " pod="openshift-image-registry/image-registry-5554df4897-v5rn8" Jul 07 22:04:46.954810 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.954787 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d2380ae8-40b7-40a7-a87e-900314d58511-installation-pull-secrets\") pod \"image-registry-5554df4897-v5rn8\" (UID: \"d2380ae8-40b7-40a7-a87e-900314d58511\") " pod="openshift-image-registry/image-registry-5554df4897-v5rn8" Jul 07 22:04:46.955400 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.955360 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d2380ae8-40b7-40a7-a87e-900314d58511-image-registry-private-configuration\") pod \"image-registry-5554df4897-v5rn8\" (UID: \"d2380ae8-40b7-40a7-a87e-900314d58511\") " pod="openshift-image-registry/image-registry-5554df4897-v5rn8" Jul 07 22:04:46.968118 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.967861 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d2380ae8-40b7-40a7-a87e-900314d58511-bound-sa-token\") pod \"image-registry-5554df4897-v5rn8\" (UID: \"d2380ae8-40b7-40a7-a87e-900314d58511\") " pod="openshift-image-registry/image-registry-5554df4897-v5rn8" Jul 07 22:04:46.968118 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.968083 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjhfm\" (UniqueName: \"kubernetes.io/projected/d2380ae8-40b7-40a7-a87e-900314d58511-kube-api-access-hjhfm\") pod \"image-registry-5554df4897-v5rn8\" (UID: \"d2380ae8-40b7-40a7-a87e-900314d58511\") " pod="openshift-image-registry/image-registry-5554df4897-v5rn8" Jul 07 22:04:46.978459 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:46.978413 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zlr8s"] Jul 07 22:04:46.983690 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:04:46.983664 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95b3f090_eee9_4d0e_9e3b_cbb74c2c43e9.slice/crio-d69b65b6c510002f9638a468b1f137bf1e39440059bb55b63889adf626b05727 WatchSource:0}: Error finding container d69b65b6c510002f9638a468b1f137bf1e39440059bb55b63889adf626b05727: Status 404 returned error can't find the container with id d69b65b6c510002f9638a468b1f137bf1e39440059bb55b63889adf626b05727 Jul 07 22:04:47.049655 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:47.049615 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mzzl4" event={"ID":"acb69b34-f020-4c76-89c2-0ab5b6e6f3d5","Type":"ContainerStarted","Data":"cc30b431c5aaf5e458b70681c9a32db9b005a2d30a59d909f3ad9e99985f39d7"} Jul 07 22:04:47.051101 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:47.051068 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zlr8s" event={"ID":"95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9","Type":"ContainerStarted","Data":"d69b65b6c510002f9638a468b1f137bf1e39440059bb55b63889adf626b05727"} Jul 07 22:04:47.054003 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:47.051855 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86f8c6bc9b-nzsj5"] Jul 07 22:04:47.056051 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:47.056023 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-88k64" event={"ID":"3fecd654-8a55-4c3d-91f6-dca0be8172e2","Type":"ContainerStarted","Data":"25ccfb5ab6e6627ba91edf181390a15a629d8e47fbcfe68303757aab31bcfec6"} Jul 07 22:04:47.057352 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:47.057331 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m7lhr" event={"ID":"6aef29a2-20f0-4804-951f-4bd876b4b8b5","Type":"ContainerStarted","Data":"c86dde7454f75fb4b78d42bc95f15382762d5a53abdbfced385712ecbf96534e"} Jul 07 22:04:47.058624 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:04:47.058594 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod277a0060_4e2d_4275_89ae_bf98a7ef7bff.slice/crio-ff5a280e14500aa2d8b30bc1d6f2ba3465f1e4214b875d0e603672dc5d49c229 WatchSource:0}: Error finding container ff5a280e14500aa2d8b30bc1d6f2ba3465f1e4214b875d0e603672dc5d49c229: Status 404 returned error can't find the container with id ff5a280e14500aa2d8b30bc1d6f2ba3465f1e4214b875d0e603672dc5d49c229 Jul 07 22:04:47.072199 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:47.072174 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-584f5b89bf-x9djv"] Jul 07 22:04:47.075746 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:04:47.075723 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3552d7fb_1996_4346_862a_d88b1ab5742a.slice/crio-bc7104668c0e2aa6d777c529bcb4c4fd150ee3cbd56d1fe6ada1b39626cd8c79 WatchSource:0}: Error finding container bc7104668c0e2aa6d777c529bcb4c4fd150ee3cbd56d1fe6ada1b39626cd8c79: Status 404 returned error can't find the container with id bc7104668c0e2aa6d777c529bcb4c4fd150ee3cbd56d1fe6ada1b39626cd8c79 Jul 07 22:04:47.089476 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:47.089403 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5554df4897-v5rn8" Jul 07 22:04:47.092386 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:47.092345 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-88k64" podStartSLOduration=17.092434755 podStartE2EDuration="24.092325932s" totalImagesPullingTime="6.999891177s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 22:04:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:04:47.091503593 +0000 UTC m=+66.919633053" watchObservedRunningTime="2026-07-07 22:04:47.092325932 +0000 UTC m=+66.920455392" Jul 07 22:04:47.247312 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:47.247277 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5554df4897-v5rn8"] Jul 07 22:04:47.251356 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:04:47.251307 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2380ae8_40b7_40a7_a87e_900314d58511.slice/crio-5483bd4d651ed13e1f78457b944c7944c3f0cff8f5924f00ccf533f62bab1b60 WatchSource:0}: Error finding container 5483bd4d651ed13e1f78457b944c7944c3f0cff8f5924f00ccf533f62bab1b60: Status 404 returned error can't find the container with id 5483bd4d651ed13e1f78457b944c7944c3f0cff8f5924f00ccf533f62bab1b60 Jul 07 22:04:47.861200 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:47.861155 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/20b4d981-ad57-4ff9-b293-48c8f4a6f43a-metrics-tls\") pod \"dns-default-hbqkg\" (UID: \"20b4d981-ad57-4ff9-b293-48c8f4a6f43a\") " pod="openshift-dns/dns-default-hbqkg" Jul 07 22:04:47.864318 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:47.864260 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/20b4d981-ad57-4ff9-b293-48c8f4a6f43a-metrics-tls\") pod \"dns-default-hbqkg\" (UID: \"20b4d981-ad57-4ff9-b293-48c8f4a6f43a\") " pod="openshift-dns/dns-default-hbqkg" Jul 07 22:04:48.063420 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:48.063381 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5554df4897-v5rn8" event={"ID":"d2380ae8-40b7-40a7-a87e-900314d58511","Type":"ContainerStarted","Data":"485c425abb0f69aa6515304fd00a21450429edf02ee65c9447c12b13a367e7bd"} Jul 07 22:04:48.063603 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:48.063492 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5554df4897-v5rn8" event={"ID":"d2380ae8-40b7-40a7-a87e-900314d58511","Type":"ContainerStarted","Data":"5483bd4d651ed13e1f78457b944c7944c3f0cff8f5924f00ccf533f62bab1b60"} Jul 07 22:04:48.063603 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:48.063558 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5554df4897-v5rn8" Jul 07 22:04:48.065156 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:48.065104 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-584f5b89bf-x9djv" event={"ID":"3552d7fb-1996-4346-862a-d88b1ab5742a","Type":"ContainerStarted","Data":"bc7104668c0e2aa6d777c529bcb4c4fd150ee3cbd56d1fe6ada1b39626cd8c79"} Jul 07 22:04:48.066478 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:48.066417 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86f8c6bc9b-nzsj5" event={"ID":"277a0060-4e2d-4275-89ae-bf98a7ef7bff","Type":"ContainerStarted","Data":"ff5a280e14500aa2d8b30bc1d6f2ba3465f1e4214b875d0e603672dc5d49c229"} Jul 07 22:04:48.085750 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:48.085674 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5554df4897-v5rn8" podStartSLOduration=2.085656099 podStartE2EDuration="2.085656099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 22:04:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:04:48.084633103 +0000 UTC m=+67.912762562" watchObservedRunningTime="2026-07-07 22:04:48.085656099 +0000 UTC m=+67.913785538" Jul 07 22:04:48.107817 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:48.107787 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kp4p7\"" Jul 07 22:04:48.115876 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:48.115787 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hbqkg" Jul 07 22:04:48.429127 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:48.429089 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hbqkg"] Jul 07 22:04:48.434684 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:04:48.434649 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20b4d981_ad57_4ff9_b293_48c8f4a6f43a.slice/crio-d8aece9a4b7839d7f9942bca7b297ce4f00a4cb5012dd655d9e885bf9aa3a1e6 WatchSource:0}: Error finding container d8aece9a4b7839d7f9942bca7b297ce4f00a4cb5012dd655d9e885bf9aa3a1e6: Status 404 returned error can't find the container with id d8aece9a4b7839d7f9942bca7b297ce4f00a4cb5012dd655d9e885bf9aa3a1e6 Jul 07 22:04:49.076315 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:49.076273 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hbqkg" event={"ID":"20b4d981-ad57-4ff9-b293-48c8f4a6f43a","Type":"ContainerStarted","Data":"d8aece9a4b7839d7f9942bca7b297ce4f00a4cb5012dd655d9e885bf9aa3a1e6"} Jul 07 22:04:49.083940 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:49.083814 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m7lhr" event={"ID":"6aef29a2-20f0-4804-951f-4bd876b4b8b5","Type":"ContainerStarted","Data":"8a679526bfa02b7eaaef69aa887b73f04a504c5893af22756f0578ec1328328a"} Jul 07 22:04:49.083940 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:49.083866 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m7lhr" event={"ID":"6aef29a2-20f0-4804-951f-4bd876b4b8b5","Type":"ContainerStarted","Data":"3551ee20aedc51a9a482241bd2952727d3101d70dafa68512bf32d3209248fb2"} Jul 07 22:04:56.108698 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:56.108664 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mzzl4" event={"ID":"acb69b34-f020-4c76-89c2-0ab5b6e6f3d5","Type":"ContainerStarted","Data":"74e25519e0b8938acb4377c49820ec1c6b2dce9f8813a1c4ba5615b043b03f14"} Jul 07 22:04:56.110217 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:56.110189 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86f8c6bc9b-nzsj5" event={"ID":"277a0060-4e2d-4275-89ae-bf98a7ef7bff","Type":"ContainerStarted","Data":"9d4316cbc2d7b934d58b9cf362884268335e3eb24e3bd599fed4bf1b2cb3e2a5"} Jul 07 22:04:56.111803 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:56.111782 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zlr8s" event={"ID":"95b3f090-eee9-4d0e-9e3b-cbb74c2c43e9","Type":"ContainerStarted","Data":"d4af4a9b4e8cab48729bb396f880c6f218db6192d2c4515d795a8ac3b7012582"} Jul 07 22:04:56.111860 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:56.111839 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zlr8s" Jul 07 22:04:56.113132 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:56.113106 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-584f5b89bf-x9djv" event={"ID":"3552d7fb-1996-4346-862a-d88b1ab5742a","Type":"ContainerStarted","Data":"5e59e7cbc5453f744257653f8bdbd489c004f7ea0657ffd7104adc21cb32baeb"} Jul 07 22:04:56.113250 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:56.113230 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-584f5b89bf-x9djv" Jul 07 22:04:56.115065 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:56.115044 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hbqkg" event={"ID":"20b4d981-ad57-4ff9-b293-48c8f4a6f43a","Type":"ContainerStarted","Data":"4887740059bbfdf4e4c015904ad913cdbd392e39c6781a981ce98df4c64e3d9b"} Jul 07 22:04:56.115182 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:56.115085 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hbqkg" event={"ID":"20b4d981-ad57-4ff9-b293-48c8f4a6f43a","Type":"ContainerStarted","Data":"4f16f90a30fa9f2e138c96fb8e62a8d74c334d619e02d2bcc74e4763b732488d"} Jul 07 22:04:56.115267 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:56.115248 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-hbqkg" Jul 07 22:04:56.115328 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:56.115317 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-584f5b89bf-x9djv" Jul 07 22:04:56.124964 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:56.124927 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-m7lhr" podStartSLOduration=73.541501259 podStartE2EDuration="1m15.124912418s" totalImagesPullingTime="1.583411159s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 22:03:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:04:49.105605554 +0000 UTC m=+68.933735014" watchObservedRunningTime="2026-07-07 22:04:56.124912418 +0000 UTC m=+75.953041879" Jul 07 22:04:56.125668 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:56.125624 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-mzzl4" podStartSLOduration=66.61210796 podStartE2EDuration="1m15.125613364s" totalImagesPullingTime="8.513505404s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 22:03:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:04:56.124742665 +0000 UTC m=+75.952872124" watchObservedRunningTime="2026-07-07 22:04:56.125613364 +0000 UTC m=+75.953742823" Jul 07 22:04:56.141724 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:56.141685 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-584f5b89bf-x9djv" podStartSLOduration=2.013694888 podStartE2EDuration="10.141673119s" totalImagesPullingTime="8.127978231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 22:04:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:04:56.141151244 +0000 UTC m=+75.969280702" watchObservedRunningTime="2026-07-07 22:04:56.141673119 +0000 UTC m=+75.969802577" Jul 07 22:04:56.160168 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:56.160124 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-hbqkg" podStartSLOduration=34.413324334 podStartE2EDuration="41.160113256s" totalImagesPullingTime="6.746788922s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 22:04:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:04:56.1598827 +0000 UTC m=+75.988012160" watchObservedRunningTime="2026-07-07 22:04:56.160113256 +0000 UTC m=+75.988242713" Jul 07 22:04:56.173268 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:04:56.173230 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-zlr8s" podStartSLOduration=66.982277682 podStartE2EDuration="1m15.173215036s" totalImagesPullingTime="8.190937354s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 22:03:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:04:56.172584962 +0000 UTC m=+76.000714422" watchObservedRunningTime="2026-07-07 22:04:56.173215036 +0000 UTC m=+76.001344493" Jul 07 22:05:06.119878 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:06.119765 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-hbqkg" Jul 07 22:05:06.136798 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:06.136749 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86f8c6bc9b-nzsj5" podStartSLOduration=12.006345284 podStartE2EDuration="20.136734165s" totalImagesPullingTime="8.130388881s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 22:04:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:04:56.186399726 +0000 UTC m=+76.014529192" watchObservedRunningTime="2026-07-07 22:05:06.136734165 +0000 UTC m=+85.964863622" Jul 07 22:05:07.093351 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:07.093316 2564 patch_prober.go:28] interesting pod/image-registry-5554df4897-v5rn8 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 07 22:05:07.093534 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:07.093366 2564 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5554df4897-v5rn8" podUID="d2380ae8-40b7-40a7-a87e-900314d58511" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 22:05:09.087034 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:09.087005 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5554df4897-v5rn8" Jul 07 22:05:12.572236 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:12.572198 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-p69r8"] Jul 07 22:05:12.590906 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:12.590879 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-p69r8" Jul 07 22:05:12.593001 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:12.592978 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 07 22:05:12.593289 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:12.593263 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-c9wff\"" Jul 07 22:05:12.593370 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:12.593313 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 07 22:05:12.593429 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:12.593262 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 07 22:05:12.593629 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:12.593612 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 07 22:05:12.593782 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:12.593621 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 07 22:05:12.593866 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:12.593721 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 07 22:05:12.636636 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:12.636600 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/91498a50-33c1-4b03-a193-83b60292acc3-root\") pod \"node-exporter-p69r8\" (UID: \"91498a50-33c1-4b03-a193-83b60292acc3\") " pod="openshift-monitoring/node-exporter-p69r8" Jul 07 22:05:12.636769 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:12.636644 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91498a50-33c1-4b03-a193-83b60292acc3-metrics-client-ca\") pod \"node-exporter-p69r8\" (UID: \"91498a50-33c1-4b03-a193-83b60292acc3\") " pod="openshift-monitoring/node-exporter-p69r8" Jul 07 22:05:12.636769 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:12.636673 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/91498a50-33c1-4b03-a193-83b60292acc3-node-exporter-tls\") pod \"node-exporter-p69r8\" (UID: \"91498a50-33c1-4b03-a193-83b60292acc3\") " pod="openshift-monitoring/node-exporter-p69r8" Jul 07 22:05:12.636769 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:12.636701 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/91498a50-33c1-4b03-a193-83b60292acc3-node-exporter-textfile\") pod \"node-exporter-p69r8\" (UID: \"91498a50-33c1-4b03-a193-83b60292acc3\") " pod="openshift-monitoring/node-exporter-p69r8" Jul 07 22:05:12.636769 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:12.636735 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/91498a50-33c1-4b03-a193-83b60292acc3-node-exporter-wtmp\") pod \"node-exporter-p69r8\" (UID: \"91498a50-33c1-4b03-a193-83b60292acc3\") " pod="openshift-monitoring/node-exporter-p69r8" Jul 07 22:05:12.636769 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:12.636761 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/91498a50-33c1-4b03-a193-83b60292acc3-node-exporter-accelerators-collector-config\") pod \"node-exporter-p69r8\" (UID: \"91498a50-33c1-4b03-a193-83b60292acc3\") " pod="openshift-monitoring/node-exporter-p69r8" Jul 07 22:05:12.637033 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:12.636796 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn8sk\" (UniqueName: \"kubernetes.io/projected/91498a50-33c1-4b03-a193-83b60292acc3-kube-api-access-sn8sk\") pod \"node-exporter-p69r8\" (UID: \"91498a50-33c1-4b03-a193-83b60292acc3\") " pod="openshift-monitoring/node-exporter-p69r8" Jul 07 22:05:12.637033 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:12.636846 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91498a50-33c1-4b03-a193-83b60292acc3-sys\") pod \"node-exporter-p69r8\" (UID: \"91498a50-33c1-4b03-a193-83b60292acc3\") " pod="openshift-monitoring/node-exporter-p69r8" Jul 07 22:05:12.637033 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:12.636878 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/91498a50-33c1-4b03-a193-83b60292acc3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p69r8\" (UID: \"91498a50-33c1-4b03-a193-83b60292acc3\") " pod="openshift-monitoring/node-exporter-p69r8" Jul 07 22:05:12.737463 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:12.737411 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91498a50-33c1-4b03-a193-83b60292acc3-sys\") pod \"node-exporter-p69r8\" (UID: \"91498a50-33c1-4b03-a193-83b60292acc3\") " pod="openshift-monitoring/node-exporter-p69r8" Jul 07 22:05:12.737618 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:12.737472 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/91498a50-33c1-4b03-a193-83b60292acc3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p69r8\" (UID: \"91498a50-33c1-4b03-a193-83b60292acc3\") " pod="openshift-monitoring/node-exporter-p69r8" Jul 07 22:05:12.737618 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:12.737501 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/91498a50-33c1-4b03-a193-83b60292acc3-root\") pod \"node-exporter-p69r8\" (UID: \"91498a50-33c1-4b03-a193-83b60292acc3\") " pod="openshift-monitoring/node-exporter-p69r8" Jul 07 22:05:12.737618 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:12.737521 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91498a50-33c1-4b03-a193-83b60292acc3-metrics-client-ca\") pod \"node-exporter-p69r8\" (UID: \"91498a50-33c1-4b03-a193-83b60292acc3\") " pod="openshift-monitoring/node-exporter-p69r8" Jul 07 22:05:12.737618 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:12.737533 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91498a50-33c1-4b03-a193-83b60292acc3-sys\") pod \"node-exporter-p69r8\" (UID: \"91498a50-33c1-4b03-a193-83b60292acc3\") " pod="openshift-monitoring/node-exporter-p69r8" Jul 07 22:05:12.737618 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:12.737548 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/91498a50-33c1-4b03-a193-83b60292acc3-node-exporter-tls\") pod \"node-exporter-p69r8\" (UID: \"91498a50-33c1-4b03-a193-83b60292acc3\") " pod="openshift-monitoring/node-exporter-p69r8" Jul 07 22:05:12.737618 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:12.737574 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/91498a50-33c1-4b03-a193-83b60292acc3-node-exporter-textfile\") pod \"node-exporter-p69r8\" (UID: \"91498a50-33c1-4b03-a193-83b60292acc3\") " pod="openshift-monitoring/node-exporter-p69r8" Jul 07 22:05:12.737618 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:12.737604 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/91498a50-33c1-4b03-a193-83b60292acc3-node-exporter-wtmp\") pod \"node-exporter-p69r8\" (UID: \"91498a50-33c1-4b03-a193-83b60292acc3\") " pod="openshift-monitoring/node-exporter-p69r8" Jul 07 22:05:12.737843 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:12.737632 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/91498a50-33c1-4b03-a193-83b60292acc3-node-exporter-accelerators-collector-config\") pod \"node-exporter-p69r8\" (UID: \"91498a50-33c1-4b03-a193-83b60292acc3\") " pod="openshift-monitoring/node-exporter-p69r8" Jul 07 22:05:12.737843 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:12.737671 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn8sk\" (UniqueName: \"kubernetes.io/projected/91498a50-33c1-4b03-a193-83b60292acc3-kube-api-access-sn8sk\") pod \"node-exporter-p69r8\" (UID: \"91498a50-33c1-4b03-a193-83b60292acc3\") " pod="openshift-monitoring/node-exporter-p69r8" Jul 07 22:05:12.738090 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:12.737601 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/91498a50-33c1-4b03-a193-83b60292acc3-root\") pod \"node-exporter-p69r8\" (UID: \"91498a50-33c1-4b03-a193-83b60292acc3\") " pod="openshift-monitoring/node-exporter-p69r8" Jul 07 22:05:12.738178 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:12.738119 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/91498a50-33c1-4b03-a193-83b60292acc3-node-exporter-wtmp\") pod \"node-exporter-p69r8\" (UID: \"91498a50-33c1-4b03-a193-83b60292acc3\") " pod="openshift-monitoring/node-exporter-p69r8" Jul 07 22:05:12.738390 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:12.738342 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/91498a50-33c1-4b03-a193-83b60292acc3-node-exporter-textfile\") pod \"node-exporter-p69r8\" (UID: \"91498a50-33c1-4b03-a193-83b60292acc3\") " pod="openshift-monitoring/node-exporter-p69r8" Jul 07 22:05:12.738526 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:12.738505 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/91498a50-33c1-4b03-a193-83b60292acc3-node-exporter-accelerators-collector-config\") pod \"node-exporter-p69r8\" (UID: \"91498a50-33c1-4b03-a193-83b60292acc3\") " pod="openshift-monitoring/node-exporter-p69r8" Jul 07 22:05:12.738584 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:12.738522 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91498a50-33c1-4b03-a193-83b60292acc3-metrics-client-ca\") pod \"node-exporter-p69r8\" (UID: \"91498a50-33c1-4b03-a193-83b60292acc3\") " pod="openshift-monitoring/node-exporter-p69r8" Jul 07 22:05:12.740334 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:12.740312 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/91498a50-33c1-4b03-a193-83b60292acc3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p69r8\" (UID: \"91498a50-33c1-4b03-a193-83b60292acc3\") " pod="openshift-monitoring/node-exporter-p69r8" Jul 07 22:05:12.740583 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:12.740563 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/91498a50-33c1-4b03-a193-83b60292acc3-node-exporter-tls\") pod \"node-exporter-p69r8\" (UID: \"91498a50-33c1-4b03-a193-83b60292acc3\") " pod="openshift-monitoring/node-exporter-p69r8" Jul 07 22:05:12.744852 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:12.744829 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn8sk\" (UniqueName: \"kubernetes.io/projected/91498a50-33c1-4b03-a193-83b60292acc3-kube-api-access-sn8sk\") pod \"node-exporter-p69r8\" (UID: \"91498a50-33c1-4b03-a193-83b60292acc3\") " pod="openshift-monitoring/node-exporter-p69r8" Jul 07 22:05:12.902419 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:12.902325 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-p69r8" Jul 07 22:05:12.916879 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:05:12.916850 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91498a50_33c1_4b03_a193_83b60292acc3.slice/crio-2939ee510db4559f495fbe6e8391ce79d07bf109fe17f58f020cf9c074a1a938 WatchSource:0}: Error finding container 2939ee510db4559f495fbe6e8391ce79d07bf109fe17f58f020cf9c074a1a938: Status 404 returned error can't find the container with id 2939ee510db4559f495fbe6e8391ce79d07bf109fe17f58f020cf9c074a1a938 Jul 07 22:05:13.174314 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:13.174191 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p69r8" event={"ID":"91498a50-33c1-4b03-a193-83b60292acc3","Type":"ContainerStarted","Data":"2939ee510db4559f495fbe6e8391ce79d07bf109fe17f58f020cf9c074a1a938"} Jul 07 22:05:13.609133 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:13.609100 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 22:05:13.613652 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:13.613625 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:05:13.628101 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:13.628073 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 07 22:05:13.628101 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:13.628087 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 07 22:05:13.628294 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:13.628077 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 07 22:05:13.628294 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:13.628195 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 07 22:05:13.629140 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:13.628420 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 07 22:05:13.629140 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:13.628464 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 07 22:05:13.629140 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:13.628480 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 07 22:05:13.629140 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:13.628565 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-45pfg\"" Jul 07 22:05:13.629140 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:13.628787 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 07 22:05:13.629140 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:13.628978 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 07 22:05:13.639411 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:13.639382 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 22:05:13.643205 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:13.643176 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6f466319-941b-4c0f-b872-f62f8c302442-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6f466319-941b-4c0f-b872-f62f8c302442\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:05:13.643325 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:13.643210 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6f466319-941b-4c0f-b872-f62f8c302442-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6f466319-941b-4c0f-b872-f62f8c302442\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:05:13.643325 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:13.643228 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6f466319-941b-4c0f-b872-f62f8c302442-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6f466319-941b-4c0f-b872-f62f8c302442\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:05:13.643325 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:13.643256 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6f466319-941b-4c0f-b872-f62f8c302442-config-out\") pod \"alertmanager-main-0\" (UID: \"6f466319-941b-4c0f-b872-f62f8c302442\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:05:13.643325 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:13.643291 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6f466319-941b-4c0f-b872-f62f8c302442-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6f466319-941b-4c0f-b872-f62f8c302442\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:05:13.643325 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:13.643318 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6f466319-941b-4c0f-b872-f62f8c302442-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6f466319-941b-4c0f-b872-f62f8c302442\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:05:13.643549 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:13.643356 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6f466319-941b-4c0f-b872-f62f8c302442-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6f466319-941b-4c0f-b872-f62f8c302442\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:05:13.643549 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:13.643389 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6f466319-941b-4c0f-b872-f62f8c302442-web-config\") pod \"alertmanager-main-0\" (UID: \"6f466319-941b-4c0f-b872-f62f8c302442\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:05:13.643549 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:13.643408 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f466319-941b-4c0f-b872-f62f8c302442-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6f466319-941b-4c0f-b872-f62f8c302442\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:05:13.643549 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:13.643464 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6f466319-941b-4c0f-b872-f62f8c302442-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6f466319-941b-4c0f-b872-f62f8c302442\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:05:13.643549 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:13.643481 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gqnq\" (UniqueName: \"kubernetes.io/projected/6f466319-941b-4c0f-b872-f62f8c302442-kube-api-access-5gqnq\") pod \"alertmanager-main-0\" (UID: \"6f466319-941b-4c0f-b872-f62f8c302442\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:05:13.643549 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:13.643498 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6f466319-941b-4c0f-b872-f62f8c302442-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6f466319-941b-4c0f-b872-f62f8c302442\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:05:13.643549 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:13.643523 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6f466319-941b-4c0f-b872-f62f8c302442-config-volume\") pod \"alertmanager-main-0\" (UID: \"6f466319-941b-4c0f-b872-f62f8c302442\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:05:13.744611 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:13.744580 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6f466319-941b-4c0f-b872-f62f8c302442-web-config\") pod \"alertmanager-main-0\" (UID: \"6f466319-941b-4c0f-b872-f62f8c302442\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:05:13.744611 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:13.744617 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f466319-941b-4c0f-b872-f62f8c302442-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6f466319-941b-4c0f-b872-f62f8c302442\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:05:13.744839 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:13.744637 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6f466319-941b-4c0f-b872-f62f8c302442-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6f466319-941b-4c0f-b872-f62f8c302442\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:05:13.744839 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:13.744652 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gqnq\" (UniqueName: \"kubernetes.io/projected/6f466319-941b-4c0f-b872-f62f8c302442-kube-api-access-5gqnq\") pod \"alertmanager-main-0\" (UID: \"6f466319-941b-4c0f-b872-f62f8c302442\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:05:13.744839 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:13.744670 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6f466319-941b-4c0f-b872-f62f8c302442-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6f466319-941b-4c0f-b872-f62f8c302442\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:05:13.744839 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:13.744693 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6f466319-941b-4c0f-b872-f62f8c302442-config-volume\") pod \"alertmanager-main-0\" (UID: \"6f466319-941b-4c0f-b872-f62f8c302442\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:05:13.744839 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:13.744719 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6f466319-941b-4c0f-b872-f62f8c302442-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6f466319-941b-4c0f-b872-f62f8c302442\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:05:13.744839 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:13.744754 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6f466319-941b-4c0f-b872-f62f8c302442-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6f466319-941b-4c0f-b872-f62f8c302442\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:05:13.744839 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:05:13.744780 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6f466319-941b-4c0f-b872-f62f8c302442-alertmanager-trusted-ca-bundle podName:6f466319-941b-4c0f-b872-f62f8c302442 nodeName:}" failed. No retries permitted until 2026-07-07 22:05:14.244754954 +0000 UTC m=+94.072884406 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/6f466319-941b-4c0f-b872-f62f8c302442-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "6f466319-941b-4c0f-b872-f62f8c302442") : configmap references non-existent config key: ca-bundle.crt Jul 07 22:05:13.744839 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:13.744825 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6f466319-941b-4c0f-b872-f62f8c302442-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6f466319-941b-4c0f-b872-f62f8c302442\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:05:13.745230 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:13.744862 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6f466319-941b-4c0f-b872-f62f8c302442-config-out\") pod \"alertmanager-main-0\" (UID: \"6f466319-941b-4c0f-b872-f62f8c302442\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:05:13.745230 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:13.744908 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6f466319-941b-4c0f-b872-f62f8c302442-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6f466319-941b-4c0f-b872-f62f8c302442\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:05:13.745230 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:13.744971 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6f466319-941b-4c0f-b872-f62f8c302442-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6f466319-941b-4c0f-b872-f62f8c302442\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:05:13.745230 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:13.744998 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6f466319-941b-4c0f-b872-f62f8c302442-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6f466319-941b-4c0f-b872-f62f8c302442\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:05:13.745684 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:13.745340 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6f466319-941b-4c0f-b872-f62f8c302442-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6f466319-941b-4c0f-b872-f62f8c302442\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:05:13.745684 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:13.745636 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6f466319-941b-4c0f-b872-f62f8c302442-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6f466319-941b-4c0f-b872-f62f8c302442\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:05:13.748103 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:13.748056 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6f466319-941b-4c0f-b872-f62f8c302442-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6f466319-941b-4c0f-b872-f62f8c302442\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:05:13.748216 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:13.748098 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6f466319-941b-4c0f-b872-f62f8c302442-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6f466319-941b-4c0f-b872-f62f8c302442\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:05:13.748216 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:13.748201 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6f466319-941b-4c0f-b872-f62f8c302442-config-volume\") pod \"alertmanager-main-0\" (UID: \"6f466319-941b-4c0f-b872-f62f8c302442\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:05:13.748391 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:13.748367 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6f466319-941b-4c0f-b872-f62f8c302442-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6f466319-941b-4c0f-b872-f62f8c302442\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:05:13.748781 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:13.748694 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6f466319-941b-4c0f-b872-f62f8c302442-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6f466319-941b-4c0f-b872-f62f8c302442\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:05:13.748947 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:13.748917 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6f466319-941b-4c0f-b872-f62f8c302442-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6f466319-941b-4c0f-b872-f62f8c302442\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:05:13.749029 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:13.748968 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6f466319-941b-4c0f-b872-f62f8c302442-web-config\") pod \"alertmanager-main-0\" (UID: \"6f466319-941b-4c0f-b872-f62f8c302442\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:05:13.749214 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:13.749196 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6f466319-941b-4c0f-b872-f62f8c302442-config-out\") pod \"alertmanager-main-0\" (UID: \"6f466319-941b-4c0f-b872-f62f8c302442\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:05:13.750530 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:13.750507 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6f466319-941b-4c0f-b872-f62f8c302442-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6f466319-941b-4c0f-b872-f62f8c302442\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:05:13.753724 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:13.753704 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gqnq\" (UniqueName: \"kubernetes.io/projected/6f466319-941b-4c0f-b872-f62f8c302442-kube-api-access-5gqnq\") pod \"alertmanager-main-0\" (UID: \"6f466319-941b-4c0f-b872-f62f8c302442\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:05:14.180047 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:14.179952 2564 generic.go:356] "Generic (PLEG): container finished" podID="91498a50-33c1-4b03-a193-83b60292acc3" containerID="ff4572bf2531f948966e0d3f4d36328dce555b2c832205cedba33a120ff722ea" exitCode=0 Jul 07 22:05:14.180047 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:14.180026 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p69r8" event={"ID":"91498a50-33c1-4b03-a193-83b60292acc3","Type":"ContainerDied","Data":"ff4572bf2531f948966e0d3f4d36328dce555b2c832205cedba33a120ff722ea"} Jul 07 22:05:14.247621 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:14.247592 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f466319-941b-4c0f-b872-f62f8c302442-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6f466319-941b-4c0f-b872-f62f8c302442\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:05:14.248581 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:14.248561 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f466319-941b-4c0f-b872-f62f8c302442-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6f466319-941b-4c0f-b872-f62f8c302442\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:05:14.525882 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:14.525789 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:05:14.660779 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:14.660744 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 22:05:14.664700 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:05:14.664664 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f466319_941b_4c0f_b872_f62f8c302442.slice/crio-46c3912d7073eca00201eaea71121ca240a44d98c541f6c34614063dbf97d71d WatchSource:0}: Error finding container 46c3912d7073eca00201eaea71121ca240a44d98c541f6c34614063dbf97d71d: Status 404 returned error can't find the container with id 46c3912d7073eca00201eaea71121ca240a44d98c541f6c34614063dbf97d71d Jul 07 22:05:15.185097 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:15.185043 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p69r8" event={"ID":"91498a50-33c1-4b03-a193-83b60292acc3","Type":"ContainerStarted","Data":"dc35e7fec597cc1df1734992f8540879508b449893316581031ef26f17e93397"} Jul 07 22:05:15.185294 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:15.185110 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p69r8" event={"ID":"91498a50-33c1-4b03-a193-83b60292acc3","Type":"ContainerStarted","Data":"17688de714cbc1b815c022090ca0fad803e8d391df4d90b04a49465d5fcf9c4c"} Jul 07 22:05:15.186266 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:15.186244 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6f466319-941b-4c0f-b872-f62f8c302442","Type":"ContainerStarted","Data":"46c3912d7073eca00201eaea71121ca240a44d98c541f6c34614063dbf97d71d"} Jul 07 22:05:15.202835 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:15.202769 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-p69r8" podStartSLOduration=2.28677137 podStartE2EDuration="3.202753656s" totalImagesPullingTime="846.81288ms" totalInitContainerRuntime="69.169406ms" isStatefulPod=true podCreationTimestamp="2026-07-07 22:05:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:05:15.201702754 +0000 UTC m=+95.029832212" watchObservedRunningTime="2026-07-07 22:05:15.202753656 +0000 UTC m=+95.030883105" Jul 07 22:05:15.535301 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:15.535269 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7946669b5f-kgqtb"] Jul 07 22:05:15.539289 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:15.539269 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7946669b5f-kgqtb" Jul 07 22:05:15.541721 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:15.541698 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 07 22:05:15.541832 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:15.541697 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-1cu0g78vc7ies\"" Jul 07 22:05:15.541832 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:15.541763 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-h2zjr\"" Jul 07 22:05:15.541991 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:15.541977 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 07 22:05:15.542029 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:15.542007 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 07 22:05:15.542239 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:15.542222 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 07 22:05:15.542316 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:15.542259 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 07 22:05:15.556926 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:15.556902 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7946669b5f-kgqtb"] Jul 07 22:05:15.658705 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:15.658666 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7d7078ff-8283-4a7e-a95d-0ed75b3cba0a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7946669b5f-kgqtb\" (UID: \"7d7078ff-8283-4a7e-a95d-0ed75b3cba0a\") " pod="openshift-monitoring/thanos-querier-7946669b5f-kgqtb" Jul 07 22:05:15.658705 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:15.658702 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d7078ff-8283-4a7e-a95d-0ed75b3cba0a-metrics-client-ca\") pod \"thanos-querier-7946669b5f-kgqtb\" (UID: \"7d7078ff-8283-4a7e-a95d-0ed75b3cba0a\") " pod="openshift-monitoring/thanos-querier-7946669b5f-kgqtb" Jul 07 22:05:15.658937 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:15.658735 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7d7078ff-8283-4a7e-a95d-0ed75b3cba0a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7946669b5f-kgqtb\" (UID: \"7d7078ff-8283-4a7e-a95d-0ed75b3cba0a\") " pod="openshift-monitoring/thanos-querier-7946669b5f-kgqtb" Jul 07 22:05:15.658937 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:15.658797 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7d7078ff-8283-4a7e-a95d-0ed75b3cba0a-secret-thanos-querier-tls\") pod \"thanos-querier-7946669b5f-kgqtb\" (UID: \"7d7078ff-8283-4a7e-a95d-0ed75b3cba0a\") " pod="openshift-monitoring/thanos-querier-7946669b5f-kgqtb" Jul 07 22:05:15.658937 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:15.658826 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgml6\" (UniqueName: \"kubernetes.io/projected/7d7078ff-8283-4a7e-a95d-0ed75b3cba0a-kube-api-access-bgml6\") pod \"thanos-querier-7946669b5f-kgqtb\" (UID: \"7d7078ff-8283-4a7e-a95d-0ed75b3cba0a\") " pod="openshift-monitoring/thanos-querier-7946669b5f-kgqtb" Jul 07 22:05:15.658937 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:15.658849 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7d7078ff-8283-4a7e-a95d-0ed75b3cba0a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7946669b5f-kgqtb\" (UID: \"7d7078ff-8283-4a7e-a95d-0ed75b3cba0a\") " pod="openshift-monitoring/thanos-querier-7946669b5f-kgqtb" Jul 07 22:05:15.658937 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:15.658868 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7d7078ff-8283-4a7e-a95d-0ed75b3cba0a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7946669b5f-kgqtb\" (UID: \"7d7078ff-8283-4a7e-a95d-0ed75b3cba0a\") " pod="openshift-monitoring/thanos-querier-7946669b5f-kgqtb" Jul 07 22:05:15.658937 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:15.658883 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7d7078ff-8283-4a7e-a95d-0ed75b3cba0a-secret-grpc-tls\") pod \"thanos-querier-7946669b5f-kgqtb\" (UID: \"7d7078ff-8283-4a7e-a95d-0ed75b3cba0a\") " pod="openshift-monitoring/thanos-querier-7946669b5f-kgqtb" Jul 07 22:05:15.759955 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:15.759917 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7d7078ff-8283-4a7e-a95d-0ed75b3cba0a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7946669b5f-kgqtb\" (UID: \"7d7078ff-8283-4a7e-a95d-0ed75b3cba0a\") " pod="openshift-monitoring/thanos-querier-7946669b5f-kgqtb" Jul 07 22:05:15.760458 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:15.759985 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7d7078ff-8283-4a7e-a95d-0ed75b3cba0a-secret-thanos-querier-tls\") pod \"thanos-querier-7946669b5f-kgqtb\" (UID: \"7d7078ff-8283-4a7e-a95d-0ed75b3cba0a\") " pod="openshift-monitoring/thanos-querier-7946669b5f-kgqtb" Jul 07 22:05:15.760458 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:15.760027 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgml6\" (UniqueName: \"kubernetes.io/projected/7d7078ff-8283-4a7e-a95d-0ed75b3cba0a-kube-api-access-bgml6\") pod \"thanos-querier-7946669b5f-kgqtb\" (UID: \"7d7078ff-8283-4a7e-a95d-0ed75b3cba0a\") " pod="openshift-monitoring/thanos-querier-7946669b5f-kgqtb" Jul 07 22:05:15.760458 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:15.760057 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7d7078ff-8283-4a7e-a95d-0ed75b3cba0a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7946669b5f-kgqtb\" (UID: \"7d7078ff-8283-4a7e-a95d-0ed75b3cba0a\") " pod="openshift-monitoring/thanos-querier-7946669b5f-kgqtb" Jul 07 22:05:15.760458 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:15.760090 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7d7078ff-8283-4a7e-a95d-0ed75b3cba0a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7946669b5f-kgqtb\" (UID: \"7d7078ff-8283-4a7e-a95d-0ed75b3cba0a\") " pod="openshift-monitoring/thanos-querier-7946669b5f-kgqtb" Jul 07 22:05:15.760458 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:15.760115 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7d7078ff-8283-4a7e-a95d-0ed75b3cba0a-secret-grpc-tls\") pod \"thanos-querier-7946669b5f-kgqtb\" (UID: \"7d7078ff-8283-4a7e-a95d-0ed75b3cba0a\") " pod="openshift-monitoring/thanos-querier-7946669b5f-kgqtb" Jul 07 22:05:15.760458 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:15.760147 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7d7078ff-8283-4a7e-a95d-0ed75b3cba0a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7946669b5f-kgqtb\" (UID: \"7d7078ff-8283-4a7e-a95d-0ed75b3cba0a\") " pod="openshift-monitoring/thanos-querier-7946669b5f-kgqtb" Jul 07 22:05:15.760458 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:15.760173 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d7078ff-8283-4a7e-a95d-0ed75b3cba0a-metrics-client-ca\") pod \"thanos-querier-7946669b5f-kgqtb\" (UID: \"7d7078ff-8283-4a7e-a95d-0ed75b3cba0a\") " pod="openshift-monitoring/thanos-querier-7946669b5f-kgqtb" Jul 07 22:05:15.760975 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:15.760917 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d7078ff-8283-4a7e-a95d-0ed75b3cba0a-metrics-client-ca\") pod \"thanos-querier-7946669b5f-kgqtb\" (UID: \"7d7078ff-8283-4a7e-a95d-0ed75b3cba0a\") " pod="openshift-monitoring/thanos-querier-7946669b5f-kgqtb" Jul 07 22:05:15.763465 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:15.763422 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7d7078ff-8283-4a7e-a95d-0ed75b3cba0a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7946669b5f-kgqtb\" (UID: \"7d7078ff-8283-4a7e-a95d-0ed75b3cba0a\") " pod="openshift-monitoring/thanos-querier-7946669b5f-kgqtb" Jul 07 22:05:15.763557 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:15.763462 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7d7078ff-8283-4a7e-a95d-0ed75b3cba0a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7946669b5f-kgqtb\" (UID: \"7d7078ff-8283-4a7e-a95d-0ed75b3cba0a\") " pod="openshift-monitoring/thanos-querier-7946669b5f-kgqtb" Jul 07 22:05:15.763557 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:15.763545 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7d7078ff-8283-4a7e-a95d-0ed75b3cba0a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7946669b5f-kgqtb\" (UID: \"7d7078ff-8283-4a7e-a95d-0ed75b3cba0a\") " pod="openshift-monitoring/thanos-querier-7946669b5f-kgqtb" Jul 07 22:05:15.764012 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:15.763992 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7d7078ff-8283-4a7e-a95d-0ed75b3cba0a-secret-thanos-querier-tls\") pod \"thanos-querier-7946669b5f-kgqtb\" (UID: \"7d7078ff-8283-4a7e-a95d-0ed75b3cba0a\") " pod="openshift-monitoring/thanos-querier-7946669b5f-kgqtb" Jul 07 22:05:15.764061 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:15.764006 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7d7078ff-8283-4a7e-a95d-0ed75b3cba0a-secret-grpc-tls\") pod \"thanos-querier-7946669b5f-kgqtb\" (UID: \"7d7078ff-8283-4a7e-a95d-0ed75b3cba0a\") " pod="openshift-monitoring/thanos-querier-7946669b5f-kgqtb" Jul 07 22:05:15.764644 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:15.764621 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7d7078ff-8283-4a7e-a95d-0ed75b3cba0a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7946669b5f-kgqtb\" (UID: \"7d7078ff-8283-4a7e-a95d-0ed75b3cba0a\") " pod="openshift-monitoring/thanos-querier-7946669b5f-kgqtb" Jul 07 22:05:15.769893 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:15.769870 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgml6\" (UniqueName: \"kubernetes.io/projected/7d7078ff-8283-4a7e-a95d-0ed75b3cba0a-kube-api-access-bgml6\") pod \"thanos-querier-7946669b5f-kgqtb\" (UID: \"7d7078ff-8283-4a7e-a95d-0ed75b3cba0a\") " pod="openshift-monitoring/thanos-querier-7946669b5f-kgqtb" Jul 07 22:05:15.851427 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:15.851354 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7946669b5f-kgqtb" Jul 07 22:05:16.229599 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:16.229569 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7946669b5f-kgqtb"] Jul 07 22:05:16.233683 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:05:16.233653 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d7078ff_8283_4a7e_a95d_0ed75b3cba0a.slice/crio-98ba10968d59906f5b855916f3f8310c3090c680cdf8fd05e028d2d0c0b61b4f WatchSource:0}: Error finding container 98ba10968d59906f5b855916f3f8310c3090c680cdf8fd05e028d2d0c0b61b4f: Status 404 returned error can't find the container with id 98ba10968d59906f5b855916f3f8310c3090c680cdf8fd05e028d2d0c0b61b4f Jul 07 22:05:17.195849 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:17.195812 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7946669b5f-kgqtb" event={"ID":"7d7078ff-8283-4a7e-a95d-0ed75b3cba0a","Type":"ContainerStarted","Data":"98ba10968d59906f5b855916f3f8310c3090c680cdf8fd05e028d2d0c0b61b4f"} Jul 07 22:05:17.197656 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:17.197627 2564 generic.go:356] "Generic (PLEG): container finished" podID="6f466319-941b-4c0f-b872-f62f8c302442" containerID="07214d1820a7807f72513515c17137820c61bd39b119466a92bac6b1c5b999c6" exitCode=0 Jul 07 22:05:17.197771 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:17.197706 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6f466319-941b-4c0f-b872-f62f8c302442","Type":"ContainerDied","Data":"07214d1820a7807f72513515c17137820c61bd39b119466a92bac6b1c5b999c6"} Jul 07 22:05:19.207917 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:19.207836 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7946669b5f-kgqtb" event={"ID":"7d7078ff-8283-4a7e-a95d-0ed75b3cba0a","Type":"ContainerStarted","Data":"624d34632a89902e00a8c82bec58cda0297a02333e106ba26d1d55da291cbd17"} Jul 07 22:05:19.208278 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:19.207919 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7946669b5f-kgqtb" event={"ID":"7d7078ff-8283-4a7e-a95d-0ed75b3cba0a","Type":"ContainerStarted","Data":"3d8f6a09ae4740bd1bfd2d441d30ac3c615041204e6e49f08eb71995d3473799"} Jul 07 22:05:19.208278 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:19.207961 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7946669b5f-kgqtb" event={"ID":"7d7078ff-8283-4a7e-a95d-0ed75b3cba0a","Type":"ContainerStarted","Data":"51fe7c43604c506702603679a08c7ccfb0a0d4a1b11dea3ec6cbde16c18cffb1"} Jul 07 22:05:20.214753 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:20.214716 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6f466319-941b-4c0f-b872-f62f8c302442","Type":"ContainerStarted","Data":"d23bf019a9f03edd24e6aff80d801fc2ff3b76ae4f5879a80f5ff5905b7cf9fe"} Jul 07 22:05:20.215200 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:20.214812 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6f466319-941b-4c0f-b872-f62f8c302442","Type":"ContainerStarted","Data":"52589d7254ebf23aae86fa9b2a1b356f6e9d49d5a0e99afa74d4e8c4951fecca"} Jul 07 22:05:20.215200 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:20.214846 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6f466319-941b-4c0f-b872-f62f8c302442","Type":"ContainerStarted","Data":"c6caae3d03ad73b22ae5c4d83b40244b874f723599a24a0faafa2c58fbb0956c"} Jul 07 22:05:20.215200 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:20.214876 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6f466319-941b-4c0f-b872-f62f8c302442","Type":"ContainerStarted","Data":"2a19268d694f7a0de841057889487456af480e0d878ab8dd69092d2f7fd68562"} Jul 07 22:05:20.215200 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:20.214918 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6f466319-941b-4c0f-b872-f62f8c302442","Type":"ContainerStarted","Data":"0e60fc74aa23d70e8d3a4ca089f8555e167505f2f9ef6ab55568196bde2ab9ba"} Jul 07 22:05:20.215200 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:20.214948 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6f466319-941b-4c0f-b872-f62f8c302442","Type":"ContainerStarted","Data":"27b427ea1e6831e81be1e46598af4204ddd0d1f74c08a935e81a3666a383b10f"} Jul 07 22:05:20.217861 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:20.217834 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7946669b5f-kgqtb" event={"ID":"7d7078ff-8283-4a7e-a95d-0ed75b3cba0a","Type":"ContainerStarted","Data":"adeb8a27706e570d3b8284223c7e37a815a2254228681471ff0372dc8267d987"} Jul 07 22:05:20.217967 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:20.217902 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7946669b5f-kgqtb" event={"ID":"7d7078ff-8283-4a7e-a95d-0ed75b3cba0a","Type":"ContainerStarted","Data":"5a0637f044bb3b6ea03ef5b4508b6890b1279827c524de78d077df4ce70dfe9a"} Jul 07 22:05:20.217967 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:20.217942 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7946669b5f-kgqtb" event={"ID":"7d7078ff-8283-4a7e-a95d-0ed75b3cba0a","Type":"ContainerStarted","Data":"c5c9f9497296b4c685881852c28f8604f543ac559568895933be2dbf029d5c3f"} Jul 07 22:05:20.218045 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:20.218028 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7946669b5f-kgqtb" Jul 07 22:05:20.243848 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:20.243797 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.373953805 podStartE2EDuration="7.243778129s" totalImagesPullingTime="3.850899875s" totalInitContainerRuntime="18.924449ms" isStatefulPod=false podCreationTimestamp="2026-07-07 22:05:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:05:20.241861846 +0000 UTC m=+100.069991304" watchObservedRunningTime="2026-07-07 22:05:20.243778129 +0000 UTC m=+100.071907589" Jul 07 22:05:20.267608 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:20.267558 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7946669b5f-kgqtb" podStartSLOduration=1.866744015 podStartE2EDuration="5.267544415s" totalImagesPullingTime="3.4008004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 22:05:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:05:20.266433687 +0000 UTC m=+100.094563145" watchObservedRunningTime="2026-07-07 22:05:20.267544415 +0000 UTC m=+100.095673873" Jul 07 22:05:26.227088 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:26.227060 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7946669b5f-kgqtb" Jul 07 22:05:27.120306 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:05:27.120267 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zlr8s" Jul 07 22:07:45.955740 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:07:45.955701 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lhwnm"] Jul 07 22:07:45.959237 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:07:45.959215 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lhwnm" Jul 07 22:07:45.961799 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:07:45.961780 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-c8np2\"" Jul 07 22:07:45.961904 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:07:45.961783 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 07 22:07:45.961904 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:07:45.961840 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 07 22:07:45.974628 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:07:45.971837 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lhwnm"] Jul 07 22:07:46.095095 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:07:46.095046 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq2lg\" (UniqueName: \"kubernetes.io/projected/2c211266-8f52-4c4f-b0f7-6b05dce41f32-kube-api-access-hq2lg\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lhwnm\" (UID: \"2c211266-8f52-4c4f-b0f7-6b05dce41f32\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lhwnm" Jul 07 22:07:46.095287 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:07:46.095108 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c211266-8f52-4c4f-b0f7-6b05dce41f32-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lhwnm\" (UID: \"2c211266-8f52-4c4f-b0f7-6b05dce41f32\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lhwnm" Jul 07 22:07:46.095287 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:07:46.095134 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c211266-8f52-4c4f-b0f7-6b05dce41f32-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lhwnm\" (UID: \"2c211266-8f52-4c4f-b0f7-6b05dce41f32\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lhwnm" Jul 07 22:07:46.196128 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:07:46.196095 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq2lg\" (UniqueName: \"kubernetes.io/projected/2c211266-8f52-4c4f-b0f7-6b05dce41f32-kube-api-access-hq2lg\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lhwnm\" (UID: \"2c211266-8f52-4c4f-b0f7-6b05dce41f32\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lhwnm" Jul 07 22:07:46.196305 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:07:46.196143 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c211266-8f52-4c4f-b0f7-6b05dce41f32-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lhwnm\" (UID: \"2c211266-8f52-4c4f-b0f7-6b05dce41f32\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lhwnm" Jul 07 22:07:46.196305 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:07:46.196166 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c211266-8f52-4c4f-b0f7-6b05dce41f32-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lhwnm\" (UID: \"2c211266-8f52-4c4f-b0f7-6b05dce41f32\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lhwnm" Jul 07 22:07:46.196521 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:07:46.196501 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c211266-8f52-4c4f-b0f7-6b05dce41f32-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lhwnm\" (UID: \"2c211266-8f52-4c4f-b0f7-6b05dce41f32\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lhwnm" Jul 07 22:07:46.196571 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:07:46.196530 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c211266-8f52-4c4f-b0f7-6b05dce41f32-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lhwnm\" (UID: \"2c211266-8f52-4c4f-b0f7-6b05dce41f32\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lhwnm" Jul 07 22:07:46.204502 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:07:46.204480 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq2lg\" (UniqueName: \"kubernetes.io/projected/2c211266-8f52-4c4f-b0f7-6b05dce41f32-kube-api-access-hq2lg\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lhwnm\" (UID: \"2c211266-8f52-4c4f-b0f7-6b05dce41f32\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lhwnm" Jul 07 22:07:46.268270 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:07:46.268177 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lhwnm" Jul 07 22:07:46.398525 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:07:46.398496 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lhwnm"] Jul 07 22:07:46.401739 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:07:46.401704 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c211266_8f52_4c4f_b0f7_6b05dce41f32.slice/crio-cefccb64fa1a16ec8c2082c94d997a1405ef813a439a5e9605197c75eb0a5188 WatchSource:0}: Error finding container cefccb64fa1a16ec8c2082c94d997a1405ef813a439a5e9605197c75eb0a5188: Status 404 returned error can't find the container with id cefccb64fa1a16ec8c2082c94d997a1405ef813a439a5e9605197c75eb0a5188 Jul 07 22:07:46.738237 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:07:46.738197 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lhwnm" event={"ID":"2c211266-8f52-4c4f-b0f7-6b05dce41f32","Type":"ContainerStarted","Data":"cefccb64fa1a16ec8c2082c94d997a1405ef813a439a5e9605197c75eb0a5188"} Jul 07 22:07:53.767673 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:07:53.767588 2564 generic.go:356] "Generic (PLEG): container finished" podID="2c211266-8f52-4c4f-b0f7-6b05dce41f32" containerID="2956f3a1db09cf16ca4060e324268194b9cab245b9be6a3c3bc9c3d1d562bb12" exitCode=0 Jul 07 22:07:53.768039 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:07:53.767671 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lhwnm" event={"ID":"2c211266-8f52-4c4f-b0f7-6b05dce41f32","Type":"ContainerDied","Data":"2956f3a1db09cf16ca4060e324268194b9cab245b9be6a3c3bc9c3d1d562bb12"} Jul 07 22:07:56.780429 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:07:56.780394 2564 generic.go:356] "Generic (PLEG): container finished" podID="2c211266-8f52-4c4f-b0f7-6b05dce41f32" containerID="3e3efd3d57b561416d6a495a78dd90123fc93678cb7b750291db8666004d5be2" exitCode=0 Jul 07 22:07:56.780818 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:07:56.780486 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lhwnm" event={"ID":"2c211266-8f52-4c4f-b0f7-6b05dce41f32","Type":"ContainerDied","Data":"3e3efd3d57b561416d6a495a78dd90123fc93678cb7b750291db8666004d5be2"} Jul 07 22:08:04.812956 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:04.812919 2564 generic.go:356] "Generic (PLEG): container finished" podID="2c211266-8f52-4c4f-b0f7-6b05dce41f32" containerID="24e48fc68768470de2d7127c9ef369948f3e15594f3e12643280f600600502d9" exitCode=0 Jul 07 22:08:04.813325 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:04.812999 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lhwnm" event={"ID":"2c211266-8f52-4c4f-b0f7-6b05dce41f32","Type":"ContainerDied","Data":"24e48fc68768470de2d7127c9ef369948f3e15594f3e12643280f600600502d9"} Jul 07 22:08:05.941365 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:05.941338 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lhwnm" Jul 07 22:08:06.064845 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:06.064810 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hq2lg\" (UniqueName: \"kubernetes.io/projected/2c211266-8f52-4c4f-b0f7-6b05dce41f32-kube-api-access-hq2lg\") pod \"2c211266-8f52-4c4f-b0f7-6b05dce41f32\" (UID: \"2c211266-8f52-4c4f-b0f7-6b05dce41f32\") " Jul 07 22:08:06.065013 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:06.064879 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c211266-8f52-4c4f-b0f7-6b05dce41f32-bundle\") pod \"2c211266-8f52-4c4f-b0f7-6b05dce41f32\" (UID: \"2c211266-8f52-4c4f-b0f7-6b05dce41f32\") " Jul 07 22:08:06.065013 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:06.064911 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c211266-8f52-4c4f-b0f7-6b05dce41f32-util\") pod \"2c211266-8f52-4c4f-b0f7-6b05dce41f32\" (UID: \"2c211266-8f52-4c4f-b0f7-6b05dce41f32\") " Jul 07 22:08:06.065631 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:06.065605 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c211266-8f52-4c4f-b0f7-6b05dce41f32-bundle" (OuterVolumeSpecName: "bundle") pod "2c211266-8f52-4c4f-b0f7-6b05dce41f32" (UID: "2c211266-8f52-4c4f-b0f7-6b05dce41f32"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 22:08:06.067283 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:06.067235 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c211266-8f52-4c4f-b0f7-6b05dce41f32-kube-api-access-hq2lg" (OuterVolumeSpecName: "kube-api-access-hq2lg") pod "2c211266-8f52-4c4f-b0f7-6b05dce41f32" (UID: "2c211266-8f52-4c4f-b0f7-6b05dce41f32"). InnerVolumeSpecName "kube-api-access-hq2lg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 22:08:06.069122 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:06.069102 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c211266-8f52-4c4f-b0f7-6b05dce41f32-util" (OuterVolumeSpecName: "util") pod "2c211266-8f52-4c4f-b0f7-6b05dce41f32" (UID: "2c211266-8f52-4c4f-b0f7-6b05dce41f32"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 22:08:06.165300 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:06.165275 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c211266-8f52-4c4f-b0f7-6b05dce41f32-bundle\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 07 22:08:06.165300 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:06.165295 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c211266-8f52-4c4f-b0f7-6b05dce41f32-util\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 07 22:08:06.165300 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:06.165304 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hq2lg\" (UniqueName: \"kubernetes.io/projected/2c211266-8f52-4c4f-b0f7-6b05dce41f32-kube-api-access-hq2lg\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 07 22:08:06.821891 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:06.821863 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lhwnm" Jul 07 22:08:06.822054 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:06.821862 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lhwnm" event={"ID":"2c211266-8f52-4c4f-b0f7-6b05dce41f32","Type":"ContainerDied","Data":"cefccb64fa1a16ec8c2082c94d997a1405ef813a439a5e9605197c75eb0a5188"} Jul 07 22:08:06.822054 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:06.821979 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cefccb64fa1a16ec8c2082c94d997a1405ef813a439a5e9605197c75eb0a5188" Jul 07 22:08:13.744550 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:13.744518 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-np5kw"] Jul 07 22:08:13.744943 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:13.744785 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c211266-8f52-4c4f-b0f7-6b05dce41f32" containerName="pull" Jul 07 22:08:13.744943 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:13.744795 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c211266-8f52-4c4f-b0f7-6b05dce41f32" containerName="pull" Jul 07 22:08:13.744943 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:13.744804 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c211266-8f52-4c4f-b0f7-6b05dce41f32" containerName="extract" Jul 07 22:08:13.744943 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:13.744810 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c211266-8f52-4c4f-b0f7-6b05dce41f32" containerName="extract" Jul 07 22:08:13.744943 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:13.744819 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c211266-8f52-4c4f-b0f7-6b05dce41f32" containerName="util" Jul 07 22:08:13.744943 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:13.744824 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c211266-8f52-4c4f-b0f7-6b05dce41f32" containerName="util" Jul 07 22:08:13.744943 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:13.744875 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c211266-8f52-4c4f-b0f7-6b05dce41f32" containerName="extract" Jul 07 22:08:13.748242 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:13.748225 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-np5kw" Jul 07 22:08:13.750646 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:13.750623 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 07 22:08:13.750789 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:13.750651 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 07 22:08:13.750789 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:13.750692 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-mqfm8\"" Jul 07 22:08:13.759175 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:13.759155 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-np5kw"] Jul 07 22:08:13.928958 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:13.928918 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3fd398b1-60ae-43a1-99e6-e51d5cca60b0-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-np5kw\" (UID: \"3fd398b1-60ae-43a1-99e6-e51d5cca60b0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-np5kw" Jul 07 22:08:13.929135 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:13.928987 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvrmz\" (UniqueName: \"kubernetes.io/projected/3fd398b1-60ae-43a1-99e6-e51d5cca60b0-kube-api-access-xvrmz\") pod \"cert-manager-operator-controller-manager-55c444bf86-np5kw\" (UID: \"3fd398b1-60ae-43a1-99e6-e51d5cca60b0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-np5kw" Jul 07 22:08:14.029862 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:14.029789 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvrmz\" (UniqueName: \"kubernetes.io/projected/3fd398b1-60ae-43a1-99e6-e51d5cca60b0-kube-api-access-xvrmz\") pod \"cert-manager-operator-controller-manager-55c444bf86-np5kw\" (UID: \"3fd398b1-60ae-43a1-99e6-e51d5cca60b0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-np5kw" Jul 07 22:08:14.029862 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:14.029854 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3fd398b1-60ae-43a1-99e6-e51d5cca60b0-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-np5kw\" (UID: \"3fd398b1-60ae-43a1-99e6-e51d5cca60b0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-np5kw" Jul 07 22:08:14.030201 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:14.030182 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3fd398b1-60ae-43a1-99e6-e51d5cca60b0-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-np5kw\" (UID: \"3fd398b1-60ae-43a1-99e6-e51d5cca60b0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-np5kw" Jul 07 22:08:14.038285 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:14.038261 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvrmz\" (UniqueName: \"kubernetes.io/projected/3fd398b1-60ae-43a1-99e6-e51d5cca60b0-kube-api-access-xvrmz\") pod \"cert-manager-operator-controller-manager-55c444bf86-np5kw\" (UID: \"3fd398b1-60ae-43a1-99e6-e51d5cca60b0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-np5kw" Jul 07 22:08:14.057155 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:14.057129 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-np5kw" Jul 07 22:08:14.188084 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:14.188053 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-np5kw"] Jul 07 22:08:14.191339 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:08:14.191291 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fd398b1_60ae_43a1_99e6_e51d5cca60b0.slice/crio-841073a46bf06e3fce39734894175c0094a49ff79e23b69aaa0901b5d8402def WatchSource:0}: Error finding container 841073a46bf06e3fce39734894175c0094a49ff79e23b69aaa0901b5d8402def: Status 404 returned error can't find the container with id 841073a46bf06e3fce39734894175c0094a49ff79e23b69aaa0901b5d8402def Jul 07 22:08:14.850083 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:14.850051 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-np5kw" event={"ID":"3fd398b1-60ae-43a1-99e6-e51d5cca60b0","Type":"ContainerStarted","Data":"841073a46bf06e3fce39734894175c0094a49ff79e23b69aaa0901b5d8402def"} Jul 07 22:08:15.857664 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:15.857629 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-np5kw" event={"ID":"3fd398b1-60ae-43a1-99e6-e51d5cca60b0","Type":"ContainerStarted","Data":"69989609c54875268ad3a023c461a08c1d371a37ea4e9cc18870b712161e506a"} Jul 07 22:08:15.876565 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:15.876387 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-np5kw" podStartSLOduration=1.325928347 podStartE2EDuration="2.876366434s" totalImagesPullingTime="1.550438087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 22:08:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:08:15.876156618 +0000 UTC m=+275.704286076" watchObservedRunningTime="2026-07-07 22:08:15.876366434 +0000 UTC m=+275.704495892" Jul 07 22:08:17.189133 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:17.189099 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcz4wx"] Jul 07 22:08:17.191685 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:17.191664 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcz4wx" Jul 07 22:08:17.193924 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:17.193906 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 07 22:08:17.194021 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:17.193921 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 07 22:08:17.194621 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:17.194607 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-c8np2\"" Jul 07 22:08:17.200631 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:17.200606 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcz4wx"] Jul 07 22:08:17.356827 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:17.356798 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r4sf\" (UniqueName: \"kubernetes.io/projected/05c251e2-819a-46bc-81c3-dacde53f661d-kube-api-access-9r4sf\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcz4wx\" (UID: \"05c251e2-819a-46bc-81c3-dacde53f661d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcz4wx" Jul 07 22:08:17.356972 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:17.356855 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05c251e2-819a-46bc-81c3-dacde53f661d-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcz4wx\" (UID: \"05c251e2-819a-46bc-81c3-dacde53f661d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcz4wx" Jul 07 22:08:17.356972 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:17.356879 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05c251e2-819a-46bc-81c3-dacde53f661d-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcz4wx\" (UID: \"05c251e2-819a-46bc-81c3-dacde53f661d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcz4wx" Jul 07 22:08:17.457633 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:17.457553 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r4sf\" (UniqueName: \"kubernetes.io/projected/05c251e2-819a-46bc-81c3-dacde53f661d-kube-api-access-9r4sf\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcz4wx\" (UID: \"05c251e2-819a-46bc-81c3-dacde53f661d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcz4wx" Jul 07 22:08:17.457633 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:17.457620 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05c251e2-819a-46bc-81c3-dacde53f661d-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcz4wx\" (UID: \"05c251e2-819a-46bc-81c3-dacde53f661d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcz4wx" Jul 07 22:08:17.457844 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:17.457640 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05c251e2-819a-46bc-81c3-dacde53f661d-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcz4wx\" (UID: \"05c251e2-819a-46bc-81c3-dacde53f661d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcz4wx" Jul 07 22:08:17.457954 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:17.457938 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05c251e2-819a-46bc-81c3-dacde53f661d-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcz4wx\" (UID: \"05c251e2-819a-46bc-81c3-dacde53f661d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcz4wx" Jul 07 22:08:17.458059 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:17.458038 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05c251e2-819a-46bc-81c3-dacde53f661d-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcz4wx\" (UID: \"05c251e2-819a-46bc-81c3-dacde53f661d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcz4wx" Jul 07 22:08:17.466036 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:17.466009 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r4sf\" (UniqueName: \"kubernetes.io/projected/05c251e2-819a-46bc-81c3-dacde53f661d-kube-api-access-9r4sf\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcz4wx\" (UID: \"05c251e2-819a-46bc-81c3-dacde53f661d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcz4wx" Jul 07 22:08:17.501022 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:17.500998 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcz4wx" Jul 07 22:08:17.638706 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:17.638680 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcz4wx"] Jul 07 22:08:17.641208 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:08:17.641184 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05c251e2_819a_46bc_81c3_dacde53f661d.slice/crio-20236ceba8259fe7c1f469a9ba25503ae1d9142e46fcdd0fe39d02958d269715 WatchSource:0}: Error finding container 20236ceba8259fe7c1f469a9ba25503ae1d9142e46fcdd0fe39d02958d269715: Status 404 returned error can't find the container with id 20236ceba8259fe7c1f469a9ba25503ae1d9142e46fcdd0fe39d02958d269715 Jul 07 22:08:17.866128 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:17.866095 2564 generic.go:356] "Generic (PLEG): container finished" podID="05c251e2-819a-46bc-81c3-dacde53f661d" containerID="91f675aee34a826b9b4bb1511405e10d607b9fe0bb99ba00809a476ce56914e5" exitCode=0 Jul 07 22:08:17.866299 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:17.866172 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcz4wx" event={"ID":"05c251e2-819a-46bc-81c3-dacde53f661d","Type":"ContainerDied","Data":"91f675aee34a826b9b4bb1511405e10d607b9fe0bb99ba00809a476ce56914e5"} Jul 07 22:08:17.866299 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:17.866210 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcz4wx" event={"ID":"05c251e2-819a-46bc-81c3-dacde53f661d","Type":"ContainerStarted","Data":"20236ceba8259fe7c1f469a9ba25503ae1d9142e46fcdd0fe39d02958d269715"} Jul 07 22:08:19.877091 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:19.877060 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcz4wx" event={"ID":"05c251e2-819a-46bc-81c3-dacde53f661d","Type":"ContainerStarted","Data":"54e3fafedd1775bcee2e5d9be744c779fdfd6d604a0583553df587973e21954f"} Jul 07 22:08:20.494249 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:20.494210 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-nfjnm"] Jul 07 22:08:20.496425 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:20.496409 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-nfjnm" Jul 07 22:08:20.498768 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:20.498734 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 07 22:08:20.498869 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:20.498763 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-mzt46\"" Jul 07 22:08:20.499291 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:20.499278 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 07 22:08:20.505040 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:20.505019 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-nfjnm"] Jul 07 22:08:20.582161 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:20.582127 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8x5n\" (UniqueName: \"kubernetes.io/projected/5ad21ae4-db3e-497a-9f7d-633ffe8f18b9-kube-api-access-j8x5n\") pod \"cert-manager-cainjector-559cdd8c87-nfjnm\" (UID: \"5ad21ae4-db3e-497a-9f7d-633ffe8f18b9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-nfjnm" Jul 07 22:08:20.582314 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:20.582186 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5ad21ae4-db3e-497a-9f7d-633ffe8f18b9-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-nfjnm\" (UID: \"5ad21ae4-db3e-497a-9f7d-633ffe8f18b9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-nfjnm" Jul 07 22:08:20.682562 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:20.682532 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8x5n\" (UniqueName: \"kubernetes.io/projected/5ad21ae4-db3e-497a-9f7d-633ffe8f18b9-kube-api-access-j8x5n\") pod \"cert-manager-cainjector-559cdd8c87-nfjnm\" (UID: \"5ad21ae4-db3e-497a-9f7d-633ffe8f18b9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-nfjnm" Jul 07 22:08:20.682700 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:20.682577 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5ad21ae4-db3e-497a-9f7d-633ffe8f18b9-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-nfjnm\" (UID: \"5ad21ae4-db3e-497a-9f7d-633ffe8f18b9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-nfjnm" Jul 07 22:08:20.690171 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:20.690145 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5ad21ae4-db3e-497a-9f7d-633ffe8f18b9-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-nfjnm\" (UID: \"5ad21ae4-db3e-497a-9f7d-633ffe8f18b9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-nfjnm" Jul 07 22:08:20.690271 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:20.690238 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8x5n\" (UniqueName: \"kubernetes.io/projected/5ad21ae4-db3e-497a-9f7d-633ffe8f18b9-kube-api-access-j8x5n\") pod \"cert-manager-cainjector-559cdd8c87-nfjnm\" (UID: \"5ad21ae4-db3e-497a-9f7d-633ffe8f18b9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-nfjnm" Jul 07 22:08:20.805536 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:20.805510 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-nfjnm" Jul 07 22:08:20.883608 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:20.883571 2564 generic.go:356] "Generic (PLEG): container finished" podID="05c251e2-819a-46bc-81c3-dacde53f661d" containerID="54e3fafedd1775bcee2e5d9be744c779fdfd6d604a0583553df587973e21954f" exitCode=0 Jul 07 22:08:20.884038 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:20.883630 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcz4wx" event={"ID":"05c251e2-819a-46bc-81c3-dacde53f661d","Type":"ContainerDied","Data":"54e3fafedd1775bcee2e5d9be744c779fdfd6d604a0583553df587973e21954f"} Jul 07 22:08:20.940868 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:20.939955 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-nfjnm"] Jul 07 22:08:21.890477 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:21.890432 2564 generic.go:356] "Generic (PLEG): container finished" podID="05c251e2-819a-46bc-81c3-dacde53f661d" containerID="15499b64d4f1180a479dd6f7cdaf617fd8e7db9a683cb6da63c9b8bab2b8704b" exitCode=0 Jul 07 22:08:21.890926 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:21.890526 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcz4wx" event={"ID":"05c251e2-819a-46bc-81c3-dacde53f661d","Type":"ContainerDied","Data":"15499b64d4f1180a479dd6f7cdaf617fd8e7db9a683cb6da63c9b8bab2b8704b"} Jul 07 22:08:21.891960 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:21.891934 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-nfjnm" event={"ID":"5ad21ae4-db3e-497a-9f7d-633ffe8f18b9","Type":"ContainerStarted","Data":"e53e310a9935b7e7133ba9df4d93d2b845aa72c9ddd66d580fbf58aee7e1e9fc"} Jul 07 22:08:23.402973 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:23.402949 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcz4wx" Jul 07 22:08:23.501964 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:23.501938 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05c251e2-819a-46bc-81c3-dacde53f661d-util\") pod \"05c251e2-819a-46bc-81c3-dacde53f661d\" (UID: \"05c251e2-819a-46bc-81c3-dacde53f661d\") " Jul 07 22:08:23.502091 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:23.501986 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05c251e2-819a-46bc-81c3-dacde53f661d-bundle\") pod \"05c251e2-819a-46bc-81c3-dacde53f661d\" (UID: \"05c251e2-819a-46bc-81c3-dacde53f661d\") " Jul 07 22:08:23.502091 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:23.502019 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9r4sf\" (UniqueName: \"kubernetes.io/projected/05c251e2-819a-46bc-81c3-dacde53f661d-kube-api-access-9r4sf\") pod \"05c251e2-819a-46bc-81c3-dacde53f661d\" (UID: \"05c251e2-819a-46bc-81c3-dacde53f661d\") " Jul 07 22:08:23.502360 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:23.502337 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05c251e2-819a-46bc-81c3-dacde53f661d-bundle" (OuterVolumeSpecName: "bundle") pod "05c251e2-819a-46bc-81c3-dacde53f661d" (UID: "05c251e2-819a-46bc-81c3-dacde53f661d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 22:08:23.504151 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:23.504127 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05c251e2-819a-46bc-81c3-dacde53f661d-kube-api-access-9r4sf" (OuterVolumeSpecName: "kube-api-access-9r4sf") pod "05c251e2-819a-46bc-81c3-dacde53f661d" (UID: "05c251e2-819a-46bc-81c3-dacde53f661d"). InnerVolumeSpecName "kube-api-access-9r4sf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 22:08:23.506769 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:23.506744 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05c251e2-819a-46bc-81c3-dacde53f661d-util" (OuterVolumeSpecName: "util") pod "05c251e2-819a-46bc-81c3-dacde53f661d" (UID: "05c251e2-819a-46bc-81c3-dacde53f661d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 22:08:23.603268 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:23.603239 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05c251e2-819a-46bc-81c3-dacde53f661d-util\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 07 22:08:23.603268 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:23.603264 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05c251e2-819a-46bc-81c3-dacde53f661d-bundle\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 07 22:08:23.603268 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:23.603273 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r4sf\" (UniqueName: \"kubernetes.io/projected/05c251e2-819a-46bc-81c3-dacde53f661d-kube-api-access-9r4sf\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 07 22:08:23.903138 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:23.903110 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcz4wx" Jul 07 22:08:23.903308 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:23.903103 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcz4wx" event={"ID":"05c251e2-819a-46bc-81c3-dacde53f661d","Type":"ContainerDied","Data":"20236ceba8259fe7c1f469a9ba25503ae1d9142e46fcdd0fe39d02958d269715"} Jul 07 22:08:23.903308 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:23.903247 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20236ceba8259fe7c1f469a9ba25503ae1d9142e46fcdd0fe39d02958d269715" Jul 07 22:08:23.904743 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:23.904722 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-nfjnm" event={"ID":"5ad21ae4-db3e-497a-9f7d-633ffe8f18b9","Type":"ContainerStarted","Data":"db00fc2b3107fb495a64739fc09fe54182955a7063cd0b788eb1bb2491f9c8da"} Jul 07 22:08:23.919463 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:23.919397 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-nfjnm" podStartSLOduration=1.4359482080000001 podStartE2EDuration="3.919384995s" totalImagesPullingTime="2.483436787s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 22:08:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:08:23.918091762 +0000 UTC m=+283.746221223" watchObservedRunningTime="2026-07-07 22:08:23.919384995 +0000 UTC m=+283.747514453" Jul 07 22:08:28.874101 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:28.874066 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-dgdjq"] Jul 07 22:08:28.874566 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:28.874336 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05c251e2-819a-46bc-81c3-dacde53f661d" containerName="pull" Jul 07 22:08:28.874566 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:28.874347 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="05c251e2-819a-46bc-81c3-dacde53f661d" containerName="pull" Jul 07 22:08:28.874566 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:28.874366 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05c251e2-819a-46bc-81c3-dacde53f661d" containerName="util" Jul 07 22:08:28.874566 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:28.874372 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="05c251e2-819a-46bc-81c3-dacde53f661d" containerName="util" Jul 07 22:08:28.874566 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:28.874382 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05c251e2-819a-46bc-81c3-dacde53f661d" containerName="extract" Jul 07 22:08:28.874566 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:28.874388 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="05c251e2-819a-46bc-81c3-dacde53f661d" containerName="extract" Jul 07 22:08:28.874566 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:28.874426 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="05c251e2-819a-46bc-81c3-dacde53f661d" containerName="extract" Jul 07 22:08:28.877430 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:28.877412 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-dgdjq" Jul 07 22:08:28.879746 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:28.879725 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-qfn7l\"" Jul 07 22:08:28.884469 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:28.884294 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-dgdjq"] Jul 07 22:08:28.946319 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:28.946292 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jph9\" (UniqueName: \"kubernetes.io/projected/bb2318b8-b92b-4ad1-a635-fb0d35c015e3-kube-api-access-8jph9\") pod \"cert-manager-55b4d954d5-dgdjq\" (UID: \"bb2318b8-b92b-4ad1-a635-fb0d35c015e3\") " pod="cert-manager/cert-manager-55b4d954d5-dgdjq" Jul 07 22:08:28.946490 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:28.946332 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bb2318b8-b92b-4ad1-a635-fb0d35c015e3-bound-sa-token\") pod \"cert-manager-55b4d954d5-dgdjq\" (UID: \"bb2318b8-b92b-4ad1-a635-fb0d35c015e3\") " pod="cert-manager/cert-manager-55b4d954d5-dgdjq" Jul 07 22:08:29.047135 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:29.047097 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jph9\" (UniqueName: \"kubernetes.io/projected/bb2318b8-b92b-4ad1-a635-fb0d35c015e3-kube-api-access-8jph9\") pod \"cert-manager-55b4d954d5-dgdjq\" (UID: \"bb2318b8-b92b-4ad1-a635-fb0d35c015e3\") " pod="cert-manager/cert-manager-55b4d954d5-dgdjq" Jul 07 22:08:29.047317 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:29.047149 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bb2318b8-b92b-4ad1-a635-fb0d35c015e3-bound-sa-token\") pod \"cert-manager-55b4d954d5-dgdjq\" (UID: \"bb2318b8-b92b-4ad1-a635-fb0d35c015e3\") " pod="cert-manager/cert-manager-55b4d954d5-dgdjq" Jul 07 22:08:29.055151 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:29.055118 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bb2318b8-b92b-4ad1-a635-fb0d35c015e3-bound-sa-token\") pod \"cert-manager-55b4d954d5-dgdjq\" (UID: \"bb2318b8-b92b-4ad1-a635-fb0d35c015e3\") " pod="cert-manager/cert-manager-55b4d954d5-dgdjq" Jul 07 22:08:29.055325 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:29.055303 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jph9\" (UniqueName: \"kubernetes.io/projected/bb2318b8-b92b-4ad1-a635-fb0d35c015e3-kube-api-access-8jph9\") pod \"cert-manager-55b4d954d5-dgdjq\" (UID: \"bb2318b8-b92b-4ad1-a635-fb0d35c015e3\") " pod="cert-manager/cert-manager-55b4d954d5-dgdjq" Jul 07 22:08:29.187404 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:29.187326 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-dgdjq" Jul 07 22:08:29.313723 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:29.313694 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-dgdjq"] Jul 07 22:08:29.316289 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:08:29.316260 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb2318b8_b92b_4ad1_a635_fb0d35c015e3.slice/crio-4aa7618d53258cb74dcf18ed034ac63794fcd391d3a5e8faf34bae554bcac322 WatchSource:0}: Error finding container 4aa7618d53258cb74dcf18ed034ac63794fcd391d3a5e8faf34bae554bcac322: Status 404 returned error can't find the container with id 4aa7618d53258cb74dcf18ed034ac63794fcd391d3a5e8faf34bae554bcac322 Jul 07 22:08:29.931274 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:29.931241 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-dgdjq" event={"ID":"bb2318b8-b92b-4ad1-a635-fb0d35c015e3","Type":"ContainerStarted","Data":"d74197fea564febc0a5fcc6b2fb8dc8a9278f0bc71174b198c4f52ecc5a3032e"} Jul 07 22:08:29.931761 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:29.931287 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-dgdjq" event={"ID":"bb2318b8-b92b-4ad1-a635-fb0d35c015e3","Type":"ContainerStarted","Data":"4aa7618d53258cb74dcf18ed034ac63794fcd391d3a5e8faf34bae554bcac322"} Jul 07 22:08:29.948554 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:29.948496 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-dgdjq" podStartSLOduration=1.9484768479999999 podStartE2EDuration="1.948476848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 22:08:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:08:29.946767111 +0000 UTC m=+289.774896571" watchObservedRunningTime="2026-07-07 22:08:29.948476848 +0000 UTC m=+289.776606304" Jul 07 22:08:40.557134 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:40.557105 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4h8zm_4b4234f5-6cda-45ea-b559-f09bcad77232/ovn-acl-logging/0.log" Jul 07 22:08:40.560483 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:40.560464 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4h8zm_4b4234f5-6cda-45ea-b559-f09bcad77232/ovn-acl-logging/0.log" Jul 07 22:08:40.561791 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:40.561774 2564 kubelet.go:1648] "Image garbage collection succeeded" Jul 07 22:08:45.246090 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:45.246059 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8f72h"] Jul 07 22:08:45.248451 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:45.248424 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8f72h" Jul 07 22:08:45.250647 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:45.250622 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 07 22:08:45.250647 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:45.250637 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 07 22:08:45.250801 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:45.250726 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-c8np2\"" Jul 07 22:08:45.257781 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:45.257756 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8f72h"] Jul 07 22:08:45.375818 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:45.375783 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5b313d93-d791-486e-8e2a-daa57b798314-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8f72h\" (UID: \"5b313d93-d791-486e-8e2a-daa57b798314\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8f72h" Jul 07 22:08:45.375818 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:45.375822 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbz4p\" (UniqueName: \"kubernetes.io/projected/5b313d93-d791-486e-8e2a-daa57b798314-kube-api-access-dbz4p\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8f72h\" (UID: \"5b313d93-d791-486e-8e2a-daa57b798314\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8f72h" Jul 07 22:08:45.376021 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:45.375907 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5b313d93-d791-486e-8e2a-daa57b798314-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8f72h\" (UID: \"5b313d93-d791-486e-8e2a-daa57b798314\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8f72h" Jul 07 22:08:45.476839 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:45.476803 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5b313d93-d791-486e-8e2a-daa57b798314-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8f72h\" (UID: \"5b313d93-d791-486e-8e2a-daa57b798314\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8f72h" Jul 07 22:08:45.477030 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:45.476863 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5b313d93-d791-486e-8e2a-daa57b798314-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8f72h\" (UID: \"5b313d93-d791-486e-8e2a-daa57b798314\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8f72h" Jul 07 22:08:45.477030 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:45.476899 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbz4p\" (UniqueName: \"kubernetes.io/projected/5b313d93-d791-486e-8e2a-daa57b798314-kube-api-access-dbz4p\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8f72h\" (UID: \"5b313d93-d791-486e-8e2a-daa57b798314\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8f72h" Jul 07 22:08:45.477190 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:45.477170 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5b313d93-d791-486e-8e2a-daa57b798314-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8f72h\" (UID: \"5b313d93-d791-486e-8e2a-daa57b798314\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8f72h" Jul 07 22:08:45.477285 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:45.477265 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5b313d93-d791-486e-8e2a-daa57b798314-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8f72h\" (UID: \"5b313d93-d791-486e-8e2a-daa57b798314\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8f72h" Jul 07 22:08:45.484830 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:45.484800 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbz4p\" (UniqueName: \"kubernetes.io/projected/5b313d93-d791-486e-8e2a-daa57b798314-kube-api-access-dbz4p\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8f72h\" (UID: \"5b313d93-d791-486e-8e2a-daa57b798314\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8f72h" Jul 07 22:08:45.557744 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:45.557712 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8f72h" Jul 07 22:08:45.703809 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:45.703777 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8f72h"] Jul 07 22:08:45.707344 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:08:45.707300 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b313d93_d791_486e_8e2a_daa57b798314.slice/crio-0491a11819470623bc92d5e10ebea9c57a021a23fd64f92404cf2fdc4c92d460 WatchSource:0}: Error finding container 0491a11819470623bc92d5e10ebea9c57a021a23fd64f92404cf2fdc4c92d460: Status 404 returned error can't find the container with id 0491a11819470623bc92d5e10ebea9c57a021a23fd64f92404cf2fdc4c92d460 Jul 07 22:08:45.996307 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:45.996221 2564 generic.go:356] "Generic (PLEG): container finished" podID="5b313d93-d791-486e-8e2a-daa57b798314" containerID="cc00ee53e29050f547e71486426ea3362d76fe91b0e9d4e9bfdf960627476e2f" exitCode=0 Jul 07 22:08:45.996307 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:45.996296 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8f72h" event={"ID":"5b313d93-d791-486e-8e2a-daa57b798314","Type":"ContainerDied","Data":"cc00ee53e29050f547e71486426ea3362d76fe91b0e9d4e9bfdf960627476e2f"} Jul 07 22:08:45.996501 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:45.996335 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8f72h" event={"ID":"5b313d93-d791-486e-8e2a-daa57b798314","Type":"ContainerStarted","Data":"0491a11819470623bc92d5e10ebea9c57a021a23fd64f92404cf2fdc4c92d460"} Jul 07 22:08:45.996774 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:45.996759 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 22:08:47.002948 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:47.002919 2564 generic.go:356] "Generic (PLEG): container finished" podID="5b313d93-d791-486e-8e2a-daa57b798314" containerID="f531de0f54ea083388d162e716f5436b07cbafe8d5e069b8dfc76e91b814fa9f" exitCode=0 Jul 07 22:08:47.003355 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:47.003011 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8f72h" event={"ID":"5b313d93-d791-486e-8e2a-daa57b798314","Type":"ContainerDied","Data":"f531de0f54ea083388d162e716f5436b07cbafe8d5e069b8dfc76e91b814fa9f"} Jul 07 22:08:48.009349 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:48.009315 2564 generic.go:356] "Generic (PLEG): container finished" podID="5b313d93-d791-486e-8e2a-daa57b798314" containerID="2ec2cba86a9b178422dd3253e5cee103e56b30eb69f5884dd1280b59440670c4" exitCode=0 Jul 07 22:08:48.009762 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:48.009372 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8f72h" event={"ID":"5b313d93-d791-486e-8e2a-daa57b798314","Type":"ContainerDied","Data":"2ec2cba86a9b178422dd3253e5cee103e56b30eb69f5884dd1280b59440670c4"} Jul 07 22:08:49.137618 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:49.137597 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8f72h" Jul 07 22:08:49.207767 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:49.207732 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbz4p\" (UniqueName: \"kubernetes.io/projected/5b313d93-d791-486e-8e2a-daa57b798314-kube-api-access-dbz4p\") pod \"5b313d93-d791-486e-8e2a-daa57b798314\" (UID: \"5b313d93-d791-486e-8e2a-daa57b798314\") " Jul 07 22:08:49.207951 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:49.207805 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5b313d93-d791-486e-8e2a-daa57b798314-bundle\") pod \"5b313d93-d791-486e-8e2a-daa57b798314\" (UID: \"5b313d93-d791-486e-8e2a-daa57b798314\") " Jul 07 22:08:49.207951 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:49.207829 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5b313d93-d791-486e-8e2a-daa57b798314-util\") pod \"5b313d93-d791-486e-8e2a-daa57b798314\" (UID: \"5b313d93-d791-486e-8e2a-daa57b798314\") " Jul 07 22:08:49.208687 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:49.208662 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b313d93-d791-486e-8e2a-daa57b798314-bundle" (OuterVolumeSpecName: "bundle") pod "5b313d93-d791-486e-8e2a-daa57b798314" (UID: "5b313d93-d791-486e-8e2a-daa57b798314"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 22:08:49.209910 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:49.209891 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b313d93-d791-486e-8e2a-daa57b798314-kube-api-access-dbz4p" (OuterVolumeSpecName: "kube-api-access-dbz4p") pod "5b313d93-d791-486e-8e2a-daa57b798314" (UID: "5b313d93-d791-486e-8e2a-daa57b798314"). InnerVolumeSpecName "kube-api-access-dbz4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 22:08:49.216281 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:49.216244 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b313d93-d791-486e-8e2a-daa57b798314-util" (OuterVolumeSpecName: "util") pod "5b313d93-d791-486e-8e2a-daa57b798314" (UID: "5b313d93-d791-486e-8e2a-daa57b798314"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 22:08:49.308407 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:49.308377 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbz4p\" (UniqueName: \"kubernetes.io/projected/5b313d93-d791-486e-8e2a-daa57b798314-kube-api-access-dbz4p\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 07 22:08:49.308407 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:49.308405 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5b313d93-d791-486e-8e2a-daa57b798314-bundle\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 07 22:08:49.308599 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:49.308414 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5b313d93-d791-486e-8e2a-daa57b798314-util\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 07 22:08:50.018684 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:50.018649 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8f72h" event={"ID":"5b313d93-d791-486e-8e2a-daa57b798314","Type":"ContainerDied","Data":"0491a11819470623bc92d5e10ebea9c57a021a23fd64f92404cf2fdc4c92d460"} Jul 07 22:08:50.018929 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:50.018697 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8f72h" Jul 07 22:08:50.018929 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:50.018704 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0491a11819470623bc92d5e10ebea9c57a021a23fd64f92404cf2fdc4c92d460" Jul 07 22:08:58.113094 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:58.113065 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-654d77bd8f-xx48n"] Jul 07 22:08:58.113510 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:58.113359 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b313d93-d791-486e-8e2a-daa57b798314" containerName="util" Jul 07 22:08:58.113510 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:58.113371 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b313d93-d791-486e-8e2a-daa57b798314" containerName="util" Jul 07 22:08:58.113510 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:58.113378 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b313d93-d791-486e-8e2a-daa57b798314" containerName="pull" Jul 07 22:08:58.113510 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:58.113383 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b313d93-d791-486e-8e2a-daa57b798314" containerName="pull" Jul 07 22:08:58.113510 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:58.113392 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b313d93-d791-486e-8e2a-daa57b798314" containerName="extract" Jul 07 22:08:58.113510 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:58.113397 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b313d93-d791-486e-8e2a-daa57b798314" containerName="extract" Jul 07 22:08:58.113510 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:58.113457 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b313d93-d791-486e-8e2a-daa57b798314" containerName="extract" Jul 07 22:08:58.117980 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:58.117959 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-654d77bd8f-xx48n" Jul 07 22:08:58.120617 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:58.120598 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 07 22:08:58.120727 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:58.120608 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 07 22:08:58.121569 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:58.121553 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-4nflv\"" Jul 07 22:08:58.121569 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:58.121562 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 07 22:08:58.121710 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:58.121562 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 07 22:08:58.121710 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:58.121585 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 07 22:08:58.133602 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:58.133580 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-654d77bd8f-xx48n"] Jul 07 22:08:58.181243 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:58.181216 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2ff22fa1-1ba5-4edb-85d3-66d93196fe38-cert\") pod \"lws-controller-manager-654d77bd8f-xx48n\" (UID: \"2ff22fa1-1ba5-4edb-85d3-66d93196fe38\") " pod="openshift-lws-operator/lws-controller-manager-654d77bd8f-xx48n" Jul 07 22:08:58.181373 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:58.181251 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk8t8\" (UniqueName: \"kubernetes.io/projected/2ff22fa1-1ba5-4edb-85d3-66d93196fe38-kube-api-access-dk8t8\") pod \"lws-controller-manager-654d77bd8f-xx48n\" (UID: \"2ff22fa1-1ba5-4edb-85d3-66d93196fe38\") " pod="openshift-lws-operator/lws-controller-manager-654d77bd8f-xx48n" Jul 07 22:08:58.181373 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:58.181283 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2ff22fa1-1ba5-4edb-85d3-66d93196fe38-metrics-cert\") pod \"lws-controller-manager-654d77bd8f-xx48n\" (UID: \"2ff22fa1-1ba5-4edb-85d3-66d93196fe38\") " pod="openshift-lws-operator/lws-controller-manager-654d77bd8f-xx48n" Jul 07 22:08:58.181373 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:58.181358 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2ff22fa1-1ba5-4edb-85d3-66d93196fe38-manager-config\") pod \"lws-controller-manager-654d77bd8f-xx48n\" (UID: \"2ff22fa1-1ba5-4edb-85d3-66d93196fe38\") " pod="openshift-lws-operator/lws-controller-manager-654d77bd8f-xx48n" Jul 07 22:08:58.281757 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:58.281720 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2ff22fa1-1ba5-4edb-85d3-66d93196fe38-cert\") pod \"lws-controller-manager-654d77bd8f-xx48n\" (UID: \"2ff22fa1-1ba5-4edb-85d3-66d93196fe38\") " pod="openshift-lws-operator/lws-controller-manager-654d77bd8f-xx48n" Jul 07 22:08:58.281757 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:58.281760 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk8t8\" (UniqueName: \"kubernetes.io/projected/2ff22fa1-1ba5-4edb-85d3-66d93196fe38-kube-api-access-dk8t8\") pod \"lws-controller-manager-654d77bd8f-xx48n\" (UID: \"2ff22fa1-1ba5-4edb-85d3-66d93196fe38\") " pod="openshift-lws-operator/lws-controller-manager-654d77bd8f-xx48n" Jul 07 22:08:58.281990 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:58.281790 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2ff22fa1-1ba5-4edb-85d3-66d93196fe38-metrics-cert\") pod \"lws-controller-manager-654d77bd8f-xx48n\" (UID: \"2ff22fa1-1ba5-4edb-85d3-66d93196fe38\") " pod="openshift-lws-operator/lws-controller-manager-654d77bd8f-xx48n" Jul 07 22:08:58.281990 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:58.281818 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2ff22fa1-1ba5-4edb-85d3-66d93196fe38-manager-config\") pod \"lws-controller-manager-654d77bd8f-xx48n\" (UID: \"2ff22fa1-1ba5-4edb-85d3-66d93196fe38\") " pod="openshift-lws-operator/lws-controller-manager-654d77bd8f-xx48n" Jul 07 22:08:58.282546 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:58.282513 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2ff22fa1-1ba5-4edb-85d3-66d93196fe38-manager-config\") pod \"lws-controller-manager-654d77bd8f-xx48n\" (UID: \"2ff22fa1-1ba5-4edb-85d3-66d93196fe38\") " pod="openshift-lws-operator/lws-controller-manager-654d77bd8f-xx48n" Jul 07 22:08:58.284578 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:58.284544 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2ff22fa1-1ba5-4edb-85d3-66d93196fe38-cert\") pod \"lws-controller-manager-654d77bd8f-xx48n\" (UID: \"2ff22fa1-1ba5-4edb-85d3-66d93196fe38\") " pod="openshift-lws-operator/lws-controller-manager-654d77bd8f-xx48n" Jul 07 22:08:58.284685 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:58.284591 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2ff22fa1-1ba5-4edb-85d3-66d93196fe38-metrics-cert\") pod \"lws-controller-manager-654d77bd8f-xx48n\" (UID: \"2ff22fa1-1ba5-4edb-85d3-66d93196fe38\") " pod="openshift-lws-operator/lws-controller-manager-654d77bd8f-xx48n" Jul 07 22:08:58.297330 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:58.297304 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk8t8\" (UniqueName: \"kubernetes.io/projected/2ff22fa1-1ba5-4edb-85d3-66d93196fe38-kube-api-access-dk8t8\") pod \"lws-controller-manager-654d77bd8f-xx48n\" (UID: \"2ff22fa1-1ba5-4edb-85d3-66d93196fe38\") " pod="openshift-lws-operator/lws-controller-manager-654d77bd8f-xx48n" Jul 07 22:08:58.426649 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:58.426575 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-654d77bd8f-xx48n" Jul 07 22:08:58.564801 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:08:58.564774 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ff22fa1_1ba5_4edb_85d3_66d93196fe38.slice/crio-dc09654941143aa2957f1c5214fd1d218b6d2d43046206e086f95db8b4d42c57 WatchSource:0}: Error finding container dc09654941143aa2957f1c5214fd1d218b6d2d43046206e086f95db8b4d42c57: Status 404 returned error can't find the container with id dc09654941143aa2957f1c5214fd1d218b6d2d43046206e086f95db8b4d42c57 Jul 07 22:08:58.569573 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:58.569552 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-654d77bd8f-xx48n"] Jul 07 22:08:59.059888 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:08:59.059849 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-654d77bd8f-xx48n" event={"ID":"2ff22fa1-1ba5-4edb-85d3-66d93196fe38","Type":"ContainerStarted","Data":"dc09654941143aa2957f1c5214fd1d218b6d2d43046206e086f95db8b4d42c57"} Jul 07 22:09:00.264515 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:00.264475 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-597db99f65-8gxnd"] Jul 07 22:09:00.266983 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:00.266968 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-597db99f65-8gxnd" Jul 07 22:09:00.269924 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:00.269901 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 07 22:09:00.270043 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:00.269930 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 07 22:09:00.270043 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:00.269959 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-wbtfm\"" Jul 07 22:09:00.270043 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:00.269906 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 07 22:09:00.270292 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:00.270275 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 07 22:09:00.288699 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:00.288665 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-597db99f65-8gxnd"] Jul 07 22:09:00.401800 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:00.401762 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpxds\" (UniqueName: \"kubernetes.io/projected/20bc9c72-6f00-403e-b62f-87093490df90-kube-api-access-jpxds\") pod \"opendatahub-operator-controller-manager-597db99f65-8gxnd\" (UID: \"20bc9c72-6f00-403e-b62f-87093490df90\") " pod="opendatahub/opendatahub-operator-controller-manager-597db99f65-8gxnd" Jul 07 22:09:00.402205 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:00.402183 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/20bc9c72-6f00-403e-b62f-87093490df90-apiservice-cert\") pod \"opendatahub-operator-controller-manager-597db99f65-8gxnd\" (UID: \"20bc9c72-6f00-403e-b62f-87093490df90\") " pod="opendatahub/opendatahub-operator-controller-manager-597db99f65-8gxnd" Jul 07 22:09:00.402335 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:00.402320 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/20bc9c72-6f00-403e-b62f-87093490df90-manifests\") pod \"opendatahub-operator-controller-manager-597db99f65-8gxnd\" (UID: \"20bc9c72-6f00-403e-b62f-87093490df90\") " pod="opendatahub/opendatahub-operator-controller-manager-597db99f65-8gxnd" Jul 07 22:09:00.402437 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:00.402424 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/20bc9c72-6f00-403e-b62f-87093490df90-webhook-cert\") pod \"opendatahub-operator-controller-manager-597db99f65-8gxnd\" (UID: \"20bc9c72-6f00-403e-b62f-87093490df90\") " pod="opendatahub/opendatahub-operator-controller-manager-597db99f65-8gxnd" Jul 07 22:09:00.402599 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:00.402585 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/20bc9c72-6f00-403e-b62f-87093490df90-tmp\") pod \"opendatahub-operator-controller-manager-597db99f65-8gxnd\" (UID: \"20bc9c72-6f00-403e-b62f-87093490df90\") " pod="opendatahub/opendatahub-operator-controller-manager-597db99f65-8gxnd" Jul 07 22:09:00.503459 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:00.503419 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/20bc9c72-6f00-403e-b62f-87093490df90-apiservice-cert\") pod \"opendatahub-operator-controller-manager-597db99f65-8gxnd\" (UID: \"20bc9c72-6f00-403e-b62f-87093490df90\") " pod="opendatahub/opendatahub-operator-controller-manager-597db99f65-8gxnd" Jul 07 22:09:00.503652 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:00.503482 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/20bc9c72-6f00-403e-b62f-87093490df90-manifests\") pod \"opendatahub-operator-controller-manager-597db99f65-8gxnd\" (UID: \"20bc9c72-6f00-403e-b62f-87093490df90\") " pod="opendatahub/opendatahub-operator-controller-manager-597db99f65-8gxnd" Jul 07 22:09:00.503652 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:00.503511 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/20bc9c72-6f00-403e-b62f-87093490df90-webhook-cert\") pod \"opendatahub-operator-controller-manager-597db99f65-8gxnd\" (UID: \"20bc9c72-6f00-403e-b62f-87093490df90\") " pod="opendatahub/opendatahub-operator-controller-manager-597db99f65-8gxnd" Jul 07 22:09:00.503652 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:00.503546 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/20bc9c72-6f00-403e-b62f-87093490df90-tmp\") pod \"opendatahub-operator-controller-manager-597db99f65-8gxnd\" (UID: \"20bc9c72-6f00-403e-b62f-87093490df90\") " pod="opendatahub/opendatahub-operator-controller-manager-597db99f65-8gxnd" Jul 07 22:09:00.503652 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:00.503591 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpxds\" (UniqueName: \"kubernetes.io/projected/20bc9c72-6f00-403e-b62f-87093490df90-kube-api-access-jpxds\") pod \"opendatahub-operator-controller-manager-597db99f65-8gxnd\" (UID: \"20bc9c72-6f00-403e-b62f-87093490df90\") " pod="opendatahub/opendatahub-operator-controller-manager-597db99f65-8gxnd" Jul 07 22:09:00.504025 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:00.504002 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/20bc9c72-6f00-403e-b62f-87093490df90-manifests\") pod \"opendatahub-operator-controller-manager-597db99f65-8gxnd\" (UID: \"20bc9c72-6f00-403e-b62f-87093490df90\") " pod="opendatahub/opendatahub-operator-controller-manager-597db99f65-8gxnd" Jul 07 22:09:00.506108 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:00.506078 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/20bc9c72-6f00-403e-b62f-87093490df90-tmp\") pod \"opendatahub-operator-controller-manager-597db99f65-8gxnd\" (UID: \"20bc9c72-6f00-403e-b62f-87093490df90\") " pod="opendatahub/opendatahub-operator-controller-manager-597db99f65-8gxnd" Jul 07 22:09:00.506226 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:00.506208 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/20bc9c72-6f00-403e-b62f-87093490df90-webhook-cert\") pod \"opendatahub-operator-controller-manager-597db99f65-8gxnd\" (UID: \"20bc9c72-6f00-403e-b62f-87093490df90\") " pod="opendatahub/opendatahub-operator-controller-manager-597db99f65-8gxnd" Jul 07 22:09:00.506271 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:00.506246 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/20bc9c72-6f00-403e-b62f-87093490df90-apiservice-cert\") pod \"opendatahub-operator-controller-manager-597db99f65-8gxnd\" (UID: \"20bc9c72-6f00-403e-b62f-87093490df90\") " pod="opendatahub/opendatahub-operator-controller-manager-597db99f65-8gxnd" Jul 07 22:09:00.519650 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:00.519597 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpxds\" (UniqueName: \"kubernetes.io/projected/20bc9c72-6f00-403e-b62f-87093490df90-kube-api-access-jpxds\") pod \"opendatahub-operator-controller-manager-597db99f65-8gxnd\" (UID: \"20bc9c72-6f00-403e-b62f-87093490df90\") " pod="opendatahub/opendatahub-operator-controller-manager-597db99f65-8gxnd" Jul 07 22:09:00.582492 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:00.582463 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-597db99f65-8gxnd" Jul 07 22:09:00.718353 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:00.718226 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-597db99f65-8gxnd"] Jul 07 22:09:00.720682 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:09:00.720656 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20bc9c72_6f00_403e_b62f_87093490df90.slice/crio-672a2da5716da3bc8ddc866237be9572841efa008a9dee4e44c93d284454009b WatchSource:0}: Error finding container 672a2da5716da3bc8ddc866237be9572841efa008a9dee4e44c93d284454009b: Status 404 returned error can't find the container with id 672a2da5716da3bc8ddc866237be9572841efa008a9dee4e44c93d284454009b Jul 07 22:09:01.072761 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:01.072728 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-597db99f65-8gxnd" event={"ID":"20bc9c72-6f00-403e-b62f-87093490df90","Type":"ContainerStarted","Data":"672a2da5716da3bc8ddc866237be9572841efa008a9dee4e44c93d284454009b"} Jul 07 22:09:02.080296 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:02.080247 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-654d77bd8f-xx48n" event={"ID":"2ff22fa1-1ba5-4edb-85d3-66d93196fe38","Type":"ContainerStarted","Data":"8ad694c3a01ca701e07fa0ed4d0fd48e71060ba09ac699edcd03318a73e786cc"} Jul 07 22:09:02.080800 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:02.080340 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-654d77bd8f-xx48n" Jul 07 22:09:02.101342 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:02.101287 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-654d77bd8f-xx48n" podStartSLOduration=1.357062163 podStartE2EDuration="4.101264819s" totalImagesPullingTime="2.744202656s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 22:08:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:09:02.099788426 +0000 UTC m=+321.927917884" watchObservedRunningTime="2026-07-07 22:09:02.101264819 +0000 UTC m=+321.929394277" Jul 07 22:09:04.091057 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:04.090969 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-597db99f65-8gxnd" event={"ID":"20bc9c72-6f00-403e-b62f-87093490df90","Type":"ContainerStarted","Data":"0e3bd7a02119e98fc658b2a640892e85a4b201ce1dab38aa985164c7cc9ff801"} Jul 07 22:09:05.096350 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:05.096318 2564 generic.go:356] "Generic (PLEG): container finished" podID="20bc9c72-6f00-403e-b62f-87093490df90" containerID="0e3bd7a02119e98fc658b2a640892e85a4b201ce1dab38aa985164c7cc9ff801" exitCode=0 Jul 07 22:09:05.096908 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:05.096409 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-597db99f65-8gxnd" event={"ID":"20bc9c72-6f00-403e-b62f-87093490df90","Type":"ContainerDied","Data":"0e3bd7a02119e98fc658b2a640892e85a4b201ce1dab38aa985164c7cc9ff801"} Jul 07 22:09:06.104416 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:06.104374 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-597db99f65-8gxnd" event={"ID":"20bc9c72-6f00-403e-b62f-87093490df90","Type":"ContainerStarted","Data":"40434c28048c6100ed2626c777f4c419915c25f2509eee1cc59e9b811d3c4acd"} Jul 07 22:09:06.104928 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:06.104515 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-597db99f65-8gxnd" Jul 07 22:09:06.126245 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:06.126199 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-597db99f65-8gxnd" podStartSLOduration=1.6592324600000001 podStartE2EDuration="6.126184081s" totalImagesPullingTime="3.456950042s" totalInitContainerRuntime="1.010001579s" isStatefulPod=false podCreationTimestamp="2026-07-07 22:09:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:09:06.125848688 +0000 UTC m=+325.953978169" watchObservedRunningTime="2026-07-07 22:09:06.126184081 +0000 UTC m=+325.954313550" Jul 07 22:09:13.088026 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:13.087992 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-654d77bd8f-xx48n" Jul 07 22:09:17.111803 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:17.111772 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-597db99f65-8gxnd" Jul 07 22:09:27.645840 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:27.645807 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-f78jq"] Jul 07 22:09:27.654037 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:27.654011 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-f78jq" Jul 07 22:09:27.656931 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:27.656901 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 07 22:09:27.657064 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:27.656983 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 07 22:09:27.657624 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:27.657604 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 07 22:09:27.658079 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:27.658059 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 07 22:09:27.658189 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:27.658166 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 07 22:09:27.658461 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:27.658425 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-gtmw7\"" Jul 07 22:09:27.658646 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:27.658631 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 07 22:09:27.671014 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:27.670984 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-f78jq"] Jul 07 22:09:27.733173 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:27.733135 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/478167a3-0a72-4e85-994e-75972b76b135-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-f78jq\" (UID: \"478167a3-0a72-4e85-994e-75972b76b135\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-f78jq" Jul 07 22:09:27.733173 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:27.733176 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/478167a3-0a72-4e85-994e-75972b76b135-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-f78jq\" (UID: \"478167a3-0a72-4e85-994e-75972b76b135\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-f78jq" Jul 07 22:09:27.733373 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:27.733206 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m9k2\" (UniqueName: \"kubernetes.io/projected/478167a3-0a72-4e85-994e-75972b76b135-kube-api-access-8m9k2\") pod \"istiod-openshift-gateway-94bb8fbfd-f78jq\" (UID: \"478167a3-0a72-4e85-994e-75972b76b135\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-f78jq" Jul 07 22:09:27.733373 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:27.733285 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/478167a3-0a72-4e85-994e-75972b76b135-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-f78jq\" (UID: \"478167a3-0a72-4e85-994e-75972b76b135\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-f78jq" Jul 07 22:09:27.733373 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:27.733316 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/478167a3-0a72-4e85-994e-75972b76b135-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-f78jq\" (UID: \"478167a3-0a72-4e85-994e-75972b76b135\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-f78jq" Jul 07 22:09:27.733373 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:27.733353 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/478167a3-0a72-4e85-994e-75972b76b135-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-f78jq\" (UID: \"478167a3-0a72-4e85-994e-75972b76b135\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-f78jq" Jul 07 22:09:27.733559 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:27.733377 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/478167a3-0a72-4e85-994e-75972b76b135-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-f78jq\" (UID: \"478167a3-0a72-4e85-994e-75972b76b135\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-f78jq" Jul 07 22:09:27.833998 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:27.833964 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/478167a3-0a72-4e85-994e-75972b76b135-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-f78jq\" (UID: \"478167a3-0a72-4e85-994e-75972b76b135\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-f78jq" Jul 07 22:09:27.834148 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:27.834010 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m9k2\" (UniqueName: \"kubernetes.io/projected/478167a3-0a72-4e85-994e-75972b76b135-kube-api-access-8m9k2\") pod \"istiod-openshift-gateway-94bb8fbfd-f78jq\" (UID: \"478167a3-0a72-4e85-994e-75972b76b135\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-f78jq" Jul 07 22:09:27.834148 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:27.834047 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/478167a3-0a72-4e85-994e-75972b76b135-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-f78jq\" (UID: \"478167a3-0a72-4e85-994e-75972b76b135\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-f78jq" Jul 07 22:09:27.834148 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:27.834073 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/478167a3-0a72-4e85-994e-75972b76b135-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-f78jq\" (UID: \"478167a3-0a72-4e85-994e-75972b76b135\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-f78jq" Jul 07 22:09:27.834148 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:27.834127 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/478167a3-0a72-4e85-994e-75972b76b135-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-f78jq\" (UID: \"478167a3-0a72-4e85-994e-75972b76b135\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-f78jq" Jul 07 22:09:27.834355 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:27.834161 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/478167a3-0a72-4e85-994e-75972b76b135-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-f78jq\" (UID: \"478167a3-0a72-4e85-994e-75972b76b135\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-f78jq" Jul 07 22:09:27.834355 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:27.834215 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/478167a3-0a72-4e85-994e-75972b76b135-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-f78jq\" (UID: \"478167a3-0a72-4e85-994e-75972b76b135\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-f78jq" Jul 07 22:09:27.834859 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:27.834832 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/478167a3-0a72-4e85-994e-75972b76b135-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-f78jq\" (UID: \"478167a3-0a72-4e85-994e-75972b76b135\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-f78jq" Jul 07 22:09:27.837152 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:27.837126 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/478167a3-0a72-4e85-994e-75972b76b135-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-f78jq\" (UID: \"478167a3-0a72-4e85-994e-75972b76b135\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-f78jq" Jul 07 22:09:27.837271 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:27.837132 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/478167a3-0a72-4e85-994e-75972b76b135-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-f78jq\" (UID: \"478167a3-0a72-4e85-994e-75972b76b135\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-f78jq" Jul 07 22:09:27.837271 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:27.837173 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/478167a3-0a72-4e85-994e-75972b76b135-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-f78jq\" (UID: \"478167a3-0a72-4e85-994e-75972b76b135\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-f78jq" Jul 07 22:09:27.837271 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:27.837197 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/478167a3-0a72-4e85-994e-75972b76b135-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-f78jq\" (UID: \"478167a3-0a72-4e85-994e-75972b76b135\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-f78jq" Jul 07 22:09:27.859399 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:27.859372 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/478167a3-0a72-4e85-994e-75972b76b135-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-f78jq\" (UID: \"478167a3-0a72-4e85-994e-75972b76b135\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-f78jq" Jul 07 22:09:27.859588 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:27.859567 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m9k2\" (UniqueName: \"kubernetes.io/projected/478167a3-0a72-4e85-994e-75972b76b135-kube-api-access-8m9k2\") pod \"istiod-openshift-gateway-94bb8fbfd-f78jq\" (UID: \"478167a3-0a72-4e85-994e-75972b76b135\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-f78jq" Jul 07 22:09:27.966582 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:27.966494 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-f78jq" Jul 07 22:09:28.127641 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:28.127616 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-f78jq"] Jul 07 22:09:28.129723 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:09:28.129682 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod478167a3_0a72_4e85_994e_75972b76b135.slice/crio-b000680a6494004d87d7878856dfa9c2a1cafdaff96ef8ceb82ac30a6c091611 WatchSource:0}: Error finding container b000680a6494004d87d7878856dfa9c2a1cafdaff96ef8ceb82ac30a6c091611: Status 404 returned error can't find the container with id b000680a6494004d87d7878856dfa9c2a1cafdaff96ef8ceb82ac30a6c091611 Jul 07 22:09:28.204505 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:28.204467 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-f78jq" event={"ID":"478167a3-0a72-4e85-994e-75972b76b135","Type":"ContainerStarted","Data":"b000680a6494004d87d7878856dfa9c2a1cafdaff96ef8ceb82ac30a6c091611"} Jul 07 22:09:30.870666 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:30.870626 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 07 22:09:30.870904 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:30.870711 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 07 22:09:31.224114 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:31.224022 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-f78jq" event={"ID":"478167a3-0a72-4e85-994e-75972b76b135","Type":"ContainerStarted","Data":"6d5547475d286d2a85f22a239e2caad57860e63dfcef3a0f6548ead6900f722e"} Jul 07 22:09:31.224366 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:31.224326 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-f78jq" Jul 07 22:09:31.225776 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:31.225739 2564 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-f78jq container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 07 22:09:31.225897 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:31.225797 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-f78jq" podUID="478167a3-0a72-4e85-994e-75972b76b135" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 22:09:31.264000 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:31.263948 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-f78jq" podStartSLOduration=1.5251450389999999 podStartE2EDuration="4.263934258s" totalImagesPullingTime="2.738789219s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 22:09:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:09:31.262088748 +0000 UTC m=+351.090218209" watchObservedRunningTime="2026-07-07 22:09:31.263934258 +0000 UTC m=+351.092063715" Jul 07 22:09:32.229812 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:09:32.229788 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-f78jq" Jul 07 22:10:36.856631 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:36.856600 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lhh6j"] Jul 07 22:10:36.859975 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:36.859958 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lhh6j" Jul 07 22:10:36.862668 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:36.862645 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 07 22:10:36.862784 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:36.862749 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-f97sq\"" Jul 07 22:10:36.863123 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:36.863107 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 07 22:10:36.869639 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:36.869614 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lhh6j"] Jul 07 22:10:36.892756 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:36.892500 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mpjj\" (UniqueName: \"kubernetes.io/projected/dbe0bc5b-dcda-463a-8764-8ad638ccb1f7-kube-api-access-5mpjj\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lhh6j\" (UID: \"dbe0bc5b-dcda-463a-8764-8ad638ccb1f7\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lhh6j" Jul 07 22:10:36.892756 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:36.892622 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dbe0bc5b-dcda-463a-8764-8ad638ccb1f7-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lhh6j\" (UID: \"dbe0bc5b-dcda-463a-8764-8ad638ccb1f7\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lhh6j" Jul 07 22:10:36.892756 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:36.892667 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dbe0bc5b-dcda-463a-8764-8ad638ccb1f7-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lhh6j\" (UID: \"dbe0bc5b-dcda-463a-8764-8ad638ccb1f7\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lhh6j" Jul 07 22:10:36.993493 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:36.993464 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dbe0bc5b-dcda-463a-8764-8ad638ccb1f7-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lhh6j\" (UID: \"dbe0bc5b-dcda-463a-8764-8ad638ccb1f7\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lhh6j" Jul 07 22:10:36.993648 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:36.993507 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mpjj\" (UniqueName: \"kubernetes.io/projected/dbe0bc5b-dcda-463a-8764-8ad638ccb1f7-kube-api-access-5mpjj\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lhh6j\" (UID: \"dbe0bc5b-dcda-463a-8764-8ad638ccb1f7\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lhh6j" Jul 07 22:10:36.993648 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:36.993562 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dbe0bc5b-dcda-463a-8764-8ad638ccb1f7-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lhh6j\" (UID: \"dbe0bc5b-dcda-463a-8764-8ad638ccb1f7\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lhh6j" Jul 07 22:10:36.993833 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:36.993812 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dbe0bc5b-dcda-463a-8764-8ad638ccb1f7-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lhh6j\" (UID: \"dbe0bc5b-dcda-463a-8764-8ad638ccb1f7\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lhh6j" Jul 07 22:10:36.993891 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:36.993872 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dbe0bc5b-dcda-463a-8764-8ad638ccb1f7-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lhh6j\" (UID: \"dbe0bc5b-dcda-463a-8764-8ad638ccb1f7\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lhh6j" Jul 07 22:10:37.001766 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:37.001742 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mpjj\" (UniqueName: \"kubernetes.io/projected/dbe0bc5b-dcda-463a-8764-8ad638ccb1f7-kube-api-access-5mpjj\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lhh6j\" (UID: \"dbe0bc5b-dcda-463a-8764-8ad638ccb1f7\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lhh6j" Jul 07 22:10:37.169588 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:37.169499 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lhh6j" Jul 07 22:10:37.300936 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:37.300908 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lhh6j"] Jul 07 22:10:37.303163 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:10:37.303134 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbe0bc5b_dcda_463a_8764_8ad638ccb1f7.slice/crio-a65483d6b7c6b582f5d24f918b06347915ba4dbb810e48c7514f7a557c056770 WatchSource:0}: Error finding container a65483d6b7c6b582f5d24f918b06347915ba4dbb810e48c7514f7a557c056770: Status 404 returned error can't find the container with id a65483d6b7c6b582f5d24f918b06347915ba4dbb810e48c7514f7a557c056770 Jul 07 22:10:37.533616 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:37.533593 2564 generic.go:356] "Generic (PLEG): container finished" podID="dbe0bc5b-dcda-463a-8764-8ad638ccb1f7" containerID="1ffa93597bd7720c434f2a6881bdf1ff3a6f0165d58859bd7b91acd724fc1b2f" exitCode=0 Jul 07 22:10:37.533750 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:37.533658 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lhh6j" event={"ID":"dbe0bc5b-dcda-463a-8764-8ad638ccb1f7","Type":"ContainerDied","Data":"1ffa93597bd7720c434f2a6881bdf1ff3a6f0165d58859bd7b91acd724fc1b2f"} Jul 07 22:10:37.533750 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:37.533730 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lhh6j" event={"ID":"dbe0bc5b-dcda-463a-8764-8ad638ccb1f7","Type":"ContainerStarted","Data":"a65483d6b7c6b582f5d24f918b06347915ba4dbb810e48c7514f7a557c056770"} Jul 07 22:10:37.648841 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:37.648804 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759phphs"] Jul 07 22:10:37.652171 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:37.652152 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759phphs" Jul 07 22:10:37.661379 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:37.661355 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759phphs"] Jul 07 22:10:37.699383 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:37.699354 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/470e5356-b27c-4300-972d-4409d4bccc45-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759phphs\" (UID: \"470e5356-b27c-4300-972d-4409d4bccc45\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759phphs" Jul 07 22:10:37.699525 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:37.699393 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txsjk\" (UniqueName: \"kubernetes.io/projected/470e5356-b27c-4300-972d-4409d4bccc45-kube-api-access-txsjk\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759phphs\" (UID: \"470e5356-b27c-4300-972d-4409d4bccc45\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759phphs" Jul 07 22:10:37.699525 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:37.699515 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/470e5356-b27c-4300-972d-4409d4bccc45-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759phphs\" (UID: \"470e5356-b27c-4300-972d-4409d4bccc45\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759phphs" Jul 07 22:10:37.800844 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:37.800766 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/470e5356-b27c-4300-972d-4409d4bccc45-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759phphs\" (UID: \"470e5356-b27c-4300-972d-4409d4bccc45\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759phphs" Jul 07 22:10:37.800844 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:37.800837 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/470e5356-b27c-4300-972d-4409d4bccc45-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759phphs\" (UID: \"470e5356-b27c-4300-972d-4409d4bccc45\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759phphs" Jul 07 22:10:37.801043 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:37.800863 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txsjk\" (UniqueName: \"kubernetes.io/projected/470e5356-b27c-4300-972d-4409d4bccc45-kube-api-access-txsjk\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759phphs\" (UID: \"470e5356-b27c-4300-972d-4409d4bccc45\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759phphs" Jul 07 22:10:37.801211 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:37.801187 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/470e5356-b27c-4300-972d-4409d4bccc45-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759phphs\" (UID: \"470e5356-b27c-4300-972d-4409d4bccc45\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759phphs" Jul 07 22:10:37.801266 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:37.801200 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/470e5356-b27c-4300-972d-4409d4bccc45-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759phphs\" (UID: \"470e5356-b27c-4300-972d-4409d4bccc45\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759phphs" Jul 07 22:10:37.811006 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:37.810986 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txsjk\" (UniqueName: \"kubernetes.io/projected/470e5356-b27c-4300-972d-4409d4bccc45-kube-api-access-txsjk\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759phphs\" (UID: \"470e5356-b27c-4300-972d-4409d4bccc45\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759phphs" Jul 07 22:10:37.961463 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:37.961420 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759phphs" Jul 07 22:10:38.051886 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:38.051820 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9j2b"] Jul 07 22:10:38.055561 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:38.055542 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9j2b" Jul 07 22:10:38.064088 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:38.064054 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9j2b"] Jul 07 22:10:38.098359 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:38.098317 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759phphs"] Jul 07 22:10:38.102311 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:10:38.102269 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod470e5356_b27c_4300_972d_4409d4bccc45.slice/crio-b37ad5e180a3522d52230f8b92816cd6be856e0e2777b0ea5b40c9380e718fd3 WatchSource:0}: Error finding container b37ad5e180a3522d52230f8b92816cd6be856e0e2777b0ea5b40c9380e718fd3: Status 404 returned error can't find the container with id b37ad5e180a3522d52230f8b92816cd6be856e0e2777b0ea5b40c9380e718fd3 Jul 07 22:10:38.102430 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:38.102368 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7121c2d1-76f0-41f8-8b8a-a693dcef10ca-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9j2b\" (UID: \"7121c2d1-76f0-41f8-8b8a-a693dcef10ca\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9j2b" Jul 07 22:10:38.102430 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:38.102405 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8mfh\" (UniqueName: \"kubernetes.io/projected/7121c2d1-76f0-41f8-8b8a-a693dcef10ca-kube-api-access-z8mfh\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9j2b\" (UID: \"7121c2d1-76f0-41f8-8b8a-a693dcef10ca\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9j2b" Jul 07 22:10:38.102692 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:38.102561 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7121c2d1-76f0-41f8-8b8a-a693dcef10ca-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9j2b\" (UID: \"7121c2d1-76f0-41f8-8b8a-a693dcef10ca\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9j2b" Jul 07 22:10:38.203354 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:38.203324 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7121c2d1-76f0-41f8-8b8a-a693dcef10ca-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9j2b\" (UID: \"7121c2d1-76f0-41f8-8b8a-a693dcef10ca\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9j2b" Jul 07 22:10:38.203354 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:38.203359 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8mfh\" (UniqueName: \"kubernetes.io/projected/7121c2d1-76f0-41f8-8b8a-a693dcef10ca-kube-api-access-z8mfh\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9j2b\" (UID: \"7121c2d1-76f0-41f8-8b8a-a693dcef10ca\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9j2b" Jul 07 22:10:38.203604 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:38.203395 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7121c2d1-76f0-41f8-8b8a-a693dcef10ca-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9j2b\" (UID: \"7121c2d1-76f0-41f8-8b8a-a693dcef10ca\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9j2b" Jul 07 22:10:38.203829 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:38.203808 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7121c2d1-76f0-41f8-8b8a-a693dcef10ca-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9j2b\" (UID: \"7121c2d1-76f0-41f8-8b8a-a693dcef10ca\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9j2b" Jul 07 22:10:38.203877 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:38.203811 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7121c2d1-76f0-41f8-8b8a-a693dcef10ca-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9j2b\" (UID: \"7121c2d1-76f0-41f8-8b8a-a693dcef10ca\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9j2b" Jul 07 22:10:38.214047 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:38.214016 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8mfh\" (UniqueName: \"kubernetes.io/projected/7121c2d1-76f0-41f8-8b8a-a693dcef10ca-kube-api-access-z8mfh\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9j2b\" (UID: \"7121c2d1-76f0-41f8-8b8a-a693dcef10ca\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9j2b" Jul 07 22:10:38.365832 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:38.365813 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9j2b" Jul 07 22:10:38.448059 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:38.448023 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vjphr"] Jul 07 22:10:38.452154 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:38.452127 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vjphr" Jul 07 22:10:38.461342 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:38.460753 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vjphr"] Jul 07 22:10:38.500069 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:38.500048 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9j2b"] Jul 07 22:10:38.502254 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:10:38.502227 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7121c2d1_76f0_41f8_8b8a_a693dcef10ca.slice/crio-48321b0dccafcf6930588abb0f563fd83c0843c195009e450e0789d205453540 WatchSource:0}: Error finding container 48321b0dccafcf6930588abb0f563fd83c0843c195009e450e0789d205453540: Status 404 returned error can't find the container with id 48321b0dccafcf6930588abb0f563fd83c0843c195009e450e0789d205453540 Jul 07 22:10:38.506277 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:38.506256 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0fa7d0f7-6fa8-43df-b565-aefc01d4855a-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vjphr\" (UID: \"0fa7d0f7-6fa8-43df-b565-aefc01d4855a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vjphr" Jul 07 22:10:38.506366 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:38.506298 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp6fd\" (UniqueName: \"kubernetes.io/projected/0fa7d0f7-6fa8-43df-b565-aefc01d4855a-kube-api-access-vp6fd\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vjphr\" (UID: \"0fa7d0f7-6fa8-43df-b565-aefc01d4855a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vjphr" Jul 07 22:10:38.506366 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:38.506348 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0fa7d0f7-6fa8-43df-b565-aefc01d4855a-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vjphr\" (UID: \"0fa7d0f7-6fa8-43df-b565-aefc01d4855a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vjphr" Jul 07 22:10:38.544916 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:38.544889 2564 generic.go:356] "Generic (PLEG): container finished" podID="dbe0bc5b-dcda-463a-8764-8ad638ccb1f7" containerID="1b9773ab82ea9e3cd2d3324e6b349bef3c1f8f68f2e0dbf4e5097a42d9a3100a" exitCode=0 Jul 07 22:10:38.545081 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:38.544979 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lhh6j" event={"ID":"dbe0bc5b-dcda-463a-8764-8ad638ccb1f7","Type":"ContainerDied","Data":"1b9773ab82ea9e3cd2d3324e6b349bef3c1f8f68f2e0dbf4e5097a42d9a3100a"} Jul 07 22:10:38.546607 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:38.546583 2564 generic.go:356] "Generic (PLEG): container finished" podID="470e5356-b27c-4300-972d-4409d4bccc45" containerID="8fe48084a1b2665574edd1d98c4eb69dea515c1ef294aae11fe75b97f20d7b92" exitCode=0 Jul 07 22:10:38.546702 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:38.546672 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759phphs" event={"ID":"470e5356-b27c-4300-972d-4409d4bccc45","Type":"ContainerDied","Data":"8fe48084a1b2665574edd1d98c4eb69dea515c1ef294aae11fe75b97f20d7b92"} Jul 07 22:10:38.546764 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:38.546721 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759phphs" event={"ID":"470e5356-b27c-4300-972d-4409d4bccc45","Type":"ContainerStarted","Data":"b37ad5e180a3522d52230f8b92816cd6be856e0e2777b0ea5b40c9380e718fd3"} Jul 07 22:10:38.548109 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:38.548082 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9j2b" event={"ID":"7121c2d1-76f0-41f8-8b8a-a693dcef10ca","Type":"ContainerStarted","Data":"48321b0dccafcf6930588abb0f563fd83c0843c195009e450e0789d205453540"} Jul 07 22:10:38.606933 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:38.606908 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0fa7d0f7-6fa8-43df-b565-aefc01d4855a-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vjphr\" (UID: \"0fa7d0f7-6fa8-43df-b565-aefc01d4855a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vjphr" Jul 07 22:10:38.607025 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:38.606959 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp6fd\" (UniqueName: \"kubernetes.io/projected/0fa7d0f7-6fa8-43df-b565-aefc01d4855a-kube-api-access-vp6fd\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vjphr\" (UID: \"0fa7d0f7-6fa8-43df-b565-aefc01d4855a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vjphr" Jul 07 22:10:38.607025 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:38.607002 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0fa7d0f7-6fa8-43df-b565-aefc01d4855a-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vjphr\" (UID: \"0fa7d0f7-6fa8-43df-b565-aefc01d4855a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vjphr" Jul 07 22:10:38.607334 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:38.607309 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0fa7d0f7-6fa8-43df-b565-aefc01d4855a-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vjphr\" (UID: \"0fa7d0f7-6fa8-43df-b565-aefc01d4855a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vjphr" Jul 07 22:10:38.607334 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:38.607321 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0fa7d0f7-6fa8-43df-b565-aefc01d4855a-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vjphr\" (UID: \"0fa7d0f7-6fa8-43df-b565-aefc01d4855a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vjphr" Jul 07 22:10:38.615631 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:38.615604 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp6fd\" (UniqueName: \"kubernetes.io/projected/0fa7d0f7-6fa8-43df-b565-aefc01d4855a-kube-api-access-vp6fd\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vjphr\" (UID: \"0fa7d0f7-6fa8-43df-b565-aefc01d4855a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vjphr" Jul 07 22:10:38.763768 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:38.763738 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vjphr" Jul 07 22:10:38.891160 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:38.891044 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vjphr"] Jul 07 22:10:38.893791 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:10:38.893769 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fa7d0f7_6fa8_43df_b565_aefc01d4855a.slice/crio-d50fa0fe0ba158af75da056ce0a5ec0d2aa24814b2fd2850cdcb8df99aeb3489 WatchSource:0}: Error finding container d50fa0fe0ba158af75da056ce0a5ec0d2aa24814b2fd2850cdcb8df99aeb3489: Status 404 returned error can't find the container with id d50fa0fe0ba158af75da056ce0a5ec0d2aa24814b2fd2850cdcb8df99aeb3489 Jul 07 22:10:39.559873 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:39.559846 2564 generic.go:356] "Generic (PLEG): container finished" podID="7121c2d1-76f0-41f8-8b8a-a693dcef10ca" containerID="85fa86569564ec5253640e2398dba5fc8d1b50e9869b063cb3d343aac119a01a" exitCode=0 Jul 07 22:10:39.560261 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:39.559928 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9j2b" event={"ID":"7121c2d1-76f0-41f8-8b8a-a693dcef10ca","Type":"ContainerDied","Data":"85fa86569564ec5253640e2398dba5fc8d1b50e9869b063cb3d343aac119a01a"} Jul 07 22:10:39.561480 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:39.561456 2564 generic.go:356] "Generic (PLEG): container finished" podID="0fa7d0f7-6fa8-43df-b565-aefc01d4855a" containerID="b7164a5183878f52ca61e23969622cda0b991f9a4a1da7ea2be6b572016652b2" exitCode=0 Jul 07 22:10:39.561570 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:39.561550 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vjphr" event={"ID":"0fa7d0f7-6fa8-43df-b565-aefc01d4855a","Type":"ContainerDied","Data":"b7164a5183878f52ca61e23969622cda0b991f9a4a1da7ea2be6b572016652b2"} Jul 07 22:10:39.561621 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:39.561602 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vjphr" event={"ID":"0fa7d0f7-6fa8-43df-b565-aefc01d4855a","Type":"ContainerStarted","Data":"d50fa0fe0ba158af75da056ce0a5ec0d2aa24814b2fd2850cdcb8df99aeb3489"} Jul 07 22:10:39.564058 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:39.564039 2564 generic.go:356] "Generic (PLEG): container finished" podID="dbe0bc5b-dcda-463a-8764-8ad638ccb1f7" containerID="8b83dec2bdcc4fe21a85197b5bdbfcb995792c0dd6930b90bdc54a7c843591d9" exitCode=0 Jul 07 22:10:39.564152 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:39.564122 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lhh6j" event={"ID":"dbe0bc5b-dcda-463a-8764-8ad638ccb1f7","Type":"ContainerDied","Data":"8b83dec2bdcc4fe21a85197b5bdbfcb995792c0dd6930b90bdc54a7c843591d9"} Jul 07 22:10:39.565850 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:39.565831 2564 generic.go:356] "Generic (PLEG): container finished" podID="470e5356-b27c-4300-972d-4409d4bccc45" containerID="1a770f8d11e2962f3472cb0f615b661ff6eefc7667d2525937e00bf919b752f2" exitCode=0 Jul 07 22:10:39.565940 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:39.565864 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759phphs" event={"ID":"470e5356-b27c-4300-972d-4409d4bccc45","Type":"ContainerDied","Data":"1a770f8d11e2962f3472cb0f615b661ff6eefc7667d2525937e00bf919b752f2"} Jul 07 22:10:40.572075 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:40.572044 2564 generic.go:356] "Generic (PLEG): container finished" podID="7121c2d1-76f0-41f8-8b8a-a693dcef10ca" containerID="94465918a53d3112cc4b9ca3ee18ce549a2fc224ac29243e471c96d6c712a971" exitCode=0 Jul 07 22:10:40.572406 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:40.572128 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9j2b" event={"ID":"7121c2d1-76f0-41f8-8b8a-a693dcef10ca","Type":"ContainerDied","Data":"94465918a53d3112cc4b9ca3ee18ce549a2fc224ac29243e471c96d6c712a971"} Jul 07 22:10:40.573975 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:40.573952 2564 generic.go:356] "Generic (PLEG): container finished" podID="0fa7d0f7-6fa8-43df-b565-aefc01d4855a" containerID="0d6d7347bd43706db3ddfa1a51b4ac8e81967b372bcfdec1bd375bd8c1aebda3" exitCode=0 Jul 07 22:10:40.574059 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:40.574025 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vjphr" event={"ID":"0fa7d0f7-6fa8-43df-b565-aefc01d4855a","Type":"ContainerDied","Data":"0d6d7347bd43706db3ddfa1a51b4ac8e81967b372bcfdec1bd375bd8c1aebda3"} Jul 07 22:10:40.576261 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:40.576241 2564 generic.go:356] "Generic (PLEG): container finished" podID="470e5356-b27c-4300-972d-4409d4bccc45" containerID="5b129e56bf7917e83435f2426608c772a6538f266cbf6d638a153b5d6412953b" exitCode=0 Jul 07 22:10:40.576371 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:40.576324 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759phphs" event={"ID":"470e5356-b27c-4300-972d-4409d4bccc45","Type":"ContainerDied","Data":"5b129e56bf7917e83435f2426608c772a6538f266cbf6d638a153b5d6412953b"} Jul 07 22:10:40.706319 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:40.706241 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lhh6j" Jul 07 22:10:40.827533 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:40.827503 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5mpjj\" (UniqueName: \"kubernetes.io/projected/dbe0bc5b-dcda-463a-8764-8ad638ccb1f7-kube-api-access-5mpjj\") pod \"dbe0bc5b-dcda-463a-8764-8ad638ccb1f7\" (UID: \"dbe0bc5b-dcda-463a-8764-8ad638ccb1f7\") " Jul 07 22:10:40.827661 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:40.827539 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dbe0bc5b-dcda-463a-8764-8ad638ccb1f7-util\") pod \"dbe0bc5b-dcda-463a-8764-8ad638ccb1f7\" (UID: \"dbe0bc5b-dcda-463a-8764-8ad638ccb1f7\") " Jul 07 22:10:40.827661 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:40.827563 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dbe0bc5b-dcda-463a-8764-8ad638ccb1f7-bundle\") pod \"dbe0bc5b-dcda-463a-8764-8ad638ccb1f7\" (UID: \"dbe0bc5b-dcda-463a-8764-8ad638ccb1f7\") " Jul 07 22:10:40.828126 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:40.828096 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbe0bc5b-dcda-463a-8764-8ad638ccb1f7-bundle" (OuterVolumeSpecName: "bundle") pod "dbe0bc5b-dcda-463a-8764-8ad638ccb1f7" (UID: "dbe0bc5b-dcda-463a-8764-8ad638ccb1f7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 22:10:40.829810 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:40.829786 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbe0bc5b-dcda-463a-8764-8ad638ccb1f7-kube-api-access-5mpjj" (OuterVolumeSpecName: "kube-api-access-5mpjj") pod "dbe0bc5b-dcda-463a-8764-8ad638ccb1f7" (UID: "dbe0bc5b-dcda-463a-8764-8ad638ccb1f7"). InnerVolumeSpecName "kube-api-access-5mpjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 22:10:40.832705 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:40.832684 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbe0bc5b-dcda-463a-8764-8ad638ccb1f7-util" (OuterVolumeSpecName: "util") pod "dbe0bc5b-dcda-463a-8764-8ad638ccb1f7" (UID: "dbe0bc5b-dcda-463a-8764-8ad638ccb1f7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 22:10:40.928835 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:40.928752 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mpjj\" (UniqueName: \"kubernetes.io/projected/dbe0bc5b-dcda-463a-8764-8ad638ccb1f7-kube-api-access-5mpjj\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 07 22:10:40.928835 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:40.928781 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dbe0bc5b-dcda-463a-8764-8ad638ccb1f7-util\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 07 22:10:40.928835 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:40.928790 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dbe0bc5b-dcda-463a-8764-8ad638ccb1f7-bundle\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 07 22:10:41.582637 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:41.582603 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lhh6j" event={"ID":"dbe0bc5b-dcda-463a-8764-8ad638ccb1f7","Type":"ContainerDied","Data":"a65483d6b7c6b582f5d24f918b06347915ba4dbb810e48c7514f7a557c056770"} Jul 07 22:10:41.583163 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:41.582680 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a65483d6b7c6b582f5d24f918b06347915ba4dbb810e48c7514f7a557c056770" Jul 07 22:10:41.583163 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:41.582615 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lhh6j" Jul 07 22:10:41.585223 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:41.585198 2564 generic.go:356] "Generic (PLEG): container finished" podID="7121c2d1-76f0-41f8-8b8a-a693dcef10ca" containerID="29484f443332e2e102242ab42c1fbc6bc56a556195a6d7a85c193515a5c464a2" exitCode=0 Jul 07 22:10:41.585357 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:41.585278 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9j2b" event={"ID":"7121c2d1-76f0-41f8-8b8a-a693dcef10ca","Type":"ContainerDied","Data":"29484f443332e2e102242ab42c1fbc6bc56a556195a6d7a85c193515a5c464a2"} Jul 07 22:10:41.587501 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:41.587480 2564 generic.go:356] "Generic (PLEG): container finished" podID="0fa7d0f7-6fa8-43df-b565-aefc01d4855a" containerID="72206b476436b5c3a327af237aa98fa822d978c256a3c6df6567d67156420d59" exitCode=0 Jul 07 22:10:41.587590 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:41.587569 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vjphr" event={"ID":"0fa7d0f7-6fa8-43df-b565-aefc01d4855a","Type":"ContainerDied","Data":"72206b476436b5c3a327af237aa98fa822d978c256a3c6df6567d67156420d59"} Jul 07 22:10:41.715885 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:41.715861 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759phphs" Jul 07 22:10:41.837490 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:41.837434 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-txsjk\" (UniqueName: \"kubernetes.io/projected/470e5356-b27c-4300-972d-4409d4bccc45-kube-api-access-txsjk\") pod \"470e5356-b27c-4300-972d-4409d4bccc45\" (UID: \"470e5356-b27c-4300-972d-4409d4bccc45\") " Jul 07 22:10:41.837683 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:41.837532 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/470e5356-b27c-4300-972d-4409d4bccc45-bundle\") pod \"470e5356-b27c-4300-972d-4409d4bccc45\" (UID: \"470e5356-b27c-4300-972d-4409d4bccc45\") " Jul 07 22:10:41.837683 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:41.837552 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/470e5356-b27c-4300-972d-4409d4bccc45-util\") pod \"470e5356-b27c-4300-972d-4409d4bccc45\" (UID: \"470e5356-b27c-4300-972d-4409d4bccc45\") " Jul 07 22:10:41.838039 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:41.838013 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/470e5356-b27c-4300-972d-4409d4bccc45-bundle" (OuterVolumeSpecName: "bundle") pod "470e5356-b27c-4300-972d-4409d4bccc45" (UID: "470e5356-b27c-4300-972d-4409d4bccc45"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 22:10:41.839596 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:41.839568 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/470e5356-b27c-4300-972d-4409d4bccc45-kube-api-access-txsjk" (OuterVolumeSpecName: "kube-api-access-txsjk") pod "470e5356-b27c-4300-972d-4409d4bccc45" (UID: "470e5356-b27c-4300-972d-4409d4bccc45"). InnerVolumeSpecName "kube-api-access-txsjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 22:10:41.843218 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:41.843186 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/470e5356-b27c-4300-972d-4409d4bccc45-util" (OuterVolumeSpecName: "util") pod "470e5356-b27c-4300-972d-4409d4bccc45" (UID: "470e5356-b27c-4300-972d-4409d4bccc45"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 22:10:41.938410 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:41.938340 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/470e5356-b27c-4300-972d-4409d4bccc45-bundle\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 07 22:10:41.938410 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:41.938362 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/470e5356-b27c-4300-972d-4409d4bccc45-util\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 07 22:10:41.938410 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:41.938371 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txsjk\" (UniqueName: \"kubernetes.io/projected/470e5356-b27c-4300-972d-4409d4bccc45-kube-api-access-txsjk\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 07 22:10:42.593936 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:42.593909 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759phphs" Jul 07 22:10:42.594386 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:42.593949 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759phphs" event={"ID":"470e5356-b27c-4300-972d-4409d4bccc45","Type":"ContainerDied","Data":"b37ad5e180a3522d52230f8b92816cd6be856e0e2777b0ea5b40c9380e718fd3"} Jul 07 22:10:42.594386 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:42.593991 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b37ad5e180a3522d52230f8b92816cd6be856e0e2777b0ea5b40c9380e718fd3" Jul 07 22:10:42.724344 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:42.724315 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9j2b" Jul 07 22:10:42.757410 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:42.757379 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vjphr" Jul 07 22:10:42.845478 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:42.845372 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7121c2d1-76f0-41f8-8b8a-a693dcef10ca-util\") pod \"7121c2d1-76f0-41f8-8b8a-a693dcef10ca\" (UID: \"7121c2d1-76f0-41f8-8b8a-a693dcef10ca\") " Jul 07 22:10:42.845478 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:42.845432 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0fa7d0f7-6fa8-43df-b565-aefc01d4855a-util\") pod \"0fa7d0f7-6fa8-43df-b565-aefc01d4855a\" (UID: \"0fa7d0f7-6fa8-43df-b565-aefc01d4855a\") " Jul 07 22:10:42.845695 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:42.845499 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vp6fd\" (UniqueName: \"kubernetes.io/projected/0fa7d0f7-6fa8-43df-b565-aefc01d4855a-kube-api-access-vp6fd\") pod \"0fa7d0f7-6fa8-43df-b565-aefc01d4855a\" (UID: \"0fa7d0f7-6fa8-43df-b565-aefc01d4855a\") " Jul 07 22:10:42.845695 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:42.845548 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z8mfh\" (UniqueName: \"kubernetes.io/projected/7121c2d1-76f0-41f8-8b8a-a693dcef10ca-kube-api-access-z8mfh\") pod \"7121c2d1-76f0-41f8-8b8a-a693dcef10ca\" (UID: \"7121c2d1-76f0-41f8-8b8a-a693dcef10ca\") " Jul 07 22:10:42.845695 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:42.845579 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7121c2d1-76f0-41f8-8b8a-a693dcef10ca-bundle\") pod \"7121c2d1-76f0-41f8-8b8a-a693dcef10ca\" (UID: \"7121c2d1-76f0-41f8-8b8a-a693dcef10ca\") " Jul 07 22:10:42.845695 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:42.845631 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0fa7d0f7-6fa8-43df-b565-aefc01d4855a-bundle\") pod \"0fa7d0f7-6fa8-43df-b565-aefc01d4855a\" (UID: \"0fa7d0f7-6fa8-43df-b565-aefc01d4855a\") " Jul 07 22:10:42.846332 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:42.846226 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fa7d0f7-6fa8-43df-b565-aefc01d4855a-bundle" (OuterVolumeSpecName: "bundle") pod "0fa7d0f7-6fa8-43df-b565-aefc01d4855a" (UID: "0fa7d0f7-6fa8-43df-b565-aefc01d4855a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 22:10:42.846332 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:42.846304 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7121c2d1-76f0-41f8-8b8a-a693dcef10ca-bundle" (OuterVolumeSpecName: "bundle") pod "7121c2d1-76f0-41f8-8b8a-a693dcef10ca" (UID: "7121c2d1-76f0-41f8-8b8a-a693dcef10ca"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 22:10:42.847846 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:42.847817 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7121c2d1-76f0-41f8-8b8a-a693dcef10ca-kube-api-access-z8mfh" (OuterVolumeSpecName: "kube-api-access-z8mfh") pod "7121c2d1-76f0-41f8-8b8a-a693dcef10ca" (UID: "7121c2d1-76f0-41f8-8b8a-a693dcef10ca"). InnerVolumeSpecName "kube-api-access-z8mfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 22:10:42.848070 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:42.848050 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fa7d0f7-6fa8-43df-b565-aefc01d4855a-kube-api-access-vp6fd" (OuterVolumeSpecName: "kube-api-access-vp6fd") pod "0fa7d0f7-6fa8-43df-b565-aefc01d4855a" (UID: "0fa7d0f7-6fa8-43df-b565-aefc01d4855a"). InnerVolumeSpecName "kube-api-access-vp6fd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 22:10:42.850973 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:42.850946 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fa7d0f7-6fa8-43df-b565-aefc01d4855a-util" (OuterVolumeSpecName: "util") pod "0fa7d0f7-6fa8-43df-b565-aefc01d4855a" (UID: "0fa7d0f7-6fa8-43df-b565-aefc01d4855a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 22:10:42.851420 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:42.851402 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7121c2d1-76f0-41f8-8b8a-a693dcef10ca-util" (OuterVolumeSpecName: "util") pod "7121c2d1-76f0-41f8-8b8a-a693dcef10ca" (UID: "7121c2d1-76f0-41f8-8b8a-a693dcef10ca"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 22:10:42.946939 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:42.946908 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7121c2d1-76f0-41f8-8b8a-a693dcef10ca-util\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 07 22:10:42.946939 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:42.946932 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0fa7d0f7-6fa8-43df-b565-aefc01d4855a-util\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 07 22:10:42.946939 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:42.946941 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vp6fd\" (UniqueName: \"kubernetes.io/projected/0fa7d0f7-6fa8-43df-b565-aefc01d4855a-kube-api-access-vp6fd\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 07 22:10:42.947141 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:42.946951 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8mfh\" (UniqueName: \"kubernetes.io/projected/7121c2d1-76f0-41f8-8b8a-a693dcef10ca-kube-api-access-z8mfh\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 07 22:10:42.947141 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:42.946959 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7121c2d1-76f0-41f8-8b8a-a693dcef10ca-bundle\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 07 22:10:42.947141 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:42.946969 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0fa7d0f7-6fa8-43df-b565-aefc01d4855a-bundle\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 07 22:10:43.601486 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:43.601458 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vjphr" Jul 07 22:10:43.601916 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:43.601481 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vjphr" event={"ID":"0fa7d0f7-6fa8-43df-b565-aefc01d4855a","Type":"ContainerDied","Data":"d50fa0fe0ba158af75da056ce0a5ec0d2aa24814b2fd2850cdcb8df99aeb3489"} Jul 07 22:10:43.601916 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:43.601534 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d50fa0fe0ba158af75da056ce0a5ec0d2aa24814b2fd2850cdcb8df99aeb3489" Jul 07 22:10:43.603784 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:43.603760 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9j2b" event={"ID":"7121c2d1-76f0-41f8-8b8a-a693dcef10ca","Type":"ContainerDied","Data":"48321b0dccafcf6930588abb0f563fd83c0843c195009e450e0789d205453540"} Jul 07 22:10:43.603917 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:43.603801 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9j2b" Jul 07 22:10:43.603917 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:10:43.603802 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48321b0dccafcf6930588abb0f563fd83c0843c195009e450e0789d205453540" Jul 07 22:12:07.583180 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:12:07.583141 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:12:07.583665 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:12:07.583487 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="470e5356-b27c-4300-972d-4409d4bccc45" containerName="util" Jul 07 22:12:07.583665 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:12:07.583502 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="470e5356-b27c-4300-972d-4409d4bccc45" containerName="util" Jul 07 22:12:07.583665 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:12:07.583513 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="470e5356-b27c-4300-972d-4409d4bccc45" containerName="pull" Jul 07 22:12:07.583665 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:12:07.583520 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="470e5356-b27c-4300-972d-4409d4bccc45" containerName="pull" Jul 07 22:12:07.583665 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:12:07.583529 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbe0bc5b-dcda-463a-8764-8ad638ccb1f7" containerName="pull" Jul 07 22:12:07.583665 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:12:07.583534 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbe0bc5b-dcda-463a-8764-8ad638ccb1f7" containerName="pull" Jul 07 22:12:07.583665 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:12:07.583544 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7121c2d1-76f0-41f8-8b8a-a693dcef10ca" containerName="pull" Jul 07 22:12:07.583665 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:12:07.583548 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7121c2d1-76f0-41f8-8b8a-a693dcef10ca" containerName="pull" Jul 07 22:12:07.583665 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:12:07.583555 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbe0bc5b-dcda-463a-8764-8ad638ccb1f7" containerName="util" Jul 07 22:12:07.583665 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:12:07.583561 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbe0bc5b-dcda-463a-8764-8ad638ccb1f7" containerName="util" Jul 07 22:12:07.583665 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:12:07.583570 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fa7d0f7-6fa8-43df-b565-aefc01d4855a" containerName="extract" Jul 07 22:12:07.583665 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:12:07.583578 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fa7d0f7-6fa8-43df-b565-aefc01d4855a" containerName="extract" Jul 07 22:12:07.583665 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:12:07.583593 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7121c2d1-76f0-41f8-8b8a-a693dcef10ca" containerName="util" Jul 07 22:12:07.583665 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:12:07.583598 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7121c2d1-76f0-41f8-8b8a-a693dcef10ca" containerName="util" Jul 07 22:12:07.583665 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:12:07.583604 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbe0bc5b-dcda-463a-8764-8ad638ccb1f7" containerName="extract" Jul 07 22:12:07.583665 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:12:07.583610 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbe0bc5b-dcda-463a-8764-8ad638ccb1f7" containerName="extract" Jul 07 22:12:07.583665 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:12:07.583615 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fa7d0f7-6fa8-43df-b565-aefc01d4855a" containerName="util" Jul 07 22:12:07.583665 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:12:07.583621 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fa7d0f7-6fa8-43df-b565-aefc01d4855a" containerName="util" Jul 07 22:12:07.583665 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:12:07.583628 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fa7d0f7-6fa8-43df-b565-aefc01d4855a" containerName="pull" Jul 07 22:12:07.583665 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:12:07.583633 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fa7d0f7-6fa8-43df-b565-aefc01d4855a" containerName="pull" Jul 07 22:12:07.583665 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:12:07.583640 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="470e5356-b27c-4300-972d-4409d4bccc45" containerName="extract" Jul 07 22:12:07.583665 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:12:07.583648 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="470e5356-b27c-4300-972d-4409d4bccc45" containerName="extract" Jul 07 22:12:07.583665 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:12:07.583657 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7121c2d1-76f0-41f8-8b8a-a693dcef10ca" containerName="extract" Jul 07 22:12:07.583665 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:12:07.583663 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7121c2d1-76f0-41f8-8b8a-a693dcef10ca" containerName="extract" Jul 07 22:12:07.584409 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:12:07.583713 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7121c2d1-76f0-41f8-8b8a-a693dcef10ca" containerName="extract" Jul 07 22:12:07.584409 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:12:07.583720 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fa7d0f7-6fa8-43df-b565-aefc01d4855a" containerName="extract" Jul 07 22:12:07.584409 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:12:07.583729 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbe0bc5b-dcda-463a-8764-8ad638ccb1f7" containerName="extract" Jul 07 22:12:07.584409 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:12:07.583737 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="470e5356-b27c-4300-972d-4409d4bccc45" containerName="extract" Jul 07 22:12:07.586504 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:12:07.586488 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-f4z8p" Jul 07 22:12:07.588697 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:12:07.588674 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 07 22:12:07.589424 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:12:07.589407 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-f97sq\"" Jul 07 22:12:07.589553 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:12:07.589420 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 07 22:12:07.589553 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:12:07.589420 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 07 22:12:07.593596 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:12:07.593508 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:12:07.626632 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:12:07.626602 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:12:07.745420 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:12:07.745385 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/41efdd54-8912-4931-b9c2-3cfae8964536-config-file\") pod \"limitador-limitador-54d6d988d5-f4z8p\" (UID: \"41efdd54-8912-4931-b9c2-3cfae8964536\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-f4z8p" Jul 07 22:12:07.745578 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:12:07.745475 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2cjr\" (UniqueName: \"kubernetes.io/projected/41efdd54-8912-4931-b9c2-3cfae8964536-kube-api-access-g2cjr\") pod \"limitador-limitador-54d6d988d5-f4z8p\" (UID: \"41efdd54-8912-4931-b9c2-3cfae8964536\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-f4z8p" Jul 07 22:12:07.846811 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:12:07.846730 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/41efdd54-8912-4931-b9c2-3cfae8964536-config-file\") pod \"limitador-limitador-54d6d988d5-f4z8p\" (UID: \"41efdd54-8912-4931-b9c2-3cfae8964536\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-f4z8p" Jul 07 22:12:07.847100 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:12:07.847072 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2cjr\" (UniqueName: \"kubernetes.io/projected/41efdd54-8912-4931-b9c2-3cfae8964536-kube-api-access-g2cjr\") pod \"limitador-limitador-54d6d988d5-f4z8p\" (UID: \"41efdd54-8912-4931-b9c2-3cfae8964536\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-f4z8p" Jul 07 22:12:07.847559 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:12:07.847534 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/41efdd54-8912-4931-b9c2-3cfae8964536-config-file\") pod \"limitador-limitador-54d6d988d5-f4z8p\" (UID: \"41efdd54-8912-4931-b9c2-3cfae8964536\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-f4z8p" Jul 07 22:12:07.854884 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:12:07.854862 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2cjr\" (UniqueName: \"kubernetes.io/projected/41efdd54-8912-4931-b9c2-3cfae8964536-kube-api-access-g2cjr\") pod \"limitador-limitador-54d6d988d5-f4z8p\" (UID: \"41efdd54-8912-4931-b9c2-3cfae8964536\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-f4z8p" Jul 07 22:12:07.898007 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:12:07.897976 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-f4z8p" Jul 07 22:12:08.030207 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:12:08.030183 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:12:08.032350 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:12:08.032323 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41efdd54_8912_4931_b9c2_3cfae8964536.slice/crio-1799e75ff195f28f8a5ec9e1df78a19ece820d656e4fee313b63f4a85af12aaf WatchSource:0}: Error finding container 1799e75ff195f28f8a5ec9e1df78a19ece820d656e4fee313b63f4a85af12aaf: Status 404 returned error can't find the container with id 1799e75ff195f28f8a5ec9e1df78a19ece820d656e4fee313b63f4a85af12aaf Jul 07 22:12:09.006165 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:12:09.006125 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-f4z8p" event={"ID":"41efdd54-8912-4931-b9c2-3cfae8964536","Type":"ContainerStarted","Data":"1799e75ff195f28f8a5ec9e1df78a19ece820d656e4fee313b63f4a85af12aaf"} Jul 07 22:12:11.017862 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:12:11.017762 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-f4z8p" event={"ID":"41efdd54-8912-4931-b9c2-3cfae8964536","Type":"ContainerStarted","Data":"ada724f53706b3ed12a3654ae0847305cf9a3b6c3cbae9788660059b6956df80"} Jul 07 22:12:11.018289 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:12:11.017907 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-f4z8p" Jul 07 22:12:11.035333 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:12:11.035283 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-f4z8p" podStartSLOduration=1.464050854 podStartE2EDuration="4.035268907s" totalImagesPullingTime="2.571218053s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 22:12:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:12:11.032623804 +0000 UTC m=+510.860753253" watchObservedRunningTime="2026-07-07 22:12:11.035268907 +0000 UTC m=+510.863398356" Jul 07 22:12:22.023637 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:12:22.023610 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-f4z8p" Jul 07 22:13:22.521244 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:13:22.521211 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:13:40.590531 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:13:40.590501 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4h8zm_4b4234f5-6cda-45ea-b559-f09bcad77232/ovn-acl-logging/0.log" Jul 07 22:13:40.592201 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:13:40.592175 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4h8zm_4b4234f5-6cda-45ea-b559-f09bcad77232/ovn-acl-logging/0.log" Jul 07 22:13:54.008419 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:13:54.008382 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:14:11.822525 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:14:11.822427 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:14:24.716993 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:14:24.716960 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-wprgj"] Jul 07 22:14:24.720279 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:14:24.720260 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wprgj" Jul 07 22:14:24.723650 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:14:24.723627 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 07 22:14:24.723758 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:14:24.723627 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-wd56t\"" Jul 07 22:14:24.723758 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:14:24.723629 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 07 22:14:24.723758 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:14:24.723629 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 07 22:14:24.730617 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:14:24.730592 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-wprgj"] Jul 07 22:14:24.739246 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:14:24.739224 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b97ea41a-1606-44a8-9a8d-0b30e0b885a7-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wprgj\" (UID: \"b97ea41a-1606-44a8-9a8d-0b30e0b885a7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wprgj" Jul 07 22:14:24.739352 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:14:24.739257 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b97ea41a-1606-44a8-9a8d-0b30e0b885a7-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wprgj\" (UID: \"b97ea41a-1606-44a8-9a8d-0b30e0b885a7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wprgj" Jul 07 22:14:24.739352 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:14:24.739280 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxqdc\" (UniqueName: \"kubernetes.io/projected/b97ea41a-1606-44a8-9a8d-0b30e0b885a7-kube-api-access-vxqdc\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wprgj\" (UID: \"b97ea41a-1606-44a8-9a8d-0b30e0b885a7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wprgj" Jul 07 22:14:24.739484 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:14:24.739372 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b97ea41a-1606-44a8-9a8d-0b30e0b885a7-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wprgj\" (UID: \"b97ea41a-1606-44a8-9a8d-0b30e0b885a7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wprgj" Jul 07 22:14:24.739484 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:14:24.739426 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b97ea41a-1606-44a8-9a8d-0b30e0b885a7-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wprgj\" (UID: \"b97ea41a-1606-44a8-9a8d-0b30e0b885a7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wprgj" Jul 07 22:14:24.739556 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:14:24.739489 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b97ea41a-1606-44a8-9a8d-0b30e0b885a7-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wprgj\" (UID: \"b97ea41a-1606-44a8-9a8d-0b30e0b885a7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wprgj" Jul 07 22:14:24.840579 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:14:24.840550 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b97ea41a-1606-44a8-9a8d-0b30e0b885a7-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wprgj\" (UID: \"b97ea41a-1606-44a8-9a8d-0b30e0b885a7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wprgj" Jul 07 22:14:24.840579 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:14:24.840585 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b97ea41a-1606-44a8-9a8d-0b30e0b885a7-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wprgj\" (UID: \"b97ea41a-1606-44a8-9a8d-0b30e0b885a7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wprgj" Jul 07 22:14:24.840821 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:14:24.840605 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxqdc\" (UniqueName: \"kubernetes.io/projected/b97ea41a-1606-44a8-9a8d-0b30e0b885a7-kube-api-access-vxqdc\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wprgj\" (UID: \"b97ea41a-1606-44a8-9a8d-0b30e0b885a7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wprgj" Jul 07 22:14:24.840821 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:14:24.840641 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b97ea41a-1606-44a8-9a8d-0b30e0b885a7-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wprgj\" (UID: \"b97ea41a-1606-44a8-9a8d-0b30e0b885a7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wprgj" Jul 07 22:14:24.840821 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:14:24.840662 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b97ea41a-1606-44a8-9a8d-0b30e0b885a7-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wprgj\" (UID: \"b97ea41a-1606-44a8-9a8d-0b30e0b885a7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wprgj" Jul 07 22:14:24.840821 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:14:24.840678 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b97ea41a-1606-44a8-9a8d-0b30e0b885a7-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wprgj\" (UID: \"b97ea41a-1606-44a8-9a8d-0b30e0b885a7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wprgj" Jul 07 22:14:24.841040 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:14:24.840958 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b97ea41a-1606-44a8-9a8d-0b30e0b885a7-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wprgj\" (UID: \"b97ea41a-1606-44a8-9a8d-0b30e0b885a7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wprgj" Jul 07 22:14:24.841040 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:14:24.841028 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b97ea41a-1606-44a8-9a8d-0b30e0b885a7-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wprgj\" (UID: \"b97ea41a-1606-44a8-9a8d-0b30e0b885a7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wprgj" Jul 07 22:14:24.841142 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:14:24.841122 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b97ea41a-1606-44a8-9a8d-0b30e0b885a7-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wprgj\" (UID: \"b97ea41a-1606-44a8-9a8d-0b30e0b885a7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wprgj" Jul 07 22:14:24.843071 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:14:24.843049 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b97ea41a-1606-44a8-9a8d-0b30e0b885a7-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wprgj\" (UID: \"b97ea41a-1606-44a8-9a8d-0b30e0b885a7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wprgj" Jul 07 22:14:24.843351 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:14:24.843331 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b97ea41a-1606-44a8-9a8d-0b30e0b885a7-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wprgj\" (UID: \"b97ea41a-1606-44a8-9a8d-0b30e0b885a7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wprgj" Jul 07 22:14:24.849013 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:14:24.848989 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxqdc\" (UniqueName: \"kubernetes.io/projected/b97ea41a-1606-44a8-9a8d-0b30e0b885a7-kube-api-access-vxqdc\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wprgj\" (UID: \"b97ea41a-1606-44a8-9a8d-0b30e0b885a7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wprgj" Jul 07 22:14:25.032311 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:14:25.032233 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wprgj" Jul 07 22:14:25.163917 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:14:25.163891 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-wprgj"] Jul 07 22:14:25.166086 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:14:25.166058 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb97ea41a_1606_44a8_9a8d_0b30e0b885a7.slice/crio-bc391ff4ab8357e3297984acf7658ec698c51aecf12c155d8995e739d7a52bb5 WatchSource:0}: Error finding container bc391ff4ab8357e3297984acf7658ec698c51aecf12c155d8995e739d7a52bb5: Status 404 returned error can't find the container with id bc391ff4ab8357e3297984acf7658ec698c51aecf12c155d8995e739d7a52bb5 Jul 07 22:14:25.167392 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:14:25.167373 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 22:14:25.687186 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:14:25.687145 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wprgj" event={"ID":"b97ea41a-1606-44a8-9a8d-0b30e0b885a7","Type":"ContainerStarted","Data":"bc391ff4ab8357e3297984acf7658ec698c51aecf12c155d8995e739d7a52bb5"} Jul 07 22:14:25.720299 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:14:25.720266 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:14:26.697758 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:14:26.697729 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wprgj" event={"ID":"b97ea41a-1606-44a8-9a8d-0b30e0b885a7","Type":"ContainerStarted","Data":"ea0bd8afdb7070e708afb5e632e9c70f1b6f106f1dcbf4515ecb372e541ec565"} Jul 07 22:14:26.715582 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:14:26.715538 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wprgj" podStartSLOduration=1.701252748 podStartE2EDuration="2.715523232s" totalImagesPullingTime="1.014270484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 22:14:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:14:26.714253627 +0000 UTC m=+646.542383097" watchObservedRunningTime="2026-07-07 22:14:26.715523232 +0000 UTC m=+646.543652692" Jul 07 22:14:32.011234 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:14:32.011202 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:14:35.032434 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:14:35.032398 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wprgj" Jul 07 22:14:35.032840 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:14:35.032475 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wprgj" Jul 07 22:14:35.045043 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:14:35.045016 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wprgj" Jul 07 22:14:35.751286 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:14:35.751255 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wprgj" Jul 07 22:14:43.915747 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:14:43.915713 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:14:55.311124 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:14:55.311081 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:15:00.132954 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:15:00.132910 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29724375-6cc8q"] Jul 07 22:15:00.137909 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:15:00.137893 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724375-6cc8q" Jul 07 22:15:00.140055 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:15:00.140038 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-8l4tr\"" Jul 07 22:15:00.142950 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:15:00.142928 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29724375-6cc8q"] Jul 07 22:15:00.250408 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:15:00.250368 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pwct\" (UniqueName: \"kubernetes.io/projected/fe0445a2-ddfe-4ff6-9d4d-87096b886097-kube-api-access-6pwct\") pod \"maas-api-key-cleanup-29724375-6cc8q\" (UID: \"fe0445a2-ddfe-4ff6-9d4d-87096b886097\") " pod="opendatahub/maas-api-key-cleanup-29724375-6cc8q" Jul 07 22:15:00.351707 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:15:00.351674 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pwct\" (UniqueName: \"kubernetes.io/projected/fe0445a2-ddfe-4ff6-9d4d-87096b886097-kube-api-access-6pwct\") pod \"maas-api-key-cleanup-29724375-6cc8q\" (UID: \"fe0445a2-ddfe-4ff6-9d4d-87096b886097\") " pod="opendatahub/maas-api-key-cleanup-29724375-6cc8q" Jul 07 22:15:00.360051 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:15:00.360015 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pwct\" (UniqueName: \"kubernetes.io/projected/fe0445a2-ddfe-4ff6-9d4d-87096b886097-kube-api-access-6pwct\") pod \"maas-api-key-cleanup-29724375-6cc8q\" (UID: \"fe0445a2-ddfe-4ff6-9d4d-87096b886097\") " pod="opendatahub/maas-api-key-cleanup-29724375-6cc8q" Jul 07 22:15:00.448720 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:15:00.448641 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724375-6cc8q" Jul 07 22:15:00.578959 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:15:00.578932 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29724375-6cc8q"] Jul 07 22:15:00.580796 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:15:00.580764 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe0445a2_ddfe_4ff6_9d4d_87096b886097.slice/crio-aa2b45e08da371d84e7253b03401e03d3efa17b97cd3d5ae6c2b6ec4a9449fc6 WatchSource:0}: Error finding container aa2b45e08da371d84e7253b03401e03d3efa17b97cd3d5ae6c2b6ec4a9449fc6: Status 404 returned error can't find the container with id aa2b45e08da371d84e7253b03401e03d3efa17b97cd3d5ae6c2b6ec4a9449fc6 Jul 07 22:15:00.860618 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:15:00.860578 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29724375-6cc8q" event={"ID":"fe0445a2-ddfe-4ff6-9d4d-87096b886097","Type":"ContainerStarted","Data":"aa2b45e08da371d84e7253b03401e03d3efa17b97cd3d5ae6c2b6ec4a9449fc6"} Jul 07 22:15:03.879296 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:15:03.879261 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29724375-6cc8q" event={"ID":"fe0445a2-ddfe-4ff6-9d4d-87096b886097","Type":"ContainerStarted","Data":"3e999e57c7a13211b217aa3d9ec8dd7a7e6e6927eebf61439577cc93f8c1580b"} Jul 07 22:15:03.894840 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:15:03.894795 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-29724375-6cc8q" podStartSLOduration=1.4738781429999999 podStartE2EDuration="3.894780197s" totalImagesPullingTime="2.420902054s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 22:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:15:03.892266586 +0000 UTC m=+683.720396041" watchObservedRunningTime="2026-07-07 22:15:03.894780197 +0000 UTC m=+683.722909646" Jul 07 22:15:04.885823 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:15:04.885788 2564 generic.go:356] "Generic (PLEG): container finished" podID="fe0445a2-ddfe-4ff6-9d4d-87096b886097" containerID="3e999e57c7a13211b217aa3d9ec8dd7a7e6e6927eebf61439577cc93f8c1580b" exitCode=0 Jul 07 22:15:04.886231 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:15:04.885877 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29724375-6cc8q" event={"ID":"fe0445a2-ddfe-4ff6-9d4d-87096b886097","Type":"ContainerDied","Data":"3e999e57c7a13211b217aa3d9ec8dd7a7e6e6927eebf61439577cc93f8c1580b"} Jul 07 22:15:06.013147 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:15:06.013122 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724375-6cc8q" Jul 07 22:15:06.204115 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:15:06.204015 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6pwct\" (UniqueName: \"kubernetes.io/projected/fe0445a2-ddfe-4ff6-9d4d-87096b886097-kube-api-access-6pwct\") pod \"fe0445a2-ddfe-4ff6-9d4d-87096b886097\" (UID: \"fe0445a2-ddfe-4ff6-9d4d-87096b886097\") " Jul 07 22:15:06.206336 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:15:06.206307 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe0445a2-ddfe-4ff6-9d4d-87096b886097-kube-api-access-6pwct" (OuterVolumeSpecName: "kube-api-access-6pwct") pod "fe0445a2-ddfe-4ff6-9d4d-87096b886097" (UID: "fe0445a2-ddfe-4ff6-9d4d-87096b886097"). InnerVolumeSpecName "kube-api-access-6pwct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 22:15:06.305595 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:15:06.305554 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pwct\" (UniqueName: \"kubernetes.io/projected/fe0445a2-ddfe-4ff6-9d4d-87096b886097-kube-api-access-6pwct\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 07 22:15:06.896578 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:15:06.896548 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724375-6cc8q" Jul 07 22:15:06.896752 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:15:06.896546 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29724375-6cc8q" event={"ID":"fe0445a2-ddfe-4ff6-9d4d-87096b886097","Type":"ContainerDied","Data":"aa2b45e08da371d84e7253b03401e03d3efa17b97cd3d5ae6c2b6ec4a9449fc6"} Jul 07 22:15:06.896752 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:15:06.896666 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa2b45e08da371d84e7253b03401e03d3efa17b97cd3d5ae6c2b6ec4a9449fc6" Jul 07 22:16:18.515334 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:16:18.515297 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:16:28.617267 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:16:28.617224 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:16:40.212387 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:16:40.212341 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:16:48.505665 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:16:48.505626 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:16:59.006143 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:16:59.006106 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:17:27.818750 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:17:27.818714 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:17:41.918959 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:17:41.918921 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:18:16.798969 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:18:16.798931 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:18:25.607269 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:18:25.607234 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:18:34.515178 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:18:34.515093 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:18:40.630513 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:18:40.630481 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4h8zm_4b4234f5-6cda-45ea-b559-f09bcad77232/ovn-acl-logging/0.log" Jul 07 22:18:40.630959 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:18:40.630709 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4h8zm_4b4234f5-6cda-45ea-b559-f09bcad77232/ovn-acl-logging/0.log" Jul 07 22:18:43.407894 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:18:43.407858 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:18:53.304328 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:18:53.304293 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:19:02.602535 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:19:02.602505 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:19:12.515050 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:19:12.515014 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:19:20.004850 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:19:20.004818 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:19:27.705791 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:19:27.705759 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:19:36.806946 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:19:36.806909 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:19:47.905839 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:19:47.905810 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:19:59.616952 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:19:59.616907 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:20:22.503127 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:20:22.503045 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:20:32.204734 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:20:32.204696 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:20:50.113541 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:20:50.113507 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:20:59.806426 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:20:59.806396 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:21:08.918933 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:21:08.918903 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:21:18.107795 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:21:18.107749 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:21:27.208369 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:21:27.208336 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:21:51.307235 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:21:51.307155 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:23:22.006990 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:23:22.006956 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:23:29.312063 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:23:29.312029 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:23:37.907699 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:23:37.907664 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:23:40.662078 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:23:40.662046 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4h8zm_4b4234f5-6cda-45ea-b559-f09bcad77232/ovn-acl-logging/0.log" Jul 07 22:23:40.662549 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:23:40.662531 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4h8zm_4b4234f5-6cda-45ea-b559-f09bcad77232/ovn-acl-logging/0.log" Jul 07 22:23:48.510136 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:23:48.510102 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:23:59.011411 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:23:59.011374 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:24:09.310183 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:24:09.310147 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:24:19.211968 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:24:19.211935 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:24:28.713094 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:24:28.713063 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:24:40.305647 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:24:40.305560 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:24:47.707243 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:24:47.707210 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:24:56.800157 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:24:56.800123 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:25:07.006554 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:25:07.006518 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:25:25.213219 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:25:25.213186 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:25:34.410317 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:25:34.410284 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:25:43.807005 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:25:43.806930 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:25:55.210935 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:25:55.210900 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:26:02.312954 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:26:02.312868 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:26:13.408917 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:26:13.408884 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:26:22.710593 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:26:22.710563 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:26:31.305342 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:26:31.305311 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:26:41.306969 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:26:41.306940 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:26:50.004866 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:26:50.004834 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:26:58.903791 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:26:58.903758 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:27:12.013587 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:27:12.013551 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:27:21.313667 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:27:21.313633 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:27:30.914268 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:27:30.914193 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:27:42.214180 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:27:42.214144 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:27:54.007732 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:27:54.007693 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:28:01.614697 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:28:01.614656 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:28:11.003593 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:28:11.003555 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:28:26.309300 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:28:26.309265 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:28:36.708002 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:28:36.707969 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:28:40.693827 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:28:40.693799 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4h8zm_4b4234f5-6cda-45ea-b559-f09bcad77232/ovn-acl-logging/0.log" Jul 07 22:28:40.696707 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:28:40.696686 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4h8zm_4b4234f5-6cda-45ea-b559-f09bcad77232/ovn-acl-logging/0.log" Jul 07 22:28:49.007285 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:28:49.007257 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:29:01.011064 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:29:01.010986 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:29:10.805644 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:29:10.805602 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:29:23.103582 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:29:23.103553 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:29:33.603943 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:29:33.603907 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:30:00.133971 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:30:00.133929 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29724390-tzl6q"] Jul 07 22:30:00.134421 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:30:00.134382 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe0445a2-ddfe-4ff6-9d4d-87096b886097" containerName="cleanup" Jul 07 22:30:00.134421 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:30:00.134396 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe0445a2-ddfe-4ff6-9d4d-87096b886097" containerName="cleanup" Jul 07 22:30:00.134555 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:30:00.134466 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe0445a2-ddfe-4ff6-9d4d-87096b886097" containerName="cleanup" Jul 07 22:30:00.137637 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:30:00.137618 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724390-tzl6q" Jul 07 22:30:00.139816 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:30:00.139795 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-8l4tr\"" Jul 07 22:30:00.143341 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:30:00.143310 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29724390-tzl6q"] Jul 07 22:30:00.234990 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:30:00.234953 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lvd7\" (UniqueName: \"kubernetes.io/projected/7586a2e1-cc6e-4262-91ba-bb7ec0dd571a-kube-api-access-9lvd7\") pod \"maas-api-key-cleanup-29724390-tzl6q\" (UID: \"7586a2e1-cc6e-4262-91ba-bb7ec0dd571a\") " pod="opendatahub/maas-api-key-cleanup-29724390-tzl6q" Jul 07 22:30:00.336313 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:30:00.336283 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lvd7\" (UniqueName: \"kubernetes.io/projected/7586a2e1-cc6e-4262-91ba-bb7ec0dd571a-kube-api-access-9lvd7\") pod \"maas-api-key-cleanup-29724390-tzl6q\" (UID: \"7586a2e1-cc6e-4262-91ba-bb7ec0dd571a\") " pod="opendatahub/maas-api-key-cleanup-29724390-tzl6q" Jul 07 22:30:00.345003 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:30:00.344979 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lvd7\" (UniqueName: \"kubernetes.io/projected/7586a2e1-cc6e-4262-91ba-bb7ec0dd571a-kube-api-access-9lvd7\") pod \"maas-api-key-cleanup-29724390-tzl6q\" (UID: \"7586a2e1-cc6e-4262-91ba-bb7ec0dd571a\") " pod="opendatahub/maas-api-key-cleanup-29724390-tzl6q" Jul 07 22:30:00.449105 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:30:00.449022 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724390-tzl6q" Jul 07 22:30:00.582060 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:30:00.581878 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29724390-tzl6q"] Jul 07 22:30:00.584272 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:30:00.584242 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7586a2e1_cc6e_4262_91ba_bb7ec0dd571a.slice/crio-3af2fc0d596112421d86d1c17a5f9461980fbf1eb8e0a61e5ca2b104004187d0 WatchSource:0}: Error finding container 3af2fc0d596112421d86d1c17a5f9461980fbf1eb8e0a61e5ca2b104004187d0: Status 404 returned error can't find the container with id 3af2fc0d596112421d86d1c17a5f9461980fbf1eb8e0a61e5ca2b104004187d0 Jul 07 22:30:01.268123 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:30:01.268031 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29724390-tzl6q" event={"ID":"7586a2e1-cc6e-4262-91ba-bb7ec0dd571a","Type":"ContainerStarted","Data":"b7d6e3340631cd63e3e057935b170caaf2450bcaa9736d2959a238382e5fa194"} Jul 07 22:30:01.268123 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:30:01.268087 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29724390-tzl6q" event={"ID":"7586a2e1-cc6e-4262-91ba-bb7ec0dd571a","Type":"ContainerStarted","Data":"3af2fc0d596112421d86d1c17a5f9461980fbf1eb8e0a61e5ca2b104004187d0"} Jul 07 22:30:01.282877 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:30:01.282832 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-29724390-tzl6q" podStartSLOduration=1.282817625 podStartE2EDuration="1.282817625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:30:01.280500463 +0000 UTC m=+1581.108629921" watchObservedRunningTime="2026-07-07 22:30:01.282817625 +0000 UTC m=+1581.110947083" Jul 07 22:30:02.274108 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:30:02.274065 2564 generic.go:356] "Generic (PLEG): container finished" podID="7586a2e1-cc6e-4262-91ba-bb7ec0dd571a" containerID="b7d6e3340631cd63e3e057935b170caaf2450bcaa9736d2959a238382e5fa194" exitCode=0 Jul 07 22:30:02.274515 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:30:02.274151 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29724390-tzl6q" event={"ID":"7586a2e1-cc6e-4262-91ba-bb7ec0dd571a","Type":"ContainerDied","Data":"b7d6e3340631cd63e3e057935b170caaf2450bcaa9736d2959a238382e5fa194"} Jul 07 22:30:03.408064 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:30:03.408042 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724390-tzl6q" Jul 07 22:30:03.460789 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:30:03.460761 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9lvd7\" (UniqueName: \"kubernetes.io/projected/7586a2e1-cc6e-4262-91ba-bb7ec0dd571a-kube-api-access-9lvd7\") pod \"7586a2e1-cc6e-4262-91ba-bb7ec0dd571a\" (UID: \"7586a2e1-cc6e-4262-91ba-bb7ec0dd571a\") " Jul 07 22:30:03.462901 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:30:03.462874 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7586a2e1-cc6e-4262-91ba-bb7ec0dd571a-kube-api-access-9lvd7" (OuterVolumeSpecName: "kube-api-access-9lvd7") pod "7586a2e1-cc6e-4262-91ba-bb7ec0dd571a" (UID: "7586a2e1-cc6e-4262-91ba-bb7ec0dd571a"). InnerVolumeSpecName "kube-api-access-9lvd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 22:30:03.561646 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:30:03.561623 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lvd7\" (UniqueName: \"kubernetes.io/projected/7586a2e1-cc6e-4262-91ba-bb7ec0dd571a-kube-api-access-9lvd7\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 07 22:30:04.285169 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:30:04.285144 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724390-tzl6q" Jul 07 22:30:04.285169 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:30:04.285150 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29724390-tzl6q" event={"ID":"7586a2e1-cc6e-4262-91ba-bb7ec0dd571a","Type":"ContainerDied","Data":"3af2fc0d596112421d86d1c17a5f9461980fbf1eb8e0a61e5ca2b104004187d0"} Jul 07 22:30:04.285420 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:30:04.285195 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3af2fc0d596112421d86d1c17a5f9461980fbf1eb8e0a61e5ca2b104004187d0" Jul 07 22:31:11.906128 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:31:11.906094 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:31:20.810125 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:31:20.810088 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:31:29.309708 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:31:29.309672 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:31:45.903898 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:31:45.903862 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:32:16.011624 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:32:16.011546 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:32:24.499413 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:32:24.499378 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:32:49.713366 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:32:49.713330 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:33:03.607808 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:33:03.607768 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:33:40.727343 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:33:40.727313 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4h8zm_4b4234f5-6cda-45ea-b559-f09bcad77232/ovn-acl-logging/0.log" Jul 07 22:33:40.731069 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:33:40.731048 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4h8zm_4b4234f5-6cda-45ea-b559-f09bcad77232/ovn-acl-logging/0.log" Jul 07 22:33:43.901885 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:33:43.901848 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:33:57.311887 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:33:57.311851 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:34:09.412367 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:34:09.412334 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:34:53.606460 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:34:53.606407 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:35:01.910194 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:35:01.910107 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:35:26.319160 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:35:26.319112 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:35:36.901209 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:35:36.901171 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:36:00.114347 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:00.114315 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-518a69/test-model-518a69-kserve-585d8745d9-s95rz"] Jul 07 22:36:00.114820 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:00.114684 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7586a2e1-cc6e-4262-91ba-bb7ec0dd571a" containerName="cleanup" Jul 07 22:36:00.114820 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:00.114696 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7586a2e1-cc6e-4262-91ba-bb7ec0dd571a" containerName="cleanup" Jul 07 22:36:00.114820 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:00.114753 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7586a2e1-cc6e-4262-91ba-bb7ec0dd571a" containerName="cleanup" Jul 07 22:36:00.117696 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:00.117679 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-518a69/test-model-518a69-kserve-585d8745d9-s95rz" Jul 07 22:36:00.120733 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:00.120706 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-518a69\"/\"openshift-service-ca.crt\"" Jul 07 22:36:00.120865 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:00.120752 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-518a69\"/\"default-dockercfg-f2pvn\"" Jul 07 22:36:00.120865 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:00.120783 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-518a69\"/\"kube-root-ca.crt\"" Jul 07 22:36:00.120865 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:00.120806 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-518a69\"/\"test-model-518a69-kserve-self-signed-certs\"" Jul 07 22:36:00.125732 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:00.125710 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-518a69/test-model-518a69-kserve-585d8745d9-s95rz"] Jul 07 22:36:00.240061 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:00.240020 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea5daa31-962e-44ae-adec-30bb870089be-dshm\") pod \"test-model-518a69-kserve-585d8745d9-s95rz\" (UID: \"ea5daa31-962e-44ae-adec-30bb870089be\") " pod="ai-tenant-e2e-inf-b-518a69/test-model-518a69-kserve-585d8745d9-s95rz" Jul 07 22:36:00.240061 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:00.240065 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsmh4\" (UniqueName: \"kubernetes.io/projected/ea5daa31-962e-44ae-adec-30bb870089be-kube-api-access-qsmh4\") pod \"test-model-518a69-kserve-585d8745d9-s95rz\" (UID: \"ea5daa31-962e-44ae-adec-30bb870089be\") " pod="ai-tenant-e2e-inf-b-518a69/test-model-518a69-kserve-585d8745d9-s95rz" Jul 07 22:36:00.240312 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:00.240145 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea5daa31-962e-44ae-adec-30bb870089be-tmp-dir\") pod \"test-model-518a69-kserve-585d8745d9-s95rz\" (UID: \"ea5daa31-962e-44ae-adec-30bb870089be\") " pod="ai-tenant-e2e-inf-b-518a69/test-model-518a69-kserve-585d8745d9-s95rz" Jul 07 22:36:00.240312 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:00.240209 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea5daa31-962e-44ae-adec-30bb870089be-tls-certs\") pod \"test-model-518a69-kserve-585d8745d9-s95rz\" (UID: \"ea5daa31-962e-44ae-adec-30bb870089be\") " pod="ai-tenant-e2e-inf-b-518a69/test-model-518a69-kserve-585d8745d9-s95rz" Jul 07 22:36:00.240312 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:00.240258 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea5daa31-962e-44ae-adec-30bb870089be-model-cache\") pod \"test-model-518a69-kserve-585d8745d9-s95rz\" (UID: \"ea5daa31-962e-44ae-adec-30bb870089be\") " pod="ai-tenant-e2e-inf-b-518a69/test-model-518a69-kserve-585d8745d9-s95rz" Jul 07 22:36:00.240312 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:00.240281 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea5daa31-962e-44ae-adec-30bb870089be-home\") pod \"test-model-518a69-kserve-585d8745d9-s95rz\" (UID: \"ea5daa31-962e-44ae-adec-30bb870089be\") " pod="ai-tenant-e2e-inf-b-518a69/test-model-518a69-kserve-585d8745d9-s95rz" Jul 07 22:36:00.341165 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:00.341128 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea5daa31-962e-44ae-adec-30bb870089be-dshm\") pod \"test-model-518a69-kserve-585d8745d9-s95rz\" (UID: \"ea5daa31-962e-44ae-adec-30bb870089be\") " pod="ai-tenant-e2e-inf-b-518a69/test-model-518a69-kserve-585d8745d9-s95rz" Jul 07 22:36:00.341367 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:00.341182 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsmh4\" (UniqueName: \"kubernetes.io/projected/ea5daa31-962e-44ae-adec-30bb870089be-kube-api-access-qsmh4\") pod \"test-model-518a69-kserve-585d8745d9-s95rz\" (UID: \"ea5daa31-962e-44ae-adec-30bb870089be\") " pod="ai-tenant-e2e-inf-b-518a69/test-model-518a69-kserve-585d8745d9-s95rz" Jul 07 22:36:00.341367 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:00.341247 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea5daa31-962e-44ae-adec-30bb870089be-tmp-dir\") pod \"test-model-518a69-kserve-585d8745d9-s95rz\" (UID: \"ea5daa31-962e-44ae-adec-30bb870089be\") " pod="ai-tenant-e2e-inf-b-518a69/test-model-518a69-kserve-585d8745d9-s95rz" Jul 07 22:36:00.341367 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:00.341276 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea5daa31-962e-44ae-adec-30bb870089be-tls-certs\") pod \"test-model-518a69-kserve-585d8745d9-s95rz\" (UID: \"ea5daa31-962e-44ae-adec-30bb870089be\") " pod="ai-tenant-e2e-inf-b-518a69/test-model-518a69-kserve-585d8745d9-s95rz" Jul 07 22:36:00.341367 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:00.341323 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea5daa31-962e-44ae-adec-30bb870089be-model-cache\") pod \"test-model-518a69-kserve-585d8745d9-s95rz\" (UID: \"ea5daa31-962e-44ae-adec-30bb870089be\") " pod="ai-tenant-e2e-inf-b-518a69/test-model-518a69-kserve-585d8745d9-s95rz" Jul 07 22:36:00.341367 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:00.341350 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea5daa31-962e-44ae-adec-30bb870089be-home\") pod \"test-model-518a69-kserve-585d8745d9-s95rz\" (UID: \"ea5daa31-962e-44ae-adec-30bb870089be\") " pod="ai-tenant-e2e-inf-b-518a69/test-model-518a69-kserve-585d8745d9-s95rz" Jul 07 22:36:00.341761 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:00.341733 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea5daa31-962e-44ae-adec-30bb870089be-tmp-dir\") pod \"test-model-518a69-kserve-585d8745d9-s95rz\" (UID: \"ea5daa31-962e-44ae-adec-30bb870089be\") " pod="ai-tenant-e2e-inf-b-518a69/test-model-518a69-kserve-585d8745d9-s95rz" Jul 07 22:36:00.341846 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:00.341767 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea5daa31-962e-44ae-adec-30bb870089be-home\") pod \"test-model-518a69-kserve-585d8745d9-s95rz\" (UID: \"ea5daa31-962e-44ae-adec-30bb870089be\") " pod="ai-tenant-e2e-inf-b-518a69/test-model-518a69-kserve-585d8745d9-s95rz" Jul 07 22:36:00.341846 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:00.341816 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea5daa31-962e-44ae-adec-30bb870089be-model-cache\") pod \"test-model-518a69-kserve-585d8745d9-s95rz\" (UID: \"ea5daa31-962e-44ae-adec-30bb870089be\") " pod="ai-tenant-e2e-inf-b-518a69/test-model-518a69-kserve-585d8745d9-s95rz" Jul 07 22:36:00.344230 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:00.344207 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea5daa31-962e-44ae-adec-30bb870089be-dshm\") pod \"test-model-518a69-kserve-585d8745d9-s95rz\" (UID: \"ea5daa31-962e-44ae-adec-30bb870089be\") " pod="ai-tenant-e2e-inf-b-518a69/test-model-518a69-kserve-585d8745d9-s95rz" Jul 07 22:36:00.344397 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:00.344380 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea5daa31-962e-44ae-adec-30bb870089be-tls-certs\") pod \"test-model-518a69-kserve-585d8745d9-s95rz\" (UID: \"ea5daa31-962e-44ae-adec-30bb870089be\") " pod="ai-tenant-e2e-inf-b-518a69/test-model-518a69-kserve-585d8745d9-s95rz" Jul 07 22:36:00.348637 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:00.348615 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsmh4\" (UniqueName: \"kubernetes.io/projected/ea5daa31-962e-44ae-adec-30bb870089be-kube-api-access-qsmh4\") pod \"test-model-518a69-kserve-585d8745d9-s95rz\" (UID: \"ea5daa31-962e-44ae-adec-30bb870089be\") " pod="ai-tenant-e2e-inf-b-518a69/test-model-518a69-kserve-585d8745d9-s95rz" Jul 07 22:36:00.427331 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:00.427239 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-518a69/test-model-518a69-kserve-585d8745d9-s95rz" Jul 07 22:36:00.556287 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:00.556264 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-518a69/test-model-518a69-kserve-585d8745d9-s95rz"] Jul 07 22:36:00.558215 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:36:00.558185 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea5daa31_962e_44ae_adec_30bb870089be.slice/crio-3f35f2ca57f762df71ae38b2f063baa68aaef2a0a7d2825e6b3d7cc6e84c3095 WatchSource:0}: Error finding container 3f35f2ca57f762df71ae38b2f063baa68aaef2a0a7d2825e6b3d7cc6e84c3095: Status 404 returned error can't find the container with id 3f35f2ca57f762df71ae38b2f063baa68aaef2a0a7d2825e6b3d7cc6e84c3095 Jul 07 22:36:01.030244 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:01.030214 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-518a69/test-model-518a69-kserve-585d8745d9-s95rz" event={"ID":"ea5daa31-962e-44ae-adec-30bb870089be","Type":"ContainerStarted","Data":"2729e4d57a6936aec009a16ab13b90d557e6eda9dceecc1aac15f908ad5e6692"} Jul 07 22:36:01.030406 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:01.030253 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-518a69/test-model-518a69-kserve-585d8745d9-s95rz" event={"ID":"ea5daa31-962e-44ae-adec-30bb870089be","Type":"ContainerStarted","Data":"3f35f2ca57f762df71ae38b2f063baa68aaef2a0a7d2825e6b3d7cc6e84c3095"} Jul 07 22:36:01.045536 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:01.045471 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-518a69/test-model-518a69-kserve-585d8745d9-s95rz" podStartSLOduration=1.045434765 podStartE2EDuration="1.045434765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 22:36:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:36:01.043820908 +0000 UTC m=+1940.871950365" watchObservedRunningTime="2026-07-07 22:36:01.045434765 +0000 UTC m=+1940.873564222" Jul 07 22:36:04.507529 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:04.507495 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:36:10.428357 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:10.428310 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-518a69/test-model-518a69-kserve-585d8745d9-s95rz" Jul 07 22:36:10.428357 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:10.428371 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-518a69/test-model-518a69-kserve-585d8745d9-s95rz" Jul 07 22:36:10.441018 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:10.440988 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-518a69/test-model-518a69-kserve-585d8745d9-s95rz" Jul 07 22:36:11.094008 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:11.093980 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-518a69/test-model-518a69-kserve-585d8745d9-s95rz" Jul 07 22:36:17.412923 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:17.412890 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:36:21.020050 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:21.019999 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-518a69/test-model-518a69-kserve-585d8745d9-s95rz"] Jul 07 22:36:33.402178 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:33.402093 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:36:36.043621 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:36.043575 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-518a69/test-model-518a69-kserve-585d8745d9-s95rz" podUID="ea5daa31-962e-44ae-adec-30bb870089be" containerName="main" containerID="cri-o://2729e4d57a6936aec009a16ab13b90d557e6eda9dceecc1aac15f908ad5e6692" gracePeriod=45 Jul 07 22:36:36.208362 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:36.208329 2564 generic.go:356] "Generic (PLEG): container finished" podID="ea5daa31-962e-44ae-adec-30bb870089be" containerID="2729e4d57a6936aec009a16ab13b90d557e6eda9dceecc1aac15f908ad5e6692" exitCode=0 Jul 07 22:36:36.208528 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:36.208393 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-518a69/test-model-518a69-kserve-585d8745d9-s95rz" event={"ID":"ea5daa31-962e-44ae-adec-30bb870089be","Type":"ContainerDied","Data":"2729e4d57a6936aec009a16ab13b90d557e6eda9dceecc1aac15f908ad5e6692"} Jul 07 22:36:36.299201 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:36.299144 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-518a69/test-model-518a69-kserve-585d8745d9-s95rz" Jul 07 22:36:36.360136 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:36.360109 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea5daa31-962e-44ae-adec-30bb870089be-home\") pod \"ea5daa31-962e-44ae-adec-30bb870089be\" (UID: \"ea5daa31-962e-44ae-adec-30bb870089be\") " Jul 07 22:36:36.360312 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:36.360150 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea5daa31-962e-44ae-adec-30bb870089be-tls-certs\") pod \"ea5daa31-962e-44ae-adec-30bb870089be\" (UID: \"ea5daa31-962e-44ae-adec-30bb870089be\") " Jul 07 22:36:36.360312 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:36.360188 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea5daa31-962e-44ae-adec-30bb870089be-tmp-dir\") pod \"ea5daa31-962e-44ae-adec-30bb870089be\" (UID: \"ea5daa31-962e-44ae-adec-30bb870089be\") " Jul 07 22:36:36.360312 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:36.360232 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea5daa31-962e-44ae-adec-30bb870089be-dshm\") pod \"ea5daa31-962e-44ae-adec-30bb870089be\" (UID: \"ea5daa31-962e-44ae-adec-30bb870089be\") " Jul 07 22:36:36.360312 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:36.360261 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea5daa31-962e-44ae-adec-30bb870089be-model-cache\") pod \"ea5daa31-962e-44ae-adec-30bb870089be\" (UID: \"ea5daa31-962e-44ae-adec-30bb870089be\") " Jul 07 22:36:36.360598 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:36.360321 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qsmh4\" (UniqueName: \"kubernetes.io/projected/ea5daa31-962e-44ae-adec-30bb870089be-kube-api-access-qsmh4\") pod \"ea5daa31-962e-44ae-adec-30bb870089be\" (UID: \"ea5daa31-962e-44ae-adec-30bb870089be\") " Jul 07 22:36:36.360598 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:36.360391 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea5daa31-962e-44ae-adec-30bb870089be-home" (OuterVolumeSpecName: "home") pod "ea5daa31-962e-44ae-adec-30bb870089be" (UID: "ea5daa31-962e-44ae-adec-30bb870089be"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 22:36:36.360598 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:36.360496 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea5daa31-962e-44ae-adec-30bb870089be-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ea5daa31-962e-44ae-adec-30bb870089be" (UID: "ea5daa31-962e-44ae-adec-30bb870089be"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 22:36:36.360598 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:36.360581 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea5daa31-962e-44ae-adec-30bb870089be-model-cache" (OuterVolumeSpecName: "model-cache") pod "ea5daa31-962e-44ae-adec-30bb870089be" (UID: "ea5daa31-962e-44ae-adec-30bb870089be"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 22:36:36.360746 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:36.360648 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea5daa31-962e-44ae-adec-30bb870089be-home\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 07 22:36:36.360746 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:36.360685 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea5daa31-962e-44ae-adec-30bb870089be-tmp-dir\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 07 22:36:36.360746 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:36.360695 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea5daa31-962e-44ae-adec-30bb870089be-model-cache\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 07 22:36:36.362639 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:36.362614 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea5daa31-962e-44ae-adec-30bb870089be-kube-api-access-qsmh4" (OuterVolumeSpecName: "kube-api-access-qsmh4") pod "ea5daa31-962e-44ae-adec-30bb870089be" (UID: "ea5daa31-962e-44ae-adec-30bb870089be"). InnerVolumeSpecName "kube-api-access-qsmh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 22:36:36.362896 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:36.362870 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea5daa31-962e-44ae-adec-30bb870089be-dshm" (OuterVolumeSpecName: "dshm") pod "ea5daa31-962e-44ae-adec-30bb870089be" (UID: "ea5daa31-962e-44ae-adec-30bb870089be"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 22:36:36.362896 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:36.362876 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea5daa31-962e-44ae-adec-30bb870089be-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ea5daa31-962e-44ae-adec-30bb870089be" (UID: "ea5daa31-962e-44ae-adec-30bb870089be"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 22:36:36.461660 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:36.461625 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsmh4\" (UniqueName: \"kubernetes.io/projected/ea5daa31-962e-44ae-adec-30bb870089be-kube-api-access-qsmh4\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 07 22:36:36.461660 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:36.461651 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea5daa31-962e-44ae-adec-30bb870089be-tls-certs\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 07 22:36:36.461660 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:36.461660 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea5daa31-962e-44ae-adec-30bb870089be-dshm\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 07 22:36:37.214539 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:37.214509 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-518a69/test-model-518a69-kserve-585d8745d9-s95rz" event={"ID":"ea5daa31-962e-44ae-adec-30bb870089be","Type":"ContainerDied","Data":"3f35f2ca57f762df71ae38b2f063baa68aaef2a0a7d2825e6b3d7cc6e84c3095"} Jul 07 22:36:37.214960 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:37.214543 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-518a69/test-model-518a69-kserve-585d8745d9-s95rz" Jul 07 22:36:37.214960 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:37.214557 2564 scope.go:117] "RemoveContainer" containerID="2729e4d57a6936aec009a16ab13b90d557e6eda9dceecc1aac15f908ad5e6692" Jul 07 22:36:37.229852 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:37.229829 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-518a69/test-model-518a69-kserve-585d8745d9-s95rz"] Jul 07 22:36:37.233201 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:37.233180 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-518a69/test-model-518a69-kserve-585d8745d9-s95rz"] Jul 07 22:36:38.707647 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:36:38.707616 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea5daa31-962e-44ae-adec-30bb870089be" path="/var/lib/kubelet/pods/ea5daa31-962e-44ae-adec-30bb870089be/volumes" Jul 07 22:37:07.705460 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:37:07.705408 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:37:22.105652 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:37:22.105615 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:37:38.006887 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:37:38.006849 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:37:52.021661 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:37:52.021620 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:38:03.209698 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:38:03.209617 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:38:15.909770 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:38:15.909739 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-41e952/auth-test-model-41e952-kserve-f74c49579-vhkwx"] Jul 07 22:38:15.910144 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:38:15.910052 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea5daa31-962e-44ae-adec-30bb870089be" containerName="main" Jul 07 22:38:15.910144 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:38:15.910063 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea5daa31-962e-44ae-adec-30bb870089be" containerName="main" Jul 07 22:38:15.910144 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:38:15.910133 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea5daa31-962e-44ae-adec-30bb870089be" containerName="main" Jul 07 22:38:15.913098 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:38:15.913077 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-41e952/auth-test-model-41e952-kserve-f74c49579-vhkwx" Jul 07 22:38:15.916021 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:38:15.915996 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-41e952\"/\"openshift-service-ca.crt\"" Jul 07 22:38:15.916162 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:38:15.916076 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-41e952\"/\"auth-test-model-41e952-kserve-self-signed-certs\"" Jul 07 22:38:15.916162 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:38:15.916079 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-41e952\"/\"kube-root-ca.crt\"" Jul 07 22:38:15.916162 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:38:15.916079 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-41e952\"/\"default-dockercfg-ntw89\"" Jul 07 22:38:15.922570 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:38:15.922544 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-41e952/auth-test-model-41e952-kserve-f74c49579-vhkwx"] Jul 07 22:38:16.006948 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:38:16.006919 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f994f535-06f7-4c28-9499-b72ae7b5ca08-model-cache\") pod \"auth-test-model-41e952-kserve-f74c49579-vhkwx\" (UID: \"f994f535-06f7-4c28-9499-b72ae7b5ca08\") " pod="ai-tenant-e2e-shared-a-41e952/auth-test-model-41e952-kserve-f74c49579-vhkwx" Jul 07 22:38:16.007093 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:38:16.006979 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f994f535-06f7-4c28-9499-b72ae7b5ca08-home\") pod \"auth-test-model-41e952-kserve-f74c49579-vhkwx\" (UID: \"f994f535-06f7-4c28-9499-b72ae7b5ca08\") " pod="ai-tenant-e2e-shared-a-41e952/auth-test-model-41e952-kserve-f74c49579-vhkwx" Jul 07 22:38:16.007093 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:38:16.007008 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f994f535-06f7-4c28-9499-b72ae7b5ca08-tmp-dir\") pod \"auth-test-model-41e952-kserve-f74c49579-vhkwx\" (UID: \"f994f535-06f7-4c28-9499-b72ae7b5ca08\") " pod="ai-tenant-e2e-shared-a-41e952/auth-test-model-41e952-kserve-f74c49579-vhkwx" Jul 07 22:38:16.007093 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:38:16.007029 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f994f535-06f7-4c28-9499-b72ae7b5ca08-dshm\") pod \"auth-test-model-41e952-kserve-f74c49579-vhkwx\" (UID: \"f994f535-06f7-4c28-9499-b72ae7b5ca08\") " pod="ai-tenant-e2e-shared-a-41e952/auth-test-model-41e952-kserve-f74c49579-vhkwx" Jul 07 22:38:16.007093 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:38:16.007048 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7kdh\" (UniqueName: \"kubernetes.io/projected/f994f535-06f7-4c28-9499-b72ae7b5ca08-kube-api-access-f7kdh\") pod \"auth-test-model-41e952-kserve-f74c49579-vhkwx\" (UID: \"f994f535-06f7-4c28-9499-b72ae7b5ca08\") " pod="ai-tenant-e2e-shared-a-41e952/auth-test-model-41e952-kserve-f74c49579-vhkwx" Jul 07 22:38:16.007093 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:38:16.007071 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f994f535-06f7-4c28-9499-b72ae7b5ca08-tls-certs\") pod \"auth-test-model-41e952-kserve-f74c49579-vhkwx\" (UID: \"f994f535-06f7-4c28-9499-b72ae7b5ca08\") " pod="ai-tenant-e2e-shared-a-41e952/auth-test-model-41e952-kserve-f74c49579-vhkwx" Jul 07 22:38:16.108225 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:38:16.108194 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f994f535-06f7-4c28-9499-b72ae7b5ca08-home\") pod \"auth-test-model-41e952-kserve-f74c49579-vhkwx\" (UID: \"f994f535-06f7-4c28-9499-b72ae7b5ca08\") " pod="ai-tenant-e2e-shared-a-41e952/auth-test-model-41e952-kserve-f74c49579-vhkwx" Jul 07 22:38:16.108225 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:38:16.108228 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f994f535-06f7-4c28-9499-b72ae7b5ca08-tmp-dir\") pod \"auth-test-model-41e952-kserve-f74c49579-vhkwx\" (UID: \"f994f535-06f7-4c28-9499-b72ae7b5ca08\") " pod="ai-tenant-e2e-shared-a-41e952/auth-test-model-41e952-kserve-f74c49579-vhkwx" Jul 07 22:38:16.108412 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:38:16.108258 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f994f535-06f7-4c28-9499-b72ae7b5ca08-dshm\") pod \"auth-test-model-41e952-kserve-f74c49579-vhkwx\" (UID: \"f994f535-06f7-4c28-9499-b72ae7b5ca08\") " pod="ai-tenant-e2e-shared-a-41e952/auth-test-model-41e952-kserve-f74c49579-vhkwx" Jul 07 22:38:16.108412 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:38:16.108290 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7kdh\" (UniqueName: \"kubernetes.io/projected/f994f535-06f7-4c28-9499-b72ae7b5ca08-kube-api-access-f7kdh\") pod \"auth-test-model-41e952-kserve-f74c49579-vhkwx\" (UID: \"f994f535-06f7-4c28-9499-b72ae7b5ca08\") " pod="ai-tenant-e2e-shared-a-41e952/auth-test-model-41e952-kserve-f74c49579-vhkwx" Jul 07 22:38:16.108412 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:38:16.108318 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f994f535-06f7-4c28-9499-b72ae7b5ca08-tls-certs\") pod \"auth-test-model-41e952-kserve-f74c49579-vhkwx\" (UID: \"f994f535-06f7-4c28-9499-b72ae7b5ca08\") " pod="ai-tenant-e2e-shared-a-41e952/auth-test-model-41e952-kserve-f74c49579-vhkwx" Jul 07 22:38:16.108412 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:38:16.108356 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f994f535-06f7-4c28-9499-b72ae7b5ca08-model-cache\") pod \"auth-test-model-41e952-kserve-f74c49579-vhkwx\" (UID: \"f994f535-06f7-4c28-9499-b72ae7b5ca08\") " pod="ai-tenant-e2e-shared-a-41e952/auth-test-model-41e952-kserve-f74c49579-vhkwx" Jul 07 22:38:16.108686 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:38:16.108654 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f994f535-06f7-4c28-9499-b72ae7b5ca08-home\") pod \"auth-test-model-41e952-kserve-f74c49579-vhkwx\" (UID: \"f994f535-06f7-4c28-9499-b72ae7b5ca08\") " pod="ai-tenant-e2e-shared-a-41e952/auth-test-model-41e952-kserve-f74c49579-vhkwx" Jul 07 22:38:16.108732 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:38:16.108699 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f994f535-06f7-4c28-9499-b72ae7b5ca08-tmp-dir\") pod \"auth-test-model-41e952-kserve-f74c49579-vhkwx\" (UID: \"f994f535-06f7-4c28-9499-b72ae7b5ca08\") " pod="ai-tenant-e2e-shared-a-41e952/auth-test-model-41e952-kserve-f74c49579-vhkwx" Jul 07 22:38:16.108732 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:38:16.108720 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f994f535-06f7-4c28-9499-b72ae7b5ca08-model-cache\") pod \"auth-test-model-41e952-kserve-f74c49579-vhkwx\" (UID: \"f994f535-06f7-4c28-9499-b72ae7b5ca08\") " pod="ai-tenant-e2e-shared-a-41e952/auth-test-model-41e952-kserve-f74c49579-vhkwx" Jul 07 22:38:16.110838 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:38:16.110817 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f994f535-06f7-4c28-9499-b72ae7b5ca08-dshm\") pod \"auth-test-model-41e952-kserve-f74c49579-vhkwx\" (UID: \"f994f535-06f7-4c28-9499-b72ae7b5ca08\") " pod="ai-tenant-e2e-shared-a-41e952/auth-test-model-41e952-kserve-f74c49579-vhkwx" Jul 07 22:38:16.110929 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:38:16.110918 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f994f535-06f7-4c28-9499-b72ae7b5ca08-tls-certs\") pod \"auth-test-model-41e952-kserve-f74c49579-vhkwx\" (UID: \"f994f535-06f7-4c28-9499-b72ae7b5ca08\") " pod="ai-tenant-e2e-shared-a-41e952/auth-test-model-41e952-kserve-f74c49579-vhkwx" Jul 07 22:38:16.115943 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:38:16.115918 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7kdh\" (UniqueName: \"kubernetes.io/projected/f994f535-06f7-4c28-9499-b72ae7b5ca08-kube-api-access-f7kdh\") pod \"auth-test-model-41e952-kserve-f74c49579-vhkwx\" (UID: \"f994f535-06f7-4c28-9499-b72ae7b5ca08\") " pod="ai-tenant-e2e-shared-a-41e952/auth-test-model-41e952-kserve-f74c49579-vhkwx" Jul 07 22:38:16.224305 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:38:16.224218 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-41e952/auth-test-model-41e952-kserve-f74c49579-vhkwx" Jul 07 22:38:16.356157 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:38:16.356133 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-41e952/auth-test-model-41e952-kserve-f74c49579-vhkwx"] Jul 07 22:38:16.358461 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:38:16.358418 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf994f535_06f7_4c28_9499_b72ae7b5ca08.slice/crio-2316b8f620b213bd0565d41d924e7d3da5c1fb972410c90ad90e712d0ed78366 WatchSource:0}: Error finding container 2316b8f620b213bd0565d41d924e7d3da5c1fb972410c90ad90e712d0ed78366: Status 404 returned error can't find the container with id 2316b8f620b213bd0565d41d924e7d3da5c1fb972410c90ad90e712d0ed78366 Jul 07 22:38:16.699733 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:38:16.699696 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-41e952/auth-test-model-41e952-kserve-f74c49579-vhkwx" event={"ID":"f994f535-06f7-4c28-9499-b72ae7b5ca08","Type":"ContainerStarted","Data":"57435272bf94c1b7b64723007d3f771160299b4a172153a1d6a7486f3843fc9c"} Jul 07 22:38:16.699898 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:38:16.699752 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-41e952/auth-test-model-41e952-kserve-f74c49579-vhkwx" event={"ID":"f994f535-06f7-4c28-9499-b72ae7b5ca08","Type":"ContainerStarted","Data":"2316b8f620b213bd0565d41d924e7d3da5c1fb972410c90ad90e712d0ed78366"} Jul 07 22:38:16.717793 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:38:16.717744 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-41e952/auth-test-model-41e952-kserve-f74c49579-vhkwx" podStartSLOduration=1.7177306479999999 podStartE2EDuration="1.717730648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 22:38:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:38:16.71508109 +0000 UTC m=+2076.543210549" watchObservedRunningTime="2026-07-07 22:38:16.717730648 +0000 UTC m=+2076.545860105" Jul 07 22:38:19.506140 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:38:19.506107 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:38:26.224535 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:38:26.224501 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-41e952/auth-test-model-41e952-kserve-f74c49579-vhkwx" Jul 07 22:38:26.225069 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:38:26.224558 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-41e952/auth-test-model-41e952-kserve-f74c49579-vhkwx" Jul 07 22:38:26.237039 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:38:26.237014 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-41e952/auth-test-model-41e952-kserve-f74c49579-vhkwx" Jul 07 22:38:26.756174 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:38:26.756146 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-41e952/auth-test-model-41e952-kserve-f74c49579-vhkwx" Jul 07 22:38:40.759810 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:38:40.759784 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4h8zm_4b4234f5-6cda-45ea-b559-f09bcad77232/ovn-acl-logging/0.log" Jul 07 22:38:40.765522 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:38:40.765502 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4h8zm_4b4234f5-6cda-45ea-b559-f09bcad77232/ovn-acl-logging/0.log" Jul 07 22:38:54.211268 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:38:54.211231 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:39:11.008401 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:39:11.008366 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:39:28.910139 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:39:28.910084 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:39:42.913862 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:39:42.913782 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:40:00.209762 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:40:00.209728 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:40:17.924617 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:40:17.924588 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:40:33.601798 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:40:33.601758 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:40:44.427190 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:40:44.427157 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-41e952/auth-test-model-41e952-kserve-f74c49579-vhkwx"] Jul 07 22:40:48.500458 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:40:48.500416 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:40:59.447539 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:40:59.447488 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-41e952/auth-test-model-41e952-kserve-f74c49579-vhkwx" podUID="f994f535-06f7-4c28-9499-b72ae7b5ca08" containerName="main" containerID="cri-o://57435272bf94c1b7b64723007d3f771160299b4a172153a1d6a7486f3843fc9c" gracePeriod=45 Jul 07 22:40:59.693142 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:40:59.693113 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-41e952/auth-test-model-41e952-kserve-f74c49579-vhkwx" Jul 07 22:40:59.797323 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:40:59.797297 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f994f535-06f7-4c28-9499-b72ae7b5ca08-home\") pod \"f994f535-06f7-4c28-9499-b72ae7b5ca08\" (UID: \"f994f535-06f7-4c28-9499-b72ae7b5ca08\") " Jul 07 22:40:59.797500 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:40:59.797358 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f994f535-06f7-4c28-9499-b72ae7b5ca08-model-cache\") pod \"f994f535-06f7-4c28-9499-b72ae7b5ca08\" (UID: \"f994f535-06f7-4c28-9499-b72ae7b5ca08\") " Jul 07 22:40:59.797500 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:40:59.797377 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f994f535-06f7-4c28-9499-b72ae7b5ca08-tls-certs\") pod \"f994f535-06f7-4c28-9499-b72ae7b5ca08\" (UID: \"f994f535-06f7-4c28-9499-b72ae7b5ca08\") " Jul 07 22:40:59.797500 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:40:59.797416 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f994f535-06f7-4c28-9499-b72ae7b5ca08-tmp-dir\") pod \"f994f535-06f7-4c28-9499-b72ae7b5ca08\" (UID: \"f994f535-06f7-4c28-9499-b72ae7b5ca08\") " Jul 07 22:40:59.797500 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:40:59.797458 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f994f535-06f7-4c28-9499-b72ae7b5ca08-dshm\") pod \"f994f535-06f7-4c28-9499-b72ae7b5ca08\" (UID: \"f994f535-06f7-4c28-9499-b72ae7b5ca08\") " Jul 07 22:40:59.797500 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:40:59.797488 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f7kdh\" (UniqueName: \"kubernetes.io/projected/f994f535-06f7-4c28-9499-b72ae7b5ca08-kube-api-access-f7kdh\") pod \"f994f535-06f7-4c28-9499-b72ae7b5ca08\" (UID: \"f994f535-06f7-4c28-9499-b72ae7b5ca08\") " Jul 07 22:40:59.797765 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:40:59.797629 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f994f535-06f7-4c28-9499-b72ae7b5ca08-model-cache" (OuterVolumeSpecName: "model-cache") pod "f994f535-06f7-4c28-9499-b72ae7b5ca08" (UID: "f994f535-06f7-4c28-9499-b72ae7b5ca08"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 22:40:59.797765 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:40:59.797645 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f994f535-06f7-4c28-9499-b72ae7b5ca08-home" (OuterVolumeSpecName: "home") pod "f994f535-06f7-4c28-9499-b72ae7b5ca08" (UID: "f994f535-06f7-4c28-9499-b72ae7b5ca08"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 22:40:59.797765 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:40:59.797678 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f994f535-06f7-4c28-9499-b72ae7b5ca08-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f994f535-06f7-4c28-9499-b72ae7b5ca08" (UID: "f994f535-06f7-4c28-9499-b72ae7b5ca08"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 22:40:59.797916 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:40:59.797797 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f994f535-06f7-4c28-9499-b72ae7b5ca08-model-cache\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 07 22:40:59.797916 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:40:59.797817 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f994f535-06f7-4c28-9499-b72ae7b5ca08-tmp-dir\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 07 22:40:59.797916 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:40:59.797839 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f994f535-06f7-4c28-9499-b72ae7b5ca08-home\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 07 22:40:59.799969 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:40:59.799914 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f994f535-06f7-4c28-9499-b72ae7b5ca08-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f994f535-06f7-4c28-9499-b72ae7b5ca08" (UID: "f994f535-06f7-4c28-9499-b72ae7b5ca08"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 22:40:59.799969 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:40:59.799928 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f994f535-06f7-4c28-9499-b72ae7b5ca08-dshm" (OuterVolumeSpecName: "dshm") pod "f994f535-06f7-4c28-9499-b72ae7b5ca08" (UID: "f994f535-06f7-4c28-9499-b72ae7b5ca08"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 22:40:59.800110 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:40:59.799999 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f994f535-06f7-4c28-9499-b72ae7b5ca08-kube-api-access-f7kdh" (OuterVolumeSpecName: "kube-api-access-f7kdh") pod "f994f535-06f7-4c28-9499-b72ae7b5ca08" (UID: "f994f535-06f7-4c28-9499-b72ae7b5ca08"). InnerVolumeSpecName "kube-api-access-f7kdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 22:40:59.898457 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:40:59.898410 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f994f535-06f7-4c28-9499-b72ae7b5ca08-tls-certs\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 07 22:40:59.898457 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:40:59.898434 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f994f535-06f7-4c28-9499-b72ae7b5ca08-dshm\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 07 22:40:59.898457 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:40:59.898457 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7kdh\" (UniqueName: \"kubernetes.io/projected/f994f535-06f7-4c28-9499-b72ae7b5ca08-kube-api-access-f7kdh\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 07 22:41:00.502919 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:41:00.502879 2564 generic.go:356] "Generic (PLEG): container finished" podID="f994f535-06f7-4c28-9499-b72ae7b5ca08" containerID="57435272bf94c1b7b64723007d3f771160299b4a172153a1d6a7486f3843fc9c" exitCode=0 Jul 07 22:41:00.502919 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:41:00.502920 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-41e952/auth-test-model-41e952-kserve-f74c49579-vhkwx" Jul 07 22:41:00.503370 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:41:00.502922 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-41e952/auth-test-model-41e952-kserve-f74c49579-vhkwx" event={"ID":"f994f535-06f7-4c28-9499-b72ae7b5ca08","Type":"ContainerDied","Data":"57435272bf94c1b7b64723007d3f771160299b4a172153a1d6a7486f3843fc9c"} Jul 07 22:41:00.503370 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:41:00.503048 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-41e952/auth-test-model-41e952-kserve-f74c49579-vhkwx" event={"ID":"f994f535-06f7-4c28-9499-b72ae7b5ca08","Type":"ContainerDied","Data":"2316b8f620b213bd0565d41d924e7d3da5c1fb972410c90ad90e712d0ed78366"} Jul 07 22:41:00.503370 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:41:00.503065 2564 scope.go:117] "RemoveContainer" containerID="57435272bf94c1b7b64723007d3f771160299b4a172153a1d6a7486f3843fc9c" Jul 07 22:41:00.512663 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:41:00.512647 2564 scope.go:117] "RemoveContainer" containerID="57435272bf94c1b7b64723007d3f771160299b4a172153a1d6a7486f3843fc9c" Jul 07 22:41:00.512941 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:41:00.512921 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57435272bf94c1b7b64723007d3f771160299b4a172153a1d6a7486f3843fc9c\": container with ID starting with 57435272bf94c1b7b64723007d3f771160299b4a172153a1d6a7486f3843fc9c not found: ID does not exist" containerID="57435272bf94c1b7b64723007d3f771160299b4a172153a1d6a7486f3843fc9c" Jul 07 22:41:00.513017 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:41:00.512950 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57435272bf94c1b7b64723007d3f771160299b4a172153a1d6a7486f3843fc9c"} err="failed to get container status \"57435272bf94c1b7b64723007d3f771160299b4a172153a1d6a7486f3843fc9c\": rpc error: code = NotFound desc = could not find container \"57435272bf94c1b7b64723007d3f771160299b4a172153a1d6a7486f3843fc9c\": container with ID starting with 57435272bf94c1b7b64723007d3f771160299b4a172153a1d6a7486f3843fc9c not found: ID does not exist" Jul 07 22:41:00.528467 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:41:00.525579 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-41e952/auth-test-model-41e952-kserve-f74c49579-vhkwx"] Jul 07 22:41:00.532885 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:41:00.532860 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-41e952/auth-test-model-41e952-kserve-f74c49579-vhkwx"] Jul 07 22:41:00.707778 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:41:00.707707 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f994f535-06f7-4c28-9499-b72ae7b5ca08" path="/var/lib/kubelet/pods/f994f535-06f7-4c28-9499-b72ae7b5ca08/volumes" Jul 07 22:41:02.210591 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:41:02.210560 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:41:33.709369 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:41:33.709330 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-a4bfe6/sub-test-model-a4bfe6-kserve-845445f84c-vl4xf"] Jul 07 22:41:33.709849 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:41:33.709661 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f994f535-06f7-4c28-9499-b72ae7b5ca08" containerName="main" Jul 07 22:41:33.709849 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:41:33.709672 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f994f535-06f7-4c28-9499-b72ae7b5ca08" containerName="main" Jul 07 22:41:33.709849 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:41:33.709732 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f994f535-06f7-4c28-9499-b72ae7b5ca08" containerName="main" Jul 07 22:41:33.712794 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:41:33.712778 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a4bfe6/sub-test-model-a4bfe6-kserve-845445f84c-vl4xf" Jul 07 22:41:33.715703 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:41:33.715678 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-a4bfe6\"/\"openshift-service-ca.crt\"" Jul 07 22:41:33.715703 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:41:33.715682 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-a4bfe6\"/\"sub-test-model-a4bfe6-kserve-self-signed-certs\"" Jul 07 22:41:33.715900 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:41:33.715689 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-a4bfe6\"/\"default-dockercfg-26nnx\"" Jul 07 22:41:33.715900 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:41:33.715729 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-a4bfe6\"/\"kube-root-ca.crt\"" Jul 07 22:41:33.721543 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:41:33.721524 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-a4bfe6/sub-test-model-a4bfe6-kserve-845445f84c-vl4xf"] Jul 07 22:41:33.879244 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:41:33.879210 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22acb4e8-9c23-4d3e-9ce0-ef3364404e20-dshm\") pod \"sub-test-model-a4bfe6-kserve-845445f84c-vl4xf\" (UID: \"22acb4e8-9c23-4d3e-9ce0-ef3364404e20\") " pod="ai-tenant-e2e-shared-b-a4bfe6/sub-test-model-a4bfe6-kserve-845445f84c-vl4xf" Jul 07 22:41:33.879478 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:41:33.879263 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22acb4e8-9c23-4d3e-9ce0-ef3364404e20-model-cache\") pod \"sub-test-model-a4bfe6-kserve-845445f84c-vl4xf\" (UID: \"22acb4e8-9c23-4d3e-9ce0-ef3364404e20\") " pod="ai-tenant-e2e-shared-b-a4bfe6/sub-test-model-a4bfe6-kserve-845445f84c-vl4xf" Jul 07 22:41:33.879478 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:41:33.879287 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb6qw\" (UniqueName: \"kubernetes.io/projected/22acb4e8-9c23-4d3e-9ce0-ef3364404e20-kube-api-access-gb6qw\") pod \"sub-test-model-a4bfe6-kserve-845445f84c-vl4xf\" (UID: \"22acb4e8-9c23-4d3e-9ce0-ef3364404e20\") " pod="ai-tenant-e2e-shared-b-a4bfe6/sub-test-model-a4bfe6-kserve-845445f84c-vl4xf" Jul 07 22:41:33.879478 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:41:33.879374 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22acb4e8-9c23-4d3e-9ce0-ef3364404e20-tls-certs\") pod \"sub-test-model-a4bfe6-kserve-845445f84c-vl4xf\" (UID: \"22acb4e8-9c23-4d3e-9ce0-ef3364404e20\") " pod="ai-tenant-e2e-shared-b-a4bfe6/sub-test-model-a4bfe6-kserve-845445f84c-vl4xf" Jul 07 22:41:33.879636 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:41:33.879517 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22acb4e8-9c23-4d3e-9ce0-ef3364404e20-tmp-dir\") pod \"sub-test-model-a4bfe6-kserve-845445f84c-vl4xf\" (UID: \"22acb4e8-9c23-4d3e-9ce0-ef3364404e20\") " pod="ai-tenant-e2e-shared-b-a4bfe6/sub-test-model-a4bfe6-kserve-845445f84c-vl4xf" Jul 07 22:41:33.879636 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:41:33.879560 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22acb4e8-9c23-4d3e-9ce0-ef3364404e20-home\") pod \"sub-test-model-a4bfe6-kserve-845445f84c-vl4xf\" (UID: \"22acb4e8-9c23-4d3e-9ce0-ef3364404e20\") " pod="ai-tenant-e2e-shared-b-a4bfe6/sub-test-model-a4bfe6-kserve-845445f84c-vl4xf" Jul 07 22:41:33.980521 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:41:33.980414 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22acb4e8-9c23-4d3e-9ce0-ef3364404e20-tmp-dir\") pod \"sub-test-model-a4bfe6-kserve-845445f84c-vl4xf\" (UID: \"22acb4e8-9c23-4d3e-9ce0-ef3364404e20\") " pod="ai-tenant-e2e-shared-b-a4bfe6/sub-test-model-a4bfe6-kserve-845445f84c-vl4xf" Jul 07 22:41:33.980521 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:41:33.980493 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22acb4e8-9c23-4d3e-9ce0-ef3364404e20-home\") pod \"sub-test-model-a4bfe6-kserve-845445f84c-vl4xf\" (UID: \"22acb4e8-9c23-4d3e-9ce0-ef3364404e20\") " pod="ai-tenant-e2e-shared-b-a4bfe6/sub-test-model-a4bfe6-kserve-845445f84c-vl4xf" Jul 07 22:41:33.980521 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:41:33.980516 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22acb4e8-9c23-4d3e-9ce0-ef3364404e20-dshm\") pod \"sub-test-model-a4bfe6-kserve-845445f84c-vl4xf\" (UID: \"22acb4e8-9c23-4d3e-9ce0-ef3364404e20\") " pod="ai-tenant-e2e-shared-b-a4bfe6/sub-test-model-a4bfe6-kserve-845445f84c-vl4xf" Jul 07 22:41:33.980798 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:41:33.980556 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22acb4e8-9c23-4d3e-9ce0-ef3364404e20-model-cache\") pod \"sub-test-model-a4bfe6-kserve-845445f84c-vl4xf\" (UID: \"22acb4e8-9c23-4d3e-9ce0-ef3364404e20\") " pod="ai-tenant-e2e-shared-b-a4bfe6/sub-test-model-a4bfe6-kserve-845445f84c-vl4xf" Jul 07 22:41:33.980798 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:41:33.980575 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb6qw\" (UniqueName: \"kubernetes.io/projected/22acb4e8-9c23-4d3e-9ce0-ef3364404e20-kube-api-access-gb6qw\") pod \"sub-test-model-a4bfe6-kserve-845445f84c-vl4xf\" (UID: \"22acb4e8-9c23-4d3e-9ce0-ef3364404e20\") " pod="ai-tenant-e2e-shared-b-a4bfe6/sub-test-model-a4bfe6-kserve-845445f84c-vl4xf" Jul 07 22:41:33.980798 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:41:33.980604 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22acb4e8-9c23-4d3e-9ce0-ef3364404e20-tls-certs\") pod \"sub-test-model-a4bfe6-kserve-845445f84c-vl4xf\" (UID: \"22acb4e8-9c23-4d3e-9ce0-ef3364404e20\") " pod="ai-tenant-e2e-shared-b-a4bfe6/sub-test-model-a4bfe6-kserve-845445f84c-vl4xf" Jul 07 22:41:33.980951 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:41:33.980905 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22acb4e8-9c23-4d3e-9ce0-ef3364404e20-model-cache\") pod \"sub-test-model-a4bfe6-kserve-845445f84c-vl4xf\" (UID: \"22acb4e8-9c23-4d3e-9ce0-ef3364404e20\") " pod="ai-tenant-e2e-shared-b-a4bfe6/sub-test-model-a4bfe6-kserve-845445f84c-vl4xf" Jul 07 22:41:33.980951 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:41:33.980892 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22acb4e8-9c23-4d3e-9ce0-ef3364404e20-tmp-dir\") pod \"sub-test-model-a4bfe6-kserve-845445f84c-vl4xf\" (UID: \"22acb4e8-9c23-4d3e-9ce0-ef3364404e20\") " pod="ai-tenant-e2e-shared-b-a4bfe6/sub-test-model-a4bfe6-kserve-845445f84c-vl4xf" Jul 07 22:41:33.980951 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:41:33.980938 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22acb4e8-9c23-4d3e-9ce0-ef3364404e20-home\") pod \"sub-test-model-a4bfe6-kserve-845445f84c-vl4xf\" (UID: \"22acb4e8-9c23-4d3e-9ce0-ef3364404e20\") " pod="ai-tenant-e2e-shared-b-a4bfe6/sub-test-model-a4bfe6-kserve-845445f84c-vl4xf" Jul 07 22:41:33.982876 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:41:33.982856 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22acb4e8-9c23-4d3e-9ce0-ef3364404e20-dshm\") pod \"sub-test-model-a4bfe6-kserve-845445f84c-vl4xf\" (UID: \"22acb4e8-9c23-4d3e-9ce0-ef3364404e20\") " pod="ai-tenant-e2e-shared-b-a4bfe6/sub-test-model-a4bfe6-kserve-845445f84c-vl4xf" Jul 07 22:41:33.983198 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:41:33.983180 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22acb4e8-9c23-4d3e-9ce0-ef3364404e20-tls-certs\") pod \"sub-test-model-a4bfe6-kserve-845445f84c-vl4xf\" (UID: \"22acb4e8-9c23-4d3e-9ce0-ef3364404e20\") " pod="ai-tenant-e2e-shared-b-a4bfe6/sub-test-model-a4bfe6-kserve-845445f84c-vl4xf" Jul 07 22:41:33.988769 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:41:33.988748 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb6qw\" (UniqueName: \"kubernetes.io/projected/22acb4e8-9c23-4d3e-9ce0-ef3364404e20-kube-api-access-gb6qw\") pod \"sub-test-model-a4bfe6-kserve-845445f84c-vl4xf\" (UID: \"22acb4e8-9c23-4d3e-9ce0-ef3364404e20\") " pod="ai-tenant-e2e-shared-b-a4bfe6/sub-test-model-a4bfe6-kserve-845445f84c-vl4xf" Jul 07 22:41:34.021848 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:41:34.021821 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a4bfe6/sub-test-model-a4bfe6-kserve-845445f84c-vl4xf" Jul 07 22:41:34.154916 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:41:34.154892 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-a4bfe6/sub-test-model-a4bfe6-kserve-845445f84c-vl4xf"] Jul 07 22:41:34.157123 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:41:34.157096 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod22acb4e8_9c23_4d3e_9ce0_ef3364404e20.slice/crio-0762280dbff295dcc1c18b6cff843f58bde84c9dc2475eeb238284f8366526e2 WatchSource:0}: Error finding container 0762280dbff295dcc1c18b6cff843f58bde84c9dc2475eeb238284f8366526e2: Status 404 returned error can't find the container with id 0762280dbff295dcc1c18b6cff843f58bde84c9dc2475eeb238284f8366526e2 Jul 07 22:41:34.672301 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:41:34.672265 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a4bfe6/sub-test-model-a4bfe6-kserve-845445f84c-vl4xf" event={"ID":"22acb4e8-9c23-4d3e-9ce0-ef3364404e20","Type":"ContainerStarted","Data":"d1d923feeb3fb435056f066535d1012065142a434f1400a60d64d65b767f1eeb"} Jul 07 22:41:34.672497 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:41:34.672309 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a4bfe6/sub-test-model-a4bfe6-kserve-845445f84c-vl4xf" event={"ID":"22acb4e8-9c23-4d3e-9ce0-ef3364404e20","Type":"ContainerStarted","Data":"0762280dbff295dcc1c18b6cff843f58bde84c9dc2475eeb238284f8366526e2"} Jul 07 22:41:34.690010 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:41:34.689951 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-a4bfe6/sub-test-model-a4bfe6-kserve-845445f84c-vl4xf" podStartSLOduration=1.689935658 podStartE2EDuration="1.689935658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 22:41:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:41:34.688420913 +0000 UTC m=+2274.516550373" watchObservedRunningTime="2026-07-07 22:41:34.689935658 +0000 UTC m=+2274.518065116" Jul 07 22:41:38.103593 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:41:38.103555 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:41:44.022681 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:41:44.022641 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-a4bfe6/sub-test-model-a4bfe6-kserve-845445f84c-vl4xf" Jul 07 22:41:44.022681 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:41:44.022687 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-a4bfe6/sub-test-model-a4bfe6-kserve-845445f84c-vl4xf" Jul 07 22:41:44.035046 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:41:44.035018 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-a4bfe6/sub-test-model-a4bfe6-kserve-845445f84c-vl4xf" Jul 07 22:41:44.742789 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:41:44.742763 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-a4bfe6/sub-test-model-a4bfe6-kserve-845445f84c-vl4xf" Jul 07 22:41:56.607389 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:41:56.607351 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:42:14.712564 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:14.712533 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:42:20.500362 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:20.500329 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-a4bfe6/sub-test-model-a4bfe6-kserve-845445f84c-vl4xf"] Jul 07 22:42:32.103295 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:32.103217 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:42:35.524130 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:35.524064 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-a4bfe6/sub-test-model-a4bfe6-kserve-845445f84c-vl4xf" podUID="22acb4e8-9c23-4d3e-9ce0-ef3364404e20" containerName="main" containerID="cri-o://d1d923feeb3fb435056f066535d1012065142a434f1400a60d64d65b767f1eeb" gracePeriod=45 Jul 07 22:42:35.559578 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:42:35.559545 2564 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod22acb4e8_9c23_4d3e_9ce0_ef3364404e20.slice/crio-d1d923feeb3fb435056f066535d1012065142a434f1400a60d64d65b767f1eeb.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod22acb4e8_9c23_4d3e_9ce0_ef3364404e20.slice/crio-conmon-d1d923feeb3fb435056f066535d1012065142a434f1400a60d64d65b767f1eeb.scope\": RecentStats: unable to find data in memory cache]" Jul 07 22:42:35.776098 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:35.776047 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a4bfe6/sub-test-model-a4bfe6-kserve-845445f84c-vl4xf" Jul 07 22:42:35.886912 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:35.886877 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22acb4e8-9c23-4d3e-9ce0-ef3364404e20-tmp-dir\") pod \"22acb4e8-9c23-4d3e-9ce0-ef3364404e20\" (UID: \"22acb4e8-9c23-4d3e-9ce0-ef3364404e20\") " Jul 07 22:42:35.886912 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:35.886919 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gb6qw\" (UniqueName: \"kubernetes.io/projected/22acb4e8-9c23-4d3e-9ce0-ef3364404e20-kube-api-access-gb6qw\") pod \"22acb4e8-9c23-4d3e-9ce0-ef3364404e20\" (UID: \"22acb4e8-9c23-4d3e-9ce0-ef3364404e20\") " Jul 07 22:42:35.887158 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:35.886953 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22acb4e8-9c23-4d3e-9ce0-ef3364404e20-dshm\") pod \"22acb4e8-9c23-4d3e-9ce0-ef3364404e20\" (UID: \"22acb4e8-9c23-4d3e-9ce0-ef3364404e20\") " Jul 07 22:42:35.887158 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:35.886968 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22acb4e8-9c23-4d3e-9ce0-ef3364404e20-home\") pod \"22acb4e8-9c23-4d3e-9ce0-ef3364404e20\" (UID: \"22acb4e8-9c23-4d3e-9ce0-ef3364404e20\") " Jul 07 22:42:35.887158 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:35.887021 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22acb4e8-9c23-4d3e-9ce0-ef3364404e20-tls-certs\") pod \"22acb4e8-9c23-4d3e-9ce0-ef3364404e20\" (UID: \"22acb4e8-9c23-4d3e-9ce0-ef3364404e20\") " Jul 07 22:42:35.887158 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:35.887042 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22acb4e8-9c23-4d3e-9ce0-ef3364404e20-model-cache\") pod \"22acb4e8-9c23-4d3e-9ce0-ef3364404e20\" (UID: \"22acb4e8-9c23-4d3e-9ce0-ef3364404e20\") " Jul 07 22:42:35.887376 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:35.887167 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22acb4e8-9c23-4d3e-9ce0-ef3364404e20-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "22acb4e8-9c23-4d3e-9ce0-ef3364404e20" (UID: "22acb4e8-9c23-4d3e-9ce0-ef3364404e20"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 22:42:35.887376 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:35.887327 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22acb4e8-9c23-4d3e-9ce0-ef3364404e20-home" (OuterVolumeSpecName: "home") pod "22acb4e8-9c23-4d3e-9ce0-ef3364404e20" (UID: "22acb4e8-9c23-4d3e-9ce0-ef3364404e20"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 22:42:35.887376 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:35.887335 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22acb4e8-9c23-4d3e-9ce0-ef3364404e20-tmp-dir\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 07 22:42:35.887376 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:35.887357 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22acb4e8-9c23-4d3e-9ce0-ef3364404e20-model-cache" (OuterVolumeSpecName: "model-cache") pod "22acb4e8-9c23-4d3e-9ce0-ef3364404e20" (UID: "22acb4e8-9c23-4d3e-9ce0-ef3364404e20"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 22:42:35.889391 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:35.889369 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22acb4e8-9c23-4d3e-9ce0-ef3364404e20-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "22acb4e8-9c23-4d3e-9ce0-ef3364404e20" (UID: "22acb4e8-9c23-4d3e-9ce0-ef3364404e20"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 22:42:35.889391 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:35.889372 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22acb4e8-9c23-4d3e-9ce0-ef3364404e20-dshm" (OuterVolumeSpecName: "dshm") pod "22acb4e8-9c23-4d3e-9ce0-ef3364404e20" (UID: "22acb4e8-9c23-4d3e-9ce0-ef3364404e20"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 22:42:35.889542 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:35.889467 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22acb4e8-9c23-4d3e-9ce0-ef3364404e20-kube-api-access-gb6qw" (OuterVolumeSpecName: "kube-api-access-gb6qw") pod "22acb4e8-9c23-4d3e-9ce0-ef3364404e20" (UID: "22acb4e8-9c23-4d3e-9ce0-ef3364404e20"). InnerVolumeSpecName "kube-api-access-gb6qw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 22:42:35.985650 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:35.985615 2564 generic.go:356] "Generic (PLEG): container finished" podID="22acb4e8-9c23-4d3e-9ce0-ef3364404e20" containerID="d1d923feeb3fb435056f066535d1012065142a434f1400a60d64d65b767f1eeb" exitCode=0 Jul 07 22:42:35.985804 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:35.985664 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a4bfe6/sub-test-model-a4bfe6-kserve-845445f84c-vl4xf" Jul 07 22:42:35.985804 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:35.985679 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a4bfe6/sub-test-model-a4bfe6-kserve-845445f84c-vl4xf" event={"ID":"22acb4e8-9c23-4d3e-9ce0-ef3364404e20","Type":"ContainerDied","Data":"d1d923feeb3fb435056f066535d1012065142a434f1400a60d64d65b767f1eeb"} Jul 07 22:42:35.985804 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:35.985718 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a4bfe6/sub-test-model-a4bfe6-kserve-845445f84c-vl4xf" event={"ID":"22acb4e8-9c23-4d3e-9ce0-ef3364404e20","Type":"ContainerDied","Data":"0762280dbff295dcc1c18b6cff843f58bde84c9dc2475eeb238284f8366526e2"} Jul 07 22:42:35.985804 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:35.985740 2564 scope.go:117] "RemoveContainer" containerID="d1d923feeb3fb435056f066535d1012065142a434f1400a60d64d65b767f1eeb" Jul 07 22:42:35.988836 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:35.988811 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb6qw\" (UniqueName: \"kubernetes.io/projected/22acb4e8-9c23-4d3e-9ce0-ef3364404e20-kube-api-access-gb6qw\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 07 22:42:35.988950 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:35.988840 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22acb4e8-9c23-4d3e-9ce0-ef3364404e20-dshm\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 07 22:42:35.988950 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:35.988854 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22acb4e8-9c23-4d3e-9ce0-ef3364404e20-home\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 07 22:42:35.988950 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:35.988867 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22acb4e8-9c23-4d3e-9ce0-ef3364404e20-tls-certs\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 07 22:42:35.988950 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:35.988883 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22acb4e8-9c23-4d3e-9ce0-ef3364404e20-model-cache\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 07 22:42:35.994951 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:35.994931 2564 scope.go:117] "RemoveContainer" containerID="d1d923feeb3fb435056f066535d1012065142a434f1400a60d64d65b767f1eeb" Jul 07 22:42:35.995189 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:42:35.995171 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1d923feeb3fb435056f066535d1012065142a434f1400a60d64d65b767f1eeb\": container with ID starting with d1d923feeb3fb435056f066535d1012065142a434f1400a60d64d65b767f1eeb not found: ID does not exist" containerID="d1d923feeb3fb435056f066535d1012065142a434f1400a60d64d65b767f1eeb" Jul 07 22:42:35.995234 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:35.995198 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1d923feeb3fb435056f066535d1012065142a434f1400a60d64d65b767f1eeb"} err="failed to get container status \"d1d923feeb3fb435056f066535d1012065142a434f1400a60d64d65b767f1eeb\": rpc error: code = NotFound desc = could not find container \"d1d923feeb3fb435056f066535d1012065142a434f1400a60d64d65b767f1eeb\": container with ID starting with d1d923feeb3fb435056f066535d1012065142a434f1400a60d64d65b767f1eeb not found: ID does not exist" Jul 07 22:42:36.005576 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:36.005553 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-a4bfe6/sub-test-model-a4bfe6-kserve-845445f84c-vl4xf"] Jul 07 22:42:36.008684 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:36.008661 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-a4bfe6/sub-test-model-a4bfe6-kserve-845445f84c-vl4xf"] Jul 07 22:42:36.707408 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:36.707376 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22acb4e8-9c23-4d3e-9ce0-ef3364404e20" path="/var/lib/kubelet/pods/22acb4e8-9c23-4d3e-9ce0-ef3364404e20/volumes" Jul 07 22:42:43.706909 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:43.706881 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:42:50.614806 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:50.614774 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-a4bfe6/rate-test-model-a4bfe6-kserve-588fb8c947-m898t"] Jul 07 22:42:50.615283 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:50.615104 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22acb4e8-9c23-4d3e-9ce0-ef3364404e20" containerName="main" Jul 07 22:42:50.615283 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:50.615115 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="22acb4e8-9c23-4d3e-9ce0-ef3364404e20" containerName="main" Jul 07 22:42:50.615283 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:50.615165 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="22acb4e8-9c23-4d3e-9ce0-ef3364404e20" containerName="main" Jul 07 22:42:50.618158 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:50.618142 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a4bfe6/rate-test-model-a4bfe6-kserve-588fb8c947-m898t" Jul 07 22:42:50.620894 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:50.620871 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-a4bfe6\"/\"default-dockercfg-26nnx\"" Jul 07 22:42:50.621037 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:50.620891 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-a4bfe6\"/\"openshift-service-ca.crt\"" Jul 07 22:42:50.621037 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:50.620946 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-a4bfe6\"/\"kube-root-ca.crt\"" Jul 07 22:42:50.621037 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:50.620894 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-a4bfe6\"/\"rate-test-model-a4bfe6-kserve-self-signed-certs\"" Jul 07 22:42:50.627615 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:50.627594 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-a4bfe6/rate-test-model-a4bfe6-kserve-588fb8c947-m898t"] Jul 07 22:42:50.709015 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:50.708986 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1962fa40-cb1c-4e8a-99da-ae8173f6172a-dshm\") pod \"rate-test-model-a4bfe6-kserve-588fb8c947-m898t\" (UID: \"1962fa40-cb1c-4e8a-99da-ae8173f6172a\") " pod="ai-tenant-e2e-shared-b-a4bfe6/rate-test-model-a4bfe6-kserve-588fb8c947-m898t" Jul 07 22:42:50.709173 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:50.709031 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1962fa40-cb1c-4e8a-99da-ae8173f6172a-home\") pod \"rate-test-model-a4bfe6-kserve-588fb8c947-m898t\" (UID: \"1962fa40-cb1c-4e8a-99da-ae8173f6172a\") " pod="ai-tenant-e2e-shared-b-a4bfe6/rate-test-model-a4bfe6-kserve-588fb8c947-m898t" Jul 07 22:42:50.709173 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:50.709046 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1962fa40-cb1c-4e8a-99da-ae8173f6172a-model-cache\") pod \"rate-test-model-a4bfe6-kserve-588fb8c947-m898t\" (UID: \"1962fa40-cb1c-4e8a-99da-ae8173f6172a\") " pod="ai-tenant-e2e-shared-b-a4bfe6/rate-test-model-a4bfe6-kserve-588fb8c947-m898t" Jul 07 22:42:50.709173 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:50.709061 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgmm4\" (UniqueName: \"kubernetes.io/projected/1962fa40-cb1c-4e8a-99da-ae8173f6172a-kube-api-access-cgmm4\") pod \"rate-test-model-a4bfe6-kserve-588fb8c947-m898t\" (UID: \"1962fa40-cb1c-4e8a-99da-ae8173f6172a\") " pod="ai-tenant-e2e-shared-b-a4bfe6/rate-test-model-a4bfe6-kserve-588fb8c947-m898t" Jul 07 22:42:50.709173 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:50.709144 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1962fa40-cb1c-4e8a-99da-ae8173f6172a-tls-certs\") pod \"rate-test-model-a4bfe6-kserve-588fb8c947-m898t\" (UID: \"1962fa40-cb1c-4e8a-99da-ae8173f6172a\") " pod="ai-tenant-e2e-shared-b-a4bfe6/rate-test-model-a4bfe6-kserve-588fb8c947-m898t" Jul 07 22:42:50.709329 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:50.709178 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1962fa40-cb1c-4e8a-99da-ae8173f6172a-tmp-dir\") pod \"rate-test-model-a4bfe6-kserve-588fb8c947-m898t\" (UID: \"1962fa40-cb1c-4e8a-99da-ae8173f6172a\") " pod="ai-tenant-e2e-shared-b-a4bfe6/rate-test-model-a4bfe6-kserve-588fb8c947-m898t" Jul 07 22:42:50.809959 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:50.809917 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1962fa40-cb1c-4e8a-99da-ae8173f6172a-dshm\") pod \"rate-test-model-a4bfe6-kserve-588fb8c947-m898t\" (UID: \"1962fa40-cb1c-4e8a-99da-ae8173f6172a\") " pod="ai-tenant-e2e-shared-b-a4bfe6/rate-test-model-a4bfe6-kserve-588fb8c947-m898t" Jul 07 22:42:50.810157 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:50.810002 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1962fa40-cb1c-4e8a-99da-ae8173f6172a-home\") pod \"rate-test-model-a4bfe6-kserve-588fb8c947-m898t\" (UID: \"1962fa40-cb1c-4e8a-99da-ae8173f6172a\") " pod="ai-tenant-e2e-shared-b-a4bfe6/rate-test-model-a4bfe6-kserve-588fb8c947-m898t" Jul 07 22:42:50.810157 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:50.810030 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1962fa40-cb1c-4e8a-99da-ae8173f6172a-model-cache\") pod \"rate-test-model-a4bfe6-kserve-588fb8c947-m898t\" (UID: \"1962fa40-cb1c-4e8a-99da-ae8173f6172a\") " pod="ai-tenant-e2e-shared-b-a4bfe6/rate-test-model-a4bfe6-kserve-588fb8c947-m898t" Jul 07 22:42:50.810157 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:50.810054 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgmm4\" (UniqueName: \"kubernetes.io/projected/1962fa40-cb1c-4e8a-99da-ae8173f6172a-kube-api-access-cgmm4\") pod \"rate-test-model-a4bfe6-kserve-588fb8c947-m898t\" (UID: \"1962fa40-cb1c-4e8a-99da-ae8173f6172a\") " pod="ai-tenant-e2e-shared-b-a4bfe6/rate-test-model-a4bfe6-kserve-588fb8c947-m898t" Jul 07 22:42:50.810157 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:50.810100 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1962fa40-cb1c-4e8a-99da-ae8173f6172a-tls-certs\") pod \"rate-test-model-a4bfe6-kserve-588fb8c947-m898t\" (UID: \"1962fa40-cb1c-4e8a-99da-ae8173f6172a\") " pod="ai-tenant-e2e-shared-b-a4bfe6/rate-test-model-a4bfe6-kserve-588fb8c947-m898t" Jul 07 22:42:50.810157 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:50.810140 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1962fa40-cb1c-4e8a-99da-ae8173f6172a-tmp-dir\") pod \"rate-test-model-a4bfe6-kserve-588fb8c947-m898t\" (UID: \"1962fa40-cb1c-4e8a-99da-ae8173f6172a\") " pod="ai-tenant-e2e-shared-b-a4bfe6/rate-test-model-a4bfe6-kserve-588fb8c947-m898t" Jul 07 22:42:50.810568 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:50.810536 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1962fa40-cb1c-4e8a-99da-ae8173f6172a-model-cache\") pod \"rate-test-model-a4bfe6-kserve-588fb8c947-m898t\" (UID: \"1962fa40-cb1c-4e8a-99da-ae8173f6172a\") " pod="ai-tenant-e2e-shared-b-a4bfe6/rate-test-model-a4bfe6-kserve-588fb8c947-m898t" Jul 07 22:42:50.810697 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:50.810651 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1962fa40-cb1c-4e8a-99da-ae8173f6172a-tmp-dir\") pod \"rate-test-model-a4bfe6-kserve-588fb8c947-m898t\" (UID: \"1962fa40-cb1c-4e8a-99da-ae8173f6172a\") " pod="ai-tenant-e2e-shared-b-a4bfe6/rate-test-model-a4bfe6-kserve-588fb8c947-m898t" Jul 07 22:42:50.810697 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:50.810679 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1962fa40-cb1c-4e8a-99da-ae8173f6172a-home\") pod \"rate-test-model-a4bfe6-kserve-588fb8c947-m898t\" (UID: \"1962fa40-cb1c-4e8a-99da-ae8173f6172a\") " pod="ai-tenant-e2e-shared-b-a4bfe6/rate-test-model-a4bfe6-kserve-588fb8c947-m898t" Jul 07 22:42:50.812349 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:50.812327 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1962fa40-cb1c-4e8a-99da-ae8173f6172a-dshm\") pod \"rate-test-model-a4bfe6-kserve-588fb8c947-m898t\" (UID: \"1962fa40-cb1c-4e8a-99da-ae8173f6172a\") " pod="ai-tenant-e2e-shared-b-a4bfe6/rate-test-model-a4bfe6-kserve-588fb8c947-m898t" Jul 07 22:42:50.812624 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:50.812608 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1962fa40-cb1c-4e8a-99da-ae8173f6172a-tls-certs\") pod \"rate-test-model-a4bfe6-kserve-588fb8c947-m898t\" (UID: \"1962fa40-cb1c-4e8a-99da-ae8173f6172a\") " pod="ai-tenant-e2e-shared-b-a4bfe6/rate-test-model-a4bfe6-kserve-588fb8c947-m898t" Jul 07 22:42:50.817845 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:50.817821 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgmm4\" (UniqueName: \"kubernetes.io/projected/1962fa40-cb1c-4e8a-99da-ae8173f6172a-kube-api-access-cgmm4\") pod \"rate-test-model-a4bfe6-kserve-588fb8c947-m898t\" (UID: \"1962fa40-cb1c-4e8a-99da-ae8173f6172a\") " pod="ai-tenant-e2e-shared-b-a4bfe6/rate-test-model-a4bfe6-kserve-588fb8c947-m898t" Jul 07 22:42:50.929097 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:50.929015 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a4bfe6/rate-test-model-a4bfe6-kserve-588fb8c947-m898t" Jul 07 22:42:51.060516 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:51.060456 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-a4bfe6/rate-test-model-a4bfe6-kserve-588fb8c947-m898t"] Jul 07 22:42:51.062987 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:42:51.062958 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1962fa40_cb1c_4e8a_99da_ae8173f6172a.slice/crio-c94991e53d43bfd5e8399a2e5ce1ffa18d7767e34197b03049fb6ec2241cc995 WatchSource:0}: Error finding container c94991e53d43bfd5e8399a2e5ce1ffa18d7767e34197b03049fb6ec2241cc995: Status 404 returned error can't find the container with id c94991e53d43bfd5e8399a2e5ce1ffa18d7767e34197b03049fb6ec2241cc995 Jul 07 22:42:52.069879 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:52.069838 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a4bfe6/rate-test-model-a4bfe6-kserve-588fb8c947-m898t" event={"ID":"1962fa40-cb1c-4e8a-99da-ae8173f6172a","Type":"ContainerStarted","Data":"45cfcfed4a207018c9565a12c0a3bdad9c6d73889141074b63c5be5ba021c0f3"} Jul 07 22:42:52.069879 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:52.069882 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a4bfe6/rate-test-model-a4bfe6-kserve-588fb8c947-m898t" event={"ID":"1962fa40-cb1c-4e8a-99da-ae8173f6172a","Type":"ContainerStarted","Data":"c94991e53d43bfd5e8399a2e5ce1ffa18d7767e34197b03049fb6ec2241cc995"} Jul 07 22:42:52.084926 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:52.084876 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-a4bfe6/rate-test-model-a4bfe6-kserve-588fb8c947-m898t" podStartSLOduration=2.084855802 podStartE2EDuration="2.084855802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 22:42:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:42:52.084195807 +0000 UTC m=+2351.912325264" watchObservedRunningTime="2026-07-07 22:42:52.084855802 +0000 UTC m=+2351.912985260" Jul 07 22:42:54.502840 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:42:54.502805 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:43:00.929672 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:43:00.929635 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-a4bfe6/rate-test-model-a4bfe6-kserve-588fb8c947-m898t" Jul 07 22:43:00.930050 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:43:00.929691 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-a4bfe6/rate-test-model-a4bfe6-kserve-588fb8c947-m898t" Jul 07 22:43:00.942109 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:43:00.942084 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-a4bfe6/rate-test-model-a4bfe6-kserve-588fb8c947-m898t" Jul 07 22:43:01.127037 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:43:01.127005 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-a4bfe6/rate-test-model-a4bfe6-kserve-588fb8c947-m898t" Jul 07 22:43:12.714208 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:43:12.714171 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:43:29.313454 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:43:29.313402 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:43:34.326650 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:43:34.326618 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-a4bfe6/rate-test-model-a4bfe6-kserve-588fb8c947-m898t"] Jul 07 22:43:40.792746 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:43:40.792720 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4h8zm_4b4234f5-6cda-45ea-b559-f09bcad77232/ovn-acl-logging/0.log" Jul 07 22:43:40.799630 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:43:40.799603 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4h8zm_4b4234f5-6cda-45ea-b559-f09bcad77232/ovn-acl-logging/0.log" Jul 07 22:43:44.307810 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:43:44.307768 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:43:49.348793 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:43:49.348736 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-a4bfe6/rate-test-model-a4bfe6-kserve-588fb8c947-m898t" podUID="1962fa40-cb1c-4e8a-99da-ae8173f6172a" containerName="main" containerID="cri-o://45cfcfed4a207018c9565a12c0a3bdad9c6d73889141074b63c5be5ba021c0f3" gracePeriod=45 Jul 07 22:43:49.602518 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:43:49.602460 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a4bfe6/rate-test-model-a4bfe6-kserve-588fb8c947-m898t" Jul 07 22:43:49.605709 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:43:49.605689 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1962fa40-cb1c-4e8a-99da-ae8173f6172a-tmp-dir\") pod \"1962fa40-cb1c-4e8a-99da-ae8173f6172a\" (UID: \"1962fa40-cb1c-4e8a-99da-ae8173f6172a\") " Jul 07 22:43:49.605840 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:43:49.605728 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1962fa40-cb1c-4e8a-99da-ae8173f6172a-dshm\") pod \"1962fa40-cb1c-4e8a-99da-ae8173f6172a\" (UID: \"1962fa40-cb1c-4e8a-99da-ae8173f6172a\") " Jul 07 22:43:49.605840 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:43:49.605765 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cgmm4\" (UniqueName: \"kubernetes.io/projected/1962fa40-cb1c-4e8a-99da-ae8173f6172a-kube-api-access-cgmm4\") pod \"1962fa40-cb1c-4e8a-99da-ae8173f6172a\" (UID: \"1962fa40-cb1c-4e8a-99da-ae8173f6172a\") " Jul 07 22:43:49.605840 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:43:49.605798 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1962fa40-cb1c-4e8a-99da-ae8173f6172a-model-cache\") pod \"1962fa40-cb1c-4e8a-99da-ae8173f6172a\" (UID: \"1962fa40-cb1c-4e8a-99da-ae8173f6172a\") " Jul 07 22:43:49.605840 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:43:49.605817 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1962fa40-cb1c-4e8a-99da-ae8173f6172a-home\") pod \"1962fa40-cb1c-4e8a-99da-ae8173f6172a\" (UID: \"1962fa40-cb1c-4e8a-99da-ae8173f6172a\") " Jul 07 22:43:49.605840 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:43:49.605839 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1962fa40-cb1c-4e8a-99da-ae8173f6172a-tls-certs\") pod \"1962fa40-cb1c-4e8a-99da-ae8173f6172a\" (UID: \"1962fa40-cb1c-4e8a-99da-ae8173f6172a\") " Jul 07 22:43:49.606100 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:43:49.605944 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1962fa40-cb1c-4e8a-99da-ae8173f6172a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1962fa40-cb1c-4e8a-99da-ae8173f6172a" (UID: "1962fa40-cb1c-4e8a-99da-ae8173f6172a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 22:43:49.606100 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:43:49.606080 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1962fa40-cb1c-4e8a-99da-ae8173f6172a-home" (OuterVolumeSpecName: "home") pod "1962fa40-cb1c-4e8a-99da-ae8173f6172a" (UID: "1962fa40-cb1c-4e8a-99da-ae8173f6172a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 22:43:49.606198 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:43:49.606097 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1962fa40-cb1c-4e8a-99da-ae8173f6172a-tmp-dir\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 07 22:43:49.606198 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:43:49.606096 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1962fa40-cb1c-4e8a-99da-ae8173f6172a-model-cache" (OuterVolumeSpecName: "model-cache") pod "1962fa40-cb1c-4e8a-99da-ae8173f6172a" (UID: "1962fa40-cb1c-4e8a-99da-ae8173f6172a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 22:43:49.607842 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:43:49.607807 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1962fa40-cb1c-4e8a-99da-ae8173f6172a-dshm" (OuterVolumeSpecName: "dshm") pod "1962fa40-cb1c-4e8a-99da-ae8173f6172a" (UID: "1962fa40-cb1c-4e8a-99da-ae8173f6172a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 22:43:49.607919 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:43:49.607897 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1962fa40-cb1c-4e8a-99da-ae8173f6172a-kube-api-access-cgmm4" (OuterVolumeSpecName: "kube-api-access-cgmm4") pod "1962fa40-cb1c-4e8a-99da-ae8173f6172a" (UID: "1962fa40-cb1c-4e8a-99da-ae8173f6172a"). InnerVolumeSpecName "kube-api-access-cgmm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 22:43:49.608004 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:43:49.607990 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1962fa40-cb1c-4e8a-99da-ae8173f6172a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1962fa40-cb1c-4e8a-99da-ae8173f6172a" (UID: "1962fa40-cb1c-4e8a-99da-ae8173f6172a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 22:43:49.706650 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:43:49.706622 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1962fa40-cb1c-4e8a-99da-ae8173f6172a-tls-certs\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 07 22:43:49.706650 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:43:49.706650 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1962fa40-cb1c-4e8a-99da-ae8173f6172a-dshm\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 07 22:43:49.706823 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:43:49.706659 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgmm4\" (UniqueName: \"kubernetes.io/projected/1962fa40-cb1c-4e8a-99da-ae8173f6172a-kube-api-access-cgmm4\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 07 22:43:49.706823 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:43:49.706668 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1962fa40-cb1c-4e8a-99da-ae8173f6172a-model-cache\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 07 22:43:49.706823 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:43:49.706676 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1962fa40-cb1c-4e8a-99da-ae8173f6172a-home\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 07 22:43:50.362115 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:43:50.362081 2564 generic.go:356] "Generic (PLEG): container finished" podID="1962fa40-cb1c-4e8a-99da-ae8173f6172a" containerID="45cfcfed4a207018c9565a12c0a3bdad9c6d73889141074b63c5be5ba021c0f3" exitCode=0 Jul 07 22:43:50.362603 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:43:50.362125 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a4bfe6/rate-test-model-a4bfe6-kserve-588fb8c947-m898t" Jul 07 22:43:50.362603 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:43:50.362137 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a4bfe6/rate-test-model-a4bfe6-kserve-588fb8c947-m898t" event={"ID":"1962fa40-cb1c-4e8a-99da-ae8173f6172a","Type":"ContainerDied","Data":"45cfcfed4a207018c9565a12c0a3bdad9c6d73889141074b63c5be5ba021c0f3"} Jul 07 22:43:50.362603 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:43:50.362169 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a4bfe6/rate-test-model-a4bfe6-kserve-588fb8c947-m898t" event={"ID":"1962fa40-cb1c-4e8a-99da-ae8173f6172a","Type":"ContainerDied","Data":"c94991e53d43bfd5e8399a2e5ce1ffa18d7767e34197b03049fb6ec2241cc995"} Jul 07 22:43:50.362603 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:43:50.362186 2564 scope.go:117] "RemoveContainer" containerID="45cfcfed4a207018c9565a12c0a3bdad9c6d73889141074b63c5be5ba021c0f3" Jul 07 22:43:50.371360 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:43:50.371342 2564 scope.go:117] "RemoveContainer" containerID="45cfcfed4a207018c9565a12c0a3bdad9c6d73889141074b63c5be5ba021c0f3" Jul 07 22:43:50.371625 ip-10-0-134-156 kubenswrapper[2564]: E0707 22:43:50.371608 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45cfcfed4a207018c9565a12c0a3bdad9c6d73889141074b63c5be5ba021c0f3\": container with ID starting with 45cfcfed4a207018c9565a12c0a3bdad9c6d73889141074b63c5be5ba021c0f3 not found: ID does not exist" containerID="45cfcfed4a207018c9565a12c0a3bdad9c6d73889141074b63c5be5ba021c0f3" Jul 07 22:43:50.371694 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:43:50.371633 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45cfcfed4a207018c9565a12c0a3bdad9c6d73889141074b63c5be5ba021c0f3"} err="failed to get container status \"45cfcfed4a207018c9565a12c0a3bdad9c6d73889141074b63c5be5ba021c0f3\": rpc error: code = NotFound desc = could not find container \"45cfcfed4a207018c9565a12c0a3bdad9c6d73889141074b63c5be5ba021c0f3\": container with ID starting with 45cfcfed4a207018c9565a12c0a3bdad9c6d73889141074b63c5be5ba021c0f3 not found: ID does not exist" Jul 07 22:43:50.389075 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:43:50.388474 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-a4bfe6/rate-test-model-a4bfe6-kserve-588fb8c947-m898t"] Jul 07 22:43:50.389075 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:43:50.388520 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-a4bfe6/rate-test-model-a4bfe6-kserve-588fb8c947-m898t"] Jul 07 22:43:50.706989 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:43:50.706915 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1962fa40-cb1c-4e8a-99da-ae8173f6172a" path="/var/lib/kubelet/pods/1962fa40-cb1c-4e8a-99da-ae8173f6172a/volumes" Jul 07 22:44:24.707876 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:24.707838 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f4z8p"] Jul 07 22:44:30.370164 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:30.370136 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29724375-6cc8q_fe0445a2-ddfe-4ff6-9d4d-87096b886097/cleanup/0.log" Jul 07 22:44:30.481107 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:30.481073 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29724390-tzl6q_7586a2e1-cc6e-4262-91ba-bb7ec0dd571a/cleanup/0.log" Jul 07 22:44:30.958971 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:30.958943 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-597db99f65-8gxnd_20bc9c72-6f00-403e-b62f-87093490df90/copy-manifests/0.log" Jul 07 22:44:30.977397 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:30.977366 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-597db99f65-8gxnd_20bc9c72-6f00-403e-b62f-87093490df90/manager/0.log" Jul 07 22:44:32.102938 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:32.102911 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759phphs_470e5356-b27c-4300-972d-4409d4bccc45/util/0.log" Jul 07 22:44:32.108417 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:32.108379 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759phphs_470e5356-b27c-4300-972d-4409d4bccc45/pull/0.log" Jul 07 22:44:32.113905 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:32.113885 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759phphs_470e5356-b27c-4300-972d-4409d4bccc45/extract/0.log" Jul 07 22:44:32.225716 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:32.225684 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lhh6j_dbe0bc5b-dcda-463a-8764-8ad638ccb1f7/util/0.log" Jul 07 22:44:32.231361 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:32.231332 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lhh6j_dbe0bc5b-dcda-463a-8764-8ad638ccb1f7/pull/0.log" Jul 07 22:44:32.236825 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:32.236800 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lhh6j_dbe0bc5b-dcda-463a-8764-8ad638ccb1f7/extract/0.log" Jul 07 22:44:32.348111 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:32.348083 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9j2b_7121c2d1-76f0-41f8-8b8a-a693dcef10ca/extract/0.log" Jul 07 22:44:32.353510 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:32.353436 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9j2b_7121c2d1-76f0-41f8-8b8a-a693dcef10ca/util/0.log" Jul 07 22:44:32.358793 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:32.358764 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9j2b_7121c2d1-76f0-41f8-8b8a-a693dcef10ca/pull/0.log" Jul 07 22:44:32.472547 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:32.472518 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vjphr_0fa7d0f7-6fa8-43df-b565-aefc01d4855a/util/0.log" Jul 07 22:44:32.479631 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:32.479611 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vjphr_0fa7d0f7-6fa8-43df-b565-aefc01d4855a/pull/0.log" Jul 07 22:44:32.486374 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:32.486354 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vjphr_0fa7d0f7-6fa8-43df-b565-aefc01d4855a/extract/0.log" Jul 07 22:44:33.327356 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:33.327324 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-f4z8p_41efdd54-8912-4931-b9c2-3cfae8964536/limitador/0.log" Jul 07 22:44:33.915240 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:33.915203 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-f78jq_478167a3-0a72-4e85-994e-75972b76b135/discovery/0.log" Jul 07 22:44:35.213059 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:35.213029 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-wprgj_b97ea41a-1606-44a8-9a8d-0b30e0b885a7/main/0.log" Jul 07 22:44:42.488253 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:42.488223 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mzzl4_acb69b34-f020-4c76-89c2-0ab5b6e6f3d5/global-pull-secret-syncer/0.log" Jul 07 22:44:42.650420 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:42.650390 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-lc54x_a38ac499-afcf-4298-9918-a825227cab29/konnectivity-agent/0.log" Jul 07 22:44:42.721598 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:42.721568 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-156.ec2.internal_7e758ef0bacc8b47cfb5ed8977696fa1/haproxy/0.log" Jul 07 22:44:47.357225 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:47.357196 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759phphs_470e5356-b27c-4300-972d-4409d4bccc45/extract/0.log" Jul 07 22:44:47.387797 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:47.387769 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759phphs_470e5356-b27c-4300-972d-4409d4bccc45/util/0.log" Jul 07 22:44:47.407832 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:47.407805 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759phphs_470e5356-b27c-4300-972d-4409d4bccc45/pull/0.log" Jul 07 22:44:47.433400 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:47.433379 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lhh6j_dbe0bc5b-dcda-463a-8764-8ad638ccb1f7/extract/0.log" Jul 07 22:44:47.461237 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:47.461214 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lhh6j_dbe0bc5b-dcda-463a-8764-8ad638ccb1f7/util/0.log" Jul 07 22:44:47.485294 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:47.485268 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lhh6j_dbe0bc5b-dcda-463a-8764-8ad638ccb1f7/pull/0.log" Jul 07 22:44:47.518096 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:47.518050 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9j2b_7121c2d1-76f0-41f8-8b8a-a693dcef10ca/extract/0.log" Jul 07 22:44:47.542099 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:47.542068 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9j2b_7121c2d1-76f0-41f8-8b8a-a693dcef10ca/util/0.log" Jul 07 22:44:47.580693 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:47.580668 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9j2b_7121c2d1-76f0-41f8-8b8a-a693dcef10ca/pull/0.log" Jul 07 22:44:47.615132 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:47.615064 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vjphr_0fa7d0f7-6fa8-43df-b565-aefc01d4855a/extract/0.log" Jul 07 22:44:47.643901 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:47.643883 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vjphr_0fa7d0f7-6fa8-43df-b565-aefc01d4855a/util/0.log" Jul 07 22:44:47.674363 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:47.674342 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vjphr_0fa7d0f7-6fa8-43df-b565-aefc01d4855a/pull/0.log" Jul 07 22:44:48.364648 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:48.364617 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-f4z8p_41efdd54-8912-4931-b9c2-3cfae8964536/limitador/0.log" Jul 07 22:44:50.050493 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:50.050460 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6f466319-941b-4c0f-b872-f62f8c302442/alertmanager/0.log" Jul 07 22:44:50.071206 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:50.071177 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6f466319-941b-4c0f-b872-f62f8c302442/config-reloader/0.log" Jul 07 22:44:50.090144 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:50.090122 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6f466319-941b-4c0f-b872-f62f8c302442/kube-rbac-proxy-web/0.log" Jul 07 22:44:50.111369 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:50.111343 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6f466319-941b-4c0f-b872-f62f8c302442/kube-rbac-proxy/0.log" Jul 07 22:44:50.130778 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:50.130753 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6f466319-941b-4c0f-b872-f62f8c302442/kube-rbac-proxy-metric/0.log" Jul 07 22:44:50.151049 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:50.151028 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6f466319-941b-4c0f-b872-f62f8c302442/prom-label-proxy/0.log" Jul 07 22:44:50.171485 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:50.171467 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6f466319-941b-4c0f-b872-f62f8c302442/init-config-reloader/0.log" Jul 07 22:44:50.547688 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:50.547652 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p69r8_91498a50-33c1-4b03-a193-83b60292acc3/node-exporter/0.log" Jul 07 22:44:50.564492 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:50.564463 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p69r8_91498a50-33c1-4b03-a193-83b60292acc3/kube-rbac-proxy/0.log" Jul 07 22:44:50.583958 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:50.583939 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p69r8_91498a50-33c1-4b03-a193-83b60292acc3/init-textfile/0.log" Jul 07 22:44:50.980415 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:50.980386 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7946669b5f-kgqtb_7d7078ff-8283-4a7e-a95d-0ed75b3cba0a/thanos-query/0.log" Jul 07 22:44:50.998406 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:50.998385 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7946669b5f-kgqtb_7d7078ff-8283-4a7e-a95d-0ed75b3cba0a/kube-rbac-proxy-web/0.log" Jul 07 22:44:51.019637 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:51.019615 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7946669b5f-kgqtb_7d7078ff-8283-4a7e-a95d-0ed75b3cba0a/kube-rbac-proxy/0.log" Jul 07 22:44:51.038011 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:51.037990 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7946669b5f-kgqtb_7d7078ff-8283-4a7e-a95d-0ed75b3cba0a/prom-label-proxy/0.log" Jul 07 22:44:51.056117 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:51.056096 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7946669b5f-kgqtb_7d7078ff-8283-4a7e-a95d-0ed75b3cba0a/kube-rbac-proxy-rules/0.log" Jul 07 22:44:51.074615 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:51.074590 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7946669b5f-kgqtb_7d7078ff-8283-4a7e-a95d-0ed75b3cba0a/kube-rbac-proxy-metrics/0.log" Jul 07 22:44:52.497847 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:52.497818 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kzlgw/perf-node-gather-daemonset-78qgz"] Jul 07 22:44:52.498217 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:52.498135 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1962fa40-cb1c-4e8a-99da-ae8173f6172a" containerName="main" Jul 07 22:44:52.498217 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:52.498146 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1962fa40-cb1c-4e8a-99da-ae8173f6172a" containerName="main" Jul 07 22:44:52.498217 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:52.498205 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1962fa40-cb1c-4e8a-99da-ae8173f6172a" containerName="main" Jul 07 22:44:52.501287 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:52.501271 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kzlgw/perf-node-gather-daemonset-78qgz" Jul 07 22:44:52.503689 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:52.503668 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kzlgw\"/\"kube-root-ca.crt\"" Jul 07 22:44:52.503824 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:52.503692 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kzlgw\"/\"default-dockercfg-7bpws\"" Jul 07 22:44:52.504656 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:52.504637 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kzlgw\"/\"openshift-service-ca.crt\"" Jul 07 22:44:52.509598 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:52.509580 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kzlgw/perf-node-gather-daemonset-78qgz"] Jul 07 22:44:52.617110 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:52.617076 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a2292131-2124-438a-b379-12ec8ab401f4-proc\") pod \"perf-node-gather-daemonset-78qgz\" (UID: \"a2292131-2124-438a-b379-12ec8ab401f4\") " pod="openshift-must-gather-kzlgw/perf-node-gather-daemonset-78qgz" Jul 07 22:44:52.617288 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:52.617178 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a2292131-2124-438a-b379-12ec8ab401f4-sys\") pod \"perf-node-gather-daemonset-78qgz\" (UID: \"a2292131-2124-438a-b379-12ec8ab401f4\") " pod="openshift-must-gather-kzlgw/perf-node-gather-daemonset-78qgz" Jul 07 22:44:52.617288 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:52.617204 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a2292131-2124-438a-b379-12ec8ab401f4-lib-modules\") pod \"perf-node-gather-daemonset-78qgz\" (UID: \"a2292131-2124-438a-b379-12ec8ab401f4\") " pod="openshift-must-gather-kzlgw/perf-node-gather-daemonset-78qgz" Jul 07 22:44:52.617288 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:52.617230 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a2292131-2124-438a-b379-12ec8ab401f4-podres\") pod \"perf-node-gather-daemonset-78qgz\" (UID: \"a2292131-2124-438a-b379-12ec8ab401f4\") " pod="openshift-must-gather-kzlgw/perf-node-gather-daemonset-78qgz" Jul 07 22:44:52.617406 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:52.617289 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xxwk\" (UniqueName: \"kubernetes.io/projected/a2292131-2124-438a-b379-12ec8ab401f4-kube-api-access-7xxwk\") pod \"perf-node-gather-daemonset-78qgz\" (UID: \"a2292131-2124-438a-b379-12ec8ab401f4\") " pod="openshift-must-gather-kzlgw/perf-node-gather-daemonset-78qgz" Jul 07 22:44:52.717942 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:52.717909 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a2292131-2124-438a-b379-12ec8ab401f4-lib-modules\") pod \"perf-node-gather-daemonset-78qgz\" (UID: \"a2292131-2124-438a-b379-12ec8ab401f4\") " pod="openshift-must-gather-kzlgw/perf-node-gather-daemonset-78qgz" Jul 07 22:44:52.717942 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:52.717939 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a2292131-2124-438a-b379-12ec8ab401f4-podres\") pod \"perf-node-gather-daemonset-78qgz\" (UID: \"a2292131-2124-438a-b379-12ec8ab401f4\") " pod="openshift-must-gather-kzlgw/perf-node-gather-daemonset-78qgz" Jul 07 22:44:52.718242 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:52.717959 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xxwk\" (UniqueName: \"kubernetes.io/projected/a2292131-2124-438a-b379-12ec8ab401f4-kube-api-access-7xxwk\") pod \"perf-node-gather-daemonset-78qgz\" (UID: \"a2292131-2124-438a-b379-12ec8ab401f4\") " pod="openshift-must-gather-kzlgw/perf-node-gather-daemonset-78qgz" Jul 07 22:44:52.718242 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:52.718084 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a2292131-2124-438a-b379-12ec8ab401f4-lib-modules\") pod \"perf-node-gather-daemonset-78qgz\" (UID: \"a2292131-2124-438a-b379-12ec8ab401f4\") " pod="openshift-must-gather-kzlgw/perf-node-gather-daemonset-78qgz" Jul 07 22:44:52.718242 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:52.718090 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a2292131-2124-438a-b379-12ec8ab401f4-proc\") pod \"perf-node-gather-daemonset-78qgz\" (UID: \"a2292131-2124-438a-b379-12ec8ab401f4\") " pod="openshift-must-gather-kzlgw/perf-node-gather-daemonset-78qgz" Jul 07 22:44:52.718242 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:52.718134 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a2292131-2124-438a-b379-12ec8ab401f4-proc\") pod \"perf-node-gather-daemonset-78qgz\" (UID: \"a2292131-2124-438a-b379-12ec8ab401f4\") " pod="openshift-must-gather-kzlgw/perf-node-gather-daemonset-78qgz" Jul 07 22:44:52.718242 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:52.718093 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a2292131-2124-438a-b379-12ec8ab401f4-podres\") pod \"perf-node-gather-daemonset-78qgz\" (UID: \"a2292131-2124-438a-b379-12ec8ab401f4\") " pod="openshift-must-gather-kzlgw/perf-node-gather-daemonset-78qgz" Jul 07 22:44:52.718242 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:52.718222 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a2292131-2124-438a-b379-12ec8ab401f4-sys\") pod \"perf-node-gather-daemonset-78qgz\" (UID: \"a2292131-2124-438a-b379-12ec8ab401f4\") " pod="openshift-must-gather-kzlgw/perf-node-gather-daemonset-78qgz" Jul 07 22:44:52.718501 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:52.718306 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a2292131-2124-438a-b379-12ec8ab401f4-sys\") pod \"perf-node-gather-daemonset-78qgz\" (UID: \"a2292131-2124-438a-b379-12ec8ab401f4\") " pod="openshift-must-gather-kzlgw/perf-node-gather-daemonset-78qgz" Jul 07 22:44:52.725417 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:52.725396 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xxwk\" (UniqueName: \"kubernetes.io/projected/a2292131-2124-438a-b379-12ec8ab401f4-kube-api-access-7xxwk\") pod \"perf-node-gather-daemonset-78qgz\" (UID: \"a2292131-2124-438a-b379-12ec8ab401f4\") " pod="openshift-must-gather-kzlgw/perf-node-gather-daemonset-78qgz" Jul 07 22:44:52.812589 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:52.812554 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kzlgw/perf-node-gather-daemonset-78qgz" Jul 07 22:44:52.942203 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:52.942078 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kzlgw/perf-node-gather-daemonset-78qgz"] Jul 07 22:44:52.944519 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:44:52.944488 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda2292131_2124_438a_b379_12ec8ab401f4.slice/crio-ca231326466b9ae6b1cfc593f7e1697632ec6263dd240b2c7e7b8954333d7b25 WatchSource:0}: Error finding container ca231326466b9ae6b1cfc593f7e1697632ec6263dd240b2c7e7b8954333d7b25: Status 404 returned error can't find the container with id ca231326466b9ae6b1cfc593f7e1697632ec6263dd240b2c7e7b8954333d7b25 Jul 07 22:44:53.679249 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:53.679212 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kzlgw/perf-node-gather-daemonset-78qgz" event={"ID":"a2292131-2124-438a-b379-12ec8ab401f4","Type":"ContainerStarted","Data":"39c948e7be77d2e36032190ca9d3882cb2dd3efc24240cda1a2616faf9afa641"} Jul 07 22:44:53.679626 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:53.679290 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kzlgw/perf-node-gather-daemonset-78qgz" event={"ID":"a2292131-2124-438a-b379-12ec8ab401f4","Type":"ContainerStarted","Data":"ca231326466b9ae6b1cfc593f7e1697632ec6263dd240b2c7e7b8954333d7b25"} Jul 07 22:44:53.679626 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:53.679342 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kzlgw/perf-node-gather-daemonset-78qgz" Jul 07 22:44:53.695650 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:53.695594 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kzlgw/perf-node-gather-daemonset-78qgz" podStartSLOduration=1.69558096 podStartE2EDuration="1.69558096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 22:44:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:44:53.692806638 +0000 UTC m=+2473.520936097" watchObservedRunningTime="2026-07-07 22:44:53.69558096 +0000 UTC m=+2473.523710417" Jul 07 22:44:55.094161 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:55.094135 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hbqkg_20b4d981-ad57-4ff9-b293-48c8f4a6f43a/dns/0.log" Jul 07 22:44:55.112042 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:55.112014 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hbqkg_20b4d981-ad57-4ff9-b293-48c8f4a6f43a/kube-rbac-proxy/0.log" Jul 07 22:44:55.131746 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:55.131725 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2gbtg_b22ba61c-8e69-4913-acd7-b58c7a92c8d4/dns-node-resolver/0.log" Jul 07 22:44:55.689674 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:55.689643 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5554df4897-v5rn8_d2380ae8-40b7-40a7-a87e-900314d58511/registry/0.log" Jul 07 22:44:55.709946 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:55.709922 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-c9npj_68c8466f-1a04-4d64-a571-35aa35b5c8b9/node-ca/0.log" Jul 07 22:44:56.841623 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:56.841588 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-f78jq_478167a3-0a72-4e85-994e-75972b76b135/discovery/0.log" Jul 07 22:44:57.788066 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:57.788036 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2bkmt_9becf7d9-4d28-4ae3-a0df-c7b1a87ddf8c/serve-healthcheck-canary/0.log" Jul 07 22:44:58.307343 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:58.307320 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-88k64_3fecd654-8a55-4c3d-91f6-dca0be8172e2/kube-rbac-proxy/0.log" Jul 07 22:44:58.330970 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:58.330950 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-88k64_3fecd654-8a55-4c3d-91f6-dca0be8172e2/exporter/0.log" Jul 07 22:44:58.352169 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:58.352125 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-88k64_3fecd654-8a55-4c3d-91f6-dca0be8172e2/extractor/0.log" Jul 07 22:44:59.695018 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:44:59.694991 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kzlgw/perf-node-gather-daemonset-78qgz" Jul 07 22:45:00.134271 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:45:00.134239 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29724405-t6z6z"] Jul 07 22:45:00.137698 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:45:00.137684 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724405-t6z6z" Jul 07 22:45:00.139760 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:45:00.139739 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-8l4tr\"" Jul 07 22:45:00.143373 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:45:00.143352 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29724405-t6z6z"] Jul 07 22:45:00.181750 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:45:00.181719 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbm6m\" (UniqueName: \"kubernetes.io/projected/cabcd91d-101a-4565-b772-eb312a1f66cf-kube-api-access-pbm6m\") pod \"maas-api-key-cleanup-29724405-t6z6z\" (UID: \"cabcd91d-101a-4565-b772-eb312a1f66cf\") " pod="opendatahub/maas-api-key-cleanup-29724405-t6z6z" Jul 07 22:45:00.282695 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:45:00.282664 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbm6m\" (UniqueName: \"kubernetes.io/projected/cabcd91d-101a-4565-b772-eb312a1f66cf-kube-api-access-pbm6m\") pod \"maas-api-key-cleanup-29724405-t6z6z\" (UID: \"cabcd91d-101a-4565-b772-eb312a1f66cf\") " pod="opendatahub/maas-api-key-cleanup-29724405-t6z6z" Jul 07 22:45:00.291058 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:45:00.291036 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbm6m\" (UniqueName: \"kubernetes.io/projected/cabcd91d-101a-4565-b772-eb312a1f66cf-kube-api-access-pbm6m\") pod \"maas-api-key-cleanup-29724405-t6z6z\" (UID: \"cabcd91d-101a-4565-b772-eb312a1f66cf\") " pod="opendatahub/maas-api-key-cleanup-29724405-t6z6z" Jul 07 22:45:00.449744 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:45:00.449656 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724405-t6z6z" Jul 07 22:45:00.577506 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:45:00.577475 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29724405-t6z6z"] Jul 07 22:45:00.581346 ip-10-0-134-156 kubenswrapper[2564]: W0707 22:45:00.581319 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcabcd91d_101a_4565_b772_eb312a1f66cf.slice/crio-361fd3c83321df4be362b28d612dbe1f44dbd4633bfa0a374ebefbf94ccb5b51 WatchSource:0}: Error finding container 361fd3c83321df4be362b28d612dbe1f44dbd4633bfa0a374ebefbf94ccb5b51: Status 404 returned error can't find the container with id 361fd3c83321df4be362b28d612dbe1f44dbd4633bfa0a374ebefbf94ccb5b51 Jul 07 22:45:00.723341 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:45:00.723264 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29724405-t6z6z" event={"ID":"cabcd91d-101a-4565-b772-eb312a1f66cf","Type":"ContainerStarted","Data":"361fd3c83321df4be362b28d612dbe1f44dbd4633bfa0a374ebefbf94ccb5b51"} Jul 07 22:45:00.868999 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:45:00.868963 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29724375-6cc8q_fe0445a2-ddfe-4ff6-9d4d-87096b886097/cleanup/0.log" Jul 07 22:45:00.889957 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:45:00.889937 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29724390-tzl6q_7586a2e1-cc6e-4262-91ba-bb7ec0dd571a/cleanup/0.log" Jul 07 22:45:01.311016 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:45:01.310988 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-597db99f65-8gxnd_20bc9c72-6f00-403e-b62f-87093490df90/manager/0.log" Jul 07 22:45:01.327317 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:45:01.327298 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-597db99f65-8gxnd_20bc9c72-6f00-403e-b62f-87093490df90/copy-manifests/0.log" Jul 07 22:45:01.729952 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:45:01.729872 2564 generic.go:356] "Generic (PLEG): container finished" podID="cabcd91d-101a-4565-b772-eb312a1f66cf" containerID="095df38c5fef34778f365c09a9a853db3a20f4fa74091043b654da2a1e6a937f" exitCode=0 Jul 07 22:45:01.729952 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:45:01.729901 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29724405-t6z6z" event={"ID":"cabcd91d-101a-4565-b772-eb312a1f66cf","Type":"ContainerDied","Data":"095df38c5fef34778f365c09a9a853db3a20f4fa74091043b654da2a1e6a937f"} Jul 07 22:45:02.874374 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:45:02.874349 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724405-t6z6z" Jul 07 22:45:02.905498 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:45:02.905460 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pbm6m\" (UniqueName: \"kubernetes.io/projected/cabcd91d-101a-4565-b772-eb312a1f66cf-kube-api-access-pbm6m\") pod \"cabcd91d-101a-4565-b772-eb312a1f66cf\" (UID: \"cabcd91d-101a-4565-b772-eb312a1f66cf\") " Jul 07 22:45:02.908235 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:45:02.908202 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cabcd91d-101a-4565-b772-eb312a1f66cf-kube-api-access-pbm6m" (OuterVolumeSpecName: "kube-api-access-pbm6m") pod "cabcd91d-101a-4565-b772-eb312a1f66cf" (UID: "cabcd91d-101a-4565-b772-eb312a1f66cf"). InnerVolumeSpecName "kube-api-access-pbm6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 22:45:02.955461 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:45:02.955406 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-654d77bd8f-xx48n_2ff22fa1-1ba5-4edb-85d3-66d93196fe38/manager/0.log" Jul 07 22:45:03.005310 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:45:03.005241 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-f78jq_478167a3-0a72-4e85-994e-75972b76b135/discovery/0.log" Jul 07 22:45:03.006644 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:45:03.006625 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbm6m\" (UniqueName: \"kubernetes.io/projected/cabcd91d-101a-4565-b772-eb312a1f66cf-kube-api-access-pbm6m\") on node \"ip-10-0-134-156.ec2.internal\" DevicePath \"\"" Jul 07 22:45:03.739920 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:45:03.739835 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29724405-t6z6z" event={"ID":"cabcd91d-101a-4565-b772-eb312a1f66cf","Type":"ContainerDied","Data":"361fd3c83321df4be362b28d612dbe1f44dbd4633bfa0a374ebefbf94ccb5b51"} Jul 07 22:45:03.739920 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:45:03.739867 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724405-t6z6z" Jul 07 22:45:03.739920 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:45:03.739888 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="361fd3c83321df4be362b28d612dbe1f44dbd4633bfa0a374ebefbf94ccb5b51" Jul 07 22:45:04.636779 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:45:04.636743 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-f78jq_478167a3-0a72-4e85-994e-75972b76b135/discovery/0.log" Jul 07 22:45:07.857418 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:45:07.857390 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-dnztd_5cd8d4d5-75db-49f2-b8e3-4291e58ec967/migrator/0.log" Jul 07 22:45:07.876373 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:45:07.876353 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-dnztd_5cd8d4d5-75db-49f2-b8e3-4291e58ec967/graceful-termination/0.log" Jul 07 22:45:09.189575 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:45:09.189547 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9z8v8_94b5ccd3-cdf2-4d52-af68-30f21b855887/kube-multus-additional-cni-plugins/0.log" Jul 07 22:45:09.207868 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:45:09.207836 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9z8v8_94b5ccd3-cdf2-4d52-af68-30f21b855887/egress-router-binary-copy/0.log" Jul 07 22:45:09.225806 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:45:09.225783 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9z8v8_94b5ccd3-cdf2-4d52-af68-30f21b855887/cni-plugins/0.log" Jul 07 22:45:09.243905 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:45:09.243882 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9z8v8_94b5ccd3-cdf2-4d52-af68-30f21b855887/bond-cni-plugin/0.log" Jul 07 22:45:09.262796 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:45:09.262778 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9z8v8_94b5ccd3-cdf2-4d52-af68-30f21b855887/routeoverride-cni/0.log" Jul 07 22:45:09.281893 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:45:09.281874 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9z8v8_94b5ccd3-cdf2-4d52-af68-30f21b855887/whereabouts-cni-bincopy/0.log" Jul 07 22:45:09.302349 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:45:09.302325 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9z8v8_94b5ccd3-cdf2-4d52-af68-30f21b855887/whereabouts-cni/0.log" Jul 07 22:45:09.640484 ip-10-0-134-156 kubenswrapper[2564]: I0707 22:45:09.640432 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-bnx62_6f9d7564-2f1c-4d78-863f-b2da1db479cf/kube-multus/0.log"